Chat with your company documents
Ask a question, get an answer from your own documents with the exact passage it came from. If they don't cover it, it says so. If two disagree, it says which is newer.
The company and its 19 documents are made up. The run in the video is real.
In the video
Returns: the current policy says 30 days, an old 2024 copy says 60. It answers 30 and shows both. Then a question no document covers, and one it answers plainly.
What I checked
- 20 questions with known answers, graded by code, not by an AI: 18 of 20 right in the latest run.
- "Not in the docs" 4 of 4, conflicts flagged 3 of 4. The wrong ones are below.
- Code checks every quote word for word against its document: 23 of 23 found.
- 3 runs of the same 20: 20, 20, 18 right.
One answer
All 20 questions
Latest run, wrong ones first. Model: gemini-3.8-flash. Downloads: eval_results.json, questions.json.
| Question | Should | Answer | Right |
|---|---|---|---|
| Can I return a cartridge I already opened? | flag the conflict | No, you cannot return an opened cartridge because replacement cartridges can be returned only if the sealed wrapper is unopened [1]. For hygiene reasons, opened cartridges cannot be taken back [2]. status answered, expected conflict | no |
| Do I need a receipt for a $20 lunch on a work trip? | answer | No, you do not need a receipt because a receipt is only required for expenses exceeding that amount [1]. answer is missing: $25 | no |
| How long does standard shipping take? | answer | Standard shipping takes 3 to 5 business days [1]. | yes |
| How much do I need to spend for free shipping? | answer | Standard shipping in the US is free on orders of $75 or more [1]. Additionally, subscription orders always ship free [2]. | yes |
| How often should I replace the Tapline cartridge? | answer | A Tapline cartridge should be replaced after filtering 100 gallons, which is about 3 months for a typical household [1]. In addition, it should be changed when a light on the housing turns red [2]. | yes |
| Does the Stillwell pitcher remove fluoride? | answer | No, the Stillwell pitcher does not remove fluoride [1]. | yes |
| What does the Undercurrent RO-4 cost? | answer | The Undercurrent RO-4 costs $329.00, which includes the faucet, tank, and a first set of cartridges [1]. | yes |
| What discount do I get with a cartridge subscription? | answer | With a cartridge subscription, you get 15% off the list price [1]. | yes |
| Is support open on Saturday, and when? | answer | Yes, support is open on Saturday from 9 am to 1 pm Eastern [1]. | yes |
| How many days of paid time off do full-time employees get? | answer | Full-time employees receive 15 days of paid time off per year [1]. | yes |
| What is the minimum order for wholesale accounts? | answer | The minimum order for wholesale accounts is 24 units, which can be mixed across products [1]. | yes |
| Can I use the WELCOME10 code on my subscription order? | answer | No, promo codes cannot be combined with the subscription discount [1]. | yes |
| Do you ship to Canada, and what does it cost? | answer | Yes, Fennick Water Co. ships to Canada [1], and shipping costs a flat $24.00 [2]. | yes |
| What is your customer support phone number? | say not in the docs | The provided documents do not state a customer support phone number. | yes |
| Does the RO-4 system work with well water? | say not in the docs | The provided documents do not mention whether the RO-4 system works with well water. | yes |
| When is the new shower filter coming out? | say not in the docs | The provided documents do not contain information about when the new shower filter is coming out. | yes |
| How many weeks of parental leave do employees get? | say not in the docs | The provided documents do not contain information about parental leave. | yes |
| How long do I have to return a Stillwell pitcher? | flag the conflict | Under the current returns policy, you can return a Stillwell pitcher within 30 days of delivery for a full refund [1], though an older policy allowed returns within 60 days of delivery [2]. | yes |
| What warranty does the Tapline faucet filter have? | flag the conflict | The Tapline faucet filter carries a 2-year limited warranty against defects in materials and workmanship from the date of delivery [1], though an earlier product sheet states that the housing comes with a 1-year warranty [2]. | yes |
| How much is the Stillwell pitcher? | flag the conflict | The Stillwell pitcher is $29.00 with 1 cartridge [1], although an older document states it is $34.00 with one cartridge [2]. | yes |
The quote check
def norm(s):
return re.sub(r"\s+", " ", s.replace("’", "'").replace("“", '"').replace("”", '"')).strip().lower()
def check_quotes(citations, passages):
by_id = {p["id"]: p for p in passages}
for c in citations:
p = by_id.get(c["id"])
c["verified"] = bool(p and c["quote"].strip() and norm(c["quote"]) in norm(p["text"]))
if p:
c.update(file=p["file"], title=p["title"], section=p["section"], updated=p["updated"])
return citations