Verify LLM answers against their source context before they reach users.
We cannot catch every hallucination — but dangerous hallucinations are specific, and specific claims can be matched as strings. So ClaimCheck checks only what it can check deterministically, and always tells you how much it did not check.
A blind retry re-sends the whole system prompt + context + question and regenerates everything. A targeted retry re-sends only the answer plus the specific failing claims. The difference is the estimate below. Run Verify first.
Structural repair only. The point is to never burn a retry on formatting. Fences, trailing commas, single quotes, raw newlines inside strings, fullwidth punctuation and Python literals are all fixed locally — retries are reserved for claims that are actually wrong.
coverageEvery result carries two independent numbers, and reporting one without the other is the most dangerous way to misuse this tool.
| field | question it answers |
|---|---|
grounding_score | Of the claims I checked, what share held up? |
coverage | What share of the answer did I check at all? |
grounding_score alone is a trap. An answer of pure vague hedging produces zero
claims and scores 1.00 on any naive groundedness metric — while being completely unverified.
A high score at low coverage means "I could not find anything to check", not "everything is fine".
| status | meaning |
|---|---|
| supported | Appears literally in the context after normalisation |
| derived | Computable from context numbers by 2–3 term arithmetic; the formula is shown |
| approximate | A near match exists — paraphrase, finer-grained date, related URL |
| unsupported | Nothing in the context backs it |
| contradicted | The context holds a near but different value |
approximate is deliberately not counted as grounded: a paraphrased quote is
still an inaccurate quote.
On the 158-case bilingual evaluation set:
answer-level recall 56/56 (100%) cases that should be flagged, and were clean answers passing 44/46 (96%) the rest were annotated, never blocked known false positives 22 (13.9%) known false negatives 3 (1.9%) JS/Python parity 158/158 (100%)
The false-positive rate is published deliberately. It is the number that decides whether anyone keeps a verifier switched on, and hiding it helps nobody.
All three false negatives are the same shape. A one-character change that reverses meaning keeps character similarity above 0.9, so a reversed quote reads as a faithful paraphrase:
| context | answer | similarity | status |
|---|---|---|---|
| 売上高は7%増加した | 売上高は7%減少した | 0.97 | approximate ❌ |
| 提案は全会一致で承認された | 提案は全会一致で否決された | 0.92 | approximate ❌ |
| revenue grew 8% | revenue declined 8% | 0.97 | approximate ❌ |
Character-level matching cannot detect negation or antonym substitution. If your domain turns on
approve/reject or increase/decrease, do not rely on quote status alone. In all three cases the
answer-level verdict was still retry, because the NUMERIC and ENTITY claims inside the quote are the
layer that catches what the quote check misses.
numeric_tolerance rather than lowering contradiction_rel.ENTITY is the noisiest check. Set {"enable_entity": false} in the policy box first
if you see too much noise, and start from annotate, never block.
ClaimCheck is one layer of defence, not a correctness guarantee.