Why our renderer deletes what the model wrote

Ask any support agent built on a language model why it reached a conclusion and you will get a fluent answer. That is the problem. Prose fluency is uncorrelated with whether the system actually checked anything, and an engineer reading an escalation cannot tell the two apart by looking.

So we stopped trying to make the model reliable at citing, and moved the guarantee somewhere a model cannot reach.

The invariant

Every claim in a diagnosis carries one or more probe_run_ids. The renderer drops any claim whose probe runs do not exist in that session, and counts the drop.

Not "should cite". Cannot ship without. The renderer is the enforcement point because it is the last thing between a model's output and a human's inbox.

The shape

A diagnosis session can only end one way, through a strict-schema tool with additionalProperties: false:

type Claim = {
  text: string            // one sentence, no hedging, no internal ids
  probe_run_ids: number[] // non-empty, and must exist in this session
}

There is no path where the model writes a conclusion outside this shape. It cannot return prose. It cannot return a claim with an empty citation array and an apology. The only exit from the session is a structured object whose citations are checkable against a table.

What the renderer does

1. Drop any claim with an empty probe_run_ids.
2. Drop any claim citing an id absent from probe_runs for this session_id.
3. Increment hallucinated_claims on every drop, tagged with session, runbook version, model and effort.
4. If dropping leaves fewer than one claim, downgrade the diagnosis to inconclusive. A diagnosis with no surviving evidence is not a diagnosis.
5. Never render a control-plane field marked dead in the tenant's field trust map, and mark stale fields inline with the reason.
6. Render probe output verbatim in the evidence block.

Rule 6 is the one people push back on, because verbatim output is ugly next to a clean summary. It stays because the summary is for the reader's first two seconds and the raw is what they read when they do not believe the summary. A system that only ships the pretty version has quietly asked to be trusted, which is the thing we are trying not to do.

Rule 4 matters more than it looks. The tempting behaviour when evidence collapses is to keep the best-sounding claim and ship a hedge. That converts a loud failure into a quiet one, and quiet failures are how a team learns to skim.

The number we gate on

hallucinated_claim_rate = dropped_claims / total_claims

Target below 0.5%, tracked split by model, effort level and runbook version.

This is a release gate, not a dashboard tile. If it rises, feature work stops until it comes down, because every other metric is then measuring a system whose outputs cannot be trusted. It is also the fastest signal available that a model swap or a prompt edit was a bad idea: the rate moves before anything a human would notice does.

Worth being precise about what the metric counts. A drop is not the model lying. It is the model asserting something the session cannot substantiate, which includes the boring case of citing a probe that timed out. We would rather over-count than build a definition with an escape hatch in it.

What this does not solve

The contract guarantees a claim is grounded in a probe that ran. It does not guarantee:

• that the probe measured the right thing. Wrong hostname, wrong tenant, warm cache. A perfectly cited claim can be about the wrong system entirely. That is the differential test's job, and it is why a control comparison is mandatory rather than optional.
• that the inference from probe to conclusion holds. "TLS handshake succeeded" and "TLS is not the cause" are different statements, and the gap between them is where a confident wrong answer lives. That is a separate adversarial verification pass.

Three defenses against three failures. The reason to keep them separate is that each catches something the others structurally cannot, and collapsing them into "the model checks its work" gets you none of the three.

Why enforce it in code

Every part of this could have been a prompt instruction, and prompt instructions for citation discipline do work most of the time. Most of the time is the problem. The failure mode is not gradual: the agent probes the wrong hostname, gets a plausible error, and produces a conclusion that reads exactly like a well-evidenced one. There is no textual signal to catch.

The moment one claim in one packet turns out to be invented, every packet becomes something an engineer verifies by hand. A packet you verify by hand is worth less than the raw ticket, because it costs you the reading time too. The whole economic argument for handing triage to an agent depends on the output being trustworthy without re-derivation, and "the model is usually careful" is not a foundation for that.

A join against a table is.