// 2026-04-22[ AGENTS ]9 min

Notes on building data agents that don't lie.

How we wire retrieval, evals, and human review so agents stay tethered to source data, and how we know when they're drifting.

A data agent lies the same way a tired analyst does: it stops looking things up and starts remembering. The model has seen a million quarterly summaries, so when you ask it to summarize yours, it produces something quarterly-summary-shaped. Most of the numbers are right because most of them came from the retrieval step. One or two are plausible inventions, and nothing in the output tells you which is which.

We build agents that read support transcripts, warehouse tables, and survey text for production teams. Our support-analytics deployment reads 100% of interactions and its numbers feed CSAT reporting that leadership acts on, so "mostly right" was never an acceptable spec. These are the mechanics we now install by default.

Confabulation is a summarization problem

The failure mode is specific. When an agent cites, it copies a value out of a retrieved row and the value is as good as the retrieval. When an agent summarizes, it compresses retrieved context through its own weights, and the weights contain every plausible-sounding number the model has ever read. Compression is where invention happens.

You cannot prompt this away. "Only use the provided context" reduces the rate; it does not bound it. The only bound we have found that holds is structural: make the agent emit claims in a format where an uncited claim is a schema violation, then reject the output mechanically before a human ever sees it.

Citation contracts

Every factual claim the agent makes must carry a pointer that our UI can resolve back to source data. Not a footnote, not a "sources" list at the bottom. A pointer per claim, validated at generation time. We call this the citation contract, and the agent's answer is structured output against it:

{
  "claim": "Refund-related tickets rose 18% week over week",
  "kind": "aggregate",
  "source": {
    "type": "sql",
    "query_id": "q_7f3a",            // logged, replayable
    "table": "analytics.ticket_daily",
    "row_ids": ["2026-04-06..2026-04-12"],
    "column": "refund_ticket_count"
  },
  "verbatim": false,
  "confidence": "computed"           // computed | retrieved | inferred
}

Three properties matter. The pointer is resolvable: the UI renders every claim as a link, and clicking it shows the rows or the document span. The pointer is replayable: query_id references a logged query we can rerun during evals. And confidence: "inferred" is an honest escape hatch: the agent may reason beyond the data, but it has to label the step, and our renderer styles inferred claims differently. A claim with no source object fails validation and the whole answer is retried or refused.

The side effect we did not expect: citation contracts make review fast. A human checking an answer clicks three pointers in thirty seconds instead of re-deriving the analysis. That is what made 100%-coverage QA affordable; on the support-analytics build it cut manual QA hours by 60%.

SQL-grounded answering beats free-text RAG for numbers

For anything quantitative we do not let the agent read rows and add them up in its head. The agent writes SQL, we execute it, and the answer is the query result. The model's job shrinks to translation and narration, which are the parts it is actually good at. Arithmetic errors drop to zero because the model never does arithmetic.

Free-text RAG still earns its place for qualitative questions: why customers churned, what a contract clause says, what themes run through open-ended survey responses. When we compressed 80,000 survey responses into a 6-page brief, the theme extraction ran over retrieved chunks, but every count in that brief ("412 respondents mentioned pricing") came from a query, not from the model counting. The rule we give teams: the model narrates; the database counts.

The two modes compose. A typical answer in our stack interleaves SQL-sourced aggregates with document-sourced quotes, and the citation contract records which mode produced each claim, so an eval can check them differently.

Eval gates: what an agent must prove before it may act

We split agent capabilities into two tiers. Read/suggest: query data, draft an answer, propose an action. Write/act: send the email, update the record, file the ticket. An agent earns the second tier per action type, by passing a gate, and the gate runs on every loop iteration in production rather than once at ship time.

fig 1 — the agent loop as we ship it. the eval gate sits between reason and act on every iteration; failed or low-confidence outputs route to human review, and corrections feed back into the reasoning step.

The gate itself is unglamorous: a battery of assertions over the structured answer. The workhorse assertion replays cited queries and compares:

def test_claims_are_tethered(answer):
    for claim in answer.claims:
        assert claim.source is not None, "uncited claim"
        if claim.source.type == "sql":
            replayed = warehouse.replay(claim.source.query_id)
            assert claim_matches(claim.value, replayed), (
                f"claim '{claim.text}' does not match replayed result"
            )
        if claim.verbatim:
            assert claim.text in resolve(claim.source), "fabricated quote"

Before an agent gets write access, it must clear the gate on a golden set of a few hundred curated tasks at a threshold we agree with the client up front, typically far stricter than what we hold read-only answers to. Until then it runs in suggest mode: same reasoning, same output, but a human clicks the button. Most of our deployments spend their first weeks in suggest mode on purpose, because that period generates the review data that calibrates the gate.

Refusal is a feature

An agent that answers every question is lying some of the time. We budget for refusal explicitly: when retrieval comes back thin, when the replayed query disagrees with the drafted claim, or when the question asks for data that does not exist, the correct output is "I can't support an answer from the data I have," plus what it looked at. Our evals score these refusals as passes. Teams initially hate this, and then the first time the agent declines to invent a churn number for a segment with nine rows, they stop hating it.

The design detail that makes refusal usable: a refusal still carries citations. It shows the queries it ran and the empty or thin results, so the human can immediately see whether the gap is in the data or in the retrieval.

Knowing when it drifts

Agents that pass at ship time degrade quietly. Schemas migrate, ticket taxonomies get renamed, a model provider silently updates weights, prompt edits accumulate. We assume drift and instrument for it two ways.

Golden sets replayed nightly. Every deployment carries a versioned set of question-answer pairs with known-correct citations. A scheduled job replays the full set against production infrastructure every night and diffs the results: pass rate, refusal rate, citation-resolution rate. A two-point drop pages us before any user notices. Because our sovereign-infrastructure stack serves open-weights models (Qwen3.5 fleets at under $0.40 per million tokens blended), the nightly replay costs almost nothing and the model under test cannot change without us changing it.

Tool-call distribution shifts. Golden sets only cover questions we thought to ask. For live traffic we watch the shape of the agent's behavior: queries issued per answer, retrieval hit rates, refusal rate, ratio of inferred to computed claims. These distributions are stable when the system is healthy. When the inferred-claim ratio creeps up, the agent is starting to summarize instead of cite, and that is drift toward confabulation even while the golden set still passes.

field note. the first drift our nightly replay caught was not the model. a warehouse migration renamed a column, retrieval went half-empty, and the agent kept answering fluently from thinner context. accuracy on the golden set fell 4 points; refusal rate should have risen and did not. the model was fine. the ground moved under it. instrument the ground.

Where humans stay

We do not staff humans as a parallel answer-checking layer; that reintroduces the cost the agent was meant to remove. Humans hold three specific positions in our deployments. They review everything the gate flags, so their attention lands only on the uncertain slice. They own the golden set, promoting real production questions into it weekly so the eval tracks what users actually ask. And they hold approval on every write-tier action until the per-action gate history earns automation, action type by action type.

On the support-analytics deployment this discipline is why the numbers held up: +22% CSAT and 35% faster resolution, sustained after the 14-week build ended, because the replay harness kept reporting whether the agent was still the same agent we shipped.

If you want to pressure-test this on your own stack, start with one afternoon of work: pick ten questions your team asks your data every week, write down the known-correct answers with the query or document that proves each one, and run them against whatever agent or copilot you currently use. Count the uncited claims. That list of ten is the seed of your golden set, and the count tells you exactly how much your agent is lying to you today.