This is part 5 of the Agent production system series. Previous: From Xbox SLAs to agent fleets. Constellation nodes: Eval gates, Search, Human approval.
Agents are fluent. Fluency is not the same as true.
Claim safety is the discipline of keeping numbers, titles, and authorship tied to evidence — especially when a model would rather sound complete than sound correct. It shows up in eval gates, in human approval, and in public artifacts like this site and Resume Builder.
If your agents can open PRs, draft blog posts, or rewrite a résumé, they can also industrialize exaggeration. The fix is not “write more carefully in the prompt.” The fix is treating unsupported claims the same way you treat a failing test: block or escalate, do not ship.
Why agents fail this by default
Language models optimize for plausible completion. Resumes, READMEs, and blog posts reward impressive completion. Put those incentives in the same loop and you get a predictable failure mode: the draft sounds confident, the metric feels right, and nobody can point at a source when asked.
The collision usually looks like this:
- Round numbers that feel right — a revenue figure rounded up, a latency win with no baseline, a “N agents in production” counter invented for rhythm
- Verb inflation — “led” where the honest word was “contributed,” “built” where the honest phrase was “extended and operates”
- Implied invention — treating an upstream open-source agent stack as if you authored it because you run it hard
- Metrics without a timestamp or source — numbers that cannot survive “show me the evidence”
- Present-tense title drift — “Senior TPM at Xbox” as a current claim when the true line is former
None of that requires malice. It requires a system that optimizes for sounding done. A production agent system that can publish must treat claim safety as a gate, not a writing preference.
Picture this: the same sentence, two outcomes
An agent drafts a line for a public page:
“I built a multi-agent framework used in production across dozens of teams, cutting publish time by 90%.”
That sentence is fluent. It is also a claim bomb if you cannot defend every clause:
| Fragment | What evidence would need to exist |
|---|---|
| “I built” | Original authorship of the framework, not integration work |
| “multi-agent framework” | A named system you actually own vs. one you extend and operate |
| “dozens of teams” | Countable org footprint you can stand behind |
| “90%” | Baseline, after state, and scope of the measurement |
In my system that draft does not get to “sounds good, ship it.” It hits a source-of-truth check. If the number is not on the allowlist, or the authorship verb overclaims, the path is amber or red — stop for a human, or rewrite inside evidence. Same muscle as Xbox publish windows: confidence is not a gate; checks are.
Rules I actually use
These are the same rules that bound About and this series. They live as a single public source of truth in the site repo (docs/claim-safe-facts.md) so agents and humans argue with a document, not with folklore:
- Verified figures only — e.g. 10+ years at Microsoft, ~$50M commerce program, 12h → 30m publish target, 20+ containers. No vanity counters without a source.
- Authorship humility — prefer “extended and operate” (or “integrated and operates”) for agent platforms; do not claim original invention of upstream open-source systems.
- Past-tense where true — former Xbox/Microsoft TPM, not a fictional current employer claim.
- No logo soup — stack claims match confidence (Azure and Docker yes; fashion-statement K8s/Terraform as confidence claims no).
- Public = stricter — if it is on davevoyles.com or a résumé, the bar is higher than a private scratch note.
- Stop and ask on disagreement — if two evidence sources conflict, do not silently pick the more impressive one.
The allowlist is deliberately short. A short list is a feature: it forces soft language (“high-stakes platform work,” “cross-team delivery”) where a hard number does not exist yet.
Evidence-backed tooling
Resume Builder exists because job search is a claim surface. AI-assisted editing without evidence discipline just industrializes exaggeration — faster bullet points, weaker truth.
The product idea is simple: role matching and language generation stay attached to what you can defend. Same philosophy as eval gates on code — different artifact. A tailored résumé that invents impact is not “optimized for ATS.” It is a future integrity problem wearing a productivity costume.
That tooling also makes the site series honest. The About metrics, the Xbox transfer post, and this post all pull from the same discipline: if a number is not allowed, it does not appear as a hard claim. Soft paraphrase is fine; inflation is not.
How this plugs into the fleet
Claim safety is not a single checkbox at the end. It is a job at every layer that touches language or numbers:
| Layer | Claim-safety job |
|---|---|
| Search / retrieval | Prefer sources over invention; cite when the system can |
| Coder / writer agents | Draft only within evidence; refuse to “complete” unsupported metrics |
| Eval gates | Block or flag unsupported metrics, present-tense employment drift, authorship theft |
| Human approval | Final call on public-facing claims that survive automation |
| Dashboards / site | Publish only what survived the funnel |
On the map: start at Eval gates and Search. Public claims also sit under Human approval for a reason — when the model is sure and the evidence is thin, a person owns the go/no-go.
Theater vs real claim safety
| Theater | Real claim safety |
|---|---|
| “Don’t make things up” in the system prompt | A source-of-truth document the agent cannot waive |
| Numbers that “sound about right” | Numbers on an allowlist with context |
| “I built X” for every integrated tool | “Extended and operates” / “integrated and operates” when true |
| Always-impressive résumé drafts | Drafts that fail closed when evidence is missing |
| Fixing one bad sentence after publish | Gates that catch the pattern before merge |
If your claim check never fails, you do not have claim safety — you have decoration. The tell is what happens when a draft wants a metric that is not on the list. A real system stops. A theatrical one rephrases until it sounds softer and ships anyway.
What this is not
- Not a ban on storytelling — anecdotes without invented metrics are fine
- Not a claim that personal ops equal Xbox global traffic
- Not permission to invent “N agents in production” or dollar figures outside the allowlist
- Not original authorship of every open-source stack I run hard
Claim safety is boring on purpose. Boring is how public trust survives contact with a fluent model.
Bottom line: impressive is easy; defensible is the product. If your agents can publish, your gates must be able to say “show me the source.”