Stopping an LLM From Narrating Changes It Never Made

The rewrite deleted the gate, the advisor/worker split, delegate_to_worker, final_response and report_blocker. All of them existed only to compensate for the split. What's left is one loop:

for iteration in range(max_iterations):              # 25 by default
    reply = await stream(model, transcript, tools, tool_choice="auto")
    transcript.append(reply)                 # text AND tool_use blocks
    if not reply.tool_uses:
        verdict = await guard.check(draft, calls, state)
        if verdict is Correct and corrections_left:
            transcript.append(verdict.feedback)   # go round once more
            continue
        break                                     # the only exit
    results = await run_tools(reply.tool_uses)    # a transaction per call
    results.append(turn_report(calls))            # facts, every iteration
    transcript.append(results)

Three properties do the work. The only way out is the model returning no tool calls. The transcript is flat and replayed for real: stored turns keep their actual tool_use and tool_result blocks (and compaction folds whole turns, so a summary never lands between a call and its result), which means the next turn sees that a write failed instead of a paraphrase of the chat. And tool_choice is back to auto: the model can ask a question whenever it needs to, because safety no longer depends on forcing its hand.

The loop now lives in a small package with no imports from the app. Everything domain-specific arrives through ports, and a test enforces that boundary with an AST scan.

Every mutating CV tool has a verifier that asks the database whether the write happened, using only the ids the tool says it touched:

async def verify_cv_write(tool_name, ctx, parsed, result):
    noun, model = _entity_for(tool_name)      # "create_skills" → Skill
    ids = _ids_from(result)                   # any "*_ids" list it returned
    found = (await ctx.db.execute(
        select(func.count()).select_from(model)
        .where(model.id.in_(ids), model.user_id == ctx.user.id)
    )).scalar()
    if tool_name.startswith("delete_"):
        ok = found == 0
    else:
        ok = found == len(ids)
    return {"ok": ok, "entity": noun, "ids": ids}   # + "error" if not ok

The verdict is folded into the tool result under a reserved _verification key, so it can never overwrite a field the tool returned. If it says ok: false, the loop treats the call as failed: the model reads an error in the same turn and can fix it before it writes a word to the user. The check is deliberately narrow (existence and ownership, not field values) because existence is what failed in production, and re-reading the whole CV after each call would cost a full fetch per write and ride along in every later iteration.

After every round of tool calls, the loop appends a report to the same message that carries the results:

<turn_report>
changes this turn: 1
  - update_profile: applied
  - create_experiences: FAILED — only 0 of 1 experiences are present in the database
This is what the database holds. Your reply must not describe anything
above as done that is not listed as applied.
</turn_report>

Two decisions matter more than the wording. The report rides on the tool-results message instead of arriving as its own, so the model reads the facts on the same pass where it decides what to say, and the transcript never has two user messages in a row, which not every provider accepts. And it appears every time, in the same shape, “changes this turn: none” included. A warning that appears only when something looks wrong is a signal, and models react to signals, often by calling a tool to make the warning go away. A constant carries no signal. It's just the state of the world.

A turn that calls no tool never gets an in-turn report, which is exactly the shape of the original bug. So each turn also opens with a report of what the previous turn changed, rendered from the stored audit trail. A bare “saved it” with nothing behind it gets confronted one turn later, for free.

The placement cost us one nasty bug. The OpenAI-compatible encoder emitted the report's text between the assistant's tool-call message and the tool messages, an ordering the API rejects with a 400. Because history replays, every later turn of that chat failed the same way. The stored rows were fine; the translation wasn't. Tool messages now go out first, and the scripted test providers run every call through the real encoder.

The original transcript had a specific shape: the agent asked for a missing year, the user answered on the next turn, and nothing in the system knew a create was pending. The obvious fix, giving the model a tool to record what it's collecting, has the weakness of every “the model must remember to” rule.

So the draft is derived. A mutating call that fails on missing input (a validation error such as “Field required”) automatically becomes a pending entry in the conversation's state, and a later success on the same section clears it. The model never maintains it; it is shown a <pending> block at the top of every call. That state drives the one guard that runs when the model tries to finish:

async def check(self, ctx, draft):
    if draft.had_successful_mutation:
        return ACCEPT
    pending = (ctx.state or {}).get("pending_entry")
    if not pending:
        return ACCEPT                 # nothing started, nothing to misreport
    if any(c.mutating for c in draft.calls):
        return ACCEPT                 # it tried, failed, and has the error
    if pending.get("corrected_once"):
        return ACCEPT                 # one confrontation per draft
    await self._mark_corrected(ctx, pending)
    return Correct(
        StateReport.render(draft.calls)
        + "<unfinished>" + json.dumps(pending)
        + "This entry is still not in the CV. Nothing was written this "
          "turn. Either call the tool that writes it now, ask the user "
          "for what is missing, or — if the user cancelled this entry — "
          "call `discard_pending_entry`. Do not describe it as saved."
        + "</unfinished>")

Each condition was earned. The guard never fires on a turn that attempted a write: that turn already has the error and the report in front of it, and in practice answers honestly. It fires at most once per draft because, measured live, a pending draft plus a chatty user meant a correction round-trip on every turn, about 12,500 extra input tokens each time. When the user cancels, the model calls discard_pending_entry, which has a read-after-write verifier like every other mutating tool. And when the guard does fire, the text the user already saw stays on screen; the correction arrives as a visible continuation, so the stored reply equals what was streamed, character for character.