Fixing a Hallucinating AI Chatbot by Deleting Most of It

The numbers

A note before we start: we broke our own rule

In April we published The Case Against Complex AI Pipelines, written from this same build. It argued for one controller, one model and one handoff rule.

Then the client's operating manual went from v2.0 to v4.4. Each version added rules and reversed some earlier ones, and we answered every new rule by adding a layer. By July the "simple" system had a classifier, a nine-mode router, a judge panel and 29 Python modules.

This article is what it cost us to learn our own advice twice.

The product

The AI answers Instagram DMs in the coach's voice and books qualified women onto a free call with her team. It runs on FastAPI, receives messages through ManyChat, and replies in English or Spanish with a human typing delay.

It sounds like a chatbot job. It is closer to a sales call, a support line and a crisis line sharing one inbox:

• The people are vulnerable. Women write after failed IVF, a miscarriage last week, or a doctor telling them donor eggs are the only option.
• The stakes are medical. The coach is not a doctor. The AI must never dose a supplement, read a lab result or promise a pregnancy.
• The business rules are strict. There are age limits, partner attendance on the call, price ranges, rules for which link goes out when, and rules for when a person takes over.
• The voice has to pass as human. One em dash, one "I hear you" or one recycled opening line, and the lead knows it's a bot.
• The rulebook keeps moving. Every manual version reversed some earlier ruling.

A conversation has four possible shapes:

1. It closes: thanks, pregnancy news, or she has stopped trying.
2. It gets a direct answer.
3. It gets nurtured without any selling.
4. It moves into qualification and the booking link.

Most of the failures below come from forcing every conversation into the fourth shape.

What "broken" looked like

Every failure here is real, taken from commit messages and client test rounds between July and October 2026.

On 2 September the client ran her own stress test and sent back 18 points. Memory, boundaries and crisis routing were fine. The voice was dry and clinical, qualification overrode the conversation, and the positioning was generic. She asked for an architecture diagnosis before any more patching.

The last two rows matter. The simplified brain wasn't perfect on day one either. The difference is that each of its mistakes had one obvious cause and was fixed within a day.

Why it hallucinated: six causes, in order of damage

1. The state was a checklist. The July brains stored the lead as slots: age, partner, priority, budget. Each turn's job was to fill the next empty one. Any answer the code had no branch for, like "very high" instead of a number, left the slot empty, and an empty slot meant ask again. That is the yes loop. Patching it means one more branch per phrasing, forever.

2. One funnel for every conversation. Anything in scope ended in a single booking waterfall, so a pregnancy announcement was treated as a lead with missing qualification data.

3. Too many words pulling in different directions. By mid-September the writer read eight layered prompt files, about 14,500 words. The reader prompt alone was 8,486 words. Each client round added rules on top, the rules contradicted each other, and the model resolved the conflicts unpredictably.

4. Example selection fed it the wrong examples. 64 example files were tagged and scored, and a few were picked per turn. The picker misfired:

• The failed-IVF example fired for a woman preparing for her first cycle.
• The partner-hesitation example fired on the word "afford".
• The social-proof file was mostly refusals, so whenever it won, it taught the AI to decline.

5. Counter-examples taught the mistake. Files had "DO NOT WRITE THIS" sections that spelled out the forbidden reply. Models copy text they are shown, including text labelled wrong. (We'd seen a version of this before. It's why we wrote Rules Don't Stick. Examples Do.)

6. Output policing made it worse. Regex backstops, banned-phrase lists and a panel of LLM judges inspected each reply. A failed check retried the same prompt at a lower temperature, which is a coin flip, not a correction. A quote checker only looked at her latest message, so a real quote from earlier in the thread was marked "fabricated". Uncertainty spiked, and the woman who had given up got silence.

A seventh cause sat outside the brain. 541 green tests built their data as Python fixtures, while the live database was five migrations behind with zero knowledge rows. The tests were green and the app knew nothing.

Seven brains in seven months

The repo has 153 commits from 4 March to 27 October 2026 and at least seven rewrites of the brain. Five of them fixed the previous version's visible failures by adding a layer. Only the last two fixed things by removing one.

What each stage taught us:

• Funnel brain: a business process is not a conversation. Qualification has to sit underneath the chat, not in front of it.
• Routed brain: more classification doesn't mean more understanding. Each new label was one more thing to get wrong.
• The reset: deleting 6,747 lines felt like losing a month. It was the most productive commit of the project.
• Brain v2: the three-stage shape was right. The weight inside each stage was wrong.
• Brain v3: the model already knew how to be warm, brief and honest. Our prompts had been stopping it.

The week that changed it

The turnaround was three commits:

• 28 September: the 64-file library became 9 complete conversations, rewritten from the coach's own line-by-line edits.
• 6 October: the reader was cut to the facts the gate actually needs, and the age and partner questions moved to just before the link.
• 8 October: the writer moved from eight layered files to one plain prompt file, plus a short knowledge file and per-turn notes. The old prompt folder was deleted the next day.

The architecture that works: read, gate, write

The working brain does three small jobs, and only two of them use an LLM. A reader writes down facts. Plain Python decides what the writer is allowed to see. A writer replies with the right examples in front of it. Nothing checks the reply after it's written.

In plain words

Imagine a new assistant answering the coach's DMs.

1. One person reads the conversation and fills in a short card: her age if she said it, her treatments, whether she asked a question, whether she sounds in danger.
2. A manager looks at the card and decides what the assistant may say this time. Is the booking link allowed? Can the price be mentioned? Must a human take over?
3. The assistant writes, after reading a dozen real conversations the coach approved.

The assistant cannot send a link it was never given. That one idea replaced most of the old safety code.

Stage 1: the reader

A small reasoning model (gpt-5-mini) reads the whole transcript and returns JSON:

• which of the four paths the conversation is on
• her intent
• her language
• the exact question she asked
• typed facts like age and IVF history
• safety flags

It writes nothing the lead sees and decides nothing. Two narrow second looks back it up. One is a safety read of her latest message. The other is a language check that runs only when the reader detects Spanish or an unsupported language, because that answer can end a conversation silently.

Stage 2: the dossier and the gate

Python merges the facts into a running dossier: what she has already said. The dossier goes back into the writer's prompt. Memory is an explicit record, not something the model re-derives from twenty messages every turn.

The gate then compares typed values and nothing else. There is no regex on her prose and no inspection of generated text. The gate decides:

• whether the booking link, the price, the masterclass or the post-booking video go into the prompt this turn
• whether to hand over to a person, and which fixed line she gets (crisis, urgent medical, medication request, or the standard review line)
• whether she's out of scope, which gets one approved line

On a handover turn, the writer isn't called at all, so it can't answer the question the system just decided it shouldn't answer.

Stage 3: the writer

gpt-4.1-mini writes in the coach's first-person voice. Its prompt has four parts:

1. one voice file: who she is and how she talks
2. a knowledge file, containing only the blocks the gate opened
3. the dossier
4. every example conversation

A short "THIS TURN" note sits last, for example: "She asked X. Answer it in this message, before anything else."

The examples

There are twelve complete conversations, each running from first message to final outcome. They cover qualification, direct answers, nurturing, money, age review, pregnancy and endings. They have no tags, no commentary and no counter-examples, and all of them go in on every turn.

Whole conversations teach what fragments can't: pacing, when to stop asking, and that some conversations correctly end with no link at all. Where the gate hasn't opened a link, its placeholder becomes "[link not available this turn]". The writer still sees the shape of the conversation, without a URL it isn't allowed to send.

Old example files opened with scoring metadata, a "WHY THIS WORKS" list, several arcs and sometimes a "DO NOT WRITE THIS" block. New ones are a heading and a transcript:

## 36, already told me everything, ready to pay, books, then gives her email

Lead: hi, im 36, we've been trying naturally for 3 years, i have pcos and my
husband's sperm analysis was normal. i'm ready to invest whatever it takes. how do i join?

Coach: Love that you're ready! The first step is a free call with my team to look at
what you need and which level fits. Here's the link:

{{booking_link}}

The gate in code

This is the whole mechanism for withholding a link. There's no output filter behind it.

# Link placeholder -> the knowledge block that must be open this turn for it to be shown.
_LINK_BLOCKS = {
    "{{booking_link}}": "booking",
    "{{masterclass_link}}": "free_resource",
    "{{replay_link}}": "post_booking",
}
_WITHHELD = "[link not available this turn]"

def render(self, *, allowed_blocks: set[str]) -> str:
    body = self.text
    for placeholder, block in _LINK_BLOCKS.items():
        if block not in allowed_blocks:
            body = body.replace(placeholder, _WITHHELD)
    return f"### EXAMPLE: {self.name}\n\n{body.strip()}"

One turn, end to end

1. Batch and wait. The webhook collects her burst of messages, then hands the whole history to the brain.
2. One-word opener. If her first message is a single word ("price?", "STOP", "hi"), one small call decides what it means and writes the acknowledgment. The reader and writer are skipped.
3. Read. The reader returns JSON, normalised against fixed enums (22 intents, 4 paths). If the reader throws an exception, the lead is paused and tagged for a human rather than the system guessing.
4. Safety look. Any flag it raises is OR-ed into the extraction.
5. Language look. Runs only for Spanish or unsupported languages.
6. Merge. Scalar slots like age replace the old value. List slots like diagnoses accumulate.
7. Gate. Returns the allowed knowledge blocks, booking allowed or not with a reason, price mode, escalation reason, handover line, or silence.
8. Handover short-circuit. If the gate escalates, she gets one fixed line, the human-review tag is applied, and the AI pauses. The writer never runs.
9. Write. The prompt is assembled from the voice file, the opened knowledge blocks, the dossier, all twelve examples and the turn notes. Opening lines recently used with other leads are listed, so the writer doesn't reuse them.
10. Clean and send. Dashes become punctuation, quantities become digits, and the reply is split into DM-sized bubbles and sent with a typing delay.

Where each kind of rule lives

The rule of the repo: behaviour lives in prompts and examples, not Python. Python only decides what the writer is given.

Models and cost

The reader runs on gpt-5-mini at low reasoning effort. It answers about twenty questions over a whole transcript, and handover flags degrade in long context without some reasoning. The writer runs on gpt-4.1-mini, because voice gains little from thinking and this model keeps temperature control. Both are switchable from an admin panel, and every turn records its token cost.

The routed brain made up to six calls per turn. The current one makes three, or four in Spanish, and none of them grade the output.

Results

The suite is 58 scripted conversations, about 140 lead turns, written from the client's own manual. It covers:

• program basics
• prices and payments
• value objections
• age review
• partner attendance
• pregnancy
• grief
• Spanish
• AI disclosure
• medical boundaries
• out-of-scope cases

Each conversation gets a pass, partial or fail and a one-line comment, judged by a person, not a model.

The dips came from new manual rules landing, such as a partner-attendance exception in v4.3. They weren't regressions. Each was fixed in a prompt or an example the same day and rerun.

What changed for the people in the DMs

• Grief gets a human answer. "I have given up" is acknowledged warmly and the conversation is allowed to end, with no silence and no sales question.
• News gets celebrated. A pregnancy announcement gets congratulations and an ending, not qualification.
• Questions get answered first, including when the honest answer is "you may not need me yet."
• Age and partner come last. They're asked once, right before the link, as part of booking rather than as an opening test.
• It's honest about being AI. Asked "is this a bot?", it says it's the coach's AI assistant and offers a person.
• No invented program details. Facts come only from the knowledge blocks the gate opened.

What changed for the team

• Fixes take minutes. A new client rule is usually an edit to one prompt file or one example conversation, not a Python deploy.
• Every turn is explainable. The trace stores the reader's JSON, the gate's decision and reason, and the cost, so a bad reply points straight at its cause.
• There's less code to break. 5,945 lines became 2,054.

The principles we now build by

Every one of these was learned by breaking it first.

1. Control what the model is given, not what it produces. One gate before the call replaced a check panel, regex backstops and retries after it.
2. Whole conversations beat clever fragments. Twelve transcripts teach pacing, restraint and endings. Sixty-four tagged snippets taught whichever one the picker grabbed.
3. Send every example, every turn. Selection logic is a second model of the conversation, and it was wrong more often than the examples were.
4. Never write out the wrong answer. A "DO NOT WRITE THIS" block is still text the model reads. Describe the shape of the mistake instead.
5. Delete the line that teaches the bug. Before fixing a reply, rebuild the writer's exact input and find the sentence that taught it. Adding a "don't do X" rule on top makes the prompt heavier and the conflict worse.
6. Measure prompts in words. Report the word count with every prompt change. If it only goes up, something is wrong.
7. Memory is a record, not a reconstruction. A typed dossier beats asking a model to re-derive her age from twenty messages.
8. Extraction and writing are different jobs. Use a reasoning model for reading and a fast model for voice.
9. A handover means the writer doesn't run.
10. Silence is a reply. A grieving woman who gets nothing has been answered, badly.
11. Test the thing that's running. Acceptance runs go against the deployed build, and the results page shows which build.
12. Happy path first. Make the common conversation boringly good, then handle the common detours, then the edge cases. Leading with edge cases is how we built the routed brain.
13. Be willing to delete a month.

Here's the opinion most AI vendors won't put in writing: for customer-facing chat, output guardrails are mostly theatre. If a reply needs a judge to catch it, the information that caused it shouldn't have been in the prompt.

Frequently Asked Questions

Why does my AI chatbot hallucinate even with a good model?

Usually because of what surrounds the model, not the model itself. Long layered prompts with conflicting rules, wrongly selected examples, and counter-examples that spell out bad replies all push it toward invented or copied answers. In our case, cutting prompt and example text by 87% fixed problems that months of extra rules had not.

Should I use few-shot examples or written rules to control an LLM's behaviour?

Use examples, and make them whole conversations rather than fragments. Rules conflict as they accumulate, while complete transcripts teach pacing, tone and when to stop. Send every example on every turn instead of selecting them per message, because selection logic misfires.

How do you stop an AI chatbot from sending links or prices it shouldn't?

Don't put them in the prompt. Have plain code decide, before the model is called, which links and facts are allowed this turn, and replace withheld ones with a placeholder. A model cannot leak information it was never given, so no output filter or retry loop is needed.

Are LLM-as-judge output checks worth it for a production chatbot?

For live conversational replies, in our experience, no. Retrying a failed reply at a lower temperature is a coin flip, not a correction, and narrow checkers create false failures that lead to worse outcomes, including silence. Gate the inputs instead, and use human-judged acceptance tests to catch regressions.

How should you test a conversational AI before going live?

Run a scripted acceptance suite against the deployed build, judged by a person. Ours is 58 conversations written from the client's own rules, each graded pass, partial or fail. Unit tests with fixture data can all pass while production is broken; ours did, with an empty knowledge database.