Memory, Self-Critique & Reliability

A stateless model starts every conversation from nothing: it has no record of what was decided last week and no obligation to be consistent with it. Memory fixes that. It says nothing, on its own, about whether the remembered content is true.

Worse: memory makes error persistent. An incorrect inference written into a long-term store becomes a premise for every later decision that retrieves it. The reliability question is therefore not only 'does it remember' but 'what does it do when the memory is wrong'.

Retrieval-augmented generation pairs a model with a search step over an external corpus: the query is used to fetch passages, and the answer is conditioned on them. The model stops relying on parametric memory for facts it can look up — which is more accurate and far more auditable, because the retrieved passage can be shown.

• Retrieval quality dominates answer quality. A perfect generator over the wrong passage produces a confident wrong answer.
• Index freshness is an operational obligation, not a one-off engineering task.
• Access control must be enforced at retrieval: the model can only cite what the user was allowed to read.

• Contamination: an unverified claim is stored as fact, then retrieved as established context.
• Supersession: a fact is updated in the world but not in the store; the model answers with the old version confidently.
• Duplication: the same conclusion stored with slight variations, one of which is wrong, with no way to tell which.
• Prompt injection: retrieved text that contains instructions, which the model may follow as if they came from the operator.

Mitigations are organisational as much as technical: a write gate with an owner, provenance on every entry, a supersession chain instead of overwrites, and a periodic audit that samples stored claims and re-checks them against sources.

1. 01A retrieved passage is misread, producing a wrong intermediate conclusion.
2. 02The conclusion is written to memory and becomes context for the next task.
3. 03Subsequent answers are internally consistent with the error, so they look more credible, not less.
4. 04Downstream summaries compress the error into a short, confident statement.
5. 05The original source is no longer consulted, so the error becomes the record.

Research on calibration found that larger models can be trained to predict whether their own answers are correct, and that this signal is reasonably informative — but systematically overconfident on hard items. Treat stated confidence as a ranking signal for human review, not as a probability of correctness.

• Useful: routing low-confidence items to a human or to a stronger verification path.
• Not useful: as a substitute for verification, because the failure mode is confident error.

Reflexion-style loops store a verbal critique and retry, which improves outcomes on tasks with verifiable rewards. But work explicitly testing intrinsic self-correction found that models asked to revise their own reasoning without external feedback often make correct answers wrong — the revision is not anchored to anything.

• Expiry by class: operational traces expire in days; decisions in quarters; definitions persist until superseded.
• Supersession over deletion: keep the chain so a decision can be explained later.
• Catastrophic interference: updating a model on new data can erase old competence, a documented property of neural networks.
• Deletion must be real: an entry removed from the index but retained in an embedding store is not forgotten.

1. 01Collect a labelled evaluation set from our own domain — not a public benchmark.
2. 02Have the system state confidence before the answer is revealed.
3. 03Bucket by stated confidence and compute observed accuracy per bucket.
4. 04Where confidence exceeds accuracy, treat that band as requiring verification.

This produces something more useful than an accuracy number: a policy. Below a threshold, the system defers; above it, it acts with a recorded justification; in a middle band, a second independent check runs.

Self-consistency samples multiple reasoning paths and takes the majority answer, which materially improves accuracy on tasks with discrete answers. Multi-agent debate extends this: agents critique one another's reasoning across rounds, and published results report improved factuality and reasoning on several benchmarks.

• Independence is the whole value: agents sharing one retrieval context share one failure.
• Cost scales linearly or worse; benefit does not. Measure the delta, not the architecture.
• Use disagreement as an escalation trigger to a human, not as a vote to be overruled.

1. 01Detect: a periodic audit re-checks a sample of stored claims against their sources and flags drift.
2. 02Quarantine: flagged entries stop being retrieved for decisions while remaining visible for investigation.
3. 03Trace: follow the dependency chain forward to every derived conclusion that cited the entry.
4. 04Correct at the root: write a superseding entry with a source and a timestamp; never silently overwrite.
5. 05Recompute: regenerate affected outputs and re-test them against the evaluation set.
6. 06Prevent: add the case to the write gate's checks so the same class of error cannot be stored again.

It is accurate to say a model has a persistent erroneous memory entry that biases later outputs. It is not accurate to say the system is 'traumatised' by an earlier interaction. The first describes a data condition with an engineering remedy; the second attributes an inner life that has no operational meaning and obscures the fix.