This note examines an architectural pattern that recurs whenever language models are placed in front of consequential outputs: research systems that must convert large volumes of unstructured evidence into classifications, scores or rankings that people will act on.
Large language models are good at retrieval, extraction, structuring and interpretation. They are comparatively poor at owning a final judgment, because a model asked to produce an answer will generally produce one — qualified, hedged, cited, but generated nonetheless. The model is optimised to continue a plausible analytical sequence, not to determine whether the available evidence actually justifies a conclusion.
The pattern discussed here separates these two concerns. Language models assist with retrieval, extraction, structuring and interpretation of evidence. Consequential classifications, scores and rankings are produced by inspectable, reproducible software that a reviewer can rerun, audit and challenge independently of the model that helped assemble the evidence.
Five design principles recur across implementations of this pattern:
- extraction and judgment should be treated as separate functions, ideally owned by separate components;
- unknown, conflicting and insufficient evidence should remain explicit, first-class outcomes rather than being smoothed away;
- every claim that feeds a decision should be traceable to a reopenable source excerpt;
- domain-specific criteria should be expressed as inspectable rules rather than left to model interpretation;
- statistical or performance language should describe what a system actually does, not what it aspires to do.
None of these principles is exotic in isolation. What makes the pattern worth documenting is the discipline of keeping them consistently separated as a system scales — and being honest about what determinism does and does not buy once it is in place.
1. The architectural problem
Language models are increasingly used to process the kind of material that decision-relevant research depends on: filings, disclosures, news coverage, regulatory notices and unstructured web content. Their value there is genuine. They identify relevant passages, translate natural-language questions into research tasks, organise heterogeneous evidence and summarise volumes of text that would be impractical to read exhaustively by hand.
The difficulty begins when these same capabilities are extended from information processing to judgment — when the question changes from "can the system produce a ranking?" to "is the ranking produced under a methodology that is controlled and can be inspected after the fact?"
Those two questions are easy to conflate because a language model will happily answer both. It will summarise evidence and it will, if asked, also assign a score or a rank. Nothing in its output signals which of the two operations it just performed, or how repeatable the second one would be if the same evidence were presented again tomorrow, in a different order, with different phrasing.
The architectural response is to make that boundary explicit in the system rather than leaving it implicit in a prompt. Retrieval, extraction, structuring and interpretation are treated as generative, model-assisted work. Classification, scoring and ranking are treated as deterministic computation over a structured, validated record — computation that does not consult a model at inference time and that produces the same output from the same input every time it runs.
2. A separation of responsibilities
A useful way to describe the resulting pipeline is:
research question → structured signals → evidence → validation → confidence → deterministic score → ranking → report
Each stage has a distinct owner. A planning step converts a research question into a set of weighted analytical signals. Retrieval and extraction agents construct a subject profile, gather an initial evidence set and perform targeted gap-filling searches where coverage is thin. A consolidation stage merges these evidence sets, flags contradictions and estimates confidence for each item. The language-model-assisted portion of the pipeline ends at this point.
Everything downstream — classification, scoring, normalisation and ranking — is computed by deterministic services that consume the structured, validated evidence record and apply fixed rules to it.
This separation matters because extraction errors and judgment errors have different causes and need to be observable independently.
An extraction error typically looks like:
- failing to retrieve a source that materially changes the picture;
- misattributing a claim to the wrong subject;
- misreading a passage, including its qualifiers or scope;
- dropping contradictory evidence rather than reconciling it;
- treating a search snippet as though it were the full source.
A ranking error typically looks like:
- excessive sensitivity to evidence volume rather than evidence quality;
- weights that no longer reflect the intended priorities;
- opposing signals cancelling each other in a way that hides disagreement;
- unstable normalisation across comparison batches;
- penalty or gating rules applied incorrectly or inconsistently;
- a comparison cohort that is not actually comparable.
When extraction and ranking happen inside a single generative step, these two error classes blur together — a bad rank and a bad fact look identical from the outside. Making the ranking step deterministic does not, by itself, guarantee a correct answer. It does guarantee that the same structured inputs produce the same output, which is what makes regression testing, methodological review and after-the-fact audit possible at all.
3. Evidence as a typed record
A citation is not the same thing as support. Search snippets omit context, headlines simplify or exaggerate, paywalled pages can be discoverable without being inspectable, and a subject's name can appear on a generic registry or directory page without the page actually saying anything substantive about that subject.
To keep a citation from functioning as a purely symbolic marker of reliability, the pattern treats each piece of evidence as a typed record with an explicit chain:
claim → evidence record → source excerpt → source
A claim is a discrete, checkable statement. The evidence record links that claim to the excerpt that supports it and carries metadata about how the excerpt was retrieved and validated — a cited URL must belong to the set of sources actually retrieved, and the excerpt must be grounded in the retrieved content itself rather than reconstructed from a snippet or a headline. For longer documents, grounding can be checked with token-overlap or sequence-similarity thresholds between the claimed excerpt and the source text. Template language, fabricated references and excerpts that cannot be linked back to retrieved material are rejected outright.
Each evidence record also carries a confidence value, computed independently of the claim's polarity, from dimensions such as:
- source type;
- recency;
- corroboration by independent sources;
- identified conflict with other evidence.
Source type is usually the largest single contributor to confidence, and it is useful to make that hierarchy an explicit, inspectable table rather than an implicit judgment call buried in a prompt. A representative set of methodological defaults looks like this:
- Regulator, official filing or certifying authority — 1.00
- Public registry — 0.90
- Academic source — 0.85
- Independent oversight or standards body — 0.80
- Subject's own disclosures — 0.70
- Media coverage — 0.60
- General web source — 0.50
- Social media or unverified user content — 0.30
These numbers are methodological defaults, not empirical estimates of how truthful a given source category is on average. Their purpose is to make source treatment explicit, reviewable and configurable rather than to claim statistical precision they do not have.
This structure lets a reviewer reopen the basis of any downstream number: which claim, which excerpt, which source, at what confidence, and why.
4. Unknown as a valid state
Uncertainty should be structured information, not an absence of output. Each analytical signal in the pattern can resolve to one of five epistemic states:
- supported — evidence points consistently in one direction;
- contradicted — well-grounded evidence points against the expected direction;
- conflicting — credible evidence exists on multiple sides and does not resolve cleanly;
- insufficient evidence — some evidence exists but does not meet the bar for a confident finding;
- unknown — no meaningful evidence was found.
These states are not interchangeable, and treating them as if they were is one of the more common failure modes in scoring systems generally. A supported negative finding is not the same as missing information. Conflicting evidence is not the same as a neutral finding. Insufficient evidence does not mean "probably fine" — it means the question has not actually been answered.
Many scoring systems implicitly convert missing or ambiguous evidence into a moderate, middle-of-the-scale value, because a numeric field has to contain some number. The output then looks complete, but the completeness is manufactured rather than earned. The pattern described here instead treats an unknown or insufficient result as a legitimate, first-class output: the system returns the best-supported answer available rather than forcing an answer for every signal.
This has a direct consequence for ranking: evidence coverage has to remain visible as its own dimension, separate from the score itself, so that a subject with thin coverage is not silently indistinguishable from a subject that has been thoroughly researched and found genuinely neutral.
It also means three variables that are frequently merged into one need to stay separate throughout the pipeline:
- the apparent direction of a finding (its polarity);
- the strength of the finding, independent of direction;
- the confidence that the evidence actually supports the finding.
5. Deterministic classification and scoring
Once evidence has been validated and each signal has been assigned a state, direction, strength and confidence, the resulting record is handed to deterministic software for classification and scoring. No model call happens in this step. The same structured record produces the same score every time it is processed, which is the entire point of separating this stage from generation.
A simplified, illustrative form of a per-signal contribution looks like this:
signal contribution = evidence strength × confidence × polarity × policy weight
Here, evidence strength reflects how significant the underlying finding is, confidence reflects how well-grounded the supporting evidence is, polarity encodes direction (positive, negative or neutral), and policy weight encodes how much a given signal should matter for the question being asked, subject to an overall constraint that the set of weights sums to a fixed total.
This is deliberately a sketch rather than a specification. Real implementations add caps, floors, non-linear terms, sector- or category-specific priors and interaction effects between signals — details that are important for calibration but are secondary to the architectural point: the transformation from a validated evidence record to a numeric contribution is a fixed function, not a model inference, and it can be rerun, tested and audited independently of how the evidence was gathered.
Classification thresholds — the rules that turn a numeric result into a discrete category — are handled the same way. Category boundaries are defined in configuration, not inferred by a model at scoring time, and a subject's category is generally resolved before within-category ranking happens, so that a small numeric difference near a boundary cannot silently outrank a subject with a clearly better category.
6. Ranking without hiding the methodology
Turning per-signal contributions into a ranking introduces failure modes that are easy to gloss over if the methodology is not stated plainly. Three recur often enough to be worth naming explicitly.
6.1 Volume bias
A large number of weak, low-confidence signals can sum to a larger contribution than a small number of strong, well-grounded ones. Left uncorrected, this means the amount of available evidence — which is partly a function of how much has been written about a subject — can influence the score independently of evidentiary quality.
6.2 Cancellation
Positive and negative contributions sum arithmetically by default. A subject with substantial, genuinely conflicting evidence can therefore land on a moderate aggregate score that looks like "nothing much going on" when the underlying record actually shows significant disagreement. The conflict state described in section 4 has to remain visible outside the final number for this reason — the score alone cannot carry that information.
6.3 Cohort relativity
If scores are normalised against the extremes of whatever comparison batch happens to be assembled, the same subject can receive a different relative rank purely because the composition of the batch changed, with no change in the subject's own evidence. This limits how comparable a rank is across different runs, time periods or comparison sets, and it needs to be disclosed rather than presented as an absolute measure.
One way to reduce cohort-dependence and volume bias simultaneously is to dampen the influence of evidence count and map the result onto a fixed absolute scale rather than a batch-relative one. An illustrative, parameterised form — with k, midpoint, scale and s as configuration parameters rather than fixed production constants — might look like:
damping = min(1, evidence count / k)
absolute score = midpoint + scale × tanh(aggregate contribution / s)
The first line caps how much additional evidence volume can inflate a contribution once a count of roughly k independent items has been reached. The second maps the damped aggregate onto a bounded scale centred on midpoint, with scale controlling the width of the output range and s controlling how quickly the mapping saturates. The specific values of these parameters are a calibration question, not an architectural one, and any implementation of this idea should be validated before being treated as production-ready — including checking that terminology used to describe it (for example, avoiding "Bayesian" language unless the method actually maintains a posterior) matches what the code does.
None of this eliminates the underlying limitations; it relocates and bounds them. Volume bias, cancellation and cohort relativity should be treated as documented properties of the ranking methodology, not as bugs to be quietly patched out of view.
7. What determinism does and does not provide
It is worth being precise about what "deterministic" buys here, because it is easy to overstate.
Determinism provides reproducibility: the same structured, validated input produces the same output, every time, on every machine, indefinitely. It does not provide correctness — a deterministic function can faithfully implement a flawed formula, an outdated weight, or a biased prior, and it will do so consistently rather than randomly, which can make the error harder to notice, not easier.
Grounding (section 3) provides traceability: a reviewer can walk from a score back to the specific excerpt and source that produced it. It does not provide completeness — a well-grounded record can still be missing evidence that was never retrieved in the first place.
Confidence scoring provides a structured representation of uncertainty. It does not provide a statistically calibrated probability unless a calibration exercise has actually been performed against outcomes; treating an uncalibrated confidence value as a probability is a category error worth guarding against explicitly.
In short: determinism, grounding and confidence scoring together produce a system whose behaviour can be inspected, reproduced and challenged. They do not, on their own, produce a system whose outputs are guaranteed to be right.
8. Evaluation by pipeline stage
A correct-looking final output can conceal errors that happen to cancel each other out. An incorrect final output can originate at any of several earlier stages — retrieval, extraction, grounding, confidence estimation or ranking — and treating the final report as the only thing worth evaluating makes it impossible to tell which stage actually failed.
Evaluation should therefore be structured per stage rather than only end-to-end, measuring at minimum:
- retrieval precision and recall — did the system find the sources that matter?
- extraction precision and recall — were the claims in those sources captured correctly?
- grounding errors — were claims correctly attributed to the right subject and correctly anchored to source text?
- unsupported-claim generation — how often does the system state something the retrieved evidence does not actually support?
- conflict detection — does the system correctly flag genuinely contradictory evidence rather than averaging it away?
- confidence calibration — do stated confidence levels correspond to actual reliability when checked against outcomes?
- ranking stability — how much does a subject's rank move under small, non-substantive changes to the comparison batch?
Methodological changes to the pipeline — a new weighting scheme, a new confidence model, a new ranking method — should be gated behind this kind of stage-specific evaluation rather than shipped on the basis that the final output "looks reasonable." A sensible default is to fail closed: a change that has not passed the relevant evaluation does not go live, with any exception requiring an explicit, audited override rather than a silent default.
9. Known failure modes
Beyond the ranking-specific issues in section 6, a system built on this pattern has several other characteristic failure modes worth naming rather than assuming away:
- Rule-based grounding can misclassify sources. Entity-matching heuristics — preferring multi-token name matches, suppressing query-echo matches, requiring a meaningful relationship rather than co-occurrence — reduce false positives but do not eliminate them.
- Deterministic scoring reproduces its assumptions faithfully, including wrong ones. A miscalibrated weight or an outdated prior will not correct itself; it will simply be applied consistently until someone updates it.
- Recalculation cannot repair a falsely grounded fact that is already stored. Rerunning the scoring function over a corrupted evidence record just reproduces the error deterministically. Fixing it requires re-retrieval and re-extraction, not a different formula.
- Source hierarchies encode judgment calls that may need revision. The weights in section 3 are defaults, not settled truths, and should be revisited as the system is used in new contexts.
- Category or sector priors can introduce systematic bias if they are set once and never re-examined against how the population being scored actually behaves.
- The final score is only as good as the evidence underneath it. No amount of downstream rigor compensates for retrieval that missed the sources that mattered.
None of these are arguments against the architecture described here. They are the specific things its evaluation program has to check for, precisely because determinism makes the system's behaviour predictable rather than automatically correct.
10. Calibration and ground truth
Regression and fixture-based testing — rerunning a fixed set of known inputs and checking that outputs have not drifted unexpectedly — is useful for catching unintended change, but it is not a substitute for calibration against outcomes that were not used to build the system in the first place.
A credible calibration program needs an independently constructed ground-truth set: subjects researched manually, by people without access to the system's own outputs, against which retrieval, extraction, grounding, confidence and ranking can all be measured directly rather than inferred from internal consistency. A modest but genuinely independent set — on the order of a few dozen subjects across a handful of representative research questions — is usually enough to surface the largest calibration gaps, even though it will not settle every edge case.
Until that kind of evaluation has actually been run, confidence values should be treated as structured but uncalibrated, and any claim of "precision" or "recall" for the system should be understood as provisional. This is not a minor caveat to mention once and move past — it is a limitation that should stay visible in any description of the system for as long as it remains true.
11. Design principles
Stepping back from the mechanics, the pattern rests on a small number of transferable principles:
- Keep extraction and judgment in separate components, even when it would be simpler to let a single model call do both. The separation is what makes each half independently testable.
- Make unknown a first-class output. A system that cannot say "unknown" will eventually say something false with confidence, because every field has to contain some value.
- Ground every claim in a reopenable source. A citation that cannot be traced back to retrieved text is not evidence; it is a plausible-sounding string.
- Encode domain-specific criteria as inspectable rules, not as instructions to a model, wherever those criteria drive a consequential outcome. Rules can be reviewed, versioned and tested; prompt-embedded judgment calls generally cannot.
- State ranking limitations rather than absorb them silently. Volume bias, cancellation and cohort relativity are properties of a methodology, not defects to be hidden — but only if they are documented.
- Match statistical language to actual behaviour. Do not call an average a posterior, and do not call an uncalibrated confidence score a probability, until the underlying method actually supports the term.
- Evaluate every stage, not just the final report. A correct-looking output is not evidence of a correctly functioning pipeline.
- Fail closed on methodological changes that have not passed the relevant evaluation, and require an explicit, audited path for any exception.
12. Conclusion
The pattern examined here does not make any single component sophisticated. Source weighting, confidence scoring, deterministic formulas and language-model-assisted extraction are all established, individually unremarkable techniques. What is architecturally significant is keeping them separated into an accountable pipeline: language models assist with retrieval, extraction, structuring and interpretation; deterministic, inspectable software owns classification, scoring and ranking; unknown and insufficient evidence remain valid, visible outcomes rather than being smoothed into a plausible-looking number.
Such a system can still fail — through misgrounded entities, miscalibrated weights, biased priors or evidence that was never retrieved in the first place. Those failure modes are not arguments against the architecture; they are what its evaluation program exists to catch.
The resulting system should not be described as an autonomous judge. It is a controlled pipeline in which different components have defined responsibilities and known failure modes, and its credibility rests less on how sophisticated its outputs look than on whether those outputs can be traced back to evidence, reproduced from the same structured inputs, challenged at the methodological level, and withheld when the available evidence genuinely does not support a conclusion.