Today, I came across an article on Alpha Architect by Elisabetta Basilico, PhD, CFA, titled AI should assist advisors, but AI is NOT a fiduciary.
That article sent me down a rabbit hole.
It brought me to John J. Nay’s paper Large Language Models as Fiduciaries — and from there straight into a question that has been sitting, somewhat implicitly, inside the systems I am building.
Nay’s title alone is provocative.
The underlying idea is better.
He starts from a problem that sits at the heart of artificial intelligence: we cannot specify every future situation an autonomous system may encounter, and therefore cannot enumerate every correct action in advance.
Law has lived with essentially the same problem for centuries.
Contracts are incomplete. Legislators cannot foresee every future case. Courts continuously have to interpret general obligations against circumstances nobody anticipated when the rule was written.
Instead of attempting to encode every contingency, law often relies on standards.
Reasonableness.
Good faith.
Due care.
Fiduciary duty.
These concepts are deliberately broader than a list of if → then instructions. They attempt to communicate not just the literal rule, but something closer to its purpose.
Nay’s proposition is that this machinery of law may offer something useful for AI alignment.
Legal standards could help communicate underspecified human goals to artificial agents because they carry context accumulated through cases, interpretation and precedent.
His wider Law Informs Code argument goes further: law may represent one of humanity’s richest repositories of attempts to translate vague values into actionable expectations.
I think that is an important idea.
And Basilico’s article is what made me stop and look at Nay’s argument again — this time not merely as an interesting piece of AI research, but against the architecture of the systems I am actually building.
Because while reading Nay, another question kept bothering me.
What happens when the fiduciary machine is wrong?
Not malicious.
Not compromised.
Not secretly pursuing somebody else’s interests.
Just wrong.
And completely convinced that it is right.
The attraction of the fiduciary standard
Fiduciary duty is particularly interesting because it exists precisely where simple rules become insufficient.
We impose fiduciary obligations on actors who exercise meaningful discretion over the interests of others.
Investment advisers are an obvious example, but the principle is broader: one party has power, knowledge or discretion that can materially affect another, and we therefore expect that power to be exercised with loyalty, care and attention to the principal’s interests.
That maps remarkably well onto the emerging world of AI agents.
The more useful these systems become, the less attractive exhaustive instruction becomes.
Imagine trying to describe every acceptable action of an AI financial adviser.
If markets fall by 12%, do this.
Unless the customer is 63.
Unless they need liquidity next year.
Unless the decline is concentrated in one sector.
Unless their tax situation makes selling undesirable.
Unless the original investment thesis has changed.
Unless there is a regulatory constraint.
Unless another asset creates a correlated risk.
Unless…
The tree never ends.
A standard such as act in the customer’s best interest compresses an enormous amount of intended behaviour into a comparatively small instruction.
Nay tested whether language models showed any ability to interpret such standards.
Using thousands of examples derived from U.S. court opinions involving fiduciary obligations, his experiments found substantial improvement across successive generations of language models.
The interesting result is not that the machine somehow acquired a conscience.
It did not.
The interesting result is that increasingly capable language models appear able, at least to some degree, to reason about standards whose application depends heavily on context.
That opens an important possibility.
Perhaps we do not need to communicate every permitted action to an artificial agent.
Perhaps we can communicate something closer to the principles by which actions should be judged.
Nay’s broader research explores precisely this territory: whether legal texts, legal reasoning and legal standards can become part of the machinery through which human objectives are communicated to artificial systems.
I find myself broadly agreeing with him.
Law may indeed provide one of the richest languages we have for communicating incomplete human expectations to machines.
But I am increasingly convinced that this solves only one part of the problem.
Inside the model
Much of the discussion around trustworthy AI understandably focuses on what happens inside the model.
Can we align it?
Can we fine-tune it?
Can we teach it rules?
Can we give it constitutional principles?
Can we expose it to legal decisions?
Can we train it from expert feedback?
Can we instruct it to behave as a fiduciary?
Can another model criticise its answer?
All of this matters.
Nay discusses several possible mechanisms through which legal standards could become stronger behavioural signals for AI systems: training on legal material, supervised learning, reinforcement learning, expert feedback and better contextualisation of the standards themselves.
But there is another layer.
The system around the model.
That distinction has become increasingly important to me because I am building a research platform that sits very close to exactly this problem.
It researches companies.
It gathers evidence.
It evaluates that evidence against explicit methodologies.
It produces scores, confidence assessments and analytical conclusions.
And because some of those conclusions may eventually influence real investment decisions, the question of why the machine believes something rapidly becomes more important than whether the prose sounds intelligent.
When we began building it, the obvious challenges seemed to be familiar ones.
Get better data.
Use better models.
Improve the prompts.
Improve retrieval.
Improve extraction.
Ask the model to reason more carefully.
And all of those things helped.
But every improvement exposed another question.
Where did that claim come from?
Why did the system trust that source?
Did three apparently independent sources all repeat the same press release?
Was the statement actually present in the source, or was it a plausible interpretation generated by the model?
Why did this evidence affect the score?
Which methodology was used?
Was that methodology the same one we used three months earlier?
Why is the system confident?
Does 90% confidence mean the evidence is strong, or merely that several sources agree?
What evidence did we never find?
What happened to contradictory information?
Could we reproduce the score?
Could we reproduce the research?
Could somebody dispute the conclusion?
Those questions progressively move the problem away from the model itself.
They turn it into an architecture problem.
The machine stopped being the system
An early AI research architecture is seductive in its simplicity:
Question
↓
Model
↓
Answer
Retrieval improves it:
Question
↓
Search
↓
Documents
↓
Model
↓
Answer
But consequential systems begin to demand something else:
Question
↓
Discovery
↓
Sources
↓
Evidence
↓
Validation
↓
Structured facts
↓
Methodology
↓
Scoring
↓
Confidence
↓
Interpretation
↓
Verification
↓
Human decision
At some point during that evolution, an important conceptual shift occurs:
The LLM stops being the system. It becomes a component of the system.
That sounds almost trivial.
It is not.
Once you accept it, the engineering objective changes.
You stop asking only:
How do I make this model produce a trustworthy answer?
And start asking:
Which responsibilities should this model be allowed to have at all?
That is a very different question.
Who may change the number?
One of the most consequential architectural decisions in the research platform I am building was to remove numeric scoring from the language model.
The model can read.
It can extract.
It can interpret messy text.
It can identify relationships.
It can help determine whether evidence appears positive, negative or relevant.
It can explain what happened.
But once structured evidence enters the methodology, numeric scoring and ranking are performed by deterministic code.
The model cannot simply decide that a company deserves 82 rather than 74.
Confidence is also calculated outside the model.
And extracted evidence has to pass deterministic validation before it can enter the register that ultimately influences scoring.
There is something satisfying about this.
Run the same calculation over the same structured inputs and the arithmetic behaves the same way.
But implementing that separation exposed another uncomfortable lesson.
Deterministic scoring does not necessarily produce deterministic judgment.
Why?
Because the model may still have influenced the facts entering the calculation.
Suppose a model reads a source and decides that a particular statement constitutes strong negative evidence.
The scoring engine may then process that classification through completely deterministic mathematics.
The resulting score is perfectly reproducible.
But the interpretation that entered the calculation was probabilistic.
The model did not choose the final number.
It influenced the premises from which the number was calculated.
This is a form of indirect epistemic authority.
And it leads to an important principle:
Making the calculator deterministic is not enough. You must also understand who is allowed to determine what enters the calculator.
This sounds obvious after the fact.
It was less obvious while building it.
Evidence is not a fact
Another distinction became equally important.
A source saying something is not the same as the thing being true.
Again, this sounds trivial when written down.
AI systems make the distinction surprisingly easy to blur.
Consider these five events:
A source contains a claim.
The system extracts the claim.
The claim passes validation.
The claim influences a methodology.
The final narrative presents a conclusion.
Those are five different epistemic events.
Yet many AI systems collapse them into a single operation called reasoning.
In the platform I am building, the source excerpt, the machine-extracted statement and the validation status are deliberately represented separately.
Even then, the word validated needs care.
Validated does not mean true.
It means that the evidence survived a defined set of integrity checks.
A deterministic validator can establish that a cited passage actually exists.
It can establish that a URL belongs to the discovered source set.
It can reject some unsupported claims.
It can detect obvious entity mismatches.
It can enforce structural rules around evidence.
What it cannot do is establish universal truth.
This is why terminology matters enormously in consequential AI.
Evidence is evidence.
Claims are claims.
Extraction is extraction.
Validation is validation.
A conclusion is a conclusion.
None of those words should quietly become a synonym for truth.
Provenance changes the architecture
Once evidence matters, provenance follows.
If the system tells me that a company received a score of 72, an explanation such as:
"The company performs reasonably well overall, although several concerns remain."
is almost useless.
I do not merely want an explanation of 72.
I want to know what caused 72.
Ideally, I should be able to walk backwards:
Conclusion
↓
Score
↓
Methodology
↓
Structured assessment
↓
Evidence
↓
Source excerpt
↓
Retrieved document
↓
Source
↓
Discovery query
Building this turned out to be considerably harder than adding citations to generated prose.
A citation tells you that the system wants to associate a statement with a source.
Provenance asks something stronger:
What was the chain of custody from discovery to conclusion?
Our current platform records searches, fetch attempts, evidence identifiers and methodology versions for modern research runs.
That makes a substantial part of the chain reconstructable.
But it is not perfect.
Some relationships still require engineering joins.
Presentation identifiers do not yet provide a perfect reverse map to acquisition identifiers.
Older runs contain less trace information.
Some parts of model configuration are not yet persisted as systematically as the methodology and evidence.
That imperfection taught me something useful.
Auditability is not one feature.
It is a stack.
Auditability is not reproducibility
We also discovered that the word reproducible hides several very different ideas.
Suppose I preserve:
- the structured evidence;
- the methodology version;
- the scoring logic;
- the configuration.
If I feed those same structured inputs into the same deterministic scoring function, I can reproduce the score.
That is computational reproducibility.
But now imagine that I repeat the research from scratch tomorrow.
The search provider may return different results.
A website may have changed.
An article may have disappeared.
The scraper may see different content.
A new source may have been published.
The model may classify an ambiguous passage differently.
That is research reproducibility.
They are not the same thing.
A research trace can preserve what happened during the original run:
Query
↓
Search result
↓
Selected source
↓
Fetch attempt
↓
Retrieved content
↓
Evidence
That makes the research more auditable.
It does not guarantee that repeating the process tomorrow will recreate the same evidence universe.
So another distinction emerged:
A trace makes research auditable. It does not make research replayable.
That matters enormously once AI starts participating in regulated or fiduciary-like decisions.
Imagine defending a recommendation two years later.
"We asked the model again and it produced approximately the same answer" is not an audit trail.
You want to know what the system saw then.
Which searches were executed.
Which sources were selected.
Which documents were retrieved.
Which claims were extracted.
Which methodology version was applied.
Which evidence contradicted the conclusion.
Which assumptions remained unresolved.
Which model and configuration participated.
That is closer to evidence custody than chatbot history.
Confidence is not what we thought it was
Then came confidence.
Almost every AI interface now seems to want a confidence score.
It looks reassuring.
Confidence: 92%.
Lovely.
But 92% of what?
In the research system I am building, production confidence is not the model announcing how sure it feels.
Model-provided confidence hints are deliberately overridden.
Instead, confidence is calculated from more tangible properties such as source characteristics, recency, corroboration, retrieval quality and conflicting evidence.
That is useful.
But it is still not a probability that the conclusion is correct.
And building it exposed one of the most interesting problems in this entire domain.
A system can be highly confident about the evidence it discovered while being ignorant of important evidence it failed to discover.
Imagine that I find four excellent independent sources supporting proposition X.
My confidence in those sources may reasonably be high.
Now imagine that my discovery process systematically failed to find six equally strong sources contradicting X.
Nothing about confidence in the found evidence necessarily tells me about the completeness of the evidence universe.
So:
Confidence in evidence is not confidence in the completeness of the evidence universe.
This distinction becomes especially important when a system says:
"We found no evidence of X."
That statement can mean several radically different things.
Perhaps X genuinely does not exist.
Perhaps the search was exhaustive.
Perhaps the search was poor.
Perhaps the relevant information exists in another language.
Perhaps it sits behind a paywall.
Perhaps nobody has published it.
Perhaps the terminology used by the system was wrong.
Perhaps the evidence disappeared from the web.
Absence of evidence is not evidence of absence.
Our system already treats many absence-style claims cautiously, but we have not solved the larger epistemic problem.
I increasingly think future consequential AI systems will need to represent at least three distinct concepts:
Evidence confidence How strong is the evidence we have?
Evidence coverage How thoroughly did we search the relevant evidence space?
Conclusion confidence Given both, how defensible is the conclusion?
Those are not the same number.
Contradictory evidence
There is another asymmetry worth considering.
Most research systems are very good at asking:
What evidence supports this proposition?
They are less naturally designed to ask:
What would prove us wrong?
This matters because language models are extraordinarily good at constructing coherent narratives from available material.
Once an emerging conclusion forms, everything downstream can begin organising itself around it.
Our current research platform detects contradictory polarity when opposing evidence appears.
It retains conflicting evidence.
Some analytical components actively search for negative events such as lawsuits, controversies or regulatory issues.
But mandatory adversarial retrieval is not yet a structural property of every research path.
That distinction matters.
Detecting contradictions is not the same as searching for them.
A fiduciary research system should probably do both.
If the emerging conclusion is positive, ask what the strongest negative case is.
If it is negative, look for credible rebuttal.
If company claims dominate the evidence, seek independent confirmation.
If every source traces back to one underlying report, treat five URLs as one information event.
This is not merely better research practice.
It is an architectural defence against confirmation bias.
Six agents do not make a court
Then there is the current fascination with multi-agent systems.
Researcher agent.
Critic agent.
Risk agent.
Judge agent.
Compliance agent.
Supervisor agent.
It sounds reassuringly institutional.
I use multiple agents myself.
And building them taught me to be very careful about what the word agent implies.
In the research platform, different agents perform different stages of the workflow.
One interprets the research question.
Others gather and extract evidence.
Another synthesises the findings.
Another produces analytical narrative.
This modularity is useful.
It improves isolation.
It creates clearer responsibilities.
It allows parts of the system to fail without necessarily destroying the entire research run.
But these agents often share the same evidence.
They operate under the same orchestrator.
They may use the same underlying model infrastructure.
One agent's output becomes another agent's context.
That is not necessarily independence.
So:
Multiple agents are not the same thing as multiple independent authorities.
And:
Separation of prompts is not separation of powers.
If one model discovers the evidence, another invocation of substantially the same model extracts it, a third sees the same context and critiques it, and a fourth produces the final judgment, we may have created useful workflow decomposition.
But we have not necessarily created epistemic independence.
True independence may require different information boundaries.
Different evidence acquisition paths.
Different methodologies.
Different model families.
Deterministic authorities.
Adversarial objectives.
Human review.
Or some combination of them.
The lesson is not that multi-agent systems are useless.
It is that institutional terminology can give technical coupling a misleading appearance of independence.
Epistemic separation of powers
And this is where Nay’s work led me somewhere I did not expect.
Nay asks what AI engineering might learn from legal standards.
Perhaps we should borrow something else from legal systems as well.
Institutional architecture.
Legal systems generally do not give one actor unrestricted authority to:
discover evidence,
determine admissibility,
write the rules,
interpret the rules,
judge the dispute,
impose the outcome,
and review the appeal.
We separate powers because the actors involved are fallible.
That is the point.
Not because every judge is malicious.
Not because every prosecutor is incompetent.
Not because legislators cannot be trusted.
Because concentrated authority magnifies error.
Consequential AI may need an analogous principle.
I have started thinking of it as:
Epistemic Separation of Powers
A consequential system should be extremely cautious about allowing the same probabilistic component to control all of the following.
Discovery
What information enters the system?
Evidence custody
What was actually retrieved?
Extraction
What do the sources supposedly say?
Methodology
How should evidence be evaluated?
Judgment
What conclusion follows?
Confidence
How certain should the system claim to be?
Explanation
How is the conclusion presented to the human?
Verification
Who checks whether the reasoning was correct?
Authority
Who decides whether the outcome becomes consequential?
The research platform I am building already separates some of these responsibilities deliberately.
Methodologies are versioned and bound to research runs.
Evidence passes deterministic validation.
Numeric scoring is performed outside the model.
Confidence is calculated outside the model.
Research traces preserve substantial parts of the acquisition process.
Narrative recommendations are constrained by computed results rather than being allowed to invent their own scores.
But the separation is incomplete.
Extraction remains probabilistic.
Discovery still determines the universe the system gets to see.
Narrative can still become more certain than the underlying evidence warrants.
Adversarial retrieval is not yet structural everywhere.
Model and prompt provenance can be improved.
Human authority is stronger at the beginning of the process than at the final conclusion.
Independent verification remains an architectural problem we have not solved.
I regard that incompleteness as useful.
Because it prevents the comforting fiction that adding a few deterministic functions suddenly transforms probabilistic AI into a perfectly trustworthy institution.
It does not.
Every boundary we introduced revealed another layer of epistemic authority underneath it.
Better prompting exposed retrieval.
Better retrieval exposed evidence quality.
Evidence quality exposed provenance.
Provenance exposed methodology.
Methodology exposed reproducibility.
Reproducibility exposed confidence.
Confidence exposed completeness.
Multiple agents exposed the difference between modularity and independence.
Deterministic scoring exposed the probabilistic judgments entering the calculation.
The engineering problem kept moving outward.
From the prompt.
To the model.
To the pipeline.
To the evidence.
To the methodology.
To governance.
And ultimately, perhaps, to something resembling an institution.
From research to advice
Now move one step further.
Imagine an AI advising someone on motor insurance.
The customer has a family.
A particular vehicle.
A budget.
Existing coverage.
A risk profile.
Specific preferences.
And the system recommends Policy A.
The conversational version might say:
"Based on your circumstances, I believe Policy A is the best choice for you."
It sounds thoughtful.
It may even be correct.
But what sits underneath that sentence?
Now imagine another system.
Internally, its recommendation looks more like this:
Customer facts
↓
Eligibility rules
↓
Product facts
↓
Coverage requirements
↓
Suitability methodology
↓
Alternatives considered
↓
Deterministic assessment
↓
Unresolved assumptions
↓
Recommendation
↓
Explanation
↓
Human authority
This is another system I am working toward.
Here, too, the language model can be extraordinarily useful.
It can conduct the conversation.
Understand natural language.
Potentially understand dialect.
Extract customer information.
Ask intelligent follow-up questions.
Explain terminology.
Compare trade-offs.
Translate.
Summarise.
Make the interaction far more accessible than traditional financial or insurance interfaces.
But once again the important question becomes:
What should the model be allowed to decide on its own?
If the system recommends one insurance product rather than another, it should ideally be possible to reconstruct:
which customer interests entered the recommendation;
which alternatives were considered;
which constraints excluded others;
which rules applied;
which methodology produced the comparison;
which assumptions remain uncertain;
which information came from the customer;
which information came from product documentation;
and which human or institutional authority ultimately owns the decision.
Then ask:
Which system deserves to be called fiduciary?
The one whose system prompt says:
Act in the customer's best interests.
Or the one that can demonstrate how the customer's interests constrained its actions?
Fiduciary behaviour inside the model
This is why I do not see this argument as contradicting Nay.
Quite the opposite.
Nay's work points in a direction I find increasingly convincing.
Rules alone will not be enough.
Programming languages alone will not capture the ambiguity of human expectations.
Natural-language prompts alone will not be sufficient either.
Legal standards encode something different: context, precedent, interpretation and accumulated institutional attempts to apply abstract human values to messy reality.
Nay’s broader Law Informs Code thesis treats law as a mechanism through which vague social expectations become more legible directives.
Other researchers have extended the fiduciary-AI idea further, exploring how principles such as identifying the principal, understanding their interests, loyalty and appropriate standards of care might be translated into the design and auditing of artificial intelligence systems.
That work reinforces rather than weakens the architectural question.
Perhaps a capable AI should indeed understand the duty of loyalty.
But the architecture should also make self-dealing difficult.
Perhaps it should understand due care.
But the system should also preserve the evidence used to exercise that care.
Perhaps it should reason about best interests.
But the methodology should make explicit whose interests are being optimised.
Perhaps the model should recognise conflicts of interest.
But the process should not depend exclusively on the model deciding whether one exists.
Perhaps it should understand uncertainty.
But it should not be allowed to manufacture its own confidence score and then use that score to justify its conclusion.
This gives us two complementary ambitions:
FIDUCIARY BEHAVIOUR
inside the model
+
FIDUCIARY ARCHITECTURE
around the model
We are going to need both.
The fiduciary machine must be able to disagree with you
There is another reason this matters.
A fiduciary is not merely somebody who makes you happy.
Sometimes acting in somebody’s interest requires saying no.
AI systems have a peculiar vulnerability here because conversational helpfulness can drift toward agreement.
A user wants confirmation.
The machine wants to help.
The user reformulates the question.
The model finds a way to accommodate the premise.
In harmless situations this can be annoying.
In consequential advice it can become dangerous.
A fiduciary machine therefore cannot optimise purely for conversational satisfaction.
Its obligations must survive pressure from the person it serves.
That again points beyond prompting.
If eligibility is deterministic, the user cannot flatter the model into changing it.
If the methodology is immutable for the run, the model cannot quietly alter the weights because the customer dislikes the result.
If evidence must be attached to material claims, persuasive prose alone cannot transform speculation into support.
If contradictory evidence is retained rather than discarded, an inconvenient source does not disappear because it damages the narrative.
If applicable duties are represented independently of the conversational model, helpfulness cannot silently override them.
These are not personality traits.
They are constraints.
Contestability
There is one more concept I increasingly think deserves to sit beside explainability.
Contestability.
Explainability asks:
Why did the system produce this answer?
Contestability asks:
What can I do if I think it is wrong?
That is a much stronger requirement.
Can I inspect the evidence?
Can I challenge a source?
Can I correct a customer fact?
Can I identify which methodological rule affected the outcome?
Can I distinguish machine extraction from source text?
Can I see contradictory evidence?
Can I request reconsideration?
Can another authority review the decision?
Can an earlier conclusion be corrected without rewriting history?
Can the system preserve both the original result and the reason it changed?
The platform I am currently building is not fully there.
We can preserve evidence.
We can reproduce deterministic calculations from structured inputs.
We can trace substantial parts of a research run.
But a first-class workflow through which a human formally disputes a consequential conclusion does not yet exist.
That gap has made me increasingly interested in contestability.
Because explanation without recourse can become little more than a beautifully formatted justification.
A machine may explain itself perfectly and still be wrong.
The real question is whether anything can challenge it.
Human authority
This also changes how I think about the phrase human in the loop.
It has become almost ceremonial.
Put a human somewhere in the process and the system is suddenly responsible.
But where?
Before the model?
After the model?
Only when confidence is low?
Only when something fails?
Can the human change the result?
Can they inspect the evidence?
Do they understand the methodology?
Does the interface encourage genuine review or merely offer an Approve button beneath an eloquent recommendation?
In the research system I am building, human authority is currently strongest at the beginning.
A person chooses or approves the research question and methodology.
The machine then performs substantial autonomous research and analysis.
The resulting report returns to a human reader.
That is meaningful human control.
But it is not the same thing as an institutional review gate over every consequential conclusion.
This matters particularly as we move from research toward advice.
A human in the loop should not merely be the final biological component in an otherwise opaque pipeline.
Human authority needs to be designed as carefully as machine authority.
Trust may be the wrong target
For years, we have talked about trustworthy AI.
The phrase is understandable.
I am beginning to wonder whether trust is actually the wrong engineering target.
Consider other consequential systems.
Aviation safety does not depend on every pilot being incapable of error.
Financial control systems do not assume every employee is incorruptible.
Courts do not assume every judge is infallible.
Science does not assume every researcher is unbiased.
Critical infrastructure does not assume every component will always behave correctly.
We construct institutions around fallibility.
We separate responsibilities.
Record actions.
Preserve evidence.
Demand reproducibility where possible.
Create independent review.
Expose conflicts.
Version rules.
Audit procedures.
Allow challenge.
Investigate failure.
Build redundancies.
Artificial intelligence should not be exempt from the same thinking simply because its reasoning is impressive.
Indeed, its fluency makes these safeguards more important.
A system that is wrong awkwardly invites inspection.
A system that is wrong beautifully invites belief.
The fiduciary machine
Nay’s central insight deserves to be taken seriously.
Legal standards may indeed become one of the mechanisms through which we communicate complex and underspecified objectives to artificial intelligence.
Teaching machines something about fiduciary duty could prove enormously valuable.
But I suspect the next step is architectural.
We should continue asking:
Can artificial intelligence understand fiduciary duty?
And:
Can legal standards improve the behaviour of artificial agents?
But we should add another family of questions.
How do we construct systems in which fiduciary behaviour remains inspectable when the model fails?
How do we distribute epistemic authority?
How do we preserve evidence?
How do we distinguish a source from a claim, and a claim from a fact?
How do we separate judgment from explanation?
How do we distinguish confidence from completeness?
How do we actively look for the evidence that could prove our conclusion wrong?
How do we make decisions contestable?
How do we prevent six agents from becoming six copies of the same mistake?
How do we determine who ultimately has authority?
These are not arguments against increasingly capable models.
They are consequences of increasingly capable models.
The better the machines become, the more consequential the responsibilities we will delegate to them.
And the more consequential that delegation becomes, the less satisfactory
"the model usually understands what we mean"
will become as an institutional design principle.
Perhaps the future of fiduciary AI therefore depends on two separate achievements.
We must teach machines something about the duties we expect them to honour.
And we must build systems around those machines that do not collapse when they misunderstand those duties.
The first is a model problem.
The second is an architecture problem.
Eventually, it may become an institutional problem.
That is the distinction between a model capable of reasoning about fiduciary behaviour—
and a fiduciary machine.
Don't build a machine you have to trust.
Build one whose work you can verify.
Sources
-
John J. Nay — Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards Nay’s original paper introducing fiduciary standards as a case study for communicating underspecified objectives to language models. arXiv: 2301.10095
-
John J. Nay — Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans Broader development of the argument that law provides structured information about human values, standards and expected behaviour that may help align artificial systems. Northwestern Journal of Technology and Intellectual Property
-
John J. Nay et al. — Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence Related empirical work examining legal reasoning, retrieval of legal authority and the limits of language-model performance compared with specialist expertise. arXiv: 2306.07075
-
Sebastian Benthall & David Shekman — Designing Fiduciary Artificial Intelligence Development of fiduciary principles as possible design and auditing criteria for AI systems, including the identification of principals, interests, loyalty and appropriate care. arXiv: 2308.02435
-
UK Financial Conduct Authority — Artificial Intelligence approach Current FCA material on AI in financial services, accountability, governance and the relationship between automated analysis and human judgment. Financial Conduct Authority — AI
-
Alpha Architect — article discussing LLMs and fiduciary reasoning The article that prompted me to return to Nay’s work and reconsider it against the systems I am currently building. Alpha Architect — LLM Fiduciary
-
Internal Architecture & Methodology Audit — Fiduciary Machine Architecture & Methodology Audit, 10 August 2026 Read-only audit of the research platform I am building, covering evidence provenance, deterministic and probabilistic boundaries, methodology versioning, scoring, confidence, research tracing, contradiction handling, reproducibility, agent independence and human authority. The audit is the basis for the engineering observations in this volume. It is not a public source.