For years, technology leaders have been asking where their software comes from.
We review dependencies. We scan packages. We track versions, verify signatures, and question anything that enters a critical environment without a clear owner. We do all this because we learned, sometimes painfully, that an application can be secure on its own and still be compromised by something it depends on.
Now we are introducing a different kind of dependency into our enterprises: AI models that don’t just execute instructions, but interpret information, make recommendations, and increasingly decide which actions to take next.
And I think we are overlooking an uncomfortable question.
We know how to verify the software we deploy. But how do we verify the intelligence we’re beginning to trust?
The dependency we rarely question
When a company adopts an AI model, the conversation usually starts with capability. How well does it reason? How fast is it? What does it cost? Can it work with our systems and data?
Those are reasonable questions. But they are not the whole conversation.
A model is the result of a long chain of decisions and dependencies: training data, fine-tuning, model weights, supporting libraries, deployment infrastructure, updates, and sometimes third-party components. In enterprise applications, its behavior is also shaped by prompts, retrieved documents, and the tools it can call.
Some of those elements are visible. Others, especially in proprietary models, are not.
That doesn’t make the model untrustworthy. It means trust cannot come from a familiar provider name alone.
As a CTO, I learned to treat critical dependencies as architecture decisions, not just procurement choices. AI deserves the same discipline, with one important difference: the dependency now influences how the system interprets the world.
A backdoor doesn’t have to look like a failure
Imagine a procurement assistant that reviews supplier documents, identifies risks, and prepares recommendations for a purchasing team.
It works well in testing. It works well for months in production. Then a particular phrase, supplier name, or pattern appears in a document, and the assistant starts downplaying a risk it would ordinarily flag.
No crash. No alert. No obviously broken response. Just a recommendation that looks plausible.
This is a hypothetical scenario, not a claim about a particular deployed model. But the underlying concern is real: attackers may try to influence a model’s behavior through the data or components used to build and operate it.
In research published in October 2025, Anthropic and its research partners showed that 250 malicious documents could introduce a specific backdoor into models they tested, up to 13 billion parameters. The demonstrated attack caused the model to produce gibberish when triggered. It did not demonstrate covert data theft, arbitrary control of a frontier model, or the supplier scenario above.
Still, the result matters. It challenges the comforting assumption that a very large model must be immune to a relatively small amount of poisoned training data.
Model size, on its own, is not a security guarantee.
The attack surface is bigger than the model
It’s tempting to picture an attacker breaking into an AI lab and secretly rewriting billions of parameters. That is one dramatic version of the story, but it is not the only one, or necessarily the most practical.
A company might fine-tune a model on compromised data. It might download a model artifact or dependency from an untrusted source. It might give an agent access to a document repository where malicious instructions have been planted. Or it might deploy a model update without understanding how its behavior has changed.
These are different attack paths and should not be confused. Poisoned training data can affect learned behavior. A malicious model package can exploit the software loading it. A prompt injection in a retrieved document can redirect an application at runtime without changing the underlying model at all.
But they share an architectural lesson: the model is only one part of the trust boundary.
The consequences also depend on what the system is allowed to do. A compromised model does not automatically give an attacker access to company data. A model connected to broad internal tools, sensitive documents, and unrestricted outbound services creates a very different level of exposure from one that can only answer questions.
This is why security can no longer stop at the model endpoint.
A signed model is not necessarily a safe model
Traditional software security gives us a useful starting point. Verify where an artifact came from. Pin the version. Scan its dependencies. Control who can change it.
We should do all of that for AI systems too.
But an intact signature tells us that an artifact came from the expected source and has not changed since it was signed. It does not prove that the model’s behavior is free from hidden weaknesses.
That is the harder part.
With software, we can inspect source code, test expected outputs, and reason through defined execution paths. AI models don’t offer the same predictability. An evaluation may reveal how a model responds to the situations we thought to test, but it cannot establish that every rare trigger or harmful behavior has been ruled out.
The answer is not to give up on verification. It’s to expand what verification means.
Industry guidance is already recognizing the distinction. The OWASP Top 10 for LLM Applications identifies supply-chain risk and data/model poisoning as separate concerns. Both belong in an enterprise AI security program.
Trust needs to be engineered
I would start with a few questions that should be familiar to every technology leader.
Do we know what we’re running? Not just the provider and model family, but the exact version, the origin of any downloaded artifacts, the fine-tuning process, and the third-party components around it. Where transparency is limited, we need to understand what evidence and assurances the provider can actually supply.
Do we test behavior when something changes? A model update should be treated as a meaningful change to a production dependency. That means repeatable evaluations, regression tests against important business scenarios, and adversarial testing designed to uncover surprising behavior, not just benchmark improvements.
Have we limited the damage a model could cause? Give an agent only the access its task requires. Separate reading information from changing it. Require approval for high-impact actions. Restrict destinations for sensitive data. The best way to reduce the impact of a compromised model is to ensure that the model cannot do everything.
Would we notice if its behavior changed? Record model versions, significant tool calls, and outcomes in ways that can be reviewed without unnecessarily exposing sensitive data. Watch for unusual patterns. Make rollback possible.
None of these controls can guarantee that a model has no hidden problem. Together, they make trust something an organization can assess, contain, and revisit rather than simply assume.
The question belongs in the boardroom
AI supply-chain security is not just another item for the security backlog.
It affects vendor selection, enterprise architecture, procurement, risk management, and the pace at which companies can safely give AI systems more autonomy. It requires technology leaders to ask not only what a model can do, but what evidence justifies allowing it to do it inside the business.
We may never be able to inspect every piece of data that shaped a foundation model. That doesn’t mean we should accept a black box with unlimited access and no meaningful checks.
The next supply-chain attack may still arrive through ordinary software. It may come through a model artifact, a training pipeline, or the information an agent consumes. The exact route matters less than the principle we need to carry forward.
We know how to verify the software we deploy. But how do we verify the intelligence we’re beginning to trust?