What AI Model Safety Testing Actually Covers, and Where It Stops

Nobody writes an AI model's abilities the way a programmer writes an application. There is no line of code that says "know how to do X." A model is built in two stages: it reads an enormous amount of text and learns to predict the next word, over and over, at a scale that ends up producing something closer to a skill than a lookup table. Then humans shape it with feedback, teaching it to follow instructions, be useful, and refuse the requests it shouldn't answer.

As models get bigger, they spontaneously get better at things nobody specifically trained them to do. Researchers call this an emergent capability, and it can include abilities nobody intended to build in, some of them risky.

Because nobody wrote the risky capability on purpose, you can't audit for it the way you'd audit code for a bug. The only way to find out what a model can actually do, good or bad, is to test it and keep pushing until you find its ceiling. That single fact is the reason the entire governance apparatus around frontier models exists.

Two documents, two jobs

Every major AI lab now runs something close to the same two-document structure, even though each company gives it a different name. Think of it as the engine and the receipt.

The first document is internal. It decides whether a model is safe enough to release, based on internal risk categories, defined thresholds, and expert review. Nobody outside the company reads it before release.

The second document is public. It exists to prove to regulators and the public that the internal process actually satisfies the law, mapping the company's practices to real requirements like California's Transparency in Frontier AI Act and the EU AI Act.

One decides. The other proves it decided correctly. They aren't the same document doing double duty, and treating them as interchangeable is where a lot of outside commentary on AI safety goes wrong.

How the internal review actually runs

The internal framework runs as a loop:

• Identify a capability that could plausibly cause severe harm
• Test the model against it
• Compare the results to a defined threshold
• Require safeguards that match whatever threshold got crossed
• Decide what happens next, via a review board
• Publish the findings alongside the model's release

Not every theoretical risk gets this treatment. A capability only earns serious attention if it clears five criteria at once:

• Plausible: a realistic path to harm, not a far-fetched scenario
• Measurable: a real test can actually be built for it
• Severe: the harm would be serious, not minor
• Net new: the model opens a path to harm that didn't already exist without it
• Irremediable: once it happens, it can't be undone

That filter exists to stop the process from crying wolf. Categories like biological and chemical risk, cybersecurity, and a model's ability to improve itself get tracked continuously. Other categories, like long-range autonomy or a model quietly underperforming on its own tests to hide a capability, are still being developed and haven't earned official status yet. One category, persuasion risk, a model being unusually good at manipulating people, gets pulled out of this framework entirely and handled through separate mechanisms. Worth remembering the next time someone points to a lab's safety report as proof a persuasion-heavy use case is covered.

The people who built the model are not the ones who decide whether it ships. A cross-functional safety review board looks at the test results and the safeguards, then makes a recommendation, anything from "go ahead" to "needs more testing." Leadership makes the final call, and the lab commits to reassessing if new evidence shows up later.

There's a soft spot worth naming honestly. If a competitor releases something high-risk without comparable safeguards, most labs have said they may loosen their own requirements to keep pace, though only after confirming that competitors' behavior actually changed the risk, saying so publicly, and checking that it doesn't meaningfully raise overall risk. That's a legitimate thing to press a vendor on.

Testing is not a quiz

It's five different formats stacked on top of each other, not a set of multiple-choice questions:

• Task-based evaluations: give the model a real objective and check whether it completes it, the AI equivalent of a hacking challenge or a dangerous chemistry walkthrough graded on whether the answer would actually work
• Agentic testing: hands the model real tools, a terminal, code execution, sandboxed internet access, and watches what it does across many steps toward a goal, which is how labs specifically probe hacking ability and self-replication
• Human expert red-teaming: brings in real specialists, virologists, chemists, professional hackers, who try to extract dangerous help using genuine domain expertise
• Comparative uplift testing: compares a person alone with the internet against a person plus the model on a risky task; the difference between those two is the actual definition of new risk
• Automated benchmark scoring: runs fast test batteries constantly, with the expensive human deep-dives saved for periodic checkpoints or flagged concerns

There's no public list of the exact test questions. That's deliberate. Publishing it would hand bad actors a roadmap.

The safety you see is not the safety a lab tests against

Front-end safety, what a regular user runs into inside a chat product, is tuned to stop an ordinary or moderately determined person from misusing it. Pre-release testing uses something else entirely: a stripped-down research version of the model, with its usual safety training deliberately removed, so testers can find the model's true ceiling under real, motivated effort. This is called capability elicitation.

The two are answering different questions. Front-end safety asks whether an ordinary user hits a wall when they ask for something they shouldn't get. Capability elicitation asks a harder question: if every wall came down, how far could the model actually go? A public chat product might refuse to walk someone through building a working exploit, and that refusal is real and worth having. But it says nothing about whether the underlying model, guardrails aside, could produce a working exploit well enough to matter. That second question is exactly what the guardrail exists to protect against, and it's the one a lab has to answer internally before anyone decides what safeguards the model actually needs.

A model can pass every one of its public-facing refusal checks while sitting at a much higher risk level underneath, once tested without the guardrails on. The stripped-down version never reaches the public, a customer, or an API. It exists purely so testers can find out what the model is actually capable of.

A refusal you can see proves the front door holds. It doesn't tell you what's on the other side.

Baked in versus bolted on

"Safety" in a deployed model is actually two mechanically different things wearing the same word.

Input and output moderation classifiers are separate, smaller AI systems that scan what a user types before the main model ever sees it, and scan the response before it reaches the user. They aren't part of the model. Think of them as independent checkpoints sitting on either side of it. Trained-in refusal behavior, by contrast, really is baked into the model itself, learned during fine-tuning from examples of harmful requests paired with refusals. It travels wherever the model gets deployed, because it's part of what the model learned.

One layer sits beside the model and can be swapped out in hours. The other is trained into the model itself and travels wherever it goes.

Why run both if the model already refuses on its own? Trained-in refusal is a learned, probabilistic behavior, not a guaranteed rule. It can be tricked through clever phrasing, role-play framing, or splitting a request into innocent-looking pieces. External classifiers are the second, independent net for exactly those moments. There's a speed advantage too: retraining a model's built-in behavior is slow and expensive, while an external classifier can be updated within hours when a new trick appears, without touching the model at all.

The underlying philosophy has a name: defense in depth. Never rely on one safeguard, because any single layer can fail.

The classifiers themselves aren't hard-coded or perfectly reliable. A classifier is still a trained AI system, still probabilistic, still capable of missing something bad or flagging something harmless. It's more consistent than the main model mostly because its job is narrower, one yes-or-no judgment instead of open-ended conversation. The genuinely rigid, hundred-percent-reliable part sits one step further down: once the classifier produces a score, the decision to block or allow is usually a hard rule based on a threshold.

Portability has limits

Skipping a chat product's interface and system prompts, by calling the model through a raw API, does not skip the model's trained safety behavior, and it doesn't skip the platform's monitoring of that traffic either. The stripped-down research version stays unreachable this way. There's no back door.

Running the model through a cloud platform is a little different. The model's trained-in refusal travels with it, because that behavior is baked into the model and the cloud platform runs the same underlying model. But the safety infrastructure around it usually isn't the lab's own. Cloud providers typically build and run their own separate content-filtering systems, which companies can configure, and in approved cases adjust, independently of anything the model's original lab does.

The model's core trained safety behavior is portable across platforms. The governance wrapper around it is not. Every platform, every company deployment, has to build and manage its own layer of control on top.

What fine-tuning quietly does to safety

When a company takes a base model and further trains it on its own data, through a fine-tuning feature, to make it better suited to a specific use case, it's changing the model's weights, the internal settings the model learned during training. The problem is that a model's refusal behavior isn't a hard-coded rule sitting off to the side. It's a learned pattern living in those same adjustable weights.

Fine-tuning a safety-trained model on completely ordinary, harmless-looking data can accidentally weaken its safety behavior, simply because the new training signal competes with, and can partly overwrite, the original safety training. Nobody has to be trying to make the model unsafe. It can happen as a side effect of making it better at something else entirely.

What doesn't get touched: external classifiers sit outside the model, so a company's fine-tuning never reaches them. They keep working exactly as before. It's specifically the baked-in refusal layer that's exposed, because it's the only safety layer that actually lives inside the part being fine-tuned.

Labs fight back with a few tools:

• Screening fine-tuning data before the job runs, to catch the most obvious attempts to train in harmful behavior
• Keeping output classifiers fully separate, untouched by any customer's fine-tuning
• Limiting how deep fine-tuning access goes on frontier models
• Ongoing usage monitoring after deployment, as a catch-all
• A re-evaluation process, in principle, though most labs don't publicly spell out a clear schedule for re-checking a model after a customer fine-tunes it

That last one is a genuinely underspecified gap in the public documentation, worth flagging plainly rather than treating as a gotcha.

A model's original safety evaluation doesn't automatically still apply once a customer has fine-tuned it on their own data. Fine-tuning is a lever a company can pull that quietly shifts the model's safety posture, without the original testing traveling along with that change.

Where the governance actually stops

The frameworks above govern the model itself, not what a company does with it after deployment. A safety report tells you what a model can do. It says nothing about how you should control it once it's embedded inside an actual business workflow.

Three examples make this concrete.

A coding assistant can quietly weaken its own company's access controls while doing completely legitimate work. Give it permission to modify internal software, a common setup, and it can introduce a subtle change that loosens access controls just by trying to make a test pass. There's no malice involved, just a model optimizing for the wrong thing. No model-level safety test catches that. The failure is operational, and it needs a deterministic outside check instead, like mandatory human review of any change touching authentication or permissions.

A customer-facing sales or negotiation product inherits a risk category the model's own lab has explicitly said it isn't tracking. Persuasion risk, pulled out of the rigorous internal framework as noted earlier, is exactly that gap, and a company building on top of the model inherits it without being told.

A European deployer has its own EU AI Act obligations that no vendor's compliance framework can satisfy for them. A lab's compliance framework maps its own obligations as a provider. A European company running that model inside something like an HR screening tool or a credit-decision system has its own, separate deployer obligations under the same law, documentation and human-oversight requirements the lab's framework doesn't and can't fulfill on the company's behalf.

One of those is operational, one is a framework's own explicit choice of what not to cover, and one is a specific legal citation. Together, they're a pattern.

The discipline nobody names

Model governance and enterprise AI governance get talked about as if they're the same problem, when they're actually two different disciplines.

Many companies are quietly assuming that a lab's own safety work substitutes for their own governance program. It doesn't, for the reasons above. As these systems move toward longer-running, more autonomous behavior, the question stops being "is the model safe" and becomes "is this specific action safe," which means permission checks, verification steps, and the ability to roll something back once it's already happened. That's the same staged-gate discipline mature software delivery has used for years, just pointed at a new kind of actor.

The lab tests the model. Somebody still has to test the company running it.

Quick glossary

• Weights: the internal numerical settings inside a model that get adjusted during training. Essentially, what the model has learned.
• Fine-tuning: additional training on top of a base model to specialize it or shape its behavior.
• Emergent capability: an ability a model develops on its own as it scales, without anyone deliberately building it in.
• Classifier: a separate, smaller AI model whose only job is a single narrow judgment, like "is this input harmful, yes or no."
• Jailbreaking: tricking a model into ignoring its trained refusal behavior through clever phrasing or framing.
• Defense in depth: a safety philosophy of using multiple independent layers of protection, so no single failure exposes the whole system.
• Provider vs. deployer (EU AI Act terms): the company that builds and supplies a model has different legal obligations than the company that puts it to use inside a real product.

Matthew Kruczek is Managing Director at EY, leading Microsoft domain initiatives within Digital Engineering. Connect with Matthew on LinkedIn to discuss enterprise AI governance and model deployment architecture for your organization.

References

1. Anthropic. "Responsible Scaling Policy." 2026.
2. OpenAI. "Preparedness Framework." 2026.
3. Google DeepMind. "Frontier Safety Framework." 2026.
4. California Legislature. "Transparency in Frontier Artificial Intelligence Act (SB 53)." 2025.
5. European Union. "EU Artificial Intelligence Act." 2024, provider and deployer obligations.
6. Matthew Kruczek. "The Immune System: How to Decide What Your Agent Does Without Asking." matthewkruczek.ai, 2026.