Switching used to be a project#
Not long ago, trying a different AI model at work meant a new vendor. A new contract, a new security review, a new integration, a new invoice. Months, if it happened at all. Most teams picked one provider and stayed.
That has changed, and mostly for the better.
The big cloud platforms now put many model makers behind a single front door. On Amazon Bedrock, a company can reach models from Anthropic, OpenAI, xAI, Moonshot and others from inside the same AWS account it already runs on (AWS, September 2026). The same permissions decide who can call them. The same logs record that they were called. One bill splits the cost by model. And one request format works across them — in AWS's own words, you can "write code once and use it with different models" (Converse API reference).
That is real progress. Nobody should miss the procurement cycle.
It also quietly removed something.
What the platform actually gives you#
It is worth being precise about what a managed platform covers, because it covers a lot — and the part it does not cover is easy to miss next to everything that it does.
One contract, many models
You buy access once, from a cloud provider you already have terms with, instead of signing with each model maker. Covers: the commercial and legal side of trying a new model.
One permission system
The same identity and access rules decide which people and systems may call which model. Covers: who is allowed to use it.
One log
Calls are recorded in the same audit logs as the rest of your cloud account. Covers: that a call happened, when, and by whom.
Data stays with the cloud you already use
Requests are processed inside the cloud provider's boundary rather than sent to each model maker, and you can choose which regions handle them. Covers: where the data travels.
Managed retrieval and agents
Ready-made pieces connect models to your documents in SharePoint, Confluence, Salesforce and the like, and run multi-step agents without much custom code. Covers: the plumbing around the model.
One request format
The same code can call a different model by changing one name. Covers: the cost of switching.
Read down the right-hand side of each box and a pattern turns up. Every one of these is about access, location and plumbing: who may call the model, where the data goes, how the call is wired. None of them is about whether the answer is right for your business. That was never the platform's job. It is yours.
The pause was doing a job#
The old, slow switch had one hidden benefit. Because changing models took months, somebody always asked the obvious question along the way: does the new one behave the same? Someone ran the old cases through it. Someone compared. The slowness forced the check.
When the switch becomes a single changed name, the check is no longer forced. It has to be chosen.
And sometimes the switch is not your choice at all. Model versions retire. On Bedrock, most models get six months' notice before they are switched off, and some get 45 days. After the end date, requests to the old version simply fail, and migration does not happen for you (Bedrock model lifecycle). So every team using these models will switch, on someone's timetable. The only question is whether the check happens first.
Why this matters more at work than in a chat window#
Most people's sense of AI comes from chat tools. You type, it answers, you read the answer. If the vendor upgrades the model overnight, you notice the next morning. It sounds different. It refuses something it used to do. You adjust.
That works because a person reads every answer. In a chat tool, you are the check.
Enterprise AI is built differently. The model's output usually goes into another system, not in front of a person. It sorts a support ticket into a queue. It reads an invoice and fills in fields. It decides whether a refund request meets the policy. It drafts the reply that goes out under your name. At the volumes these systems run, nobody reads most of the outputs. That is the point of building them.
So when the model changes, nobody notices the morning after. The request still goes through. The response still comes back in the right format. Nothing errors. The new model just makes slightly different calls — a little more willing to approve, a little stricter on a borderline case, a different reading of an ambiguous field — and those calls flow straight into refunds, queues and records.
The difference in one line
In a chat tool, a changed model changes what a person reads. In a business system, it changes what the business does — and no one reads it first.
What the risk really is#
The risk is not that the new model is worse. It may well be better on average. Newer models usually are.
The risk is that "better on average" and "the same on your cases" are different claims, and only the second one protects you. A model that is better at general reasoning can still read your refund policy differently, treat your missing fields differently, or be more confident on exactly the borderline cases where your old model hesitated. None of that shows up on a public leaderboard, because the leaderboard was not built from your work.
It also does not show up in the platform's logs. The log says a call was made at 10:42 by the claims service to a given model. It does not say what the claim said, what the model decided, or whether last month's model would have decided the same.
Pharmacy has a rule for this#
When a pharmacist hands you a generic instead of the brand-name drug, they are making a switch you did not ask for. It is allowed for one reason: someone first showed that the substitute acts the same way in the body.
Not that it has the same label, or the same active ingredient on paper. In the US, the maker has to measure how much of the drug actually reaches the bloodstream, and how fast, and show that the whole likely range of difference sits inside a tight band around the original. In practice, approved generics land very close — one FDA review found an average difference of about 3.5% (Pharmacy Times).
The principle is the useful part: equivalence is shown on the effect, not the label, and it is shown before the swap, not after.
Model switches today mostly run the other way. Same interface, same format, same "label" — and the effect is checked, if at all, after it is in production.
The swap test#
You do not need a research team for this. You need four things, and the first one is the one most teams do not have.
- A record of real inputs. Not test cases somebody wrote. The actual requests your system sent last month, with the context that went with them — the document retrieved, the fields filled in, the ones left empty.
- The decisions the current model made on them. What it chose, approved, routed, extracted. Decisions, not prose. Two models will always word things differently; that is not the question.
- The same inputs through the new model. Replay them and record what it decides.
- A count of where the decisions differ. Then a person reads those cases — not all of them, just the ones that changed.
A rule of thumb for how many to replay: if you run 300 real cases and find no differences, you can say with reasonable confidence that the difference rate is under about 1 in 100 — not that it is zero. To get that down to 1 in 1,000, you need around 3,000 cases. The arithmetic is simple: divide 3 by the number of cases. It tells you the worst rate your test could actually rule out.
What an Inspector does about it#
Step one is the hard one, and it is the one no platform gives you by default: a faithful record of what your system actually sent and what came back. Not "a call happened", but the content of the call, the context around it, and the decision that followed — kept on your own systems.
That is what an Inspector keeps in Watch mode. It sits in front of the AI already running in your work, records what goes in and what comes out, and changes nothing while it does. When the next model switch arrives — your choice or the provider's — that record is the set of cases you replay. Without it, there is nothing to compare against, and "we tested the new model" quietly means "we tried a few prompts and it seemed fine."
What survives the switch
Models will keep changing underneath you, faster than procurement ever allowed. The thing that carries over from one to the next is the record: what went in, what came back, and what your business did about it.
Five questions for your next review#
These work for any model switch — a new provider, a newer version, or a retirement notice you did not choose.
“The last time we changed models, what did we rerun before it shipped?”
If the honest answer is “a few prompts”, that is the finding. Everything else here is a way of making the answer better.
“Do we have last month’s real inputs, with the context that went with them?”
Platform logs show that a call happened. The replay needs what the call contained.
“Which of our models is in its retirement window, and when does it end?”
It is shown on the model card. Six months goes faster than a migration plan.
“Do we compare decisions, or do we compare writing?”
Two models will always phrase things differently. What matters is whether they approve, route and extract the same way.
“How many cases would we need to rule out a 1‑in‑100 difference?”
About 300 with no differences found. If your test was smaller, it could not see that far.
What this does not tell you#
It does not tell you which model to choose, or that managed platforms are a bad idea. They remove real risk: fewer contracts to sign, fewer places data travels, one set of permissions to get right instead of five. For most companies, that is a better starting point than wiring up each provider on its own.
It also does not tell you that new models are worse. Usually they are better. The point is narrower than that. The platform made the switch cheap, and the check used to be paid for by the cost of the switch. Now it has to be paid for on purpose — and it can only be paid for by a team that kept a record of what it actually sent.
Related
