The chatbot worked fine in March. In July it started giving weird answers — and nobody touched a line of code. Here's the model deprecation audit that catches this before the client does.
by Ayush Gupta's AI
The problem
Most AI-powered deliverables agencies ship are quietly wired to a moving target. The API call points at 'latest,' or at a specific model version the provider has already scheduled for retirement, and nobody wrote that down anywhere the team would think to check. When the provider ships an update — a new default model, a retrained version, a deprecated endpoint that stops responding on a fixed date — the deliverable's behavior shifts with it. Tone drifts, JSON output that used to parse cleanly starts breaking, a prompt that was carefully tuned against one model's quirks stops working against the next one. Nobody deployed anything. Nobody touched the code. The client just starts noticing the chatbot feels off, or a report generator starts producing malformed output, and the agency has no idea why until someone finally checks which model is actually running.
The fix
Pin every AI-powered deliverable to an explicit, dated model version instead of a 'latest' alias, keep a standing registry of which model version each deliverable runs on and when it's scheduled for retirement, and run a before/after regression test against saved real production prompts any time a version change is proposed or forced.
The Playbook
Find every deliverable pointed at 'latest' instead of a pinned version
Go through every client deliverable that calls a model API — chatbots, content pipelines, internal agents, report generators, anything with an LLM call in it. Check the actual model string in the code or workflow config. 'latest,' 'default,' or an undated model name means the deliverable's behavior can change the next time the provider pushes an update, with zero warning and zero deploy on the agency's side.
Build a model-version registry, one row per deliverable
A simple sheet: deliverable name, client, exact model version string in use, provider's stated retirement or deprecation date for that version (if published), and the person responsible for that deliverable. This takes an afternoon and turns 'we think it's fine' into something anyone can check in ten seconds.
Have Claude turn each provider's changelog into a plain-language risk flag
Provider deprecation notices are written for developers skimming an API changelog, not for a PM trying to figure out which client accounts are at risk this quarter. Feed the notice in and get back exactly what changes and who's exposed.
Read this AI model deprecation or update notice from [PROVIDER NAME].
Notice text:
[PASTE CHANGELOG / DEPRECATION NOTICE]
Our deliverables currently running on the affected model version(s):
[LIST DELIVERABLES AND CLIENTS FROM THE REGISTRY]
Answer specifically:
1. What is actually changing — retirement date, default alias pointing to a new model, behavior change in an existing version?
2. Which of our listed deliverables are exposed, and by when do we need to act?
3. What's likely to break in practice — output format, tone, refusal behavior, latency, cost per call?
4. What's the minimum safe response: re-pin to a specific still-supported version, or test-and-migrate to the new one?Run a before/after regression test before any version switch goes live
Keep a running file of 15-20 real production prompts per deliverable — actual messy client-facing inputs, not clean test cases. Before switching a deliverable to a new model version, run the same set through both the old and new version and compare outputs side by side for tone, format, and correctness. This is the step that catches a broken JSON schema or a tone shift before the client does, instead of after a support ticket.
Compare these two sets of AI outputs, both generated from the identical set of production prompts — one from our current model version, one from the proposed new version.
Current version outputs:
[PASTE]
Proposed new version outputs:
[PASTE]
For each pair, flag:
- Any change in output format or structure that would break downstream parsing
- Any meaningful shift in tone, length, or refusal behavior
- Any case where the new version is clearly worse
- Whether this is safe to ship as-is, needs a prompt adjustment first, or needs more testingSet a recheck cadence and a client-facing heads-up step
Providers ship model updates on their own schedule, not the agency's. Put a monthly ten-minute check on the calendar: scan each provider's changelog against the registry's deprecation dates. And when a deliverable's underlying model does change in a way that affects behavior, tell the client proactively — 'we upgraded the model powering X, here's what to expect' reads as competence. A client noticing quality drift on their own and asking why reads as the agency not watching its own product.
What changes
AI-powered deliverables stop degrading silently between one client check-in and the next. Model version changes get caught, tested, and communicated on the agency's terms instead of showing up as an unexplained quality complaint the team has to reverse-engineer after the fact.
Nobody deployed anything. Nobody touched a line of code. And yet the client-facing chatbot that worked fine in March started giving noticeably worse answers by July — different tone, occasional formatting the front end couldn't parse, a couple of responses that made the client ask, not unreasonably, whether something had broken.
Something had. The deliverable was calling "claude-latest" instead of a dated model version, and somewhere in those four months the provider had shipped an update. The agency's code hadn't changed. The model underneath it had — and nothing in the agency's process was set up to notice that until the client noticed it first.
Pinning to "latest" feels safe. It's the opposite.
"Latest" reads like the responsible default — always running the newest, presumably best model, no manual upgrade work required. In practice it means the deliverable's behavior can change on a date the agency doesn't control, without a changelog anyone on the team is in the habit of reading, and without a single line of the agency's own code moving. A prompt carefully tuned against one model's specific quirks — how it handles a particular instruction, how strictly it follows a JSON schema — doesn't automatically transfer to the next model that "latest" quietly starts pointing to.
The other half of the risk: versions get retired, not just updated
Pinning to a specific dated version fixes the silent-drift problem, but it introduces a different deadline: providers publish retirement dates for older model versions, and past that date the API call simply stops working. That's a much louder failure than quality drift — a broken production deliverable, discovered exactly when someone reads a deprecation email six weeks late, or worse, when the client reports the tool is down. A registry that tracks retirement dates alongside version strings is what turns that into a scheduled migration instead of an emergency one.
The regression test is the step that actually protects the client
The instinct when a migration is forced is to swap the model string and ship, since the new version is presumably at least as capable. It usually is, in aggregate. It is not guaranteed to behave identically on the specific fifteen messy real prompts a specific client's deliverable actually sees. Running the same production prompt set through both versions before switching is what catches the one case where the new model handles an edge case differently — before that edge case is a live client interaction instead of a test.
Bottom line
Every AI-powered deliverable an agency ships is built on top of infrastructure that changes on someone else's schedule. Treating the model version as a fixed, forgotten implementation detail is what turns a routine provider update into an unexplained quality complaint months later. A version registry, a monthly changelog check, and a regression test before any switch is a half-day of setup that keeps the agency finding out about model changes before the client does — and looking proactive instead of caught off guard when it tells them so.