Pin the Model, Gate the Upgrade: Catch Agent Drift Before Customers Do
Yes—pin production agents to explicit model IDs. Floating aliases change behavior with no deploy. Upgrade only through a golden-set gate you actually run.
Yes — pin the model version for production agents, and upgrade only through a golden-set gate. Floating aliases (gpt-5.6, latest, bare family nicknames) can change weights and defaults under you with no deploy. Customers notice first as “the agent got weird.” You notice last as a support spike.
This spoke sits under the Agentic Systems Operating Manual. It assumes evaluators exist so the gate has teeth, and observability can show drift online when offline scores still look fine.
The short answer
- Pin explicit model IDs in prod. Store them in config, not string literals scattered across services.
- Floating aliases are a drift channel. Behavior changes without a PR.
- Upgrade = config PR + golden-set gate + canary. Never “swap the alias on Friday.”
- Drift is not only model weights. Prompts, tools, retrieval indexes, and input mix move too — schedule checks even when nothing deployed.
- Providers do not keep snapshots forever. Pinning buys stability until deprecation/retirement; watch their notices.
What model / prompt drift looks like for agents
| Drift type | What changed | Symptom |
|---|---|---|
| Model alias float | Provider pointed the alias at new weights | Tool-choice shifts; tone/verbosity jumps; cost spikes |
| Prompt edit | System prompt or tool descriptions edited | Same model, new failure modes |
| Tool schema drift | Enums/fields changed | Argument errors, loops |
| Retrieval / memory drift | Index or memory policy changed | Confident wrong citations |
| Input-distribution drift | New ticket types, seasons, locales | Offline golden set still green; online revision rate up |
Agents amplify small drifts: one extra tool call per turn compounds cost; one wrong enum compounds retries. Chatbots hide drift in prose. Tool agents deposit it in CRM notes.
Why floating aliases silently change behavior
Aliases exist for convenience. Production needs identity.
Examples of float risk (as of August 2026 docs):
- OpenAI’s
gpt-5.6routes togpt-5.6-soltoday — fine until the routing policy or Sol snapshot behind an alias changes and your evals never ran. - Convenience aliases on older Claude lines resolved to “latest dated snapshot” for a minor version — a different contract than pinned IDs.
- Managed-agent defaults can move (Google has already moved Managed Agents defaults to newer Flash models without you editing business logic).
If the model string in prod is not the string you evaluated, you are A/B testing on customers.
Which IDs to pin today (verified August 2026)
Prefer the most specific ID your provider documents as a pinned snapshot / stable model ID. Re-check provider model pages before you copy these into a new deploy months later — training data and this table both go stale.
OpenAI — GPT-5.6 family
Sources: OpenAI model guidance, GPT-5.6 Sol model page (checked 2026-08-07).
| Role | Pin this ID | Notes |
|---|---|---|
| Flagship / hard agent reasoning | gpt-5.6-sol | gpt-5.6 alias routes here — do not use the alias in prod |
| Balanced worker | gpt-5.6-terra | Cost/quality middle |
| High-volume / latency-sensitive | gpt-5.6-luna | Volume tier |
If the model page lists a more specific snapshot ID than the tier alias, prefer the snapshot for regulated or high-stakes agents. Do not invent dated strings that are not on the page.
Anthropic — Claude 5-era pins
Sources: Anthropic model IDs and versions, model deprecations (checked 2026-08-07). From the 4.6 generation onward, dateless IDs are pinned snapshots, not evergreen pointers. Convenience aliases (opus, sonnet) can move — avoid them in prod.
| Role | Pin this ID | Tentative retirement floor (Anthropic table) |
|---|---|---|
| Long-horizon / highest capability | claude-fable-5 | Not sooner than 2027-06-09 |
| Complex agent / coding | claude-opus-5 | Not sooner than 2027-07-24 |
| General production | claude-sonnet-5 | Not sooner than 2027-06-30 |
| Fast classify / extract | claude-haiku-4-5-20251001 | Not sooner than 2026-10-15 |
Google — Gemini Flash workhorse
Sources: Gemini 3.6 Flash model docs (GA noted Jul 2026).
| Role | Pin this ID | Notes |
|---|---|---|
| Agentic / coding workhorse | gemini-3.6-flash | Documented stable ID; no separate dated suffix on the public model card as of Jul 2026 |
| Cost / latency Lite class | gemini-3.5-flash-lite | Use when the job earned the cheaper tier |
Honest limit: Gemini’s public card for 3.6 Flash does not expose a YYYY-MM-DD-style snapshot the way older stacks did. Pin the stable ID you evaluated, track Google’s model changelog, and re-run the golden set when they announce a replacement — do not pretend a dated pin exists if the docs do not list one.
How long do providers keep pinned snapshots?
Do not invent retention SLAs. Use the published notice policies and watch the deprecation tables.
| Provider | What they commit (as of Aug 2026 docs) | Practical meaning |
|---|---|---|
| OpenAI | For GA models, at least 6 months notice before retirement after deprecation announcement; specialized variants ≥3 months; previews may be ~2 weeks (deprecations) | Pinning ≠ forever. Calendar the shutdown date when it appears. |
| Anthropic | At least 60 days notice before retirement for publicly released models; retired IDs fail (model deprecations) | Weights may be preserved internally; API access is not. Migrate before the retirement date. |
| Google Gemini | Publish model versions / shutdowns on model cards and release notes; notice windows vary by model | Treat changelog + golden-set canaries as the control, not a assumed multi-year pin. |
Hedged on purpose: third-party blogs sometimes claim “12 months” or other windows. Prefer the provider deprecation page over folklore.
Staging on floating vs prod on pinned
Good pattern:
| Environment | Model string | Job |
|---|---|---|
| Local / spike | Alias OK | Explore |
| Staging | Candidate pin or intentional float + nightly eval | Detect provider moves |
| Production | Pinned ID only | Stable behavior |
Staging-on-float only works if something reads the nightly eval. A floating staging env nobody watches is theater.
Upgrade procedure that does not lie
- Pick candidate ID from provider docs (not Twitter)
- Freeze prompts, tools, and stubs
- Run full golden set + cost band vs current pin
- Diff failure codes — not only pass rate
- Canary a small online % with observability panels watched
- Merge config PR that changes the pin
- Keep the old pin in config comments / rollback map for 48–72 hours
Gate rule example (tune to your risk):
| Signal | Ship | Hold |
|---|---|---|
| Offline pass rate | ≥ baseline − 1 pt | Drop > 1 pt |
| Argument accuracy | ≥ baseline | Any drop on write tools |
| Cost per pass | ≤ baseline × 1.15 | Above |
| New failure codes | None critical | Any policy / wrong_tool surge |
What belongs in SemVer for prompts vs model pins
Keep two version axes:
| Axis | Example | Bumps when |
|---|---|---|
prompt_version | support-triage@3.2.0 | Wording, tool descriptions, rubrics |
model_id | claude-sonnet-5 | Pin change |
tools_version | crm-tools@1.7.0 | Schema / handler contract |
eval_suite | golden@2026-03-12 | Cases added/removed |
Never bury the model id inside an opaque “agent version 42.” Incident response needs to answer “which weights?” in one query.
Scheduling drift checks when nothing deployed
Calendar, not vibes:
- Weekly: online sample pass rate, revision rate, cost per pass vs trailing baseline
- On provider emails / changelog: open an upgrade ticket the same day
- Monthly: staging float vs prod pin bake-off on the golden set
- After any tool or prompt PR: suite run — model pin unchanged still needs the gate
Input-distribution drift shows up online first. Offline-only teams learn from angry humans.
Worked failure: the quiet alias weekend
What broke: Prod used gpt-5.6 “so we always get the best.” A routing/default change shifted tool verbosity and doubled average tool calls on a support agent. Pass rate dipped two points; cost per pass jumped ~40%. No deploy in git.
Cost: Budget alerts, weekend rollback to an explicit gpt-5.6-terra pin, emergency golden-set triage.
Instead: Prod pin gpt-5.6-sol or gpt-5.6-terra by role; staging tracks alias; upgrade only through the gate.
The model did not “get dumber.” Your identity string did.
What breaks if you never unpin
- You hit a retirement date and production hard-fails (Anthropic retired IDs return errors; OpenAI shutdown dates are published on the deprecations page)
- You accumulate prompt patches that only work around old-model quirks, then a forced migration becomes a rewrite
- Your competitors ship on newer pins you never evaluated
Pinning without a migration calendar is just delaying a worse incident. Pair pins with a quarterly “candidate upgrade” ritual.
Config shape that survives review
# models.yml — reviewed in PRs
agents:
support_triage:
provider: anthropic
model_id: claude-sonnet-5 # pinned snapshot ID
prompt_version: support-triage@3.2.0
tools_version: crm-tools@1.7.0
last_eval_passed: 2026-03-10
rollback_model_id: claude-sonnet-4-6CI fails if model_id matches a denylist of aliases (latest, gpt-5.6, sonnet, opus).
Pilot minimum
A Spurlock $1,500 · 5-day agentic pilot ships:
- One pinned
model_idper agent role in config - Golden-set gate wired so a pin change is a scored PR
- Online sampling panel that would have caught the quiet alias weekend
You do not need multi-provider routing on day one. You need identity and a gate.
FAQ
Which OpenAI / Anthropic / Gemini IDs should I pin today?
As of August 2026 docs: OpenAI gpt-5.6-sol / gpt-5.6-terra / gpt-5.6-luna by role (avoid the gpt-5.6 alias in prod); Anthropic claude-fable-5, claude-opus-5, claude-sonnet-5, or dated claude-haiku-4-5-20251001; Gemini gemini-3.6-flash (and Lite variants when earned). Re-verify on provider model pages before hardcoding months later.
How long do providers keep pinned snapshots?
Until they deprecate and retire them — not forever. OpenAI documents at least six months’ notice for GA model retirement after announcement; Anthropic documents at least sixty days before retirement for public models. Gemini notice windows vary by model card/changelog. Track the deprecation tables; do not assume multi-year API access.
Staging on floating vs prod on pinned — good pattern?
Yes, if staging evals run on a schedule and someone owns failures. Prod stays pinned. Staging float without alerts is how you learn about provider changes from Twitter instead of CI.
What belongs in SemVer for prompts vs model pins?
Version prompts/rubrics (prompt_version), tool contracts (tools_version), and the eval suite separately from model_id. A model pin change is a config change that must pass the golden-set gate even when the prompt SemVer does not bump.
How does online sampling catch input-distribution drift?
Offline goldens freeze yesterday’s tickets. Online samples score today’s mix with the same evaluator. When online pass/revision rates diverge from offline, you are seeing distribution drift — not necessarily a bad pin. Investigate before you “fix” the model.
What breaks if I never unpin?
Forced retirement outages, prompt debt that only works on the old pin, and a painful big-bang migration. Pin for stability; schedule candidate upgrades so unpinning is a controlled PR, not an incident.
CTA
Want pins, gates, and drift panels on a real agent job in five days? Start at /agentic or /contact?intent=agentic-pilot.
Keep reading.
AI Agents Why Pass Rate Lies: Revision Rate, Trajectories, and Coverage
Pass rate flatters bad agents. Gate deploys on revision rate, trajectory scores, eval coverage, and cost per successful task—not a single green percentage.
AI Agents Golden Sets from Production Failures: Turn Bad Runs into Regression Fuel
Turn a bad production agent run into a regression test by harvesting the trace into a stubbed fixture with a clear pass criterion.
AI Agents LLM-as-Judge Reliability: Calibrate the Scorer Before You Trust the Score
An LLM judge is a noisy instrument, not ground truth. Calibrate against human labels, kill position and verbosity bias, re-check after model or rubric changes.