Migrating an agent with a reference set and a tested return path

Behavior changes often surface only after a client asks why their case was handled differently. Automations built in sunsetting tools may have no tests or documentation, leaving the team without a written definition of correctness for the migration.

Our migration program starts by extracting a definition of correctness from the old system. New code comes after that baseline exists.

A reference set from old behavior

In our agent migration case, a saved record supplied 336 cases with the source system's inputs and expected decisions. The controlled case uses a generated catalog and contains no client data or credentials. The old system's actual responses form the reference set and the first written specification of its behavior.

The comparison report checks the new agent against that set case by case and lists 288 strict matches and 48 discrepancies. Human reviewers classify every discrepancy as an old-system defect, an intended change, or an error to fix. At measurement time, 41 discrepancies carried a closed decision and 7 remained open.

Open discrepancies block the full cutover. Each unresolved item leaves a hole in the definition of correctness, regardless of whether review eventually accepts or fixes it.

Building a reference set worth trusting

Historical behavior is a starting point, but not every old output should become the target. Segment the sample by intent, channel, customer type, tools used, refusals, and exceptions. Rare cases with serious consequences need deliberate inclusion because random sampling may miss them entirely.

Each case should retain the input, relevant process state, expected decision, permitted actions, and user-visible outcome. Separate behavior that must match from wording or presentation that may change safely. Character-for-character comparison creates noise when the important invariants are the decision, tool call, and transferred data.

The process owner should review known legacy defects before the set becomes binding. A defect can remain as a documented intended change, but it should not silently become the new system's expected answer.

Running the discrepancy register

Every difference needs an owner, category, impact assessment, and decision. Useful categories include replacement defect, legacy defect, intended change, neutral variation, and insufficient evidence. The closed decision should specify which behavior now becomes the regression target.

A headline match percentage is weak without this register. One wrong payment decision may matter more than hundreds of harmless wording differences. Cutover criteria should weigh the type, severity, and count of open items.

Once decisions close, the reference set becomes the new system's regression suite. Model changes and code releases should continue to run against it. Migration evidence then becomes a durable release control.

Cutover and return planning

The target-environment check should cover connections, permissions, limits, observability, and exception handling. Next, run the replacement in shadow mode on current traffic without letting its outputs affect cases. Review differences while the legacy system remains authoritative.

Move a small, reversible traffic segment first and define an observation window. The return plan names the decision owner, trigger signals, switching procedure, treatment of in-flight cases, and reconciliation after return. Measuring the switch itself does not prove that partially completed work and data will reconcile correctly.

Reading the evidence and cost

Reference-set agreement demonstrates saved cases. The target check demonstrates selected intents in one environment. A return trial demonstrates one technical path. None of them alone establishes capacity, resilience to dependency failure, or correctness on current production traffic.

Model cost per case helps estimate regression-run expense. The migration budget must also include building the set, reviewing discrepancies, parallel operation, two environments, and staff availability during cutover. Expert time spent deciding correct behavior is often the largest component.

Migration-readiness checklist

1. Does the reference set cover core intents, refusals, and rare high-impact exceptions?
2. Are known legacy defects labeled before comparison?
3. Which fields and decisions require exact agreement, and which may vary?
4. Does every discrepancy have an owner, category, impact, and decision?
5. Which open differences block cutover regardless of the overall match rate?
6. Has the replacement run in shadow mode on current traffic?
7. What happens to in-flight cases during a return?
8. Who orders the return, and which signals trigger it?

A target-environment check and a tested return

Comparison with a saved record does not exercise the replacement environment. The second check therefore sends two cases for each of six intents through that environment, with a review boundary and one read-only tool. All 12 cases followed the expected intent, review route, and scope, and every audit chain verified. The recorded run cost USD 0.030373 in total, or USD 0.002531 per case.

The third check covers the return to the old system. One local trial directed traffic to the new inventory, confirmed the target, and returned to the source in 2.086 ms. The team tests that return before cutover and records the result.

Evidence for the migration decision

Together, these layers give the automation owner a reference set, a discrepancy list with decisions, a record from the target environment, and proof of the way back. One screen shows which conditions are complete and which still block the traffic switch.

Seven open decisions still block the limited traffic switch. The 2.086 ms return check shows that the local traffic controller returned to the source once; repeated checks and parallel traffic belong in the production cutover plan.

Details are on the case page. If your critical flow runs on a sunsetting tool, a free process scan will help plan the reference set before migration begins.