How to compare agent frameworks under the same conditions

Choosing an agent framework can consume weeks of team time. Many public comparisons use different models, instructions, and tasks at once, making it hard to identify what caused the result.

We built a workbench that holds those variables constant. It sends 50 generated cases through a local service that returns predetermined responses and contains no client data or credentials. Only the framework differs between variants.

Predetermined responses remove model variation

The method uses a local service in place of a model provider. It returns an answer based only on the case number, with identical recorded usage and a price of PLN 0 for every variant. This lets the comparison isolate adapter overhead.

Six implementations of the same task, using LangGraph, Claude Agent SDK, OpenAI Agents SDK, CrewAI, n8n, and plain Python at fixed versions, receive the same fifty cases, including twelve expected refusals. A separate test runner compares each result with the reference set and records timing, so no framework grades itself.

What the benchmark found

Across 300 observations, every implementation matched the gateway contract and all 72 expected refusals fired. With the deterministic gateway holding task behavior constant, the visible difference was adapter overhead.

The highest time overhead belonged to the Claude Agent SDK adapter from our own default stack. In the local run, the 95th percentile for the full path was 304 ms, compared with 1 to 13 ms for the other adapters. The measurement combines framework, process, command-line, container, and transport overhead, so it cannot assign the gap to one cause. We still keep the slowest result in the report.

Publishing the losing dimension makes the comparison useful. A result table engineered to flatter its author belongs in sales material.

Where the benchmark stops

The seeded gateway removes model behavior from the comparison, so the bench covers no real streaming, provider retries, live tool calling, or external rate limits. This local execution supplies a method and a shortlist. A pilot of the finalists on one real company workflow comes next.

Requirements before candidates

A framework should fit the system's operating model. Before selecting candidates, state whether the process needs durable state, long-running work, human intervention, retries, parallel execution, streaming, permission controls, and deployment into existing infrastructure. Include the team's language, licensing constraints, release cadence, and trace export.

This produces a shortlist of two or three candidates. Comparing six tools across a complete process is expensive, so a broad test can check the adapter contract while deep pilots focus on finalists.

A reproducible method

1. Define the task contract: input, permitted tools, output, and refusal conditions.
2. Create a reference set with common, boundary, and invalid cases.
3. Pin framework, runtime, and dependency versions.
4. Build thin adapters that execute the same contract.
5. Separate deterministic overhead tests from real-model trials.
6. Warm the system, repeat runs, and retain raw observations.
7. Use an external runner so no framework grades itself.
8. Publish configuration, missing measures, and limits beside the result.

Median describes typical timing, while the 95th percentile reveals slower runs that affect production. Longer tasks also need success rate, retries, recovery time, cost, and trace completeness.

Sources of a misleading ranking

Different process counts, a cold start for one candidate, a busy shared machine, or different network settings can move the result. A command-line adapter may pay process startup overhead that an in-process library avoids. The result then describes the whole tested configuration and requires diagnosis before assigning a cause.

Adapter correctness matters as much as the runner. A benchmark can measure an inefficient use of a framework. Public adapter code and review by someone familiar with the tool reduce this risk.

Pilot finalists on the real workflow

After the technical bench, run the best two candidates through the same slice of the company process. Use controlled data, identical tool policies, and one acceptance set. Include diagnostic ergonomics: time to locate a bad step, resume work, and explain a refusal to an operator.

The decision record should contain criteria, result, limits, owner, and review triggers. A model change, major framework upgrade, or new process requirement may justify another measurement.

Technology decision checklist

• Does every candidate execute the same contract?
• Are versions, machine, and startup conditions recorded?
• Does an external set score correctness?
• Is adapter overhead separated from model behavior?
• Are median, 95th percentile, errors, and raw observations available?
• Do unmeasured fields remain empty?
• Have finalists run on a real workflow?
• Can the team maintain and diagnose the selected tool?

That is how we use the workbench in practice. A client comparing LangGraph and CrewAI gets one shared measurement method, then a pilot on the actual process. A separate small-model connection check using gpt-5-mini cost about PLN 0.0019 and confirmed trace capture, but it does not affect the framework comparison.

The method, the adapters, and the full run report are described on the case page. If you are facing an agent-platform choice, a free process scan is a good place to test this measurement against your case.