The team tests six technologies on the same tasks before choosing a foundation for the new system.
Each technology receives the same 50 cases and the same response rules. A separate program measures time, errors, and required refusals. The result helps the team choose two options for a pilot on the company's real process.
What we measured
The comparison covers six approaches performing the same fixed task. The result shows differences but does not replace a pilot on the client’s actual workflow.
- Recorded measurement
- What we measured
- Six approaches · shared task set · required refusals
- Agent technology comparison
- Verified
- public benchmark
Problem, solution, and result
The problem
Technology comparisons often use different data and rules, making them a weak basis for a decision. A poor choice may surface late and force an expensive rewrite.
How it works
Six implementations receive the same fifty cases and response rules. An independent program measures overall execution time and records refusals and failures. The result becomes a starting point for a pilot on the buyer’s process.
What the recorded run covered
The comparison covers six approaches performing the same fixed task. The result shows differences but does not replace a pilot on the client’s actual workflow.
For whom
This test fits teams choosing agent technology from vendor opinions when changing the foundation after implementation starts would cost weeks of work.
Golden set → 6 adapters → measurement
- 01Six implementations get identical cases and one shared gateway
- 02The gateway removes the model, leaving the framework to be measured
- 03The team picks the stack after a pilot on its own workflow
- Business type
- Software and automation teams selecting an agent platform
- Input
- The same fixed set of 50 cases for each approach, including 12 required refusals
- Boundary
- Anything unmeasured reads "Not measured"; the bench cannot approve a stack without a pilot on your own workflow
- Cost
- About PLN 0.0019 for the separate recorded connection check.
- Security
- Each approach receives the same cases and refusal rules. Differences remain visible in the result.
- Speed
- We compare execution time on one fixed set. A result from one machine still needs confirmation in a pilot.
- Trace
- The history contains the comparison conditions, results, and the version of each tested approach.
- Similar system build
- from €6,000 net (approx. $6,500) · 6–16 weeks
The conclusions' boundary
This test does not choose the technology for the team
The comparison measures fixed tasks on one machine. It does not cover production traffic, failures in external services, or every integration. The technical lead chooses a solution only after a pilot on the company’s workflow and data.
- Boundary
- Anything unmeasured reads "Not measured"; the bench cannot approve a stack without a pilot on your own workflow
- Cost
- About PLN 0.0019 for the separate recorded connection check.
- Security
- Each approach receives the same cases and refusal rules. Differences remain visible in the result.
- Speed
- We compare execution time on one fixed set. A result from one machine still needs confirmation in a pilot.
The result includes technology we use ourselves
The comparison showed clear execution-time differences across the six approaches, including technology we use ourselves. One run cannot explain the cause of the gap. The result helps select two candidates for a pilot on the client’s actual process.
Estimated impact
Calculate the impact at your volume
This estimate uses the stated volume. Enter your own numbers to assess the possible impact in your company. The result is confirmed only during a pilot.
Today
Separate-PoC scenario: 24 days
With the system
Shared-bench scenario: 8 days
Time or cost saved
Modelled scenario: 12-20 days/decision, base 16 days
- Volume
- Modelled scenario: 6 variants
- Formula
- 24 days - 8 days
- Calculation status
- low
Data in the screenshots. The names, amounts, and documents shown in the screenshots are synthetic. Client data remains private. Measurements describe the demonstrator and are kept separate from production outcomes.
Working surfaces
The tech lead compares six implementations by one method and decides which go to a pilot on the team's own process.
The team receives one comparison method: a shared task set and a report that evaluates every approach in the same way. Unmeasured fields remain blank. A client pilot must still test production load, failure handling, and limits in external services.
Frame/Lab: method, implementations, and the shared run.
Method
The set, the gateway, and the rubric written down before the results; that order is what makes the measurement repeatable.
Six implementations
Adapter code at pinned versions, each with an identical task and input.
The shared run
300 observations with timings, refusals, and overhead against plain Python.
System screens
See how the system works in practice
These desktop and mobile screenshots come from the running application. They show the process and the points where a person makes the decision.
- Screens
- 11
- px
- 1440 · 390
- 021440×3315
Run history with its run ID, 300-observation scope, and trace. - 031440×1100
A controlled refusal framed on a phone.
Open the remaining screen archive (8)
- 041440×1100
A matching-answer case - 051440×1100
A controlled-refusal case - 061440×1100
Measurement method - 071440×1100
Implementations - 081440×1100
LangGraph: version, result, and code - 091440×1100
Claude Agent SDK: adapter result and code - 101440×1100
How to repeat the measurement - 111440×1100
Completed measurement history
Technology stack
The measuring layer is deliberately simpler than what it measures.
The adapter comparison uses a deterministic gateway to isolate platform overhead from model behavior. A separate run uses a live model and records the provider trace. The two measurements answer different questions and are not mixed into one ranking.
- Python 3.13 runner
- walks the set through the adapters, grades from outside, and writes the raw observations
- Deterministic gateway (FastAPI)
- returns an answer keyed only to the case number, with identical usage and a price of PLN 0
- Six adapters at pinned versions
- LangGraph 1.2.7, Claude Agent SDK 0.2.126, OpenAI Agents SDK 0.7.0, CrewAI 1.15.8, n8n 2.32.6, and Python 3.13.7
- PostgreSQL 17
- stores runs, steps, decisions, and the verified audit trail
- Next.js
- renders the report straight from the generated artifact, with no hand-entered values
The bench is public in full: the golden set, the code of all six adapters, the runner, and the raw run artifact, so anyone can rerun it or dispute it. Everything runs locally under Docker Compose and touches no client information or credentials.
Technical details and measurements
The working loop
Method before result. Otherwise the result is an opinion with a table.
First the frozen set, the shared gateway, and the grading rubric; only then the run. That order removes the option of fitting the measurement to a thesis, including ours.
A shared set: 50 cases, 12 of them expected refusals
Six adapters at pinned versions: LangGraph, Claude SDK, OpenAI SDK, CrewAI, n8n, plain Python
One shared deterministic test gateway
The runner grades from outside and times the full path
A report: contract agreement, per-adapter p95, and "Not measured" fields
System architecture
Golden set, six adapters, gateway, and runner.
Frozen cases, six equal implementations, deterministic responses, and a judge placed outside the variants under test.
- 01
The set
Cases and refusals are frozen.
Fifty cases with an expected transport contract, twelve refusals among them, exist before the run. No variant receives an easier version of the task.
- 02
Adapters
Six implementations of the same task.
LangGraph, the Claude Agent SDK, the OpenAI Agents SDK, CrewAI, n8n, and plain Python at pinned versions solve an identical problem, each written in its own native shape. Only the framework layer differs.
- 03
Gateway
The model replaced with a deterministic echo.
A local endpoint returns a result that depends only on the case number, with equal usage and a price of PLN 0. The measurement stops being about a model provider and starts being about the framework.
- 04
Runner
The judge stands outside the competitors.
Agreement with the gateway contract, time, cost, and trace are recorded by the runner. The report shows the agreement spread and each adapter's p95 relative to plain Python.
Plain Python provides the framework-free control
The control is an implementation with no framework at all. If a framework does not beat it on a measured dimension, its benefit must appear on another measured dimension, and the report should make that visible. For the same reason we publish the highest measured overhead, in the Claude Agent SDK lane we use ourselves, together with the limit on reading it: this run does not decompose the overhead into framework, process, container, and transport.
- The golden set and rubric exist before the run
- The deterministic gateway isolates framework overhead
- Plain Python serves as the no-framework control
- A result against our own stack goes into the report too
Want to assess a similar process in your company?
- A 30-minute call with the engineer who would lead the work.
- A review of the processes that cost you the most time and money.
- A written summary of what to automate first and the likely cost range.
Times are shown in your own time zone. We work with clients across time zones.
The scan is free and creates no obligation. If automation is unlikely to pay off, the written recommendation will say so.