Implementation comparison: how the workflow works

What we really compare

Technology comparisons often use different data and rules for each option, so it is hard to base a decision on them. A wrong choice shows up late and means rewriting the system after several weeks of work. That is why we first write down the tasks and the cost of an error, and only then run the options on the same input.

The fixed answers of the test endpoint separate integration time from model time. A difference in time then comes from the framework and the adapter, because the model always answers the same way. The required refusals check whether the framework carries a „do not perform” decision to the end of the run or loses it on the way.

How to read an agent framework benchmark

First you look at the refusals. A framework that performed the task where a refusal was required drops out, whatever its time. Then at answer format compliance, because every format error means extra handling code in the system. You compare p95 time last, between the technologies that passed the first two thresholds. A „Not checked” field means no measurement and does not enter the average.

According to METR (2025), experienced developers with AI tools took 19% longer, although they expected a speed-up. A team's impression after a few days with a framework easily misleads, so we record the result on a fixed set before the decision.

What one run does not tell you

The result showed clear differences in run time between the six approaches, including in solutions we use ourselves. One run does not explain the cause of a difference. Fixed answers remove the influence of the model, and they also hide how frameworks behave with real streaming, retries, tool calls and provider limits. We do not present such a result as a vendor benchmark.

Anthropic, in Building effective agents, advises starting with the simplest solution and adding complexity only when the result needs it. Plain Python in the set serves exactly that purpose: it shows what a framework adds over ordinary code. According to Gartner (2025), of the thousands of agentic AI vendors only about 130 offer real agentic solutions.

The first trial in a company

For a pilot you need examples with an agreed correct result, a decision owner with acceptance criteria, and data sources with a defined scope of access. We build agent systems as part of AI agents for €3,000 to €17,500 net, in 6 to 16 weeks. For the free process review, a description of the process and a list of the technologies the team is considering is enough.