Local LLM: how to test its Polish

A model that writes fluently can turn “does not apply” into “applies”, confuse a product name or add a condition that was never in the source. These errors only show up on a test set that contains them. Anthropic describes building evaluations from specific tasks and grading criteria.

The test set should contain the hard parts of Polish

Prepare short texts with an unambiguous answer and a place where it can be checked. In a fictional example the sentence reads: “Delivery by Friday has not been confirmed; we will set the expected date after the supplier replies”. The summary “delivery on Friday” loses the most important information.

Add inflected forms of names, an industry abbreviation and two similar company names. Check whether the model tells the parties apart and keeps the uncertainty. A separate test should hold a question the document does not answer. A useful result points out the gap and leaves the likely value out.

Compare the same task configuration

Give the candidates the same text and the required format. Keep the model version, the settings and the input file. If you change the way the material is fed in, record it as a separate variant. Otherwise it is hard to tell whether the model or the data preparation caused the difference.

The grading can cover whether the finding is correct, the source, whether gaps are kept and the time spent checking. Do not merge factual errors and style preferences into one score without explaining it. A shorter answer that kept the meaning can be better than a long summary with one false detail.

Hardware and the way the model runs are part of the local LLM test

Test the model in the environment planned for the users. The response time for one request does not describe behaviour when the team works at the same time. Write down the waiting time, the document length and the cases where the tool did not finish the task.

LM Studio describes working with downloaded models and documents without the internet, and separately the network operations, such as downloading models and updates. That is a property of one specific way of using it. It does not prove that any app with a local model sends no data. LM Studio documentation, checked on 9 October 2026.

When is the result enough for the next decision?

Keep the failed attempts too, and grade new texts after you have chosen a candidate. A small set helps you reject weak configurations; it does not justify a promise of zero errors. This article proposes a method and does not present a new model benchmark.

Syntalith can discuss a private LLM together with the task, the environment and the acceptance criteria. The scope and the price need to be agreed. Before you choose the infrastructure, also read the comparison of a local model and an API. For the first call, it is enough to describe the type of documents and the errors the recipient cannot accept.

What a local model for Polish documents costs and when it pays back

Where to start

In the free process review we choose one task and 20 to 100 documents the team has already worked on, and within 2 working days you get a written summary with a price. We roll out the local model as part of private LLMs, and the app around it as an AI app.