Polish company RAG: two studies, two different winners

A “company brain” is not a model with a document folder poured into it. It is a system that must retrieve the right source, apply permissions, use the effective document version and expose evidence. Only then should the team choose Qwen, Bielik, PLLuM or an API model as the generator.

Two 2026 papers show why that component cannot be selected by brand. Bielik led a RAG study over technical documentation. PLLuM led another study over Polish Wikipedia. Both results are real and cover different checkpoints and pipelines.

Study one: real technical documentation

Evaluation of Two Leading Polish Language Models in a Real-world RAG Scenario used real technical documentation for a low-code platform, split into about 1,200 chunks. The evaluation contained 146 reference questions selected as likely user questions, with employees helping to build the reference answers.

The best vector retrieval setup, using OrlikB/KartonBERT-USE-base-v1, produced:

The generator received five documents. Mean scores on a 1–5 scale were:

The study also exposed judge bias. When PLLuM's answer appeared first, Bielik won 81.5% of comparisons. After answer order was reversed, Bielik's share fell to 54.3%. Pairwise evaluation without order randomisation can therefore manufacture a large apparent margin.

Study two: Polish Wikipedia

Evaluating Cost-Efficiency of LLMs in a RAG Setup on Polish Wikipedia used 1,000 PolQA questions. Its pipeline retrieved candidates from Polish Wikipedia, reranked them and sent the selected passages to the generator. GPT-4o provided pairwise judgments.

The Bradley-Terry ratings were:

PLLuM led this study. That does not invalidate the first paper. The Bielik checkpoint, corpus, questions, retriever, reranker and judging procedure all changed. “The best Polish RAG model” is not a useful claim without those details.

Where Qwen3.8-27B fits

The official Qwen3.8-27B model card describes a dense 27B model with a native 262,144-token window. In our home experiment, Qwen recalled every checked fact from prompts containing 50,059, 115,074 and 230,085 tokens. The longest run took 563.4 seconds and peaked at 23,623 MiB of VRAM.

That result places Qwen on the shortlist for long-document testing. It does not settle the generator choice for Polish RAG. Our Qwen run only checked whether the model could retrieve three facts from a long input; the RAG scores above belong only to the checkpoints used in those papers.

A knowledge system has at least two results

The first result is retrieval: did the right passage rank high enough? The second is generation: did the model preserve the source meaning, citation, version and justified refusal?

A single aggregate score hides the failure location. When an answer is wrong, the team should be able to determine whether:

• parsing lost a table or heading;
• chunking separated a rule from its exception;
• retrieval missed the effective document;
• an access filter removed an allowed source or admitted a forbidden one;
• generation ignored the supplied passage;
• the index retained an obsolete version.

Evidence retained with every answer

An answer without a source trace is difficult to accept and difficult to repair. An operational system should retain document identity and version, ranked passages, user identity and access-filter result, exact checkpoint and profile, final answer and reviewer decision.

That record allows the case to be repeated after a parser, embedding or model change. Without it, an upgrade becomes an impression contest.

A pre-deployment evaluation

Build the set from questions employees actually ask. The process owner identifies the governing source, effective version, acceptable answer and refusal cases. Access-control cases belong in the same set rather than being added after model selection.

Compare exact profiles on the same index. If retrieval changes, score retrieval separately from the generator. The report should show every rejected answer and its reason. An average alone is insufficient.

What Syntalith can build and transfer

We can start with a source audit and question set, then implement parsing, indexing, role filters, citations, a review panel and an upgrade gate. We compare local models with a sensible API alternative. If Qwen wins on the buyer's task, we deploy Qwen. If Bielik, PLLuM or a simpler non-LLM system wins, the recommendation should say so.

See our AI process audit and custom AI applications, or begin with a free process scan. Delivery includes implementation, acceptance evidence, documentation and training for the people who will own the system.