Polish company RAG: two studies, two different winners

An employee asks which procedure applies. The assistant must find the right document, check its version and the employee's permissions, then answer with a source. Qwen can generate the answer, but retrieval and access controls must be built around it.

Evaluate the whole task when choosing the model. Two 2026 papers reached different results: Bielik led a RAG study on technical documentation, while PLLuM led one on Polish Wikipedia. They used different checkpoints, documents, retrievers and evaluation methods. Neither result settles what will work on your company's files.

Study one: real technical documentation

The paper Evaluation of Two Leading Polish Language Models in a Real-world RAG Scenario used real technical documentation for a low-code platform, split into about 1,200 chunks. The evaluation contained 146 reference questions selected as likely user questions, with employees helping to prepare the reference answers.

The best vector retrieval setup, using OrlikB/KartonBERT-USE-base-v1, produced:

Here, precision is the share of retrieved passages matching the reference set under the paper's definition, while recall is the share of relevant passages that retrieval found. F1 is the harmonic mean of those two measures, and NDCG accounts for ranking by giving more credit to relevant passages near the top. These values describe retrieval quality, separate from the generator's answer quality.

The generator received five documents. The authors used quantitative scoring and pairwise comparisons, with three separate automated evaluator models: gpt-oss-20b, Mistral-Small-3.2-24B-Instruct-2506 and Qwen3-30B-A3B-Instruct-2507. Mean answer scores on a 1-5 scale were:

The study also exposed an order effect. When PLLuM's answer appeared first, Bielik won 81.5% of comparisons. After answer order was reversed, Bielik's share fell to 54.3%. Pairwise evaluation needs randomised order and checks for evaluator bias.

On this scale, 5 meant fully correct and comprehensive, while 1 meant incorrect or irrelevant.

Study two: Polish Wikipedia

The paper Evaluating Cost-Efficiency of LLMs in a RAG Setup on Polish Wikipedia used 1,000 PolQA questions. Its pipeline retrieved candidates from Polish Wikipedia, reranked them and sent the selected passages to the generator. GPT-4o provided the pairwise judgments as a separate automated evaluator.

The authors used a Bradley-Terry model, which converts pairwise wins into a shared ranking. A higher rating means higher estimated quality under this procedure. The ± value is the uncertainty reported by the authors.

PLLuM led this procedure. The first paper evaluated a different Bielik checkpoint, corpus, question set, retriever, reranker and judging procedure. “The best Polish RAG model” is meaningful only when those details are specified.

Where Qwen3.8-27B fits

The official Qwen3.8-27B model card describes a dense 27B model with a native 262,144-token window. In our home experiment, Qwen recalled every checked fact from prompts containing 50,059, 115,074 and 230,085 tokens. The longest run took 563.4 seconds and peaked at 23,623 MiB of VRAM.

This result justifies testing Qwen as a candidate for long-document work. It does not settle the generator choice for Polish RAG. Our Qwen run checked three facts planted in a long input, while the RAG scores above belong only to the checkpoints used in the cited papers.

A knowledge system has at least two results

The first result concerns retrieval: did the right passage rank high enough? The second concerns generation: did the model preserve the source meaning, citation, version and justified refusal? Track them separately because each points to a different failure.

A single aggregate score hides the failure location. When an answer is wrong, the team should be able to determine whether:

• parsing lost a table or heading;
• chunking separated a rule from its exception;
• retrieval missed the effective document;
• an access filter removed an allowed source or admitted a forbidden one;
• generation ignored the supplied passage;
• the index retained an obsolete version.

Evidence retained with every answer

An answer without a source trace is difficult to check and repair. An operational system should retain document identity and version, ranked passages, user identity and access-filter result, exact checkpoint and configuration, final answer and reviewer decision.

That record allows the case to be repeated after a parser, embedding or generator change. Without it, an upgrade becomes an impression contest.

A pre-deployment evaluation

Build the set from questions employees actually ask. The process owner identifies the source, effective version, acceptable answer and refusal cases. Include access-control cases from the beginning.

Compare exact configurations on the same index. If retrieval changes, score retrieval separately from the generator. The report should show every rejected answer and its reason. An average alone is insufficient.

What Syntalith can build and transfer

Syntalith builds private assistants over company documents. We review sources, prepare evaluation questions and compare models. Deployment covers file parsing, indexing, permissions, citations and answer review.

If the model still confuses internal categories or formats despite good sources, we can assess fine-tuning. Tell us about your documents and the team's questions so we can scope a trial. One department and one dataset are enough to begin.