Polish company RAG: two studies, two different winners

A company knowledge assistant is a complete knowledge workflow in which the language model serves as the generator. The system must retrieve the right source, apply permissions, use the effective document version and retain source evidence. The team should choose Qwen, Bielik, PLLuM or an API model after defining those requirements.

Two 2026 papers show why a model name cannot replace an evaluation. Bielik led a RAG study over technical documentation. PLLuM led another study over Polish Wikipedia. Each result belongs to a particular checkpoint, corpus, retriever and evaluation procedure.

Study one: real technical documentation

The paper Evaluation of Two Leading Polish Language Models in a Real-world RAG Scenario used real technical documentation for a low-code platform, split into about 1,200 chunks. The evaluation contained 146 reference questions selected as likely user questions, with employees helping to prepare the reference answers.

The best vector retrieval setup, using OrlikB/KartonBERT-USE-base-v1, produced:

Here, precision is the share of retrieved passages matching the reference set under the paper's definition, while recall is the share of relevant passages that retrieval found. F1 is the harmonic mean of those two measures, and NDCG accounts for ranking by giving more credit to relevant passages near the top. These values describe retrieval quality, separate from the generator's answer quality.

The generator received five documents. The authors used quantitative scoring and pairwise comparisons, with three separate automated evaluator models: gpt-oss-20b, Mistral-Small-3.2-24B-Instruct-2506 and Qwen3-30B-A3B-Instruct-2507. Mean answer scores on a 1–5 scale were:

The study also exposed an order effect. When PLLuM's answer appeared first, Bielik won 81.5% of comparisons. After answer order was reversed, Bielik's share fell to 54.3%. Pairwise evaluation needs randomised order and checks for evaluator bias.

On this scale, 5 meant fully correct and comprehensive, while 1 meant incorrect or irrelevant.

Study two: Polish Wikipedia

The paper Evaluating Cost-Efficiency of LLMs in a RAG Setup on Polish Wikipedia used 1,000 PolQA questions. Its pipeline retrieved candidates from Polish Wikipedia, reranked them and sent the selected passages to the generator. GPT-4o provided the pairwise judgments as a separate automated evaluator.

The authors used a Bradley-Terry model, which converts pairwise wins into a shared ranking. A higher rating means higher estimated quality under this procedure. The ± value is the uncertainty reported by the authors.

PLLuM led this procedure. The first paper evaluated a different Bielik checkpoint, corpus, question set, retriever, reranker and judging procedure. “The best Polish RAG model” is meaningful only when those details are specified.

Where Qwen3.8-27B fits

The official Qwen3.8-27B model card describes a dense 27B model with a native 262,144-token window. In our home experiment, Qwen recalled every checked fact from prompts containing 50,059, 115,074 and 230,085 tokens. The longest run took 563.4 seconds and peaked at 23,623 MiB of VRAM.

This result justifies testing Qwen as a candidate for long-document work. It does not settle the generator choice for Polish RAG. Our Qwen run checked three facts planted in a long input, while the RAG scores above belong only to the checkpoints used in the cited papers.

A knowledge system has at least two results

The first result concerns retrieval: did the right passage rank high enough? The second concerns generation: did the model preserve the source meaning, citation, version and justified refusal? Track them separately because each points to a different failure.

A single aggregate score hides the failure location. When an answer is wrong, the team should be able to determine whether:

• parsing lost a table or heading;
• chunking separated a rule from its exception;
• retrieval missed the effective document;
• an access filter removed an allowed source or admitted a forbidden one;
• generation ignored the supplied passage;
• the index retained an obsolete version.

Evidence retained with every answer

An answer without a source trace is difficult to check and repair. An operational system should retain document identity and version, ranked passages, user identity and access-filter result, exact checkpoint and configuration, final answer and reviewer decision.

That record allows the case to be repeated after a parser, embedding or generator change. Without it, an upgrade becomes an impression contest.

A pre-deployment evaluation

Build the set from questions employees actually ask. The process owner identifies the source, effective version, acceptable answer and refusal cases. Include access-control cases from the beginning.

Compare exact configurations on the same index. If retrieval changes, score retrieval separately from the generator. The report should show every rejected answer and its reason. An average alone is insufficient.

What Syntalith can build and transfer

We prepare the source audit and question set, implement parsing, indexing, role filters, citations, a review panel and an upgrade gate, then compare local models with a sensible API alternative. If Qwen wins on the buyer's task, we deploy Qwen. If Bielik, PLLuM or a simpler non-LLM system wins, we recommend that option.

See our AI process audit and custom AI applications, or begin with a free process scan. We deliver implementation, acceptance evidence, documentation and training for the people who will own the system.