Fine-tuning, RAG or training your own model?

A model answers from an outdated policy. Or it knows the policy but keeps assigning complaints to the wrong department. These failures look similar to a user. One concerns access to information; the other concerns how the task is performed. That distinction helps choose between RAG and fine-tuning.

At Syntalith, we begin by recording errors made by the existing model. A few concrete examples usually define the work better than a request to train AI on every company document.

RAG supplies information at query time

RAG combines retrieval with answer generation. An application finds document passages, checks permissions and supplies them to the model with the question. The model weights remain unchanged. A revised instruction can therefore become available without retraining.

This fits policies, product information and operating procedures. Retrieval quality matters: if the search misses the relevant document, a larger generator can still answer incorrectly. We evaluate retrieved sources separately from the final answer. A citation should lead to a passage that supports the statement.

Fine-tuning teaches a task through examples

Supervised adaptation pairs an input with the expected output. For a ticket, that might be department, priority and escalation reason. For a document, it could be fields in an agreed schema, with missing information represented explicitly.

LoRA trains additional matrices while freezing base weights. QLoRA combines that approach with a quantized base model. This reduces training resource requirements compared with updating every weight. Inference still needs the base model and a compatible way to apply the adapter.

Fine-tuning is worth evaluating when an existing model repeatedly mishandles internal categories, writing style or output structure. Information that changes often or must be deleted is easier to govern in a separate source than in model weights.

When a small model is enough

Choosing one of a dozen departments may be handled by a classifier, an embedding model or a small decision model. Free-form text can add latency when the required output is simply a correct category.

Projects inspired by Jev illustrate this direction. We compare them with simple rules or classifiers first. If a small model meets the acceptance threshold, more of the budget can go into data controls and integration.

Training from scratch brings broader responsibility for data and compute costs. It can make sense for a small, narrowly defined model. Building a general LLM from scratch needs a separate economic case; a company knowledge base does not automatically justify it.

Training data requires human decisions

Examples should represent actual work, including rare and ambiguous cases. If two reviewers label the same case differently, clarify the rule first. A model cannot resolve an inconsistent departmental policy.

We separate training, validation and final evaluation sets. Related messages from one case stay in the same partition to prevent leakage into the test. We check data rights, minimize personal information and record dataset versions. For distillation, we also check whether the teacher model provider permits using its outputs for training.

What the comparison should deliver

Evaluate the base model, a stronger prompt or RAG setup, and the adapted model. Use examples unseen during training, then measure completion time and error costs. Record regressions as well as improvements.

Syntalith selects and fine-tunes private models, prepares datasets and deploys the chosen version. Describe the task, expected output format and several current errors. We can establish whether training is justified and how to measure its effect.