Is your company's archive ready for fine-tuning?

A correct note with no visible basis

A company wants to prepare notes on the state of its projects. The archive holds a polished update: „Integration work completed, documentation awaiting review.” The author used the task changes and team messages of the time, but those materials were not saved with the note. A model trained on such an example receives an answer with no basis and learns to write confident sentences without a source.

A person who knows the project can help establish whether the information available to the author at that moment can be recovered, because today's state of the tasks does not recreate that situation. If the basis cannot be confirmed, the example needs completing or stays out of the set. Experts also differ in their judgements, so part of the preparation is agreeing what the model should reproduce.

How much a simpler change gives, and how much training adds

In this count a clear instruction with sources gives most of the effect for a fraction of the cost. Training makes sense at a larger scale or when the remaining errors are expensive. We count today's error rate on a sample of the archive assessed by experts, before anyone starts training a model.

The set of training examples has to be maintained: when the process, the naming or the base model changes, the evaluation is repeated on the same test set. We include that cost in the quote together with the training itself.

First check what a ready-made model cannot do

Fine-tuning means further training of an existing model on examples of the expected behaviour. A clear instruction and the right information are often enough for a good result. OpenAI's fine-tuning best practices stress correct and representative examples, consistent judgements and the presence of the information needed for the answer. No single number of records settles readiness for every project. When product data change often, consider RAG, as described in the article on fine-tuning or RAG for changing data.

Where to start

The first scope is a limited review of the archive with a person who can confirm whether the examples are correct. For the call, describe the task and the kind of examples kept, without sending the whole archive. We adapt models as part of private LLMs, and we build the app where an employee sees the result next to the source as part of AI apps. For the free analysis, a few examples of an error that keeps coming back and a note on who corrects it today are enough.