Fine-Tuning for Freight Document Extraction

Start by identifying the recurring error

Fine-tuning means adapting an existing model to a task on examples with correct answers approved by the company. In transport documents a typical problem looks like this: a carrier uses its own consignment note layout, and the model puts the loading date in the delivery date field or mixes up the order number with the registration number. Before you order training, collect a few dozen such mistakes and check whether they come from the layout, the image or the rules. The name of the method says little about what you buy and what it costs. What matters is whether a recurring kind of mistake can be reduced, and whether the improvement outweighs the cost of checking the results and maintaining the example set.

What a useful result should preserve

For every value, the reader sees which field of the document it comes from, and the system flags a missing basis when the document does not contain one. An operations employee checks the assignment, and the data owner decides whether an error comes from recognition, mapping or a rule. Without such a trail, nobody can say whether fine-tuning improved anything. That is why we first build the trail for the base model and count errors per field, and decide on training afterwards. It often turns out that most errors come from two or three fields that a validation rule can fix, for example comparing the delivery date with the loading date and the route schedule.

When fine-tuning for freight documents pays

Training is a one-off cost, but the example set has to be maintained. When categories or document layouts change every week, maintaining the example set costs more than the one-off training. Google describes training Document AI processors on labelled documents and evaluating them on a test set.

How to evaluate the purchase

The choice of cases also covers documents that are rare but expensive to get wrong, for example shipments with a late-delivery surcharge. A model adapted to the most common layout can do worse on rare documents, so the evaluation covers both kinds. After rollout, the same evaluation set serves to check the model after every change at the provider. First check whether a legibility gate removes most problems more cheaply. We adapt models as part of our private LLMs, and the cost of errors at different thresholds is covered in the post on pricing a model's errors.