When should you fine-tune AI to group complaints?

Three descriptions of a drawer

Suppose a furniture maker receives these messages from three customers: „The drawer does not close all the way”, „After I push it in, the drawer stays slightly open” and „I cannot close the drawer, and the handle is loose”. For its report, the quality team agreed the categories „drawer closing problem” and „loose handle”.

The first two descriptions would go into the same group, although the customers used different words. The third contains both reported symptoms. When an employee has to choose only one category, the report loses either the handle problem or the closing problem. A model that helps with this work should be able to assign more than one category to a complaint.

Counted this way, three complaints contain three reports of a closing problem and one report of a loose handle. The symptoms add up to four, although there were three customers. The person reading the report has to know what the numbers mean. Otherwise better-ordered descriptions can lead to a wrong conclusion about the number of complaints.

These messages do not yet show why the drawer does not close. A later inspection of the product can establish the cause. Keep the customer's original description next to the assigned categories, and record the findings of the inspection separately. Then the analyst can go back to what was actually reported and compare it with the later diagnosis.

First a shared meaning for the categories

Useful grouping needs a fairly simple agreement in the team: what do we want to count together? If one person marks every drawer problem as „mechanism” and another separates difficulty opening from difficulty closing, their reports will not be comparable. Changing the model will not resolve that.

You do not need a detailed breakdown of every possible problem at once. Start with the ones the team already discusses at meetings and can recognise in messages. For descriptions such as „the drawer is broken”, you can keep a general label or send the case back for more detail, as your way of working says. A more precise category needs more precise information.

Old assignments can be useful, but they need a review. They may follow an earlier breakdown or include the service technician's knowledge that was not in the customer's message. If the model is to group only the complaint descriptions, assess the examples from that point of view. Otherwise we expect it to recognise information it never received.

Do you need fine-tuning?

With a small number of complaints, a shared category list and manual labelling in the current tool may be enough. If customers choose the problem in a form, check whether those answers already give the breakdown you need. Rules that look for typical phrases can also help, although in the drawer example they have to cover the different ways of describing the same symptom.

Another option is a ready model that has been told what the categories mean. A trial on real complaints will show whether it recognises other wordings and keeps both problems from the third message. If the employee has few corrections to make, extra training may bring no benefit that justifies maintaining it.

Fine-tuning means further training of an existing model on examples that show the correct answers. Here the team would teach it to map the ways customers describe problems to its own categories. Google lists classification among the uses of this kind of adaptation. It recommends first refining the instructions for the model, checking recurring errors and making sure the examples are labelled well.

So consider adaptation when the team has consistent categories and a ready model still regularly confuses certain descriptions or misses the second symptom. You need examples where employees can explain the correct assignment.

What will show that the report is better

The comparison should cover later complaints that were not used to adapt the model. A quality employee then checks whether the proposals match the customers' descriptions and how much they have to correct before preparing the report. Set this result against the ready model and against the simpler method the team uses today.

A model may miss loose handles while recognising frequent closing problems well, especially when the customer mentions the handle as a second fault. So the person ordering the solution should see what happens to each group and to messages that contain several problems. In our example, missing the handle would remove that symptom from the whole report.

A new product or a change in how complaints are broken down will require the work to be checked again. When you buy, agree who assesses new mistakes and who decides on changes to the categories. Grouping for quality analysis is a separate task from deciding which team should handle a case. We describe that second topic in the article on fine-tuning to route support tickets.

How Syntalith can help prepare the report

Syntalith builds applications and adapts AI models. For a quality team, we suggest starting with the report that today needs complaints read again. We check how they are grouped and compare simpler methods with a ready model. If repeatable errors remain, an adaptation trial shows whether further work on the model is justified.

The chosen solution should show the employee the proposed categories next to the original message, so they can correct the assignment before it goes into the report. For the first call, it is enough to describe the report and an example of a problem that is hard to count today. You do not need a specification or customer data. How to work with examples can be agreed later; details of the offer are on the Syntalith pricing page.

What it costs

Where to start

In the free review we choose one process and 20 to 100 cases the team has already handled. We compare the simpler change with an adapted model on cases the model has not seen, and within 2 working days you receive a written summary with a price. We adapt models as part of private LLMs, and build the application as part of AI apps.