In our recent look at machine-learning (ML) text classifiers, we showed how a business can cut costs by orders of magnitude by using AI solutions that efficiently solve problems with smaller, cheaper models. A custom-trained ML model can perform as well or even better than more-powerful large language models (LLMs) for specific business use cases. But how do you create a small, cheap model that can perform as well as an LLM?
You may have heard about the process of distillation, where you can "teach" a smaller model with a larger model. Geoffrey Hinton and his co-authors mentioned the analogy of larval insects in the landmark 2015 paper where they introduced the concept:
Many insects have a larval form that is optimized for extracting energy and nutrients from the environment and a completely different adult form that is optimized for the very different requirements of traveling and reproduction.
They point out that the requirements for a model during training are different from the requirements at runtime, when efficiency becomes more important than the ability to identify patterns and to generalize. The distillation process that they developed transfers knowledge from large, cumbersome models to "student" models that are efficient enough to use at runtime.
For large language models, the distillation process is generally a form of fine-tuning that connects the outputs of the larger model to the smaller model for training in a sort of mind-meld. Our goal is to replace the LLM with a simple ML model, not a smaller LLM, so we can't use the distillation process. But there's a related idea that's even simpler.
Rather than using the teacher model in a fine-tuning process for a smaller LLM, we can connect the teacher model to a student model through training data. This method was described in early 2023 by researchers at Amazon:
To enable models that are lightweight enough for runtime use, even when real training data is scarce, we propose teaching via data (TvD), in which we use an LLM-based “teacher” model to generate synthetic training data for a specific task, then use the generated data to fine-tune a smaller “student” model.
It's a simple process: You use the teacher model to perform the task that you want a smaller model to be able to do, using real data. That data then becomes a labeled training set for training a small, new ML model from scratch.
In our recent case study on optimizing an AI/ML solution for cost efficiency, we created a text classifier that can recognize inquiries about Spanish immigration law by using GPT 3.5 to generate a set of labeled sentences that either did or did not meet the criteria. Then we used that as training data to create an ML text classifier using logistic regression.
We generated data that looked like this:
| label | sentence |
|---|---|
| 0 | The cat sat on the mat. |
| 1 | How long does the visa application process take for Spain? |
| 0 | I love sunny days in the park. |
| 1 | What is the cost of a student visa for Spain? |
Okay, but, "The cat sat on the mat"?
That's an actual sentence that GPT 3.5 generated for our training data set for that classifier. And it did work really well. But we can hypothesize that we probably could get better accuracy with real-world classification using more realistic data for training.
The problem is that we're trying to do prompt engineering to create a training dataset that will work, but we're driving blind. The LLM can only guess at what kinds of sentences it should include as examples. It doesn't know anything about the typical length of the sentences that the classifier will process at runtime, or typical topics for the non-matching sentences, or anything. And we could make subtle mistakes in our prompt engineering that could mis-align our training data with our problem.
It would be better to eliminate any prompt engineering from the process entirely so that we can replace an LLM classifier with an ML classifier using only its own output. We can do that by simply using the LLM classifier to process some real-world sentences, then treating those classifications as labels in a training dataset.
Do we need millions of examples for training a model? Luckily, no. Not for task-specific ML models like this. We got great results from only a thousand records in our previous text classifier example, and there are plenty of examples of training useful classifiers with only hundreds of records.
Continuing scoping work for a call center project, we can't deploy any text classifiers until we do some groundwork first. We have transcriptions of call center recordings that are labeled Speaker A and Speaker B, but we don't know if Speaker A is the call center agent or the contact. Before we can deploy any classifiers for detecting useful things in the conversations, we need to identify the speakers. We'll use this speaker classifier to test the idea of using TvD to train a classifier based on an ML model using an LLM classifier.
We need to transform each call center transcript into a prompt to an LLM that we can use for classifying the speaker. A simple option is to create a transcript that shows only the utterances from one speaker, masking the other speaker. For example:
Speaker 0: Thank you for calling ABC Spanish Immigration Law Consultants, how can I help you?
Speaker 1: ...
Speaker 0: Certainly, I'll be happy to help you look into that.
Speaker 1: ...
We would also get a second masked transcript from the same call:
Speaker 0: ...
Speaker 1: Hi, yes, I live in Florida but I want to learn what I would have to do to move to Barcelona?
Speaker 0: ...
Speaker 1: Okay, cool. Thanks.
You can see from the example that it's easy to see whether Speaker A is the call center agent or the contact from just a couple of utterances. So, let's only include the first three utterances from each side in our transcripts.
We'll create a prompt from each masked transcript by adding instructions about classifying the transcript by speaker, giving the LLM a function to call for recording the classification labels.
The following transcript represents one side of a call center phone call transcript. We do not know whether this side of the call is from the call center agent or the contact (customer). Look at the transcript and classify Speaker 0 as either the agent or the contact. Respond by calling the
classify_agent_or_contactfunction.
Here's that function definition:
{
"type": "function",
"function": {
"name": "classify_agent_or_contact",
"description": "Identify the speaker as either the call center
agent or the contact/customer",
"parameters": {
"type": "object",
"properties": {
"classification": {
"type": "enum[agent, contact]",
"description": "Either \"agent\" or \"contact\".",
},
},
"required": ["classification"],
},
}
}
[PENDING]
[PENDING]
[PENDING]