Call Criteria runs quality assurance for call centers: human experts listen to calls and score them against detailed scorecards, one scorecard per client. By 2023 they'd already put AI on the problem. An outside scoring service ran custom fine-tuned LLM classifiers against their calls, and it worked. It just couldn't grow with them. Every new scorecard meant another round of coordinating changes, evaluating them, and getting them deployed through an outside relationship, and each round was slow and expensive. Scaling to every client meant either paying the vendor's price on every scorecard or waiting in the vendor's queue. They wanted the technology in their own hands.
They came to us with one ask: bring it in-house so they could scale it up, and take control of what it cost and how fast it moved.
We built them an in-house classifier lab on Plexus, our MLOps platform, deployed in their own AWS account so the models, the data, and the record belonged to them. Then we operated it for them, for years. A classifier lab is a running thing, and the value comes from running it well month after month.
Before any of that could work, the measurements had to. When we arrived, scorecards were evaluated for "accuracy" against historic human labels, and that number was lying in two ways. First, the human QA scores behind it weren't consistent with each other on a lot of questions, so the historic labels made a poor dataset to build anything on. Second, most of their classification tasks are badly imbalanced: on a question where nearly every call is a "Yes", a classifier that answers "Yes" every time evaluates at 99 percent accuracy while doing nothing at all. We moved them to a metric built for imbalanced data, Gwet's AC1, so a scorecard's number meant what it said.
Then we changed where the labels came from. Instead of training against history, the classifier scores real calls first, reviewers audit a share of the results, correct the ones that are wrong, and, most important, say why they changed or kept each label. A metacognitive loop reads those explanations, forms hypotheses about what the scorecard is missing, and hill-climbs from there against held-out review. Once a scorecard is aligned, the reviewers step back and the human labelling effort drops, with ongoing feedback keeping it aligned rather than rebuilding it. We made that a repeatable process and ran it many times over, one scorecard after another. It is the same pattern we now run on decision models like Jev, where one approved question lifted accuracy ten points with the model unchanged.
The first thing a lab lets you do is stop treating every scorecard the same. Some of their scorecards had to run on language ML models rather than LLMs, so we built many of those, and Plexus works with them like any other classifier backend. Then we ran the evaluations across the whole portfolio to sort it: which scorecards could move to an ML classifier that costs orders of magnitude less and scores just as well, which ones needed an LLM fine-tuned to the task, and which ones a hosted model handled fine. That is the maximize profit, not intelligence argument applied one scorecard at a time, with the evaluation numbers deciding rather than the vendor's default.
On top of that sorting we established the self-aligning optimization loop: evaluations test a hypothesis about a scorecard, a change that scores better on held-out review is promoted, one that doesn't is dropped, and the process hill-climbs accuracy version by version with the record to show for it.
With the measurement right and the labels coming from feedback, the lab turned QA into a loop rather than a queue.
That is reinforcement learning from human feedback as a daily operating procedure, and a data flywheel in the plain sense: the more the reviewers corrected, the less they had to.
Onboarding changed too. A new client scorecard used to take weeks of manual configuration. The lab reads the client's scoring criteria, generates a first classifier tuned to them, deploys it in days, and starts the loop immediately.
Running hundreds of scorecards this way meant running models every way there is. We served LLMs and fine-tuned LLMs with vLLM on AWS, served small language models through SageMaker Serverless Inference, and used every kind of hosted model where it fit. We trained and fine-tuned on our own GPUs, in SageMaker training jobs, and through hosted fine-tuning like OpenAI's. Which one a scorecard got was an evaluation result, and the lab made switching a routine change.
The reviewers didn't go away, and that was the design. Their judgment is what the system learns from; the lab's job is to spend their attention where it teaches the model the most.
The lab we built for Call Criteria became the product we run for everyone. Plexus grew out of this engagement, and the loop above is the one we offer as the classifier lab today: start scoring now, review continuously, let the corrections align the model, pick the cheapest backend that clears the bar, gate every change, and own the result.
We now run the same loop on the new class of decision models like Jev. Plexus treats a decision model as one more classifier backend, and the Jev-Flywheel repository shows the alignment loop in miniature: reviewer feedback calibrating a hosted decision model's confidence, then one approved question lifting its accuracy ten points with the model unchanged. If you have a high-volume judgment task and a review team that can't keep up, here's how an engagement works.