Primus
An MLOps platform built from a running lab
Primus is our platform for training, evaluating, deploying, and monitoring classifiers, the work known as MLOps. We built it as the core infrastructure for the lab we built for Call Criteria, where hundreds of call-scoring classifiers were built, measured and corrected by reviewers, and we have run it for years across millions of scored interactions. It manages the complete machine learning lifecycle for classifiers and AI agents at production scale.
Built for feedback at scale
Primus was designed for lab work: creating classifiers, measuring them, and refining them from reviewer feedback. At its core it runs a reinforcement learning from human feedback (RLHF) loop continuously in production, so each scorecard's verdicts track its reviewers' judgment more closely over time.
What it manages
- Model development: creating and experimenting with AI agents and classification models
- Training pipeline: training workflows that incorporate reviewer feedback
- Evaluation: held-out tests on every change, with metrics built for imbalanced data such as Gwet's AC1
- Deployment: a new version serves only when it clears held-out review on the slices that matter
- Monitoring: accuracy and drift tracked in production
- Continuous learning: retraining from reviewer corrections, with a record of what changed and who approved it
The feedback loop
- A classifier scores content in production.
- Reviewers audit a share of the verdicts, agree or disagree, and say why.
- Every disagreement is a labelled example aimed at the cases the model was least sure about, and the explanation becomes stated policy.
- The model refits on the corrections and a new version is evaluated against held-out review.
- The version that scores better is promoted, on the record.
That loop has run in production for years. Once a scorecard is aligned, the reviewers step back and the human labelling effort on it drops, with ongoing feedback keeping it aligned rather than rebuilding it.
Any model as a backend
- Multi-model: hosted LLMs, fine-tuned LLMs served with vLLM, small language models on SageMaker Serverless Inference, and decision models like Jev, each treated as one more classifier backend
- Human in the loop: reviewer feedback is a first-class input, not an afterthought
- Serverless on AWS: deployed in the client's own AWS account so the models, the data, and the record belong to them
- Production scale: hundreds of classifiers running at once, each with its own gate and its own reviewers
What it has produced
For Call Criteria, Primus has managed hundreds of classification models and AI agents that have scored millions of calls, with accuracy climbing month over month on the record. The same loop is what we offer today as the classifier lab, and what we run on the new decision models. If you have a high-volume judgment task, the loop is open source as Cyclotron, and here's how an engagement works.
The product site is primus.anth.us.