The Third Building Block: Classifiers Between Code and AI

A message lands in the shared inbox. Is it a question, a complaint, or somebody selling you something? Is it urgent? Does it need a lawyer?

You could write rules for that. People have been writing them for forty years, and they’re always almost right — they catch “invoice” and miss “the attached,” they flag everything from the one domain that gave you trouble in 2023. Or you could hand the message to a language model, wait two seconds, pay for a paragraph of reasoning you immediately throw away, and get back a word you still have to parse and validate.

Neither is a good answer, and it’s not because either tool is bad. It’s because neither is the right shape. A while back I argued that the best systems combine AI and deterministic code — AI for understanding, code for enforcement. I still think that’s right. But it’s a two-block picture, and there’s a third block sitting between them that I left out.

The decisions that are neither

Look at what software actually spends its day doing and a surprising amount of it isn’t “write me a report” and isn’t if amount > 1000 either.

Which department handles this ticket. Does this passage support this claim. Is this page trustworthy enough to cite. Is this message an ask, a commitment, or an FYI. Should this get a human’s attention right now. Is this paragraph off-topic for the section it’s in.

Every one of those has the same shape: a small, known set of possible answers, and a judgement that needs to read a sentence. Rules can’t read the sentence. A generative model can, but it answers in prose, and prose is not a decision — it’s something you then have to turn into one, which is where the bugs live.

That shape is common enough to deserve its own primitive. It mostly hasn’t had one.

What a classifier is, as a building block

Three properties, and none of them belong to an ordinary model call.

The answer space is fixed before you ask. You define the options up front, and the output is constrained to them. That makes an out-of-schema answer mechanically impossible rather than statistically unlikely. It’s a different kind of guarantee from “the model usually returns valid JSON” — the same difference as between a type system and a careful team.

The output is a distribution, not a token. Not “urgent” but “urgent 0.62, normal 0.31, spam 0.07.” That distinction matters more than it sounds. A single label throws away the fact that the decision was nearly a coin flip, and the near-coin-flip is exactly the case you wanted to treat differently.

It can decline. This is the property almost nobody implements, and it’s the interesting one. “I’m torn between two labels” and “I don’t have enough here to tell you” are completely different situations that produce the same 50/50 split. The first wants both options offered to a human. The second wants a clarifying question, or more context, or nothing at all. If your system can’t tell them apart, it will confidently do the wrong thing in one of the two cases forever.

The clean formulation: the answer tells you what, and the confidence tells you whether to act. Two axes. Most systems ship one and quietly pretend it’s both.

Why the category got a name in 2026

In September, a company called TypeSafe launched a model called Jev that does nothing but this. You give it state and a set of questions with defined answer spaces. It gives back typed answers with probabilities. It cannot write you a sentence, and that’s the point — it’s the fast, intuitive counterpart to the slow, deliberate frontier model, which is where the “System One” framing comes from.

The published numbers are striking: on their own workflow benchmark, roughly 190 times faster and 440 times cheaper than a frontier model doing the same job. Read those carefully, though. They’re vendor-run, on an evaluation the vendor built, and the post says plainly that the workflows were written by people on its own model-capabilities team.

Two things happened next, and both are more informative than the launch.

First, the open-source community rebuilt it within days — at least seven independent implementations, most of them small, running open models on hardware that ranges from a laptop to a single gaming GPU. That tells you the mechanism isn’t exotic. It’s a readout, not a breakthrough.

Second — and this is the part worth holding onto — not one of them reproduced the calibration. Calibration is the actual product claim: that when the model says 0.7 it’s right about 70% of the time. There’s no published reliability diagram for Jev, no Brier score, no expected calibration error, and no independent reproduction. The central claim of the category is the one with the least evidence attached to it.

So take the category seriously and the numbers as marketing. That’s not cynicism, it’s just the order the evidence arrived in.

Five ways to build one, and the question that picks between them

There are basically five ways to get a bounded answer out of a machine:

	Approach	Schema known	Gives a distribution	Natively calibratable
A	Generative model with a typed output	at runtime	no	no
B	Read the probabilities of the option labels	at runtime	yes	after the fact
C	Score each label independently	at runtime	yes	after the fact
D	A small model trained on your labels	at training time	yes	yes
E	A rule table over extracted features	in code	no	only by refitting

Here’s the thing most write-ups get wrong: the axis that picks between these is not accuracy. It’s when you know the schema.

If the label set is fixed — five ticket categories, three urgency levels, and they’ll still be the same next year — a small trained model wins on every axis that matters. It’s cheaper, it’s faster, and it’s the only option on that list you can properly calibrate, because calibration is a fit between predicted probabilities and observed outcomes on a fixed set of outcomes.

But the moment you want a general classifier — one where somebody supplies the labels in a workflow step at runtime — option D is gone by definition. You can’t have trained on labels that didn’t exist when you trained. That single constraint eliminates the most accurate and most calibratable approach from the general case, and it’s why “a general classifier” is a much harder product than it sounds.

The uncomfortable part

Which leads to the thing I’d want someone to tell me before I built one.

A general classifier is the one classifier you can’t calibrate in advance. It’s not a gap in the tooling, it’s structural. Calibration requires a fixed schema and a history of outcomes. The general primitive’s entire value proposition is that the schema shows up at runtime with no history at all.

So an honest general classifier ships a distribution, a confidence derived from how peaked that distribution is, and an abstention. It does not ship a calibrated probability, because it doesn’t have one. A specific deployment of it earns a calibrated probability later, the boring way, by accumulating its own outcomes and fitting against them.

And if you put an unfitted number on the screen and call it confidence, you’ve built something worse than no number at all. A confidence value that was never checked against reality looks exactly like one that was. It reads as verified. The reader — including the future you — stops asking. That’s the failure mode to design against, and it’s a naming problem as much as an architecture one. Call the field a distribution. Don’t call it a probability until it’s earned the word.

The defect nobody mentions

Here’s a concrete, measurable reason to care which of B or C you pick.

The cheap approach — B, presenting the options to the model and reading their probabilities — has a bias nobody advertises. Change the order of the options and the answer changes. One of the open-source implementations flipped 10 of 36 predictions when the options were reordered. The commercial model flipped zero. And this isn’t an implementation slip. It’s a well-documented property of asking a model to pick from a list: one study tested twenty models across three benchmarks and found all of them vulnerable to option position changes, because the model assigns probability mass to the option labels themselves — A, B, C — before it has considered what the options say.

There are two known fixes and both cost you something. You can estimate the bias by permuting the options across a sample of questions and subtracting it out — which needs a sample of questions from that same schema, and a schema you’re seeing for the first time doesn’t have one. Or you can ask k times with the options rotated and average — which multiplies your cost by k and throws away the speed advantage that made the approach attractive.

Or you pick C, and score each label independently. There’s no option ordering, so there’s no position to be biased toward. It costs more forward passes, but they share almost all their input and they batch, so it’s nothing like k times the work.

The part I find genuinely useful: you can test for this without any labelled data at all. Reorder the options, run it again, count how often the answer moves. No ground truth, no annotation budget, no corpus. It takes an afternoon, it’s the first thing you should run, and it’s approximately the last thing anyone does — because it doesn’t look like an evaluation, it looks like a sanity check.

It also silently invalidates every other number you might measure. Agreement rates, accuracy, anything — all of it was measured under one particular option order. If the predictions move when the order moves, those numbers are artifacts.

Where this leaves the architecture

A model may categorise. Only code should decide.

That’s the same argument as AI plus deterministic execution, one level down. A classifier’s job isn’t to be the decision. It’s to hand deterministic code a better input than it had before — a bounded label and a number, instead of a paragraph to parse or a regex to maintain. The budget check, the jurisdiction routing, the escalation threshold, the approval gate: those stay in code, where you can read them, test them, and point at the line that did it.

What the third block actually buys you isn’t autonomy. It’s that every threshold in your system is a number somebody typed, and a classifier makes those numbers checkable. The 0.7 that decides whether a human gets interrupted is a claim about the world. Right now, in most systems, nobody has ever checked it — and in most systems, nobody can, because the number was never written down anywhere when the decision was made.

That’s the unglamorous prerequisite. Before the better model, before the trained head, before any of it: log the decision and the confidence and what happened next. Everything interesting is downstream of that, and it’s a log line, not a research project.

The part that surprised me

Reading the option probabilities isn’t a feature of a model. It’s a feature of the thing serving the model — the inference engine, which either hands you those numbers or doesn’t.

Which means you can only have it if you run the engine. The open serving stacks expose it. Closed APIs mostly don’t, or give you a truncated slice of it, and you get whatever they decide to return.

So for once the self-hosting decision isn’t a tax you pay for privacy or control. It’s a precondition for a capability that the API-only shop simply cannot buy. That’s an unusual shape and I don’t think it’s been priced in yet.

What to do now

1. Go find the thresholds. Grep your codebase for hard-coded confidence cutoffs — every 0.5, 0.7, 0.8 compared against a score. Make the list. It’ll be longer than you expect, and it’s the map of everywhere this applies.
2. For each one, ask who computed the number. If the answer is “the model reported it about itself,” you have a problem: models are unreliable at reporting their own certainty and reliable at categorical judgements. Take the category from the model, compute the number in code.
3. Log the decision before you improve it. Which label, what the alternatives were, what the confidence was, what happened afterwards. You cannot calibrate what you never wrote down, and this is a log line rather than a project.
4. Run the reorder test. Shuffle the option order, re-run, count the flips. No labels needed. If it’s more than a percent or two, every other number you’ve measured is an artifact of one particular ordering.
5. Pick the approach from when your schema is known, not from a benchmark. Fixed labels and real volume means train a small model. Labels that arrive at runtime means score each one independently and don’t pretend you’re calibrated.
6. Name the field honestly. distribution, not probability, until a reliability check has earned the second word.
7. Keep the decision in code. The classifier supplies the input. The gate, the threshold, and the audit trail stay somewhere you can point at.
8. Check what your serving stack gives you. If you’re on a closed API, find out whether you can get option probabilities at all. The answer shapes which of the five approaches are even available to you.

None of this needs a new model. Most of it needs a log line and an afternoon.