Decision Models Are Not Calculators

Jev is a decision model: give it a description and a list of possible answers, and it returns a choice and probabilities for the options. We asked it which face had shown on a fair six-sided die rolled out of sight. Jev chose one and assigned it 88% probability. With no information about the roll, every face still had the same one-in-six chance. Why did the model favor one so strongly?

We had a hypothesis inspired by Benford's Law. First we looked at Jev's answer. Then we moved one around the list of choices and asked two other decision models the same question. The results took us somewhere less tidy than our initial explanation.

A hypothesis about ones

In The Dominance of Ones, I looked at Benford's Law: one appears as the leading digit more often than other digits in certain observed numerical datasets. Maybe patterns like that in training data leave a model inclined to pick 1 when a prompt provides no evidence for any answer.

That's a possible explanation for a model's behavior. The die stays fair. We didn't inspect Jev's training data, and Benford's Law alone doesn't tell us how often its training corpus contained the token 1. We could measure the prediction pattern. We couldn't yet name its cause.

The preliminary result

Our first request described a fair die that had already been rolled, but whose result nobody saw. The model had no clue about which face showed. Here's the ordinary-order request we sent to Jev, one of the exact requests preserved in the public experiment:

{
  "model": "jev-latest",
  "state": "A fair six-sided die was rolled once. The result is unknown.",
  "questions": {
    "die_result": {
      "type": "choice",
      "instructions": "Which face showed on the roll?",
      "criteria": {
        "1": "the die face marked 1",
        "2": "the die face marked 2",
        "3": "the die face marked 3",
        "4": "the die face marked 4",
        "5": "the die face marked 5",
        "6": "the die face marked 6"
      }
    }
  }
}

Jev selected one. The linked example preserves its complete response:

Taken alone, this answer looks like support for our hypothesis.

But one was the first-listed choice in this request. Maybe Jev was following the top of a list. The recorded request belongs to our full test matrix. Viewed alone, it shows why we needed the rest of the test.

Does moving one change the answer?

Six choices have 720 possible orders, so we sent Jev every one. We used digits (1 through 6) and words (one through six), then repeated both sets. Every face occupied every list position exactly 120 times per label form and pass. The preregistered protocol fixed that test before inference.

Jev chose one in all 2,880 requests. Face one moved through all six list positions; Jev still chose it 120 times from each position in each digit pass. It made one the unique highest-probability face in all 1,440 digit-labelled replies and chose one in every word-labelled reply. Across the first digit pass, its mean returned P(1) was 89.7%, against 4.4% for face six in the released summary.

One followed its label around the list. That rules out a simple “always pick the first option” account of Jev's choices. But the result still doesn't tell us what Jev saw during training, or why it learned to respond this way.

The other models changed the story

Would the same preference show up elsewhere? We gave the identical question matrix to Kev and Laya, changing only the model identifier in each request. All 8,640 planned responses across the three models were valid, as the release manifest records. The complete summary gives the comparison:

Kev selected one in 49.0% of its replies. That's a preference, but nothing like Jev's. Its mean digit probabilities were much closer to one-sixth, and in the first digit pass it chose the first-listed option in 426 of 720 orders. Laya went the other way: it selected six in 2,610 replies, or 90.6%. With word labels, its mean reported P(six) reached 91.8% in the first pass. None of those patterns reveals a training-data cause.

The chart groups probabilities by the face named in each answer, even when that answer moved to another position. The dashed line marks the fair-die baseline stated in the prompt, independent of the models' outputs.

Choice order still mattered

The ordering test did more than rule out a first-slot shortcut for Jev. In the first digit pass, Laya never selected the last-listed option. Kev's first-position preference was much stronger than a position-neutral pattern across this balanced matrix. Spelling the same faces as words changed Kev's selected answer in 324 of 720 paired orders and Laya's in 129. Jev kept choosing one.

Exact repeat requests selected the same face in every case for all three models. Kev and Laya returned identical probability vectors on repeat; Jev's probabilities moved slightly without changing its choice. Those repeats check stability of the model response. They aren't new die rolls.

The build notes pin the three systems. Jev's hosted replies identified jev-1.13.0; Kev and Laya used pinned local builds. We compare their returned choice probabilities, without treating the separate confidence fields as calibrated probabilities of correctness. Jev also rounds its probabilities to two decimal places; the measurement note records that limitation.

What the result leaves open

Benford's Law might help explain Jev's distribution of predictions. That's the limit of the claim. We didn't inspect training corpora, count their leading digits or numeral tokens, or retrain a model with those frequencies changed. Jev's pattern is consistent with the idea; the experiment doesn't establish the mechanism.

Kev's partial preference for one and Laya's strong preference for six don't fit the same simple story. Choice order and spelling affect them, but we can't explain their distributions from these measurements.

If you need a fair die, use a random-number generator. If you need a decision model, test its answers under reordered choices and equivalent labels before using its reported probabilities to make a consequential decision. The MIT-licensed experiment includes the exact requests and replies, complete response archives, code, and figures.