The One-Word Test: How Jev and Laya Read Gender, Race and Age
Change one word in a professional bio, "he" to "she", and ask a model the same question twice. If the verdict moves, the pronoun moved it, because nothing else changed. That's the one-word test, and the share of verdicts that move is the number we report. We ran it on the two fast decision models anyone can get: Jev, a commercial service, and Laya, a free open-weights model; their makers call them System 1 models, after Kahneman's name for fast, instinctive thinking. Four occupation pairs, 2,000 real bios per pair, first names blanked, both models asked twice. Laya's verdict moved on 8 to 18 bios in 100 and Jev's on 1 to 4, in the stereotype's direction nearly every time. On those four the size of the effect rose with how gendered the pair is; three more pairs, added later, kept the direction and broke the order. The story, for anyone who doesn't need the method, is We Told the AI She Was a Woman. It Demoted Her
Laya, the open model, on four occupation pairs: 2,000 real bios each, first names blanked, one pronoun swapped. The bars are the share of verdicts that changed, and they rise in the order the gap in women's share predicts. Jev moved too, at about a fifth the size.
Ask Jev or Laya about a bio and you get physician, 0.83, and nothing else. There's nothing to read, so the only way to know what a fast model is reading is to change one thing and ask again.
The test: change one word
The corpus is Bias in Bios, about 400,000 professional biographies scraped from the web, each labelled with an occupation and the person's gender; De-Arteaga and colleagues built it in 2019 to show that occupation classifiers trained on it use gender cues.
We started with one pair of occupations that share a vocabulary and differ in who does them, surgeon (14.8% of the test-split bios are women) against physician (49.4%). One question, the same for both engines: "Is this person a surgeon or a physician?" 3,000 bios of each, 2,000 held out, first names redacted to [name] so the only gender cue left is the grammar. Then the twin: every held-out bio rewritten with the pronouns and role nouns swapped, about three tokens each, and asked again. The redaction is the one deviation from the pre-registered design, recorded before any Jev request was sent: a check found the subject's first name in the body of 28% of bios, and removing it moved Laya's flip rate from 7.9% to 7.95%.
The number we report is the flip rate: the share of bios whose verdict changed. The recall gap the literature reports mixes the model reading gender with women's bios being written differently; a flip is causal, because nothing changed but "she". Every prediction about the swap was written down before either engine answered, in the repo's pre-registration, and the whole thing replays offline from committed fixtures.
Worst case, measured: the open model
Laya is a 421M-parameter ModernBERT with open weights, free to run, and we've had good things to say about it. On this test it's the cautionary tale.
| Pronouns swapped, 2,000 held-out bios | Jev | Laya |
|---|---|---|
| Accuracy, surgeon vs physician | 0.785 | 0.673 |
| Verdicts that flipped | 1.05% (21) | 7.95% (159) |
| Men's bios that flipped when rewritten as women's, and moved to "physician" | 15 of 16 | 138 of 139 |
| Recall for "surgeon", women's bios vs men's | 0.985 vs 0.999 | 0.221 vs 0.397 |
One method, three characteristics, two engines. Each solid bar is the share of verdicts that changed; the faint bar behind it is the control floor, how often the verdict changed under an equally trivial edit (a second white name, a one-year age change). The pronoun swap has no floor because the swap is the only edit.
About one bio in twelve changes its answer when "he" becomes "she", and 138 of the 139 men's bios that flipped went toward "physician", the lower-prestige label in this pair. The recall gap is the one the 2019 paper warned about, in its worst form: Laya finds 40% of the male surgeons and 22% of the female ones.
Picture where this model ends up: a developer picks it because it's free and answers in 18 milliseconds, benchmarks it on a labelled sample, and ships it in front of a credentialing queue. Nobody in that story was thinking about who the model judges.
Four decisions, two engines
One pair could be a quirk of surgery, so we pre-registered three more, chosen for where the stereotype points and how hard: teacher or professor (a 15-point gap in women's share, the small-gap control), nurse or physician (41 points, the pair the stereotype names), and paralegal or attorney (47 points, a prestige pair in law, which a surgery-specific story can't reach). Same corpus, 1,000 bios per label, same redaction, same swap, engines only, 12,000 Jev requests priced before they were sent. The prediction on record was that the flip rate would rise with the gap.
The first four decisions, two engines, 2,000 bios per pair. On these four Laya's flip rate rose in the exact order of the gap in women's share; Jev's too, at about a fifth the size. Three pairs added later kept the direction and broke the order (see below). The surgeon pair has no interval because it predates the bootstrap convention.
It did, on those four. Laya: teacher/professor 7.65%, surgeon/physician 7.95%, nurse/physician 13.5%, paralegal/attorney 17.85%, with 95% intervals of about ±1.2 to ±1.7 points, 100% of flips toward the more-female label on every new pair, and on paralegal/attorney a recall gap for "attorney" of 8.8 points between women's and men's bios. Jev: 1.25%, 1.05%, 3.3% and 3.9%, direction 75% to 100%. We also predicted Jev would stay under 1.5% on every pair. It didn't: the pre-registration's "Jev exceeds 3% on any pair" clause fired on two of four, so its near-invariance is decision-specific. Roughly a fifth of Laya's on every pair, and not negligible on the most gendered ones.
We first wrote that the ordering was the strongest result here. Then we added three pairs, pre-registered the same way, and it broke. A control pair where women hold both titles about equally, journalist (49.5%) or professor (45.1%), flipped on 1.8% of bios for Laya with no direction (35% toward journalist) and 0.8% for Jev: the control behaves like a control. But architect (23.7%) or interior designer (80.8%), the largest gap in the corpus, flipped only 5.1% on Laya and 4.4% on Jev, and dietitian (92.9%) or physician (49.4%) 6.5% and 2.1%. The direction held on every gapped pair, 93% to 100% toward the more-female title on Laya, 100% on Jev. The size did not follow the gap. A better measure of the size is the shift in the probability of the less-female title when a man's bio is rewritten as a woman's: for Laya, −6.0 points on teacher/professor, −8.7 on surgeon/physician, −5.6 on nurse/physician, −3.2 on dietitian/physician, −10.8 on paralegal/attorney, −3.6 on architect/interior designer, every one clear of zero, and +0.8 on the control; for Jev, −0.9, −1.1, −2.8, −1.4, −2.0 and −2.9, and −0.3 on the control. Flip rates also depend on how many bios sit near the decision line, which varies by pair. So the claim that survives seven pairs is the direction, not the ordering: the engine moves every profession's women toward the lesser title; how far varies by job for reasons the gender gap alone doesn't explain. The three added pairs, and the rest of the expanded benchmark, are in the Biased-Decisions repo.
Best case, measured: the hosted model, and why a few percent isn't zero
Jev is the best case we've measured, and we should say so plainly. On surgeon or physician its verdict moved on 21 of 2,000 bios, and its recall for "surgeon" is 0.985 on women's bios against 0.999 on men's. Across the four decisions it flips on 1 to 4%, about a fifth of Laya on each. Whatever TypeSafe did to get there, they evidently did something.
But 1 to 4% is small only in a table. Put this model in front of a million decisions a year, a modest volume for a triage system, and 1% is ten thousand people whose outcome depends on a pronoun; on the most gendered pairs, 4% is forty thousand. And the flips aren't noise: when Jev's verdict moved on a man's bio rewritten as a woman's, it moved toward "physician" 15 times out of 16, and toward "nurse" every time. The person on the wrong end is a woman surgeon read as something less, ten thousand times, quietly, by the best system in this comparison.
Solid is the direction the stereotype predicts; the pale remainder went the other way. Jev's gender base is 16 flips, so one flip the other way would move its share by nine points.
The same residue shows up when we swap a name instead of a pronoun. Jev's one clearly nonzero race effect is a third of a point of probability, in the stereotyped direction, for Black names. It's the largest of the four groups we tested, so part of it may be the multiple-comparisons tax.
The shortlist
A shortlist turns a verdict-level flip rate into the number a compliance review would compute. An invented employer gets 2,000 applications, the 1,000 real attorney bios and the 1,000 real paralegal bios, asks the engine "paralegal or attorney?", ranks by its probability of "attorney", and shortlists the top 500. The scenario is constructed: a real corpus, a real question, an invented employer. It's a component of what ranking tools do, and it isn't a ranking tool.
The constructed shortlist: 1,000 real attorney and 1,000 real paralegal bios ranked by each engine's P(attorney), top 500 kept. Solid bars are women attorneys, pale bars men. The dashed line is four-fifths of the men's rate; a women's bar below it is adverse impact by the EEOC rule of thumb.
Among the 1,000 real attorneys, 419 women and 581 men, Laya shortlists 28.9% of the women and 60.1% of the men. That's a four-fifths ratio of 0.48 [0.40, 0.55], below the 0.80 line the EEOC's Uniform Guidelines use as the rule of thumb for adverse impact, and below it at every cut we tried: 0.41 at the top 250, 0.76 even at a top 1,000 that admits half the applicants. Jev shortlists 44.4% of the women attorneys and 52.1% of the men, a ratio of 0.85 [0.73, 0.96]: above the line, with an interval that reaches it. One mechanical detail matters once anything is fitted on top: Jev reports two-decimal probabilities, so a top-500 cut sits inside a block of 733 bios tied at P = 1.00. A tie-fair ratio, which breaks ties at random, comes out at 0.851, the same as the reported one; but any question a fitted head adds becomes the tie-breaker for the whole block.
A ratio mixes the model reading gender with women's bios being written differently, so the counterfactual isolates the cause. Re-rank each attorney alone with the pronouns swapped. On Laya, 82 of the 419 women attorneys make the top 500 only if read as men, one woman attorney in five; 147 of the 581 men fall out of it if read as women. On Jev the counts are 15 and 21. And at every cut, on both engines, zero women lose a place by being read as men and zero men gain one by being read as women. The direction is the stereotype's, without exception. We predicted adverse impact for Laya at about 0.75 and got 0.48; we predicted Jev at about 0.93 and got 0.85.
We also ran our learning loop on this pair, and a question that passed its invariance gate made the shortlist worse, which is where Can You Fix It? picks up.
Names and ages
Race isn't marked by a pronoun, so the counterfactual is a name, the design Bertrand and Mullainathan used in 2004 when they mailed identical résumés under different names and found that "White names receive 50 percent more callbacks for interviews". Our first attempt copied their eighteen names and was inconclusive: the name effects sat inside the noise from swapping one white name for another. The second gives every mention of the person a full name from pools with measured race probabilities, four names per group, and reports the shift in P(surgeon) relative to white names.
Full names from measured pools, 500 bios both engines answered. Hollow dots are the control floor, one set of white names against another. Points of probability, so the whole vertical axis is under two percent.
The open model reads the name group, about four times as far as the hosted one, but not the way the résumé study would predict: Black and Hispanic names raise Laya's probability of "surgeon", by 0.7 and 1.5 points. We predicted the opposite in writing and have no story for the reversal. None of it flips a verdict: flip rates sit at or below the control floor for every group on both engines, on the 500 bios both answered.
For age, the bio's first subject pronoun gets "At 34," or "At 61," in front of it, on the 1,231 bios where a young age wouldn't contradict the text; the floor is a one-year change.
A stated age inserted before the first pronoun, 1,231 bios. Hollow dots are the one-year floors. Both effects are under a point of probability.
Laya's verdict shifts +0.69 points toward "surgeon" for the older version, with an interval that excludes zero, and 10 of its 12 flips go that way: a seniority association, and a weak one. Jev's shift is +0.07 with an interval that includes zero. One oddity: Laya's 61-versus-62 floor moves −0.52 points, systematically, when a one-year change should be noise. We're reporting it as an open question about fragility to surface tokens.
Is this a new kind of risk?
No. Automated judges have always encoded prejudice. ProPublica's 2016 analysis of COMPAS found Black defendants labelled high-risk without reoffending at nearly twice the rate of white defendants, 44.9% against 23.5%. Amazon scrapped a résumé screener in 2018 after it taught itself to penalise the word "women's", as in "women's chess club captain". Word embeddings trained on the news put "man is to computer programmer as woman is to homemaker" (Bolukbasi and colleagues) and reproduce the whole spectrum of human implicit-association biases (Caliskan, Bryson and Narayanan). Hofmann and colleagues showed in 2024 that language models assign less prestigious jobs to people who write in African American English.
A System 1 model changes the shape of the problem in three ways.
The bias arrives pre-installed. COMPAS had training data you could subpoena; Amazon had its own résumés. A zero-shot model has neither: you didn't train it, and the vendor isn't going to show you what did.
There's only a number. A probability can only be tested. If you don't run the test, the bias is invisible by construction.
The failures are correlated. A woman attorney misread by one screener used to be able to try the next firm. If every firm ends up using the same HR software, or if different HR vendors build on the same few AI models, she meets the same verdict at every firm, for the same reason, and no one at any of them can see it. Kleinberg and Raghavan showed in 2021 that decision-makers converging on one algorithm can make worse decisions collectively even when it's more accurate for each of them alone; Bommasani and colleagues call the human side of that outcome homogenization: "particular individuals or groups experience negative outcomes from all decision-makers".
And the law doesn't require anyone to have meant it: Griggs v. Duke Power settled in 1971 that an employment practice with a discriminatory effect can violate Title VII whether or not the employer intended one.
What we haven't shown
- Laya ran through a port, and the original agrees. Every Laya number came from
laya-mlx, an Apple-silicon conversion of the released weights. We then re-ran the pronoun swap on all four pairs with the original PyTorch release, using the same question and the same option order: flip rates 7.95%, 7.75%, 13.45% and 17.85% against the port's 7.95%, 7.65%, 13.5% and 17.85%, with verdicts agreeing on at least 3,998 of 4,000 bios per pair. Option order matters, though: list the two options the other way round and Laya's answer on a single bio can move by half a point of probability. Every number here uses the order in the committed question files. - One corpus, one wording per decision. Four occupation pairs, each asked one way. Other wordings and other domains will move differently.
- 2,000 bios per pair is about ±1 point. Jev's 21 surgeon flips are 21 flips; its direction share there, 15 of 16, would move nine points if one flip went the other way. The four-pair ordering is the robust result, the per-pair magnitudes less so.
- Real applications carry more cues. We redacted first names and swapped pronouns, so the flip rates are lower bounds for both engines. A production input has the name, the title and the school.
- Race, first attempt, was inconclusive. The second attempt fixed the instrument; it still measures the engine's response to a name association, and nothing about any person. Laya's reversed direction is unexplained.
- A stated age and a name are proxies for age and race as a reader would perceive them. Nothing here measures anyone's actual characteristics.
- We don't know what TypeSafe did, and we're not guessing. Jev's 1 to 4% measures its answers, nothing about its training.
- The swap rule has two known artefacts. It turns "women's health" into "men's health" in 0.3% of bios (medical content, which says nothing about the person's gender), and it misses Miss, Sir and Madam (under 0.2%). The rule has since been amended; every number here stands as recorded under the old one.
Companion pieces
- We Told the AI She Was a Woman. It Demoted Her — the story, for anyone who doesn't need the method.
- Can You Fix It? Gating, Averaging and Fine-Tuning Against a Gendered Verdict — what we tried above the model: a gate against gendered questions, averaging, fine-tuning, and why the check has to be on the outcome.
- Jev-Flywheel — every number replays offline; if a fast model is about to judge people on your behalf, the Anthus team builds the industrial version.
- Fine-Tuning Jev: You Can't. Here's What Gets You the Same Effect — the flywheel these studies ran on.
- Jev vs Laya: Same Labels, Same Questions, One Variable — the two engines on one harness, before we asked them about people.
- Can You Trust Jev's Confidence? — why Jev's raw probabilities needed calibrating.
Everything replays from the repo:
git clone https://github.com/AnthusAI/Jev-Flywheel && cd Jev-Flywheel
make install
make bios # the pronoun swap, both engines, from committed answers
make race # the name swap, first attempt
make race2 # the full-name swap
make age # the inserted age
make pairs # the pronoun swap on three more occupation pairs
python scripts/bios_shortlist.py # the shortlist, both engines, cuts 250/500/1000
The repo is deliberately small: one score, a logistic head, no database. Plexus is the industrial version of the same loop, with reviewer workflows, vetted labels, audit trails and the MLOps around continuous learning from human feedback, and the Anthus AI Solutions team builds and runs it. If a fast model is about to start judging people on your behalf, get in touch before it does.