Deep dives / How people are evaluating Jev-like models
How people are evaluating Jev-like models
Two public leaderboards, a pile of single-job tests and a few honest calibration studies. They disagree.
By Jim Thornton Updated 7 min read Stub
Stub · to be expanded
Three weeks after Jev, here is what each approach measures, what the Decision Index currently says, and what we use instead. It goes with the briefing Decision models.
Teal: measured by the named evaluatorAmber: vendor-reported
Chapter 01 · The problem
Why this is hard
A decision model returns a probability for every answer you allow. So "is it good" has at least four parts: is the top answer right, does a 0.9 mean 90% (calibration), how much can you hand off at an error rate you accept (selective risk), and what does each decision cost in time and money. Most published numbers report the first and skip the rest. Author-reported scores routinely sit 20 to 40 points above the independent board's numbers for the same model (Oct 5).7
Chapter 02 · The board
The Decision Index
How edition 0.3.1 is computed
Board file generated Oct 7, 2026, 18:25 UTC.3,2
- A frozen public suite of 110,201 requests over 42 benchmarks in five areas: knowledge and reasoning, language, retrieval and classification, tools and automation, arts and human taste.
- Each benchmark's native metric is multiplied by coverage (unanswered counts as wrong) and chance-corrected, so guessing scores 0 and perfect scores 100.
- The headline Full score = 0.20 × public index + 0.50 × private tests of the same skills + 0.30 × private tests from new domains. The private parts are mapped onto the public scale using untuned stock models. Entries within 0.9 points share a rank.
- Every entrant runs its own published inference code with no prompt tuning. Anything slower than 1,000 ms median on the maintainer's single RTX PRO 6000 is excluded as "no longer Jev-like", which keeps reasoning decision models off the main board.
- Calibration (ECE and Brier, on 32 benchmarks with a per-field right answer, a 1-in-6 sample) is published for each entrant but does not enter the score.
- Edition 0.3 rebuilt GSM8K's options so option-pattern rules stop scoring, retired ForecastBench, and added a 21-entrant vision board; a reasoning board is announced for 0.3.x.9
Current numbers: 0.3.1, Oct 7, 115 entries
| Rank | Entrant | Full | Public part | ECE | Median latency | Site page |
|---|---|---|---|---|---|---|
| 1 | Perplexity Decider v1.127B, open, Qwen3.8-27B | 62.75 | 62.25 | 0.071 | 104 ms | not yet; v1 has a page |
| 2 | Fastino GLiDE no-thinking28B, weights promised | 60.21 | 59.06 | 0.040 | 98 ms | none |
| 3 | Jev 1.13hosted | 60.11 | 57.96 | 0.074 | 524 ms round trip | none (closed) |
| 4 | Torchcast Decision 27B | 59.91 | 65.10 | 0.024 | 100 ms | none |
| 5 | deck31bstock Gemma 4 31B, readout only | 59.03 | 58.19 | 0.204 | 219 ms | base: Gemma 4 31B |
| 6 | Kev 27B | 58.78 | 56.69 | 0.022 | 135 ms | none |
| 10 | Surogate Rune 26B-A4B v3 | 57.43 | 58.28 | 0.120 | 121 ms | model page |
| 19 | Cloudflare Clef27B | 53.08 | 61.71 | 0.040 | 102 ms | model page |
| 31 | Cloudflare Clef Flash9B | 47.61 | 56.15 | 0.025 | 53 ms | none |
| 106 | Laya0.42B encoder | 4.43 | 6.19 | 0.140 | 5.8 ms | model page |
All 25 rows in our extract (the top 20 plus selected lower ranks)
| Rank | Entrant | Kind | Full | Public | Public rank | ECE | Latency |
|---|---|---|---|---|---|---|---|
| 1 | Perplexity Decider v1.1 (27B) | full fine-tune | 62.75 | 62.25 | 2 | 0.071 | 104 ms |
| 2= | Fastino GLiDE no-thinking (28B) | full fine-tune | 60.21 | 59.06 | 5 | 0.040 | 98 ms |
| 3= | Jev | closed | 60.11 | 57.96 | 10 | 0.074 | 524 ms round trip |
| 4= | Torchcast Decision 27B | LoRA | 59.91 | 65.10 | 1 | 0.024 | 100 ms |
| 5= | deck31b (Gemma 4 31B) | inference technique | 59.03 | 58.19 | 8 | 0.204 | 219 ms |
| 6= | Kev 27B | full fine-tune | 58.78 | 56.69 | 14 | 0.022 | 135 ms |
| 7= | Quyet-1.0-Large | LoRA | 58.71 | 60.79 | 4 | 0.078 | 105 ms |
| 8= | Blink v0.3 26B-A4B (NVFP4) | full fine-tune | 57.76 | 58.21 | 7 | 0.040 | 25 ms |
| 9= | Decider chat · Gemma-4-31B | inference technique | 57.62 | 57.79 | 11 | 0.047 | 109 ms |
| 10= | Surogate Rune 26B-A4B v3 | full fine-tune | 57.43 | 58.28 | 6 | 0.120 | 121 ms |
| 11= | GEV-26B-Decide | LoRA | 56.17 | 58.03 | 9 | 0.072 | 129 ms |
| 12= | Xor 26B-A4B | LoRA | 55.95 | 56.30 | 15 | 0.040 | 110 ms |
| 13= | vLLM-SR Decision 2.0 Vega 27B | LoRA + head | 55.88 | 56.97 | 13 | 0.086 | 120 ms |
| 14= | JEV-27B | full fine-tune | 55.64 | 53.64 | 19 | 0.076 | 102 ms |
| 15= | frontier-infra Jebadiah 27B | LoRA | 55.63 | 55.11 | 18 | 0.014 | 110 ms |
| 16= | JADE | LoRA | 55.56 | 53.42 | 20 | 0.238 | 133 ms |
| 17= | Eikos-27B-FP8 Qwen3.8-27B LoRA | LoRA | 55.34 | 52.85 | 21 | 0.057 | 130 ms |
| 18 | Bespoke Nimble 9B v3 | LoRA | 54.67 | 57.19 | 12 | 0.123 | 74 ms |
| 19 | Cloudflare clef | full fine-tune | 53.08 | 61.71 | 3 | 0.040 | 102 ms |
| 20= | simple-jev · Qwen3.8-27B (featherless) | inference technique | 51.79 | 56.05 | 17 | 0.113 | 373 ms |
| 31= | Cloudflare clef-flash | full fine-tune | 47.61 | 56.15 | 16 | 0.025 | 53 ms |
| 48= | Bespoke Nimble 9B v2 | LoRA | 40.95 | 39.90 | 53 | 0.024 | 77 ms |
| 66= | Together Tev1 Qwen3.5-4B | full fine-tune | 34.14 | 29.51 | 73 | 0.104 | 36 ms |
| 93= | Together Tev1 Qwen3.5-0.8B | full fine-tune | 12.02 | 13.10 | 99 | 0.126 | 29 ms |
| 106= | Laya | full fine-tune | 4.43 | 6.19 | 109 | 0.140 | 5.8 ms |
All 59 decision models on this site, with their 0.2.1 figures: the decision-model list.
How the leader's score has moved
Jev read 59.5 on 0.1, 51.67 on 0.2, 57.91 on 0.2.1 (board file Sep 28) and 60.11 on 0.3.1 (Oct 7, a different formula). A "beats Jev" claim made against one edition does not carry over.5,6,3
0.1 and 0.2: reported 0.2.1 and 0.3.1: board files
What it does not tell you
- 80% of the Full score is private and cannot be re-run, even for open weights. The public part alone ranks differently: Clef is 19th on Full but scores 61.71 on the public part, above Jev's 57.96.3
- It mostly measures base-model breadth. A model built for one narrow job can be excellent at it and score near zero here; our fine-tuned small model is the example (0.893 on our keyword job, Laya base 4.43 on the board).17,3
- Calibration is shown, not scored. Perplexity's v1 had ECE 0.018 on 0.2.1; v1.1 has 0.071 on 0.3.1 while its rank rose.6,4
- Latency is single-request on one GPU, and hosted entrants are measured over the network, so their figures are round trips.4
Chapter 03 · Elsewhere
Other ways people evaluate these models
Eight approaches beside the Decision Index, each with the date we read it and its main caveat.
| Approach | Example | What it measures | Date | Caveat |
|---|---|---|---|---|
| Composite of accuracy, calibration, speed and cost | JevBench v1.6.1, Benchmark Heaven8 | 1,500 decisions per system, open and sealed items; Capability = mean of Intelligence and Calibration; hosted APIs add speed and cost | Oct 6 | Ranks the same checkpoints tens of points apart from the Decision Index (Kev 27B is 6th on the Decision Index and scores 5.8 on JevBench's composite)9 |
| Head-to-head on one job against classifiers and LLM judges | Red Hat AI Safety, guardrails10 | Prompt injection: a 200M classifier 89.01% at 54 ms, Jev 86.35%. Content safety: Jev first at 86.20% | Oct 2, updated Oct 5 | Two tasks, English only, no intervals; Red Hat ships the classifiers |
| Calibration on your own labels, then a temperature fit | Verne Systems, 1,232 Banking77 messages11 | Jev ECE 0.086; 0.059 after a temperature fitted on 50 labels; answers stated at 0.6 to 0.7 were right 44% of the time | Oct 1 | One public dataset |
| Coverage at a fixed error budget | jev-benchmarks; jev-eval12,13 | Share of items accepted at 5% error or less, with paired bootstrap intervals; recommended cut-points | checked Oct 9 | Small studies |
| Domain test against a frontier reference | Jev on four medical benchmarks vs GPT-6 Sol14 | Accuracy, calibration, selective prediction, unanswerable questions | Oct 1 | Preprint |
| Human disagreement as the target | Jevify jev-bench15 | Accuracy, ECE and distance to human label spreads; Jev 0.733 accuracy, ECE 0.113 | checked Oct 9 | Author-run |
| A protocol for your own traffic | Evalgent, decision APIs on call data16 | 400 to 900 double-labeled decisions per type; accuracy with Wilson intervals, ECE, Brier, coverage, option-order and repeat flips, p95 latency, cost | Oct 4, updated Oct 7 | A guide, not a result |
| Your own jobs, pre-registered, blind key | Our own studies17 | Accuracy with intervals, ECE, share handled alone at 0.90 on a 2-fold cutoff, cost per 1,000, McNemar for wording changes | Oct 1 to 9 | Keys other than keyword buckets are frontier-model labels |
What we take from it
The boards are useful for a shortlist. They disagree with each other about the same checkpoints, they mostly measure breadth, and only one of them scores calibration. The number that decides a deployment comes from a few hundred of your own labeled decisions: accuracy, calibration, the share you can hand off at your error budget, and cost per 1,000. That is the method in Chapter 03 of the briefing.
Sources
Dates are the page's own date or the day we checked it. Board numbers are the maintainer's runs; 80% of the Full score comes from private tests.
- 1 Decision Index (Hugging Face Space, multimodalart/jev-decision-index) · checked 2026-10-09
- 2 Decision Index reproduction kit, README (apolinario/decision-index, MIT) · checked 2026-10-09
- 3 Decision Index 0.3.1 board data, v03.json · generated 2026-10-07T18:25:58Z
- 4 Decision Index 0.3.1 index.json (latency, ECE, base models) · generated 2026-10-07T18:25:58Z
- 5 Jev News: Decision Index editions 0.1 and 0.2 · observed 2026-10-05
- 6 Decision Index 0.2.1 board file (generated Sep 28) · observed 2026-10-04
- 7 Decisions Per Second (Jim Thornton, field guide) (unpublished) · Oct 5, 2026
- 8 JevBench v1.6.1 methodology (Benchmark Heaven) · Oct 6, 2026
- 9 Tom Rochette: Decision Index note (0.3 summary) · Oct 8, 2026
- 10 Canberk Kilicarslan on Red Hat's guardrail benchmark · 2026-10-02, updated 2026-10-05
- 11 Verne Systems: Jev calibration on 1,232 banking messages · Oct 1, 2026
- 12 jev-benchmarks (AbdelStark) · checked 2026-10-09
- 13 jev-eval (PyPI) · checked 2026-10-09
- 14 Jev on four medical benchmarks vs GPT-6 Sol, arXiv 2609.34024 · Oct 1, 2026
- 15 Jevify jev-bench (uspraveen) · checked 2026-10-09
- 16 Evalgent: benchmark decision APIs on your own call data · 2026-10-04, updated 2026-10-07
- 17 Decision Model Field Notes (our own studies) (unpublished) · 2026-10-01 to 2026-10-09