What the numbers actually say

The pricing facts, which are not in dispute#

Six multipliers, one dataset#

These all describe the same body of vendor-run evaluation. They differ by which end of the range is quoted and which baseline is in mind.

If you quote one number, quote TypeSafe's: 40×–200× faster, with the homepage's 193.6×/444.6× flagged as a best case that TypeSafe itself hedges.

The accuracy-matched table, which is the real story#

TypeSafe evaluated four production-shaped workflows — security incidents, observability, invoices, customer service — grading against the average of GPT-6 Astra and Fable 5.1 at high reasoning settings.

Read against its nearest-accuracy peer, Jev is roughly 76× cheaper and 25× faster at statistically indistinguishable accuracy. Read against the best baseline, it is 6.3 accuracy points behind. Neither of those is 193× or 445×, and both are genuinely good product stories.

Four methodology caveats worth holding#

1. Reference answers are model-generated. Grading against an average of GPT-6 Astra and Fable 5.1 caps the measurable ceiling at "agrees with two frontier models", and TypeSafe acknowledges it biases results toward OpenAI's and Anthropic's outputs.
2. The workflows are TypeSafe's own. They say the content was not chosen to flatter the model and is outside the training distribution — and also that it was written by their model-capabilities team, "so some bias could exist."
3. Conditions are vendor-run. By their own footnote, evals were generally run from laptops on the West Coast, where the service is hosted.
4. The baselines were wrapped. LLM comparisons ran through TypeSafe's own System One wrapper to force compatible structured decisions, which they say is the most accurate method and also "slower and more expensive than giving decisions without probabilities."

The 0% hallucination bar is an axiom#

On TypeSafe's hallucination chart, Jev sits at 0%. Their footnote explains why: "Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots."

That is true and it is trivial. The output space is the type, so an out-of-type value is impossible. It says nothing about whether the in-type value is correct. The honest claim is "never makes type errors", which TypeSafe's docs do say, rather than "never hallucinates", which the marketing says. The full argument is here.

What independent testers found#

TypeSafe published no standard benchmark table — a deliberate policy, which Diogo Almeida defends in an essay called Lies, Damned Lies, and Benchmarks: "We are choosing the clean slate: no standard benchmark table in our model releases." The community filled the gap within days. The picture that emerged is more interesting than either the launch post or the sceptics predicted.

The retracted benchmark worth reading#

One developer published a claim that Jev was 5,000× cheaper than Claude at filling web forms, then retracted it himself. The correction is the most useful post of launch week:

His fair rerun: "Median 168s vs 61s and $0.49 vs $0.25, same quality. So the hybrid is 2.7× slower and 1.9× more expensive. In the hybrid arm Jev itself is about 2 percent of the cost, the rest is Claude planning and repairing."

Benchmark it yourself#

Jev was added to OpenRouter on 18 September 2026, which makes independent comparison straightforward for the first time. Two experiments are worth your afternoon.

1. The calibration test — now partly run#

This is the central claim and the easiest thing in machine learning to check: if the value is 0.9 for a thousand answers, roughly nine hundred should be correct. A pre-registered community study has now run it and got a split verdict — well calibrated on one dataset, overconfident on another. That makes running it on your data more important, not less. See calibration, honestly for the numbers.

from collections import defaultdict

BINS = [(i / 10, (i + 1) / 10) for i in range(10)]

def reliability(predictions):
    """predictions: list of (probability, was_correct)

    Returns per-bin (mean predicted, observed rate, n) — the raw material
    of a reliability diagram. Perfect calibration puts observed == predicted.
    """
    buckets = defaultdict(list)
    for p, correct in predictions:
        for lo, hi in BINS:
            if lo <= p < hi or (hi == 1.0 and p == 1.0):
                buckets[(lo, hi)].append((p, correct))
                break

    out = []
    for b in BINS:
        rows = buckets[b]
        if not rows:
            continue
        mean_p = sum(p for p, _ in rows) / len(rows)
        observed = sum(c for _, c in rows) / len(rows)
        out.append((mean_p, observed, len(rows)))
    return out

def expected_calibration_error(predictions):
    total = len(predictions)
    return sum(
        (n / total) * abs(observed - mean_p)
        for mean_p, observed, n in reliability(predictions)
    )

Run it over a labelled set from your own domain, not a public one — distribution shift is the thing that breaks calibration, and a public benchmark tells you about someone else's distribution. One community tool found that an optimal threshold moved from 0.67 to 0.37 between two datasets, with "nothing measured on the first dataset" predicting the second.

2. Cost per resolved request, not per call#

The routing layer is the cheap part. Price the whole pipeline:

JEV_PER_MTOK = 0.042
LLM_PER_MTOK_IN = 2.00      # your fallback model
HUMAN_PER_REVIEW = 4.50     # loaded cost of a minute of someone's time

def cost_per_resolved(volume, tokens_per_call,
                      llm_share, human_share, llm_tokens):
    jev = volume * tokens_per_call / 1e6 * JEV_PER_MTOK
    llm = volume * llm_share * llm_tokens / 1e6 * LLM_PER_MTOK_IN
    human = volume * human_share * HUMAN_PER_REVIEW
    return (jev + llm + human) / volume

# A router that sends 8% to a model and 4% to a person:
print(cost_per_resolved(1_000_000, 1_000, 0.08, 0.04, 3_000))

Run it twice — once with your current design, once with the cascade — and watch what dominates. In most realistic settings it is the human review share, which means the variable worth optimising is not price per token but how often your criteria are ambiguous enough to force an escalation.

What to tell your team#

• The pricing is published, transparent, and TypeSafe concedes it cannot prove it is not subsidised.
• The speed and cost multipliers are vendor-measured and quoted at the top of their own range. Independent tests land far lower and vary by task. Quote 40×–200× attributed to TypeSafe, and measure your own workload.
• Accuracy is roughly at parity with a mid-tier frontier model and several points behind the best one — on TypeSafe's own numbers.
• The type-safety guarantee is real and structural. The "never hallucinates" framing is not.
• Calibration is the central claim. TypeSafe has still published no metric of its own; independent tests report it as dataset-dependent — good on some, overconfident on others.

None of which is a reason not to use it. It is a reason to write down what you expect, measure it against your own data, and keep the fallback path warm.

Next#

Continue to Shipping it: errors, retries and version pinning.