Confidence-gated routing

Where confidence comes from#

Choice and Score answers carry a probabilities map. The shape of that distribution is the uncertainty signal: concentrated on one outcome means a confident answer, spread out means an uncertain one. The confidence field collapses that shape into a single number from 0 to 1 so you can threshold on it without doing the statistics yourself.

TypeSafe has not published the formula. Their docs say only that confidence is "a statistic computed from the probability distribution the answer already gives you", and that the precise computation is a topic for a future cookbook. Anthony Maio's summary is fair: it is derived from distribution shape, but TypeSafe has not said which statistic. If you need a specific one — entropy, margin between top two, max probability — compute it yourself from probabilities, which is exactly why they return it.

Thresholds scale with risk#

The central idea is that a confidence threshold is not one number for your application. It is one number per action, set by what happens when that action is wrong. TypeSafe's own worked example:

response = client.system_one(
    state=user_message,
    questions={
        "action": Choice(
            instructions="What is the user trying to do?",
            criteria={
                "check_balance": "View account balance",
                "approve_transfer": "Approve the pending withdrawal request",
                "support": "Get help with an issue",
            },
        ),
    },
)

action = response.answers["action"]
confidence = action.confidence

if confidence < 0.5:
    # Model is genuinely unsure. Don't guess.
    route_to_human(user_message)

elif action.choice == "check_balance":
    # Low stakes. Showing the wrong screen is recoverable.
    show_balance(account_id)

elif action.choice == "approve_transfer":
    if confidence > 0.9:
        # High stakes, high confidence. Proceed with confirmation.
        confirm_then_execute(account_id)
    else:
        # High stakes, moderate confidence. Verify first.
        ask_user_to_confirm(account_id)

Two thresholds, doing different jobs. The 0.5 floor catches anything the model reports as genuinely uncertain, whatever the action. Above that floor, the bar for acting without confirmation is higher for the operation that moves money than for the one that renders a screen. Maio puts the principle in one sentence: "Routing a help request and authorizing an irreversible transaction should not share a confidence threshold."

The three-band policy#

A good starting structure, straight from TypeSafe's confidence page, is to divide confidence into three ranges that produce three system behaviours:

The part people skip is the third row's second half. A low-confidence answer is not only a signal about this input — it is often a signal about your question.

Low confidence usually means your criteria are wrong#

TypeSafe names the diagnosis directly: low confidence on a Choice often means no option is a clear winner, and low confidence on a Score often means the levels are ambiguous, multi-dimensional, or the state does not contain enough to go on.

• Overlapping options. If billing and account both plausibly cover "I can't update my card", the model is right to be unsure. Tighten the descriptions or merge the options.
• A level that measures two things. "Frustrated but civil" mixes intensity and register. Split it into two Scores and weight them in code.
• A missing option. If the true answer is not in your list, the probability mass has to land somewhere. Add other.
• Evidence that isn't in the state. The model cannot judge a policy you did not send it.

What calibration actually promises#

Calibration, in TypeSafe's own definition, is a property of groups of predictions: outcomes assigned 0.2 should occur about 20% of the time, outcomes assigned 0.8 about 80% of the time. And immediately after, the qualifier that matters most: "These rates describe groups of predictions, not a guarantee about any single answer."

Three consequences for your code:

1. You cannot read one answer's confidence as a correctness probability for that answer. You can only build a policy whose aggregate error rate you are willing to accept.
2. Calibration is not usefulness. Maio's sharpest line: "A model that always predicts the base rate can be perfectly calibrated and contribute nothing to any individual decision." Calibrated and informative are different properties.
3. Calibration does not survive composition automatically. Individually calibrated judgments run through thresholds, weights and branches do not yield a calibrated workflow — and correlated mistakes survive composition.

Thresholds are coupled to a model version#

The moment you tune 0.85 against your own data, that number encodes the shape of one model's distributions. jev-latest is an alias. When it moves, your threshold is silently re-pointed at a distribution nobody checked it against.

# Once a threshold is tuned, pin the version it was tuned against.
client = TypeSafeClient(model="jev-1.13.0")

And log the model field from every response, which reports the versioned ID that actually answered. When your auto-approval rate shifts next quarter, that log is how you find out whether the world changed or the model did.

A complete gate#

from dataclasses import dataclass

@dataclass(frozen=True)
class Gate:
    """Per-action policy. Values live in code so they can be reviewed."""
    auto: float       # act without asking above this
    assist: float     # ask the user to confirm above this
    # below assist: hand it to a person

GATES = {
    "check_balance":    Gate(auto=0.60, assist=0.40),
    "open_ticket":      Gate(auto=0.70, assist=0.50),
    "issue_refund":     Gate(auto=0.92, assist=0.75),
    "approve_transfer": Gate(auto=0.97, assist=0.85),
}

def decide(answer):
    gate = GATES[answer.choice]
    if answer.confidence >= gate.auto:
        return "auto"
    if answer.confidence >= gate.assist:
        return "confirm"
    return "human"

The reason to write it this way is not elegance. It is that GATES is a table a risk owner can read, a test can assert against, and a pull request can diff. That is the actual argument for putting AI decisions behind a typed API: the policy stops being prose.

Next#

Confidence gates a single answer. The next pattern combines several answers into one judgment with weights you control. Continue to Composite scoring.