sys1-bench
Which decision model should I trust for yes-or-no calls?
sys1-bench is an eval suite for OpenRouter decision models: small models that return a choice, a score or a probability instead of text. It runs them over a dataset of structured judgments and compares accuracy, calibration, cost and latency.
What is sys1-bench?
sys1-bench measures how well OpenRouter Decisions API models make small structured decisions: judging, classifying and scoring. These models return a choice with calibrated probabilities, bill only input tokens, and have no chat completions path. The cases come from my Bropilot and Protopilot projects, so they reflect decisions those tools actually need to make.
The dataset
18 categories, 231 cases. Examples: is this requirement ambiguous, does this quoted passage support the claim, is this journal entry valid under the stated rulebook, does this one step in a proof follow from the lines before it, which of 28 node kinds should this description become. Binary questions map to the noul primitive, unordered multi-option ones to choice, and ordered scales such as clarity score and priority triage to score.
Some cases are templated: variables are drawn with a seeded RNG per sample and the expected answer can be an expression over those variables. A validator script checks every category file against the spec.
Running it
The app is a Vite and React SPA with a headless CLI. Pick models, review or edit cases, start a run, then compare results. Each run reports accuracy, Brier score, confidence on correct versus incorrect answers, cost and latency, with per-label probabilities for every case. The model list is fetched live from OpenRouter so new decision models appear automatically.
Published results
A GitHub Pages workflow publishes a results-only page from the latest run. The October 2026 run covered all 22 listed decision models at a total cost of $0.138, with the top model at 99% accuracy. The only errors came from known option caps and a model that accepts only binary questions.