Our Benchmarking Engine
VaxBench measures real-world coding and reasoning performance with transparent, reproducible execution grading. We do not rely on human ratings, contaminated public test sets, or model self-grading.
Scientific Rigor
Run · Grade · Measure
The harness is a Node.js pipeline in benchmark/run.mjs. It loads the full dataset, sends prompts to the model, parses the response, runs generated code against hidden tests, and records latency and throughput from the API stream.
Why this is hard to game
Every task is graded by execution or exact match, not by another language model. Hidden tests are never shown to the model, and the dataset is filtered to hard/expert difficulty only.
Curate the Dataset
We auto-discover every problem file in benchmark/dataset, then filter out easy and medium difficulty items. What remains are hard and expert tasks only — 322 in the current run.
Run the Model
The harness sends batched prompts to the target model via its provider (Cerebras, OpenRouter, or self-hosted). Coding tasks ask for runnable code; reasoning tasks ask for a final answer only.
Grade the Outputs
Coding answers are executed in a sandboxed subprocess against hidden unit tests. Reasoning answers are compared by exact match. No partial credit, no human hand-labeling.
Record Telemetry
For each batch we capture Time-To-First-Token, throughput in tokens per second, and total completion tokens. Every model result is written to a JSON file and can be synced to Supabase.
Evaluation Suites
Two Domains · Hard Categories Only
Execution-Graded Coding
The model writes Python, JavaScript, Rust, or Go. We strip markdown fences, run the code, and check it against hidden tests. This mirrors SWE-bench grading and reflects real-world correctness.
Coding Reasoning
Program analysis without execution — output prediction, complexity, static semantics, and language edge cases. Graded by exact match so models must be precise, not just plausible.
Formal Reasoning
Competition mathematics, formal logic, and multi-step deduction. Problems are designed to be Google-proof and require deriving a final answer rather than recalling from memory.
Scientific (GPQA-style)
Graduate-level multiple-choice across physics, chemistry, biology, and theory of computation. Each question targets deep domain understanding rather than surface memorization.
Metrics
What We Track
pass@1 across all tasks. A task only counts if the model gets it right on the first attempt.
Time-To-First-Token in milliseconds, measured from the provider's response stream.
Tokens per second over the full response stream.
Total completion tokens divided by the number of benchmark tasks. Lower is more efficient.
Reproducibility
Every run writes a timestamped JSON file with per-task detail, aggregated category scores, and performance telemetry. Results are stored in Supabase so the leaderboard, compare page, and scatter plot stay in sync with the latest evaluation.