Plan and run
A run starts from a frozen plan that fixes its dataset, targets, sampling and limits. This page covers the manifest, the run commands and their limits, native output capacity, and answer grading. It assumes a prepared dataset from Prepare reusable tasks and targets.
Write a manifest
Create a JSON or YAML manifest with the prepared dataset's path and sha256,
its matching profile/seed, and the targets to compare. A minimal shape is:
version: sr-bench-1.0
name: Development comparison
mode: live
profile: quick
seed: 20260918
dataset:
path: /absolute/path/from/prepared/manifest/cases.jsonl
sha256: REPLACE_WITH_PREPARED_DATASET_SHA256
targets:
- id: single-a
kind: single
base_url: http://model-gateway.example:8000/v1
model: served-model-name
api_key_env: MODEL_API_KEY
cost_policy: capability_only
limits:
concurrency: 1
total_timeout_s: 180
idle_timeout_s: 30
max_output_tokens: 4096
max_run_seconds: 1800
sampling:
temperature: 0
top_p: 1
max_tokens: 4096
This example explicitly permits unknown prices, so it can't prove cost
savings. For a cost comparison, supply complete four-bucket target prices and
set cost_policy: require_priced (the default).
Other limits get their recorded defaults during plan creation; inspect the frozen result.
Plan and start a run
vllm-sr benchmark plan --manifest candidate.yaml --output frozen.json
vllm-sr benchmark run --manifest frozen.json --detach --idempotency-key iteration-1
vllm-sr benchmark runs
vllm-sr benchmark show RUN_ID
vllm-sr benchmark report RUN_ID --output report.json
- Idempotency. A repeated idempotency key binds to the same frozen plan, and doesn't resend model requests.
- Waiting. Without
--detach, the CLI waits for completion; interrupting that wait leaves the worker running. - Stopping. Stop actual work with
vllm-sr benchmark cancel RUN_ID.
Limits
| Scope | Limits |
|---|---|
| Each call | Absolute and idle timeouts, output and repetition limits |
| Run and task | Elapsed time and call counts |
| Priced runs | Reserve estimated spend before dispatch, and stop on reported actual spend |
- Not a hard USD cap. The spend limits aren't a provider-enforced universal hard cap: missing usage, inaccurate prices or unsupported backend accounting can leave cost unknown. Unknown usage never becomes zero, or evidence of savings.
- Quality only (
capability_only) disables USD-budget stopping; request, case and run time, call and output limits still apply. Known spend is recorded, but missing prices or usage don't support savings claims.
Native output capacity
Set output_policy: native to explore each model's available output
capacity, without a shared generation-token cap:
- Register
native_limitson every selected target, keyed by the physical response model, with verifiedcontext_windowandmax_output_tokensvalues. - Keep model-specific reasoning settings in
request_params. - Omit
max_tokensfrom both sampling and target overrides.
How each target kind sets its budget:
- Single model. The adapter uses the vLLM Chat render API to count the actual prompt, then generates once with the smaller of the configured output capacity and the remaining context.
- MoM. The recipe must use
request_params.default_max_tokens: autoon every reachable decision, without a smaller output limit. Its Router response records the actual selected-model input and output budget. - Evidence, never a guess. Missing or inconsistent evidence stops qualification; the harness doesn't guess a budget from generated token usage.
- Requirements. Native mode requires physical response identities to match the selected model identities, and one fully accounted dispatch.
The plan derives its evidence token ceiling from the frozen native limits:
- Allow for it. Give enough call and run time and output storage for that capacity; idle, cancellation and repetition controls stay active.
- A ceiling, not a target. Model maximum output and total context are different values, normal end-of-answer stopping stays enabled, and native capacity doesn't force a model to fill its context.
- Like for like. Compare native candidates against native baselines with the same model limits. Offline replay is unavailable when equivalent per-call native budget evidence can't be established.
Answer grading
The MMLU-Pro and GPQA adapters use sr-bench-mcq-final-v2. Capability
scoring extracts an unambiguous answer from the visible final channel:
- Accepted: a leading answer line, an explicit answer declaration, or a boxed choice. Markdown emphasis doesn't change the answer.
- Left unparsed: conflicting declarations, and prose without an explicit answer. The grader doesn't guess from isolated letters, or use hidden reasoning.
This is a conservative sr-bench adaptation, not an exact reproduction of the upstream extraction heuristics.
- Format compliance is reported separately from correctness, and truncated responses still count as output failures.
- Frozen graders. Adapter versions are frozen in each plan, so different graders can't silently share a comparison, and existing run scores stay unchanged.
- Regrading.
benchmark regrade RUN_IDreturns a separate, versioned result from saved final answers, without generating or rewriting them.