Skip to main content
Version: Latest (main)

Plan and run

A run starts from a frozen plan that fixes its dataset, targets, sampling and limits. This page covers the manifest, the run commands and their limits, native output capacity, and answer grading. It assumes a prepared dataset from Prepare reusable tasks and targets.

Write a manifest​

Create a JSON or YAML manifest with the prepared dataset's path and sha256, its matching profile/seed, and the targets to compare. A minimal shape is:

version: sr-bench-1.0
name: Development comparison
mode: live
profile: quick
seed: 20260918
dataset:
path: /absolute/path/from/prepared/manifest/cases.jsonl
sha256: REPLACE_WITH_PREPARED_DATASET_SHA256
targets:
- id: single-a
kind: single
base_url: http://model-gateway.example:8000/v1
model: served-model-name
api_key_env: MODEL_API_KEY
cost_policy: capability_only
limits:
concurrency: 1
total_timeout_s: 180
idle_timeout_s: 30
max_output_tokens: 4096
max_run_seconds: 1800
sampling:
temperature: 0
top_p: 1
max_tokens: 4096
caution

This example explicitly permits unknown prices, so it can't prove cost savings. For a cost comparison, supply complete four-bucket target prices and set cost_policy: require_priced (the default).

Other limits get their recorded defaults during plan creation; inspect the frozen result.

Plan and start a run​

vllm-sr benchmark plan --manifest candidate.yaml --output frozen.json
vllm-sr benchmark run --manifest frozen.json --detach --idempotency-key iteration-1
vllm-sr benchmark runs
vllm-sr benchmark show RUN_ID
vllm-sr benchmark report RUN_ID --output report.json
  • Idempotency. A repeated idempotency key binds to the same frozen plan, and doesn't resend model requests.
  • Waiting. Without --detach, the CLI waits for completion; interrupting that wait leaves the worker running.
  • Stopping. Stop actual work with vllm-sr benchmark cancel RUN_ID.

Limits​

ScopeLimits
Each callAbsolute and idle timeouts, output and repetition limits
Run and taskElapsed time and call counts
Priced runsReserve estimated spend before dispatch, and stop on reported actual spend
  • Not a hard USD cap. The spend limits aren't a provider-enforced universal hard cap: missing usage, inaccurate prices or unsupported backend accounting can leave cost unknown. Unknown usage never becomes zero, or evidence of savings.
  • Quality only (capability_only) disables USD-budget stopping; request, case and run time, call and output limits still apply. Known spend is recorded, but missing prices or usage don't support savings claims.

Native output capacity​

Set output_policy: native to explore each model's available output capacity, without a shared generation-token cap:

  1. Register native_limits on every selected target, keyed by the physical response model, with verified context_window and max_output_tokens values.
  2. Keep model-specific reasoning settings in request_params.
  3. Omit max_tokens from both sampling and target overrides.

How each target kind sets its budget:

  • Single model. The adapter uses the vLLM Chat render API to count the actual prompt, then generates once with the smaller of the configured output capacity and the remaining context.
  • MoM. The recipe must use request_params.default_max_tokens: auto on every reachable decision, without a smaller output limit. Its Router response records the actual selected-model input and output budget.
  • Evidence, never a guess. Missing or inconsistent evidence stops qualification; the harness doesn't guess a budget from generated token usage.
  • Requirements. Native mode requires physical response identities to match the selected model identities, and one fully accounted dispatch.

The plan derives its evidence token ceiling from the frozen native limits:

  • Allow for it. Give enough call and run time and output storage for that capacity; idle, cancellation and repetition controls stay active.
  • A ceiling, not a target. Model maximum output and total context are different values, normal end-of-answer stopping stays enabled, and native capacity doesn't force a model to fill its context.
  • Like for like. Compare native candidates against native baselines with the same model limits. Offline replay is unavailable when equivalent per-call native budget evidence can't be established.

Answer grading​

The MMLU-Pro and GPQA adapters use sr-bench-mcq-final-v2. Capability scoring extracts an unambiguous answer from the visible final channel:

  • Accepted: a leading answer line, an explicit answer declaration, or a boxed choice. Markdown emphasis doesn't change the answer.
  • Left unparsed: conflicting declarations, and prose without an explicit answer. The grader doesn't guess from isolated letters, or use hidden reasoning.

This is a conservative sr-bench adaptation, not an exact reproduction of the upstream extraction heuristics.

  • Format compliance is reported separately from correctness, and truncated responses still count as output failures.
  • Frozen graders. Adapter versions are frozen in each plan, so different graders can't silently share a comparison, and existing run scores stay unchanged.
  • Regrading. benchmark regrade RUN_ID returns a separate, versioned result from saved final answers, without generating or rewriting them.

Next​