01 · Define useful work
One complete invoice.
Extract five fields, including dates, currency, exact monetary amounts and an absent purchase order. Every required field must pass. Partially correct outputs fail the task.
BEP Research · benchmark development · v0.1
What does a completed AI task actually cost?
A reproducible starting point for measuring useful work: correct answers, delivered on time, with the cost of failures included.
The decision
For your workload, which complete serving configuration meets the required quality and response time at an acceptable cost? GPU-hour prices and token throughput help explain the result. The primary economic outcome is the cost of successful original tasks.
02 · Keep the clock
Correct and on time.
Record each original task and completion time. The development target is ten seconds. Timeouts remain censored; they do not become completed tasks at the deadline.
03 · Count the full bill
Failures still cost money.
Include the entire serving window or all metered tokens. Divide by tasks that pass quality and finish within the deadline. Zero successes or missing cost evidence gives an undefined result.
Primary economic metric
Cost per 1,000 successful tasks
1,000 × total serving cost ÷ original tasks passing quality and the deadline.
Two separate ledgers.
Deployment economics: measured usage valued at a dated ordinary-access quote. Actual expenditure: the paid invoice, including credits and other charges. Sponsored access does not establish a zero-cost deployment.
A favorable cost ratio does not establish an acceptable service. Quality floors, latency requirements, confidence and workload representativeness must also pass review.
Implemented now
Inspect the machinery.
- 24 fictional BEP-authored development invoices and deterministic exact-field grading.
- Offline replay with intentional errors, a timeout and a late answer.
- An adapter for an explicitly configured chat-completions endpoint, with sequential requests and no retries.
- Per-task traces, CSV, source fingerprints, modeled cost accounting and time/cost admission limits.
- Failure tests for scoring, denominators, missing usage and censored timing.
Before measured comparisons
Earn the performance claim.
- Pin model, tokenizer, precision, serving software and complete system configuration.
- Freeze independently reviewed, licensed holdout tasks and buyer-relevant quality requirements.
- Add controlled load, repeat sessions and case-aware uncertainty estimates.
- Validate the real endpoint and reconcile prices with permissioned billing evidence.
- Independently reproduce results and disclose sponsor support and publication terms.
Run the offline validation
Unzip the download and run these commands in its folder with Python 3.10 or later. No GPU, API key or external service is needed.
python3 -m unittest discover -s . -p 'test_*.py' python3 runner.py --output /tmp/bep-extraction-demo
Choose a new output directory for each run. The replay uses invented fixture timing and prices, prominently labeled in every summary. Those outputs validate accounting and failure handling; they are not benchmark findings. The real endpoint path requires an explicit execution flag and a completed configuration. This package does not provision compute or manage provider billing.
A research platform, one measured question at a time
Bring a workload worth testing.
We can scope a study around an actual compute decision. Hardware access, case rights, review, budget and publication terms are agreed first. Sponsor support, when obtained, will be disclosed; no sponsorship is represented as committed here.