Synthetic coding evaluation
One-task coding-agent evaluation pilot
A privately held, original synthetic evaluation task for teams building or benchmarking coding agents. Offered as a fixed-fee, limited, non-exclusive pilot — only after written scope, agreed price, and acceptance terms. There is no set price and no instant checkout on this page.
Who this is for
- AI agent benchmark and evaluation teams
- Coding-tooling teams needing an extra deterministic test task
- Researchers who want a small, self-contained reconciliation problem with a reference solution
What the sample contains
- One original fictional event-journal reconciliation problem: UTC cutoffs, revisions, duplicates, tombstones, and invalid-data validation
- 15 deterministic Python standard-library unittest checks
- A reference solution passing 15/15 checks locally
- A deliberately flawed starter passing 5/15 checks locally
- A written task specification
The 15/15 and 5/15 figures are local deterministic test results on the supplied code — not AI model results, benchmark scores, or capability claims.
What a preview discussion covers
- High-level description of the task and its reproducibility
- The scope of rights and permitted use on offer
- Format and delivery options
The full task specification, tests, flawed starter, and reference solution stay private until terms are agreed. No code, answer keys, or grader are published here.
Stated plainly
- An earlier 10-task / 59-check evaluation archive was not recovered; this is a new, separate one-task sample created November 10, 2026
- Produced with AI assistance; no claim of exclusive rights or sole human authorship
- No model benchmark runs, no independent buyer validation, and no revenue have resulted from this sample
- No guarantee of automated grading acceptance in any harness other than standard Python unittest
- No guaranteed capability measurement or model discrimination
- No employer or customer data is involved
Commercial terms
A potential fixed-fee, limited, non-exclusive license to this one task — discussed and agreed in writing before anything is delivered or paid. Any engagement requires:
- A written scope and an agreed price
- Agreed acceptance tests
- Agreed rights, licensing, and permitted AI assistance
- A buyer payment arrangement settled before delivery
There is no buy-now link and no speculative checkout for this pilot. To start a conversation, email with your grading harness, licensing needs, accepted format, budget, and timeline: