Design note · pilot offer
Measure the agent before you rely on it.
AI coding agents write statistical programs quickly. The regulated question is different: how often is the code right, on which kinds of task, and how would you know? This note sets out how an evaluation could work and what it can and cannot tell you.
Status, plainly. A small evaluation harness exists as a proof of concept. It has been tested with hand-written stand-in code, not with a real AI agent, and there are no results from real agents to report yet. Evaluations are offered as a pilot.
What an evaluation contains
- A task bank built from your own specifications. Real tasks from your standards and analysis plans, not generic puzzles.
- A definition of a pass, written down before any result is seen. Including tolerances and the exact output format.
- Several attempts per task. Agents are not deterministic, so one success proves little.
- A controlled place to run the code. Agent-written code is untrusted, so it runs in a scratch environment, not on a laptop with credentials.
- A comparison with a reference solution, with failures sorted by type: wrong values, wrong shape, error, no result.
- An audit log of every attempt, so the evaluation itself can be reviewed and replayed.
- A named human who reads the failures and decides. That decision is recorded, not automated.
- A re-run when something changes: a new model version, a new prompt, a new tool.
Mistakes worth testing for
The tasks that separate careful from careless code are small and specific. Examples used in the proof of concept: a strict versus a non-strict comparison against a median, a mean that silently ignores or propagates missing values, keeping the wrong record when subjects repeat, the clinical rule that study day has no day 0, and a Welch versus a pooled t-test. Each looks plausible when wrong, which is the point.
What it does not tell you
A pass rate on a small task bank is a measurement, not a qualification. It says how the agent did on those tasks on that day. It does not show the agent is safe for other tasks, or that it will behave the same next month. People stay accountable for the code that is used, and nothing here replaces your quality system.
What a pilot looks like
One agent or tool you already use or are considering, one set of tasks from your specifications, a fixed time box, and a short report your QA group can read, with the audit log behind it. Scope is agreed together.
The earlier idea on this page, package-based agents that draft qualification documents, is on hold. This evaluation work came first because you need to be able to measure an agent before you let it draft anything.
Using, or considering, an AI coding agent for statistical programming?
A short conversation can show what an evaluation would look like for your tasks and standards.