Design note · pilot offer
AI as the second programmer, people as the judges.
Independent programming QC is a time-consuming routine step in clinical statistical programming. An AI coding agent could write the second program. The value is in what happens next: finding every difference, and having a person decide who is right.
Status, plainly. A small proof of concept exists. It has been tested with hand-written stand-in programs and synthetic data, not with a real AI agent and not on a real study. There are no results from real work to report yet. This is offered as a pilot.
The idea
- The production programmer writes the derivation as usual.
- A second program is written independently from the specification only, without seeing the production code. Here, an AI agent writes it.
- A script runs both on the same data and lists every difference.
- A named person adjudicates each difference: production is wrong, the QC program is wrong, the specification is unclear, or it is not an issue.
- Everything is recorded in a tamper-evident audit log: the hashes of the data and both programs, each difference, each decision and who made it.
What the demonstration shows
On a small synthetic dataset with five derived variables, the comparison found three differences, and each had a different right answer:
- A boundary mistake in the production program. Age exactly 65 was put in the wrong group.
- A mistake in the QC program. A BMI of 24.97 was categorised on its rounded value, 25.0, instead of the unrounded value the specification asks for.
- A gap in the specification. The rule for treatment duration said nothing about subjects still on treatment. The QC program assumed a cut-off date. The right fix is to amend the specification.
That is the main point. A difference does not say who is right, and one of the three was nobody's bug. Agents tend to fill silent gaps with plausible assumptions, which makes them good at exposing underspecified rules, as long as a person reads the result.
What it does not do
- It does not prove either program is correct. Two programs can agree and still share a misreading of the specification.
- It does not replace the QC lead's judgement. The adjudication is deliberately manual.
- It does not remove the need for your own SOPs on independent programming, which decide what counts as acceptable QC in your organisation.
- The demonstration used a synthetic dataset. Real work needs agreed rules for rounding, sorting and metadata, and agent-written code has to be run in a controlled environment.
What a pilot looks like
One deliverable you already QC the traditional way, for example a derived dataset or a table. We run the AI-assisted comparison next to your normal QC, so you can see what each approach finds and what each costs in effort. The output is a short report and the audit log. Nothing replaces your existing QC during the pilot.
Curious how it would work on one of your deliverables?
A short conversation is enough to tell whether a pilot makes sense for your team.