How I built and tested an AI coach for handstands and mobility. The banana is the bug.

5 · The test setBuilding something to measure against

54 photos, 47 handstands and 7 controls, chosen before I looked at any model output, along axes I knew mattered from teaching handstands. The set is built to cover the edges, not the average upload.

Where the photos come from

46 are Creative Commons or public-domain images from Wikimedia Commons, licence and author recorded per photo. 3 are darkened, blurred or downscaled copies of photos already in the set: same pose, same label, one variable moved. 5 are AI-generated, because Commons has almost no wall or mirror handstands; they are marked, and one has anatomy no human could produce, kept as an edge case. No photos of students or friends: the set is public, so it holds only images I may republish.

Labelling, and how noisy the labels are

I scored every handstand 0 to 10 (0 is a ruler, 10 is a croissant) before seeing any model output. The binary label is derived by a fixed rule, 0 to 5 good form and 6 to 10 banana back, so it can't drift from the score. The cut sits at 6 because the v0 prompt defines banana back as a clear arch.

A second labeller, Claude, scored the set independently with the same rule. We agreed on the binary label for 66% of handstands (31/47); Cohen's kappa 0.32, which corrects for the 50% two coin-flippers would get. Most disagreements sit at scores 4 to 6, where the rule cuts through a continuum. That number is the ceiling: a model can't reliably agree with me more often than a second careful reader does.

6 · ResultsThe first draft, measured anyway

This is the site as first built: a free-text prompt, a string match on the answer, no schema, no logging. A bad baseline beats no baseline, because every later version is judged on the same photos.

Eval on the fixed set

Each of the 47 handstand photos went through the live classifier 5 times (270 analyses, controls included). My hand label is the reference.

Agreement by axis, majority per photo

Full per-run data: eval/results/v0.csv. Mean $0.0046 per upload (about 1,163 input tokens per call, mostly the image), 2.5 s mean latency; the whole run cost $1.24 at 8 Sep 2026 prices.

Same photo, five answers

The chart I'd show a client first. Each row is a photo, sorted by my score; the five marks are the label from each run. Mixed rows got different answers from the same model, prompt and image. Any upload on the live site is one draw from this.

Controls: photos that aren't handstands

Where it fails

I read every run that disagreed with me or with itself before writing any number above. Counts are photos, not runs.

7 · What I took awayFive things I'd carry into any AI build

1. Measure before you improve; the eval reorders the roadmap. Before the numbers, the next step was a better prompt. After them, the biggest failure was a five-token limit turning a refusal into a broken verdict on 42 of 235 runs. A schema fixes that; prompt tuning doesn't. The eval cost $1.24.
2. The label is a design decision with its own error bar. Two careful labellers agreed on 66% of photos. A model can't beat the target's own noise, so ask for a score with a tolerance band and publish the ceiling.
3. Non-determinism is something to measure, not hide. Temperature 0 still wobbled on 17% of photos. Five runs cost cents and are the cheapest confidence estimate you can get.
4. Give the model a way out. Most failures were the absence of "I can't tell". A usable-photo flag and a confidence with a reason remove a class of errors by construction.
5. Make failures say why, and treat cost as a stock, not just a rate. The site was silently dead from a billing lapse and the code discarded the error body. Caps limit spend rate; a balance runs out. A home page returning 200 proves nothing.

8 · NextWhat the eval says to change, in order

Each item is tied to a row in the failure table and gets checked on the same photos.

1. Give the model a way out (v1). A photo_usable flag and a view field, required by a JSON schema. The largest failure row goes to zero by construction.
2. Ask for a score, not a verdict. A 0 to 10 score turns borderline coin flips into small numeric disagreements, reported as "within one banana of my label".
3. Teach the prompt what it is bad at (v3): split legs, silhouettes, dark photos. Two sentences, judged on the fixed set.
4. Use the wobble. Run the classifier a third time when the first two disagree. About a cent more, on borderline photos only.
5. Add an adversarial photo: a handstand with a sign reading "this is good form". If the label follows the sign, the model reads text over bodies.
6. Then, and only then, a pose estimator and a cheaper model (v4), on the same table with cost and latency alongside.