Action Assay: certifying that robot training video does what its labels say

A synthetic training episode for a robot policy is two files describing one event: a video and an action track. The video shows what happened; the track says what was commanded. Everything downstream assumes agreement. When the video comes out of a world model, the assumption can fail silently: the render drops a grasp, drifts an arm, or plays a plausible motion that was never commanded, and the file still looks fine. A policy trained on disagreeing pairs learns wrong associations one corrupted episode at a time. Watching the video does not catch it; the failure is a video that looks right.

Action Assay is our instrument for that failure. A small inverse dynamics model, trained only on real robot data, reads the actions back out of the pixels; a calibrated gate checks that what the video does agrees with what the label says, to within the agreement real trajectories have with themselves. In goes a video plus its action track; out comes a certificate: pass or fail, per clip and per segment, with the numbers attached. It ships as a Python package, an HTTP API, and one reference instrument for the bimanual AgileX Piper (14-D joint actions). This post covers the validation, and the case where the obvious metric hid the failure the gate exists to catch.

An auditor that reads actions off pixels#

The auditor must be independent of the generator and cheap to retrain per robot. Ours is a 0.24M-parameter network (237,422 parameters): two consecutive frames in, the 14-D action that produced the transition out, through a spatial-softmax keypoint head that localizes arms and grippers rather than reading a global feature blob.¹ We trained it on 260 real episodes from a 2,301-episode public dataset of bimanual Piper pick-and-place, 16 epochs over 44,913 frame pairs, 29 minutes on one A10G: about $0.65 of GPU time.

On 30 held-out real episodes it recovers commanded actions at R² 0.945 across the twelve arm dimensions and 0.988 on the two grippers. The weakest dimension is left-arm joint 3 at 0.68; every other dimension sits at or above 0.90. We publish the per-dimension table; aggregates hide soft spots.

Grading on the right curve#

A gate needs a threshold from somewhere defensible. We calibrate two bands. The real band is the envelope of self-agreement on the 30 held-out real episodes: how well the instrument reads real pixels against their own labels. The render band is the same construction over re-renders of real episodes driven by their recorded actions, the population the validation document designates as the faithful reference.² The two differ by about 0.24 in pooled correlation (0.97 real, 0.73 faithful renders): an appearance penalty. Generation artifacts depress the reading on every render, faithful or not; a real-calibrated gate run on renders works as a render detector and says nothing about obedience.

One control proves it. A certified-faithful render from that population, driving the exact actions of real episode 45, scores corr 0.794 and mae 0.491: it fails the real band and passes the render band. Same commanded motion, different curve. Grading renders against the render band isolates the question that matters, whether the video obeyed its commands.

Sabotage as validation#

A gate validated only on positives proves little, so we attacked ours. Temporally shuffling an episode's own actions collapses pooled correlation to 0.16: rejected. Pairing a video with the mirrored wrong-arm trajectory scores corr -0.52: rejected. Real episodes pass; the faithful render above passes on its own band.

One control we publish as a limitation. Swapping in the action track of a different episode of the same task (episode 46, another banana pick-and-place) scores corr 0.837, close to the threshold: every episode of the task approaches, grasps, lifts, and places along roughly the same arc, and pooled correlation is fooled by trajectory stereotypy. The error stays high (mae 0.448) and the per-frame series separates it; clip-level correlation alone does not. Gross disobedience is caught cleanly. A same-task swap sits on the boundary, and we say so.

The average that hid a collapse#

The gate earns its keep on spliced episodes we call chimeras: take the first part of episode A's action track, continue with episode B's, and have the world model render the whole thing. Seven exist, three from an early splice recipe (v1), four from a more aggressive one (v2). Scored as whole clips against the render band, all seven pass.

That verdict is accurate and misleading at once. In the per-frame error series, the v1 renders genuinely obey their spliced commands: post-splice error runs 0.81 to 0.94 times the pre-splice level. The four v2 renders track the A-segment, decouple at the splice, and stay wrong: post-splice error a sustained 2.0 to 3.3 times pre-splice. The clip mean dilutes that collapse below threshold; the well-tracked first half pays for the disobedient second half. This is the failure the gate was built to catch, and the first metric anyone reaches for, one number per clip, passed all four.

The shipped gate therefore scores per segment as well as per clip: a segment that sustains error beyond the band fails the episode even when the clip mean survives.

From gate to service#

The gate now ships as a package and a service. The CLI takes a video plus an action track and emits a certificate; the HTTP API exposes the same certify operation, embodiment listing, and an experimental relabel endpoint that writes the recovered actions back out. Each certificate carries a deterministic id derived from the input hashes and the instrument's weight hash, so a verdict names exactly which video, labels, and weights produced it.

We checked the port against the research code before shipping. Held-out episode 16 through the packaged CLI reads corr 0.9560 against the published 0.9556: a delta at the fourth decimal, explained by the video round-trip the exported frames go through.³ The certificate reads CERTIFIED, 12 of 12 segments passing, id ac_0f0d814b4cc4fe73. Thirteen tests pin the geometry, normalization, band thresholds, and certificate ids.

Scope, honestly#

What a certificate means, and what it does not. An instrument is per-embodiment and per-camera-rig: this one reads a bimanual Piper through one composite camera layout; a new robot or rig needs a new instrument. Render bands are additionally per-generator, because the appearance penalty depends on the renderer. Certification proves the video obeyed its labels. It does not prove a policy trained on the certified data improves; that claim needs a training run we have not made. The relabel endpoint is experimental; its outputs are labels to validate downstream, not certified data. Onboarding a new embodiment repeats the protocol: about 250-300 real demonstrations (we used 260 train, 30 held out), two calibrated bands, and a five-control battery (real positive, render-domain positive, wrong-arm, shuffle, same-task) before any verdict is trusted.

Take it home#

• The page: alakazam.gg/action-assay, with bands, controls, and chimera series.
• Technical report: alakazam.gg/action-assay/report.pdf, every number here with its source artifact.
• Code: private during the pilot program, access on request.
• Contact: hugo@alakazam.gg.

Provenance#

The instrument comes from our own Piper augmentation work: we were generating synthetic training video and needed to know whether it obeyed its labels before training on it, so we built the auditor first. Every number here traces to a logged artifact: training logs, calibration files, gate reports. The validation document ships with the instrument, weakest dimension and boundary cases included.

Alakazam, July 2026.