16. Real-Robot Inference, Evaluation, and Data Iteration
Chapter 16 of the SO-ARM Physical AI Beginner's Course — how inference works on the real SO-ARM, run your trained ACT policy with lerobot-record and lerobot-rollout, evaluate by success rate, analyze failures, and iterate on data.
Inference moves the real arm at full torque with no human in the loop. Clear the workspace within 1 meter, keep hands away, and be ready to cut power before starting.
16.1 Inference Data Flow
16.1 Inference Data Flow
Recall Chapter 9: inference is the mirror image of data collection. The loop per frame:
- Read the two cameras → Observation;
- Read the follower's 6 joints → State;
- Normalize with the dataset's stored statistics;
- The policy predicts an Action Chunk (k × 6);
- Execute the first few actions of the chunk on the servos;
- When the executed portion is consumed, re-observe and re-predict.
The leader arm is now unused — the model has replaced the human as the decision-maker. Everything else (cameras, serial bus, safety limits) is identical to recording, which is why the camera keys and --robot.id must match the recording session exactly.
16.2 Preprocessing and De-normalization
16.2 Preprocessing and De-normalization
Your dataset stored per-feature statistics (mean/std) in meta/stats (Chapter 13). At inference:
- Input side: images and joint states are normalized with the training statistics before entering the network.
- Output side: the network's normalized actions are de-normalized back into real joint values before being sent to the servos.
This is why the evaluation command must use the same camera keys (front, side) as recording — a mismatched key means missing statistics, and you'll see:
mean is infinity. You should either initialize with stats as an argument or use a pretrained model
16.3 Starting Real-Robot Inference
16.3 Starting Real-Robot Inference
Method A: lerobot-record with a policy (recommended for beginners)
The record tool accepts a policy checkpoint. This records an evaluation dataset while the policy drives the arm — 10 evaluation episodes:
lerobot-record \
--robot.type=so101_follower \
--robot.port=/dev/ttyACM0 \
--robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30, fourcc: "MJPG"}, side: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30, fourcc: "MJPG"}}" \
--robot.id=my_awesome_follower_arm \
--display_data=false \
--dataset.repo_id=seeedstudio123/eval_test123 \
--dataset.single_task="Put the object from Point A into Point B" \
--policy.path=outputs/train/act_so101_test/checkpoints/last/pretrained_model
Notes:
--policy.pathis the Chapter 15 checkpoint path. If you uploaded the model to the Hub, a repo id works too (e.g.${HF_USER}/act_so101_test).- The evaluation
dataset.repo_idconventionally starts witheval_— evaluation videos and data are saved separately, e.g.seeedstudio123/eval_test123. - If you hit
File exists: '.../eval_xxxx', delete the old evaluation folder first and rerun. - Camera keys (
front,side) must exactly match those used during dataset recording (see 16.2).
Method B: lerobot-rollout real-time deployment
For a live run without recording (supported by policies like SmolVLA / ACT):
lerobot-rollout \
--strategy.type=base \
--policy.path=${HF_USER}/act_policy \
--robot.type=so101_follower \
--robot.port=/dev/ttyACM0 \
--robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30, fourcc: "MJPG"}}" \
--display_data=true \
--task="Your task description" \
--duration=60
16.4 Action Chunk Execution
16.4 Action Chunk Execution
Chapter 14's chunk/horizon distinction becomes visible here:
- The model predicts k=100 actions per forward pass;
- Only the first portion executes open-loop before the loop re-observes;
- The overlap between the tail of the old chunk and the head of the new one is smoothed (temporal ensembling), so motion looks continuous instead of stepping.
If the arm moves in visible "bursts" — a strong pause every half second — the effective horizon or inference speed is the problem: check GPU utilization, camera fps, and consider asynchronous inference (running the policy as a separate server process, see Stage 4's deployment chapter for the idea).
16.5 Safety: Limits, Speed Limits, E-Stop
16.5 Safety: Limits, Speed Limits, E-Stop
- Joint limits from calibration constrain every commanded action.
- Keep a hand on the power switch during the first runs of any new checkpoint; cutting power is the fastest stop (the servos go limp).
- ESC in a
lerobot-recordevaluation session stops the episode cleanly. - Never let the arm run unattended near people, cables, or the edge of the table.
16.6 Evaluation: Success Rate and Completion Time
16.6 Evaluation: Success Rate and Completion Time
Use the Chapter 11 success criteria — objectively determinable, not "looks about right":
- Define success: "the object ends up inside the marked Point B area."
- Vary starts: place the object at the 5 marked points, and repeat each point N times.
- Count: successes / total = success rate. Also record average completion time — a policy that succeeds slowly or with retries is telling you something.
Example result table to keep per checkpoint:
| Start point | Runs | Successes | Rate |
|---|---|---|---|
| Center | 4 | 4 | 100% |
| Top | 4 | 3 | 75% |
| Left | 4 | 2 | 50% |
| ... |
16.7 Generalization Testing
16.7 Generalization Testing
After the in-distribution test passes, probe the boundaries deliberately (one change at a time):
- Move the object slightly outside the marked area;
- Change the object (different size, same category);
- Change ambient lighting (daylight vs. evening lamp).
Expect graceful degradation inside the data distribution and hard failure outside it — that's Chapter 8's distribution principle showing up on hardware. Every generalization failure you want to fix must be answered with data covering that situation, not with more training steps.
16.8 Failure Type Analysis
16.8 Failure Type Analysis
Sort every failed episode into one of four buckets — the bucket dictates the fix:
| Failure type | Looks like | Fix |
|---|---|---|
| Perception failure | Grips 3 cm left of the object | Data lacks coverage for that position/lighting → collect more there |
| Strategy failure | Approaches from the wrong side consistently | Inconsistent demonstrations → re-record with one strategy |
| Execution failure | Right approach, object slips | Gripper force/speed inconsistent in data; check gripper calibration |
| System failure | Arm stutters, pauses, camera lag | Not a model problem: fps, USB bandwidth, calibration, servo temperature |
16.9 Data Iteration: Failure-Driven Data Collection
16.9 Data Iteration: Failure-Driven Data Collection
The loop that ends Stage 3's core path:
- Evaluate (16.6) and bucket the failures (16.8);
- For every perception/strategy failure bucket, collect 10-20 new episodes covering exactly that situation (Chapter 12 workflow);
- Clean and merge into the dataset with the Chapter 13 tools;
- Retrain (Chapter 15) — resume or fresh run;
- Re-evaluate. Repeat until the success rate satisfies your definition of done.
This failure-driven loop — not any single training run — is the daily practice of real robot learning projects.