Skip to main content
Stage 3 · Chapter 9 · Get Your Own VLA Model

9. Collect Data from the Real Environment

Test the cameras, record synchronized two-camera episodes of the task with lerobot-record, and verify the LeRobot v3 dataset before any training happens.

9.1 Test the Cameras​

Before recording

9.1 Test the Cameras

Chapter 8 mounted and previewed both cameras; repeat the check now — /dev/video* indices change after reconnecting a camera or rebooting:

lerobot-find-cameras opencv

Map the previews to indices (the file name contains the device index), then re-export:

export CAMERA_GRIPPER=4
export CAMERA_EXTERNAL=6

Also confirm the arm exports from Chapter 7 are set in this terminal:

echo "$TELEOP_PORT $TELEOP_ID $ROBOT_PORT $ROBOT_ID"

9.2 Record Episodes​

lerobot-record

9.2 Record Episodes

With both arms calibrated and the studio ready, run:

export HF_USER=local   # or your Hugging Face username

lerobot-record \
--robot.type=so101_follower \
--robot.port=$ROBOT_PORT \
--robot.id=$ROBOT_ID \
--robot.cameras="{
front: {type: opencv, index_or_path: $CAMERA_GRIPPER, width: 640, height: 480, fps: 30},
external: {type: opencv, index_or_path: $CAMERA_EXTERNAL, width: 640, height: 480, fps: 30}
}" \
--teleop.type=so101_leader \
--teleop.port=$TELEOP_PORT \
--teleop.id=$TELEOP_ID \
--dataset.repo_id=so101_vials_real \
--dataset.root=$HOME/rebot-arm-dli-isaacsim/datasets/so101_vials_real \
--dataset.single_task="Put the three vials into the vial rack" \
--dataset.num_episodes=10 \
--dataset.push_to_hub=false \
--display_data=false \
--play_sounds=false

Follow the terminal prompts to complete each episode: pick up a vial with the leader, place it in the rack, release, and let the scene settle.

Key flags:

FlagWhy this value
--robot.cameras keys front and externalMatches the simulated dataset's feature names exactly (Chapter 12 co-training)
--dataset.single_taskMust be this exact sentence everywhere — training prompts and episode labels depend on it
--dataset.num_episodesStart with 10; 5 is NVIDIA's workshop minimum, 50 is a solid policy
--dataset.push_to_hub=falseKeep everything local; upload only if you want a backup
--display_data=falseAvoids rendering every frame on the recording machine
Start with slow, complete demonstrations

A real episode that aborts halfway teaches the policy to stop halfway. Finish the placement, then press the right arrow. Five clean trajectories beat twenty interrupted ones.

9.3 Recording Controls and Safety​

Keyboard

9.3 Recording Controls and Safety

KeyAction
Right arrowFinish the current episode early and move on
Left arrowDiscard the current episode and re-record it
EscStop the session, encode videos, and finish the dataset

If the follower moves unexpectedly:

  1. Press Ctrl+C first;
  2. Cut the follower's 12 V power if it keeps moving;
  3. Return the arm to its middle pose by hand;
  4. Re-check the calibration and the leader ID before restarting.
Keyboard does not respond

On some Ubuntu installations pynput needs to be pinned. Inside the LeRobot environment:

pip install pynput==1.6.8

9.4 Data Storage, Checkpointing and Resuming​

Where data lives

9.4 Data Storage, Checkpointing and Resuming

  • Data is stored using the LeRobotDataset format and is written to disk during recording, at the path given by --dataset.root;
  • With --dataset.push_to_hub=false, nothing is uploaded;
  • Checkpoints are created automatically during recording. To resume after an interruption, re-run the same command with --resume=true;
  • To start recording from scratch, delete the dataset directory.
Resuming

When resuming, set --dataset.num_episodes to the number of additional episodes to record, not the targeted total.

If the arm freezes mid-session, increase the episode time with --dataset.episode_time_s=90 (default 60) and record again.

9.5 Verify the Dataset​

Inspect before training

9.5 Verify the Dataset

Check the metadata:

python3 -c "
import json
info = json.load(open('$HOME/rebot-arm-dli-isaacsim/datasets/so101_vials_real/meta/info.json'))
print('fps:', info['fps'], 'episodes:', info['total_episodes'], 'frames:', info['total_frames'])
print({k: v['shape'] for k, v in info['features'].items()})
"

Expected:

fps: 30 episodes: 10 frames: ...
{'observation.state': [6], 'observation.images.front': [480, 640, 3],
'observation.images.external': [480, 640, 3], 'action': [6], ...}

Two camera streams and six joint dimensions are the contract with the rest of the pipeline. If a shape differs, delete the dataset directory and re-record with the same flags.

Count the videos:

find $HOME/rebot-arm-dli-isaacsim/datasets/so101_vials_real/videos -name "*.mp4" | wc -l

Ten episodes × two cameras = 20 files. A quick visual check:

lerobot-dataset-viz \
--repo-id so101_vials_real \
--root $HOME/rebot-arm-dli-isaacsim/datasets/so101_vials_real \
--episode-index 0 \
--display-compressed-images true

9.6 Data Quality Checklist​

Watch for these

9.6 Data Quality Checklist

Before moving to training, review a few episodes end to end:

  • Every episode completes the task — vial released into the rack;
  • Movements are smooth; no long pauses, no violent corrections;
  • The vials start at varied positions across episodes, not the same spot every time;
  • The wrist view shows the jaws and the grasped vial; the external view shows the rack;
  • No dropped frames (glitchy video) and no autofocus hunting;
  • The task string is identical in every episode.

A dataset that fails any of these points produces a policy that fails in an annoyingly specific way during deployment. Fix the data now; training will not fix it for you.

Next: Chapter 10 — Train the GR00T N1.7 Policy.

Loading Comments...