12. Collect Data from Isaac Sim
Teleoperate the white SO-ARM101 digital twin with the same leader arm, record episodes in the exact same dataset schema, and validate the whole loop without occupying the real robot.
It is recommended to run the complete real-world loop first (collect data → train → evaluate in the real environment, Chapters 9–11). Simulation data does not occupy the real robot and can scale scene and lighting diversity at low cost, so it works well as a supplement once the real pipeline is verified.
12.1 Learning Objectives and Data Flow
12.1 Learning Objectives and Data Flow
- Learn about the NVIDIA Isaac Sim simulation tool;
- Learn the twin's camera layout: the wrist-mounted
frontcamera and the overheadexternalcamera — the same two streams the real dataset uses; - Generate a complete VLA training dataset in the twin.
The data link of the SO-ARM101 project is as follows:
SO-101 Leader (or keyboard IK)
| USB serial
▼
LeRobot Python bridge -- UDP 127.0.0.1:5015 --> Isaac Sim white SO-ARM101
|
+--------------------+--------------------+
| |
▼ ▼
action/state front + external cameras
| |
+--------------------+--------------------+
|
▼
LeRobot Dataset v3
The leader only provides manual control commands; the simulated arm does not autonomously execute grasps. The system simultaneously records:
action: the six leader joint commands;observation.state: the six simulated joint values in leader space;observation.images.front: the camera that moves with the wrist;observation.images.external: the overhead view above the workspace;- the task text (
Put the three vials into the vial rack), randomization parameters, and task success status.
The default acquisition frequency is 30 FPS and both recorded cameras are 640 × 480 — identical to the real dataset of Chapter 9.
12.2 Camera Views
12.2 Camera Views
After pressing R to reset the scene, the two camera views look like the
real robot's: front is the wrist-mounted stream that moves with the arm,
and external is the fixed overhead view — both are the streams recorded
into the dataset.
The gripper and camera mount visible in the lower part of the front view
are intentional — the wrist camera sits at the front of the lens and looks
along the arm, exactly like the real wrist bracket from Chapter 8. Use
./run_so101.sh --single-viewport to keep only the main viewport and hide
the floating camera windows; recording is unaffected.
12.3 Environmental Requirements
12.3 Environmental Requirements
- Ubuntu 22.04;
- NVIDIA RTX GPU — NVIDIA recommends 12 GB+ VRAM for Isaac Sim work;
- Isaac Sim 6.1 standalone installed and launchable;
Install Isaac Sim once:
# Download the Isaac Sim 6.1 standalone zip for Linux from NVIDIA,
# then extract and launch it once:
mkdir -p ~/isaacsim && cd ~/isaacsim
unzip ~/Downloads/isaac-sim-standalone-6.1.0-linux-x86_64.zip
./python.sh -c "print('isaac ok')"
- The LeRobot environment from Chapter 7 (
~/rebot_lerobot/.venv), with the dataset extras so recorded episodes convert:uv pip install -e "./lerobot[dataset,viz]". Without them, recorded episodes fail to convert and the terminal prints[dataset-error]; git-lfsinstalled — the project's USD assets are stored with Git LFS.
On 8 GB GPUs (most laptop GPUs) the render load can starve the desktop compositor and freeze the machine, even with the project's built-in mitigations (Real-Time rendering, 640×360, 30 Hz cap). Use the mitigation flags the project ships, or run headless collection instead.
12.4 Download the Source Code
12.4 Download the Source Code
sudo apt update
sudo apt install -y git-lfs
cd ~
git clone https://github.com/yuyoujiang/rebot-arm-dli-isaacsim.git
cd rebot-arm-dli-isaacsim
git lfs install
git lfs pull
Without git lfs pull the clone only contains small pointer files, and
Isaac Sim cannot open the scene.
12.5 Launch Isaac Sim Teleoperation
12.5 Launch Isaac Sim Teleoperation
cd ~/rebot-arm-dli-isaacsim
./run_so101.sh --leader-port /dev/ttyACM0
If there is only one serial device, the program detects it automatically:
./run_so101.sh
If Isaac Sim is installed in a different directory:
ISAACSIM_ROOT=/path/to/isaacsim ./run_so101.sh --leader-port /dev/ttyACM0
The very first launch builds scenes/so101_vials_sorting.usda and compiles
shaders, which takes a few minutes; this is normal.
No leader hardware? The keyboard IK mode drives the end effector directly:
- Click the 3D viewport once (keyboard focus);
- Press
Y— the terminal prints[teleop] Control source: KEYBOARD (IK: WASD ...); W/S/A/Dmove in the table plane, arrow keys move up and down,Z/Cpitch the tool,Gtoggles the gripper.
12.6 Capture an Episode
12.6 Capture an Episode
Ensure the Isaac Sim viewport has keyboard focus, then:
- Press
Rto reset the scene. The three vials are repositioned, and randomization of objects and lighting is applied; - Check the two camera views and the initial vial positions;
- Press
S(leader mode) orT(keyboard mode) to start recording:
[record] Episode started at 30 FPS -> .../datasets/so101_vials_front_side
- Operate the leader and put the three vials into the vial rack; the
grasp assist attaches a vial when the gripper closes near it
(
[GRASP]in the terminal) and releases it when you open the gripper over the rack ([RELEASE]); - Press
S/Tagain to finish and save; - Wait for the background encoding:
[dataset] Episode saved; total completed: 1
The system converts temporary JPEGs, actions and states into LeRobot Dataset v3 in the background; you can start the next episode during encoding.
Keyboard shortcuts:
| Key | Function |
|---|---|
R | Reset the three vials and re-randomize them |
S / T | Start recording; press again while recording to end and save |
C | Cancel the current episode without writing it to the dataset |
Y | Toggle between leader mode and keyboard IK mode |
B | Toggle slow/fast keyboard step size |
If a vial falls, a grasp fails, or the motion quality is poor, press C
to discard the episode, then R and record again.
12.7 Inspect and Visualize the Dataset
12.7 Inspect and Visualize the Dataset
cd ~/rebot_lerobot
source .venv/bin/activate
lerobot-dataset-viz \
--repo-id local/so101_vials_front_side \
--root ~/rebot-arm-dli-isaacsim/datasets/so101_vials_front_side \
--episode-index 0 \
--display-compressed-images true
Check episodes in sequence: actions, states, and both video streams must be synchronized, with no black frames, frozen frames, or severe occlusion.
12.8 Replaying Actions in Isaac Sim
12.8 Replaying Actions in Isaac Sim
First start an Isaac Sim receiver that does not connect to the leader, does not record, and has randomization disabled:
cd ~/rebot-arm-dli-isaacsim
./run_so101.sh --headless --no-start-bridge --no-recording --no-dr &
Then play back episode 0 from a second terminal:
cd ~/rebot-arm-dli-isaacsim
PYTHONPATH=scripts ~/rebot_lerobot/.venv/bin/python \
scripts/replay_episode_so101.py --episode 0
Loop playback: add --loop. The replay script sends the dataset's
action column to check the arm's trajectory; it is not a video player —
inspect camera content with the dataset visualizer above. The terminal
prints [teleop:LIVE] while actions stream and [SUCCESS] when all three
vials settle inside the rack for 0.35 s.
12.9 Verify Without the Leader (Optional)
12.9 Verify Without the Leader (Optional)
If the leader arm is not available yet, the built-in scripted recorder validates physics, cameras, and dataset writing headlessly:
cd ~/rebot-arm-dli-isaacsim
./run_so101.sh --headless --auto-record 2
This records two short episodes into datasets/so101_vials_front_side and
exits. The final [dataset] Episode saved message means the whole chain —
scene build, physics, cameras, dataset conversion — works with no hardware.
12.10 Configuration and Reference Materials
12.10 Configuration and Reference Materials
Main configuration files of the SO-ARM101 project:
config/teleop_config_so101.json: serial port, UDP, joint mapping and grasping assistance;config/domain_randomization_so101.json: object, lighting, camera and dataset parameters;config/grasp_config_so101.json: scene objects, physical parameters and task success conditions.
Continue with Chapter 13 — Datasets Augment (Optional) or return to the real-robot path at Chapter 14 — Setup Your Jetson Edge Device.