Skip to main content
Stage 3 · Chapter 12 · Optional · Sim data

12. Collect Data from Isaac Sim

Teleoperate the white SO-ARM101 digital twin with the same leader arm, record episodes in the exact same dataset schema, and validate the whole loop without occupying the real robot.

This chapter is optional

It is recommended to run the complete real-world loop first (collect data → train → evaluate in the real environment, Chapters 9–11). Simulation data does not occupy the real robot and can scale scene and lighting diversity at low cost, so it works well as a supplement once the real pipeline is verified.

12.1 Learning Objectives and Data Flow​

Data link

12.1 Learning Objectives and Data Flow

  • Learn about the NVIDIA Isaac Sim simulation tool;
  • Learn the twin's camera layout: the wrist-mounted front camera and the overhead external camera — the same two streams the real dataset uses;
  • Generate a complete VLA training dataset in the twin.

The data link of the SO-ARM101 project is as follows:

SO-101 Leader (or keyboard IK)
| USB serial
▼
LeRobot Python bridge -- UDP 127.0.0.1:5015 --> Isaac Sim white SO-ARM101
|
+--------------------+--------------------+
| |
▼ ▼
action/state front + external cameras
| |
+--------------------+--------------------+
|
▼
LeRobot Dataset v3

The leader only provides manual control commands; the simulated arm does not autonomously execute grasps. The system simultaneously records:

  • action: the six leader joint commands;
  • observation.state: the six simulated joint values in leader space;
  • observation.images.front: the camera that moves with the wrist;
  • observation.images.external: the overhead view above the workspace;
  • the task text (Put the three vials into the vial rack), randomization parameters, and task success status.

The default acquisition frequency is 30 FPS and both recorded cameras are 640 × 480 — identical to the real dataset of Chapter 9.

12.2 Camera Views​

front + external

12.2 Camera Views

After pressing R to reset the scene, the two camera views look like the real robot's: front is the wrist-mounted stream that moves with the arm, and external is the fixed overhead view — both are the streams recorded into the dataset.

Wrist hardware in view is expected

The gripper and camera mount visible in the lower part of the front view are intentional — the wrist camera sits at the front of the lens and looks along the arm, exactly like the real wrist bracket from Chapter 8. Use ./run_so101.sh --single-viewport to keep only the main viewport and hide the floating camera windows; recording is unaffected.

12.3 Environmental Requirements​

GPU and install

12.3 Environmental Requirements

  • Ubuntu 22.04;
  • NVIDIA RTX GPU — NVIDIA recommends 12 GB+ VRAM for Isaac Sim work;
  • Isaac Sim 6.1 standalone installed and launchable;

Install Isaac Sim once:

# Download the Isaac Sim 6.1 standalone zip for Linux from NVIDIA,
# then extract and launch it once:
mkdir -p ~/isaacsim && cd ~/isaacsim
unzip ~/Downloads/isaac-sim-standalone-6.1.0-linux-x86_64.zip
./python.sh -c "print('isaac ok')"
  • The LeRobot environment from Chapter 7 (~/rebot_lerobot/.venv), with the dataset extras so recorded episodes convert: uv pip install -e "./lerobot[dataset,viz]". Without them, recorded episodes fail to convert and the terminal prints [dataset-error];
  • git-lfs installed — the project's USD assets are stored with Git LFS.
8 GB laptop GPUs

On 8 GB GPUs (most laptop GPUs) the render load can starve the desktop compositor and freeze the machine, even with the project's built-in mitigations (Real-Time rendering, 640×360, 30 Hz cap). Use the mitigation flags the project ships, or run headless collection instead.

12.4 Download the Source Code​

Project clone

12.4 Download the Source Code

sudo apt update
sudo apt install -y git-lfs

cd ~
git clone https://github.com/yuyoujiang/rebot-arm-dli-isaacsim.git
cd rebot-arm-dli-isaacsim
git lfs install
git lfs pull
USD assets use Git LFS

Without git lfs pull the clone only contains small pointer files, and Isaac Sim cannot open the scene.

12.5 Launch Isaac Sim Teleoperation​

First build

12.5 Launch Isaac Sim Teleoperation

cd ~/rebot-arm-dli-isaacsim
./run_so101.sh --leader-port /dev/ttyACM0

If there is only one serial device, the program detects it automatically:

./run_so101.sh

If Isaac Sim is installed in a different directory:

ISAACSIM_ROOT=/path/to/isaacsim ./run_so101.sh --leader-port /dev/ttyACM0

The very first launch builds scenes/so101_vials_sorting.usda and compiles shaders, which takes a few minutes; this is normal.

No leader hardware? The keyboard IK mode drives the end effector directly:

  1. Click the 3D viewport once (keyboard focus);
  2. Press Y — the terminal prints [teleop] Control source: KEYBOARD (IK: WASD ...);
  3. W/S/A/D move in the table plane, arrow keys move up and down, Z/C pitch the tool, G toggles the gripper.

12.6 Capture an Episode​

R · S · C

12.6 Capture an Episode

Ensure the Isaac Sim viewport has keyboard focus, then:

  1. Press R to reset the scene. The three vials are repositioned, and randomization of objects and lighting is applied;
  2. Check the two camera views and the initial vial positions;
  3. Press S (leader mode) or T (keyboard mode) to start recording:
[record] Episode started at 30 FPS -> .../datasets/so101_vials_front_side
  1. Operate the leader and put the three vials into the vial rack; the grasp assist attaches a vial when the gripper closes near it ([GRASP] in the terminal) and releases it when you open the gripper over the rack ([RELEASE]);
  2. Press S/T again to finish and save;
  3. Wait for the background encoding:
[dataset] Episode saved; total completed: 1

The system converts temporary JPEGs, actions and states into LeRobot Dataset v3 in the background; you can start the next episode during encoding.

Keyboard shortcuts:

KeyFunction
RReset the three vials and re-randomize them
S / TStart recording; press again while recording to end and save
CCancel the current episode without writing it to the dataset
YToggle between leader mode and keyboard IK mode
BToggle slow/fast keyboard step size

If a vial falls, a grasp fails, or the motion quality is poor, press C to discard the episode, then R and record again.

12.7 Inspect and Visualize the Dataset​

Dataset visualizer

12.7 Inspect and Visualize the Dataset

cd ~/rebot_lerobot
source .venv/bin/activate

lerobot-dataset-viz \
--repo-id local/so101_vials_front_side \
--root ~/rebot-arm-dli-isaacsim/datasets/so101_vials_front_side \
--episode-index 0 \
--display-compressed-images true

Check episodes in sequence: actions, states, and both video streams must be synchronized, with no black frames, frozen frames, or severe occlusion.

12.8 Replaying Actions in Isaac Sim​

Closed sim loop

12.8 Replaying Actions in Isaac Sim

First start an Isaac Sim receiver that does not connect to the leader, does not record, and has randomization disabled:

cd ~/rebot-arm-dli-isaacsim
./run_so101.sh --headless --no-start-bridge --no-recording --no-dr &

Then play back episode 0 from a second terminal:

cd ~/rebot-arm-dli-isaacsim
PYTHONPATH=scripts ~/rebot_lerobot/.venv/bin/python \
scripts/replay_episode_so101.py --episode 0

Loop playback: add --loop. The replay script sends the dataset's action column to check the arm's trajectory; it is not a video player — inspect camera content with the dataset visualizer above. The terminal prints [teleop:LIVE] while actions stream and [SUCCESS] when all three vials settle inside the rack for 0.35 s.

12.9 Verify Without the Leader (Optional)​

Headless check

12.9 Verify Without the Leader (Optional)

If the leader arm is not available yet, the built-in scripted recorder validates physics, cameras, and dataset writing headlessly:

cd ~/rebot-arm-dli-isaacsim
./run_so101.sh --headless --auto-record 2

This records two short episodes into datasets/so101_vials_front_side and exits. The final [dataset] Episode saved message means the whole chain — scene build, physics, cameras, dataset conversion — works with no hardware.

12.10 Configuration and Reference Materials​

Config files

12.10 Configuration and Reference Materials

Main configuration files of the SO-ARM101 project:

  • config/teleop_config_so101.json: serial port, UDP, joint mapping and grasping assistance;
  • config/domain_randomization_so101.json: object, lighting, camera and dataset parameters;
  • config/grasp_config_so101.json: scene objects, physical parameters and task success conditions.

Continue with Chapter 13 — Datasets Augment (Optional) or return to the real-robot path at Chapter 14 — Setup Your Jetson Edge Device.

Loading Comments...