Get started with Lightwheel LeIsaac — Open-source workflow combining Hugging Face LeRobot x GR00T N1.5 x Isaac Sim!
Interduction
This wiki will follow the document of leisaac, show how to teleoperate SoArm101 robot arm in IsaacLab by leisaac. Additionally, we will demonstrate the deployment of the fine-tuned Nvidia Isaac GR00T N1.5 model within the Isaac Lab simulation environment, using data collected from Isaac Lab. Primary projects employed in this wiki:
- LeIsaac provides teleoperation functionality in IsaacLab using the SO101Leader (LeRobot), including data collection, data conversion, and subsequent policy training.
- NVIDIA Isaac™ Lab is an open-source, unified framework for robot learning designed to help train robot policies.
- SO-ARM101 is a low-cost, open-source 3D-printable robotic arm kit. Designed to work seamlessly with the open‑source LeRobot library.
- NVIDIA Isaac GR00T N1.5 is an open foundation model for generalized humanoid robot reasoning and skills.
Requierment
- Ubuntu PC
- SoArm101 Leader Arm
The computer used in this wiki has an NVIDIA RTX 3080 GPU and operates on Ubuntu 22.04 LTS.
Simulation Environment Setup
Use the following command in the terminal to install the Isaac Lab runtime environment:
# Create and activate environment
conda create -n leisaac python=3.10
conda activate leisaac
# Install cuda-toolkit
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
# Install PyTorch
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu118
# Install IsaacSim
pip install --upgrade pip
pip install 'isaacsim[all,extscache]==4.5.0' --extra-index-url https://pypi.nvidia.com
# Install IsaacLab
git clone https://github.com/isaac-sim/IsaacLab.git
sudo apt install cmake build-essential
cd IsaacLab
# fix isaaclab version for isaacsim4.5
git checkout v2.1.0
./isaaclab.sh --install
If your computer does not have conda installed, please refer to this guide for installation.
If you are using 50 series GPU, we recommend to use isaacsim5.0 and isaaclab with feature/isaacsim_5_0 branch.
Install LeIsaac
Clone LeIsaac repository and install it as dependency.
cd ..
git clone https://github.com/LightwheelAI/leisaac.git
cd leisaac
pip install -e source/leisaac
pip install pynput pyserial deepdiff feetech-servo-sdk
Please ensure installation within the leisaac conda virtual environment.
Asset Preparation
LeIsaac provide an example USD asset—a kitchen scene. We can download related scene here and extract it into the assets directory. The directory structure should look like this:
<assets>
├── robots/
│ └── so101_follower.usd
└── scenes/
└── kitchen_with_orange/
├── scene.usd
├── assets
└── objects/
├── Orange001
├── Orange002
├── Orange003
└── Plate
Collect Dataset
Connect the SO-ARM101 leader to an Ubuntu computer via USB cable, then use commands to grant serial port permissions.
ls /dev/ttyACM*
sudo chmod 666 /dev/ttyACM0
If everything works properly, you should see similar log output.
Run teleoperation tasks with the following script to collect dataset:
python scripts/environments/teleoperation/teleop_se3_agent.py \
--task=LeIsaac-SO101-PickOrange-v0 \
--teleop_device=so101leader \
--port=/dev/ttyACM0 \
--num_envs=1 \
--device=cpu \
--enable_cameras \
--record \
--dataset_file=./datasets/dataset.hdf5
After entering the IsaacLab window, press the b key on your keyboard to start teleoperation. You can then use the specified teleop_device to control the robot in the simulation. If you need to reset the environment after completing your operation, simply press the r or n key. r means resetting the environment and marking the task as failed, while n means resetting the environment and marking the task as successful.
Dataset Replay
After teleoperation, you can replay the collected dataset in the simulation environment using the following script:
python scripts/environments/teleoperation/replay.py \
--task=LeIsaac-SO101-PickOrange-v0 \
--num_envs=1 \
--device=cpu \
--enable_cameras \
--dataset_file=./datasets/dataset.hdf5 \
--episode_index=0
Data Conversion
Collected teleoperation data is stored in HDF5 format in the specified directory. If using this data to train a proxy model, the dataset must be converted to LeRobot-compatible format using LeIsaac's conversion scripts.
This script must be executed within the LeRobot virtual environment. Please create a new LeRobot environment by following the steps in this wiki.
cd ..
git clone https://github.com/huggingface/lerobot.git
cd lerobot
conda create -y -n lerobot python=3.10
conda activate lerobot
pip install -e .
Then, we can modify the parameters in the script and run the following command:
cd ../leisaac
python scripts/convert/isaaclab2lerobot.py
If you modified the dataset storage path during data collection, you must update the corresponding path in the conversion script prior to execution.
After the program completes execution, the converted dataset can be found at: ~/.cache/huggingface/lerobot/.
We can also inspect the converted data using LeRobot's built-in dataset visualization toolkit.
cd ../lerobot
python -m lerobot.scripts.visualize_dataset --repo-id EverNorif/so101_test_orange_pick --episode-index 0
Policy Training
In this wiki, we'll fine-tune NVIDIA Isaac GR00T N1.5. Let's begin by setting up the Isaac-GR00T virtual environment:
cd ..
git clone https://github.com/NVIDIA/Isaac-GR00T
cd Isaac-GR00T
conda create -n gr00t python=3.10
conda activate gr00t
pip install --upgrade setuptools
pip install -e .[base]
pip install --no-build-isolation flash-attn==2.7.1.post4
The installation of flash-attn involves package compilation, which can be extremely slow. It is recommended to download the precompiled package version matching your system environment and install it locally using the command: pip install ./package_name.
Run the following command in terminal to launch training:
cd <path-to-Isaac-GR00T>
cp ./getting_started/examples/so100_dualcam__modality.json ~/.cache/huggingface/lerobot/EverNorif/so101_test_orange_pick/meta/modality.json
python scripts/gr00t_finetune.py \
--dataset-path ~/.cache/huggingface/lerobot/EverNorif/so101_test_orange_pick \
--num-gpus 1 \
--output-dir ./so101-checkpoints \
--max-steps 10000 \
--data-config so100_dualcam \
--video-backend torchvision_av \
--batch_size 2
Policy Inference
At this stage, we can deploy our fine-tuned NVIDIA Isaac GR00T N1.5 model to control the SO-ARM101 robotic arm in Isaac Lab. The deployment architecture of Isaac-GR00T adopts a decoupled design between the inference endpoint and control endpoint:
- Inference Endpoint (Server): Dedicated solely to executing model inference tasks.
- Control Endpoint (Client): Responsible for acquiring robotic arm states and orchestrating motion control.
Server Open a new terminal window and run:
conda activate gr00t
cd <path-to-Isaac-GR00T>
python scripts/inference_service.py --server --model_path ./so101-orange-checkpoints/checkpoint-10000 --embodiment_tag new_embodiment --data_config so100_dualcam --port 5555
Client Open a new terminal window and run:
conda activate leisaac
cd <path-to-leisaac>
python scripts/evaluation/policy_inference.py \
--task=LeIsaac-SO101-PickOrange-v0 \
--policy_type=gr00tn1.5 \
--policy_host=localhost \
--policy_port=5555 \
--policy_timeout_ms=5000 \
--policy_action_horizon=16 \
--policy_language_instruction="Pick up the orange and place it on the plate" \
--device=cuda \
--enable_cameras
If you encounter ZMQ-related errors, run pip install pyzmq to resolve them.
The final trained model failed to control the SOArm101 robotic arm to pick the orange. This is because I only collected three sets of data during the experiment. If more data could be gathered, the model's accuracy would be significantly improved!
Tech Support & Product Discussion
Thank you for choosing our products! We are here to provide you with different support to ensure that your experience with our products is as smooth as possible. We offer several communication channels to cater to different preferences and needs.