Stage 4 · Chapter 21 · Practice

21. Fine-tuning the SO-ARM with Isaac GR00T

Chapter 21 of the SO-ARM Physical AI Beginner's Course — set up the Isaac-GR00T environment, fine-tune GR00T N1.5 on your SO-ARM dataset with the so100_dualcam config, then run inference with the GR00T server and the SO-100 robot control client, on a workstation or Jetson AGX Thor.

EnvironmentFine-tuneInference

21.1 Choose Your Training Path

Paths

21.1 Choose Your Training Path

This chapter follows the Isaac-GR00T path used by NVIDIA's official SO101 fine-tuning guide — the same recipe Seeed uses in the SO101 + GR00T on Jetson Thor tutorial. Three ways to get the compute:

Path	Hardware	Notes
Local workstation	RTX 4090 / A6000 class (Ampere+, ~25GB VRAM)	What this chapter assumes
NVIDIA Brev cloud	Any browser	Same commands in a cloud notebook; export the checkpoint and download it
No GPU at all	—	Skip to Chapter 22: the no-code SenseCraft Robotics path

warning

GR00T training or fine-tuning requires a GPU with Ampere or newer architecture (e.g. RTX A6000, RTX 4090). V100 (Volta) is not supported.

21.2 Environment Setup

Environment

21.2 Environment Setup

Create a dedicated environment (separate from your lerobot env is fine):

git clone https://github.com/NVIDIA/Isaac-GR00T
cd Isaac-GR00T
conda create -n gr00t python=3.10
conda activate gr00t
pip install --upgrade setuptools
pip install -e .[base]
pip install --no-build-isolation flash-attn==2.7.1.post4

flash-attn is the step most likely to fail — it must match your PyTorch/CUDA build. If the build errors out: check python -c "import torch; print(torch.version.cuda)" against the wheel you need, install ninja and packaging first, and retry.

The pretrained foundation model downloads automatically from Hugging Face on first use: nvidia/GR00T-N1.5-3B.

21.3 Fine-tuning on Your SO-ARM Dataset

Fine-tune

21.3 Fine-tuning on Your SO-ARM Dataset

With the Chapter 20 dataset ready (here ./demo_data/so101-table-cleanup/ — replace with your dataset path):

python scripts/gr00t_finetune.py \
   --dataset-path ./demo_data/so101-table-cleanup/ \
   --num-gpus 1 \
   --output-dir ./so101-checkpoints  \
   --max-steps 10000 \
   --data-config so100_dualcam \
   --video-backend torchvision_av

Key parameters:

• --dataset-path: your LeRobot dataset folder (the one prepared in Chapter 20);
• --data-config so100_dualcam: the standard two-camera SO-ARM configuration;
• --max-steps: 10,000 is a solid first run for ~50 episodes;
• --video-backend torchvision_av: video decoding backend;
• To fine-tune from a locally stored base model instead of auto-downloading, add --base-model-path ./pretrained/GR00T-N1.5-3B.

note

The default fine-tuning settings require ~25GB of VRAM. If you don't have that much, add --no-tune_diffusion_model — it freezes the DiT action head and trains less, but fits smaller cards.

Monitor loss as in Chapter 15; expect fine-tuning to take noticeably longer per step than ACT — the model is ~3B parameters.

Exporting from a cloud notebook (Brev path)

The Jupyter sidebar cannot navigate into training folders directly. Compress the checkpoint folder into a .zip or .tar.gz first, then download the archive through the notebook interface.

21.4 Running Inference: Server + Robot Control Client

Inference

21.4 Running Inference: Server + Robot Control Client

⚠️

Safety check before running

Inference moves the real arm with no human in the loop. Clear the workspace, keep hands away, and be ready to cut power.

To achieve the best performance, replicate the data-collection setup as closely as possible — same cameras, same positions, same lighting. The model's generalization is limited.

Terminal 1 — launch the GR00T inference service:

python scripts/inference_service.py --server \
    --model_path ./so101-checkpoints \
    --embodiment-tag new_embodiment \
    --data-config so100_dualcam \
    --denoising-steps 4

Terminal 2 — start the robot control client (the official SO-100 example, written for this arm):

python examples/SO-100/eval_lerobot.py \
    --robot.type=so100_follower \
    --robot.port=/dev/ttyACM0 \
    --robot.id=my_awesome_follower_arm \
    --robot.cameras="{ wrist: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, front: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \
    --policy_host=0.0.0.0 \
    --lang_instruction="Grab pens and place into pen holder."

Notes:

• Replace index_or_path with your camera indexes — find them with ls /dev/video* or lerobot-find-cameras opencv. They must be the same views (front + side/gripper) as in the dataset.
• Change --lang_instruction to your dataset's exact task sentence — this is language conditioning live (Chapter 18).
• The client and server can run on the same machine, or the server on a networked GPU box / Jetson Thor.

21.5 Edge Deployment on Jetson AGX Thor

Edge

21.5 Edge Deployment on Jetson AGX Thor

Jetson AGX Thor provides enough compute to run GR00T N1.5 inference at the edge. All GR00T dependencies ship in a dedicated Docker image:

sudo docker run --rm -it \
  --network=host \
  -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,video,graphics \
  --runtime nvidia \
  --privileged \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  -v /etc/X11:/etc/X11 \
  --device /dev/nvhost-vic \
  -v /dev:/dev \
  <image_name>:<tag>

Inside the container:

git clone https://github.com/NVIDIA/Isaac-GR00T.git
cd Isaac-GR00T
pip install --upgrade setuptools
pip install -e .[thor]

Fine-tuning (optional, if you skipped it in the cloud) uses the same gr00t_finetune.py command as 21.3; then run the same server + client pair as 21.4. For the complete Jetson Thor walkthrough — flashing, environment, data collection, Brev training, deployment — see the full tutorial: Fine-tune GR00T N1.5 for LeRobot SO-ARM and deploy on Jetson Thor.

21.6 Alternative: The LeRobot groot Policy Path

Alternative

21.6 Alternative: The LeRobot groot Policy Path

If you prefer staying inside the LeRobot toolchain from Stage 3, the --policy.type=groot policy works too (it also requires flash-attn and a CUDA GPU):

pip install "lerobot[groot]"

accelerate launch \
  --num_processes=1 \
  $(which lerobot-train) \
  --output_dir=outputs/train/groot_so101 \
  --save_checkpoint=true \
  --batch_size=64 \
  --steps=20000 \
  --policy.type=groot \
  --policy.repo_id=${HF_USER}/groot_so101 \
  --dataset.repo_id=seeedstudio123/test \
  --wandb.enable=true

Expect higher environment requirements than ACT/Pi0 — the official advice holds: get ACT running end-to-end first, then try GR00T. Evaluation afterwards uses the familiar lerobot-record ... --policy.path recipe from Chapter 16.

21.7 Evaluation and Iteration

Evaluate

21.7 Evaluation and Iteration

Reuse Chapter 16's framework verbatim — success criteria, varied starts, success rate, failure buckets — with two VLA-specific additions:

1. Instruction matrix: evaluate each task sentence your dataset contains; a VLA should switch behavior when you switch the sentence;
2. Same-weights new-instruction probe: try a sentence you didn't record — where it succeeds, that's language generalization; where it fails, that's Chapter 18's distribution boundary showing up. Answer failures with data, not with hope.

21.8 Troubleshooting FAQ

FAQ

21.8 Troubleshooting FAQ

Problem	Fix
Model download failure / slow	Download nvidia/GR00T-N1.5-3B manually and pass --base-model-path
flash-attn build fails	Check PyTorch/CUDA match; install ninja + packaging first; use the pinned 2.7.1.post4
CUDA OOM during fine-tuning	Add --no-tune_diffusion_model; reduce batch size
V100 rejected	Expected — Ampere+ required
Client can't reach server	--policy_host must be the server's reachable IP; check firewall/port
Arm moves but task ignored	Camera keys or instruction sentence don't match the dataset — Chapters 19-20 rules