Skip to main content
Stage 3 · Chapter 15 · Practice

15. Training Your First ACT Policy

Chapter 15 of the SO-ARM Physical AI Beginner's Course — understand batch size, learning rate, and training steps, then train your first ACT policy on the SO-ARM dataset with lerobot-train, monitor loss and GPU, and manage checkpoints.

Three configsCheckpointsStart training

15.1 Three Key Configs: Batch Size, Learning Rate, Steps​

Configs

15.1 Three Key Configs: Batch Size, Learning Rate, Steps

Batch Size​

How many frames the model sees per update. Bigger batches train more stably but need more VRAM. If you hit CUDA out-of-memory, lower the batch size first. Start with the ACT default and only touch it when you must.

Learning Rate​

How far each update moves the parameters. Too high — loss oscillates or explodes; too low — training crawls. ACT's default (1e-5 range) is well tested for tabletop tasks; don't change it for your first run.

Training Steps​

How many updates in total. ACT's default is 100k-300k steps for SO-ARM-scale tasks; more steps help harder tasks but risk overfitting. Judge by the loss curve: when validation loss stops improving, stop.

tip

First-run strategy: keep all defaults, train 100k steps, evaluate on the real robot (Chapter 16), then decide what to tune. Beginners lose more time tuning blind than training twice.

15.2 Saving and Managing Checkpoints​

Checkpoints

15.2 Saving and Managing Checkpoints

No manual saving needed: checkpoints are saved automatically every save_freq steps plus a final last one. It's fine to set a large step count — you can pick the model from lower steps and discard under-trained or over-fitted ones.

outputs/train/act_so101_test/
├── train_config.json              ← Full config for this run (needed to resume)
└── checkpoints/
    ├── 0020000/pretrained_model/  ← Model archive at each step count
    ├── 0040000/pretrained_model/
    ├── ...
    └── last/pretrained_model/     ← Last checkpoint, used in Chapter 16
  • Disk usage: each checkpoint is a complete model weight file; dozens of them add up. Once training stabilizes, early/mid checkpoints can be deleted, keeping only last.
  • During inference, --policy.path points to checkpoints/last/pretrained_model.

15.3 Starting Training​

Training

15.3 Starting Training

All checks passed — start training (in the conda lerobot environment). For a local dataset (recorded in Chapter 12), make sure the repo_id matches the one used during data collection and add --policy.push_to_hub=false:

lerobot-train \
  --dataset.repo_id=seeedstudio123/test \
  --policy.type=act \
  --output_dir=outputs/train/act_so101_test \
  --job_name=act_so101_test \
  --policy.device=cuda \
  --wandb.enable=false \
  --policy.push_to_hub=false \
  --steps=300000

For a dataset on the Hugging Face Hub, use your username:

lerobot-train \
  --dataset.repo_id=${HF_USER}/so101_test \
  --policy.type=act \
  --output_dir=outputs/train/act_so101_test \
  --job_name=act_so101_test \
  --policy.device=cuda \
  --wandb.enable=false \
  --steps=300000

What each argument does:

  • Dataset specification: --dataset.repo_id points at your Chapter 12 dataset.
  • Training steps: --steps=300000. The algorithm defaults higher; adjust based on task difficulty and the loss curve.
  • Policy type: --policy.type=act loads the ACT configuration. The policy automatically adapts to your robot's motor states, motor actions, and the number of cameras — this information is already stored in your dataset. Other options include diffusion, pi0, pi0fast, smolvla, groot.
  • Device selection: --policy.device=cuda for NVIDIA GPUs; mps works on Apple Silicon.
  • Visualization tool: re-enable --wandb.enable=true to chart training with Weights and Biases (run wandb login first).
tip

RTX 50-series GPUs must append --dataset.video_backend=pyav — it bypasses missing APIs in the preview torchvision. Full command:

lerobot-train \
  --dataset.repo_id=seeedstudio123/test \
  --dataset.video_backend=pyav \
  --policy.type=act \
  --output_dir=outputs/train/act_so101_test \
  --policy.device=cuda \
  --wandb.enable=false \
  --policy.push_to_hub=false \
  --steps=300000

15.4 Monitoring Loss and GPU Status​

Monitor

15.4 Monitoring Loss and GPU Status

  • Loss: printed to the terminal every log_freq steps. Expect a fast drop early, then a long shallow decline — that's healthy. Loss rising or oscillating wildly usually means the learning rate is wrong for your setup.
  • GPU: watch VRAM and utilization with nvidia-smi (or watch -n 1 nvidia-smi). Utilization near 0% for long stretches means the data pipeline (disk/CPU) is the bottleneck, not the GPU.
  • Time expectations: as a rough reference, training ACT on 50 episodes takes about 6 hours on a laptop RTX 3060 8GB, and about 2-3 hours on an RTX 4090 or A100. Actual time depends on dataset size, image resolution, batch size, and hardware.

15.5 Resuming Interrupted Training​

Resume

15.5 Resuming Interrupted Training

To resume from the last checkpoint, feed back the saved config:

lerobot-train \
  --config_path=outputs/train/act_so101_test/checkpoints/last/pretrained_model/train_config.json \
  --resume=true

Uploading Policy Checkpoints (Optional)​

Once training is done, upload the latest checkpoint:

huggingface-cli upload ${HF_USER}/act_so101_test \
  outputs/train/act_so101_test/checkpoints/last/pretrained_model

15.6 Common Training Errors​

Troubleshooting

15.6 Common Training Errors

ErrorFix
TypeError: stack(): argument 'tensors' must be tuple of Tensors, not Columnpip install datasets==2.19
CUDA out of memoryLower batch size; close other GPU processes; use --dataset.video_backend=pyav
No valid stream found in input file during data loadingconda install ffmpeg=7.1.1 -c conda-forge (Chapter 5's fix)
Dataset not foundFor local datasets, re-check the exact repo_id from recording; the folder lives under ~/.cache/huggingface/lerobot/

With a trained checkpoints/last/pretrained_model in hand, the next chapter puts the policy on the real arm.

Loading Comments...