Skip to main content
Stage 3 · Chapter 13 · Optional · Augmentation

13. Datasets Augment

Real data is expensive and narrow. Use NVIDIA Cosmos Transfer to re-render recorded episodes in new scenes while keeping the robot's motion — then co-train on the widened dataset.

This chapter is optional advanced material

It is recommended to complete the real-world loop first (collect → train → evaluate, Chapters 9–11). The workflow applies to both the real dataset from Chapter 9 and the simulated dataset from Chapter 12.

13.1 Why Augment​

Limits of real data

13.1 Why Augment

Collecting data in real-world scenarios entails high costs and significant limitations:

  • Limited environments: data is typically collected across only a few fixed scenes;
  • Background overfitting: policy networks may learn background features irrelevant to the task (wall color, lighting, object placement), causing the policy to fail to generalize;
  • High collection costs: each additional scene requires redeployment, recalibration, and re-collection.

We leverage the state-of-the-art Cosmos Transfer model to modify video scenes, expanding dataset diversity to enhance model robustness.

13.2 How Cosmos Transfer Works​

Video-to-video

13.2 How Cosmos Transfer Works

NVIDIA Cosmos is a foundational AI platform designed to accelerate the development of Physical AI (robotics and autonomous vehicles). Cosmos Transfer is a video-to-video generative model. It takes three inputs:

InputPurposeExample
Source video (vision)Provides temporal dynamic informationSO-101 external-camera manipulation video
Control signals (controls)Constrains spatial structure, preserves robot morphologyCanny edge map + SAM2 segmentation map
Text promptDescribes the target scene's appearance"A modern laboratory with white lab benches..."

The control signals (edges + segmentation) lock the robot's shape and motion in place; the text prompt drives what the new background looks like.

Control TypePurposePre-computation Required?
EdgePreserves object outlines, prevents subject deformationOptional (framework can auto-compute Canny)
SegmentationPreserves semantic regions (robot/object/background)Required (depends on SAM2)
BlurPreserves global layout, allows texture variationOptional
DepthPreserves spatial depth relationshipsRequired (depends on DepthAnything)
Action labels carry over

Transfer keeps the robot's motion trajectory, so the original actions (joint angles, gripper state) work as-is for the augmented videos. High edge weight (0.9) is what makes this reuse safe.

13.3 Environment Setup​

GPU and install

13.3 Environment Setup

Hardware requirements:

ModelMin GPU VRAMRecommended GPU
Cosmos-Nano~24 GBSingle RTX 4090 / A5000 / Jetson AGX Thor
Cosmos-Super~80 GB × 44× A100 / H100

Installation:

# Clone the Cosmos repository
git clone https://github.com/NVIDIA/Cosmos.git
cd Cosmos

# Navigate to the framework directory
cd packages/cosmos3

# Create a virtual environment
uv venv --python 3.13
source .venv/bin/activate

# Install dependencies
uv pip install -e ".[inference]"

# Log in to Hugging Face and download the model weights
huggingface-cli login
hf download nvidia/Cosmos-Nano --local-dir /path/to/models/Cosmos-Nano

13.4 Prepare Your Data​

Controls and prompts

13.4 Prepare Your Data

Directory structure:

your_project/
├── controls/
│ └── episode_000/
│ ├── vision.mp4 # Source video (robot recording)
│ ├── control_edge.mp4 # Canny edge map
│ └── control_seg.mp4 # SAM2 segmentation map
├── prompts/
│ ├── lab_prompt.json # Target-scene prompt
│ └── workshop_prompt.json
├── specs/
│ └── my_lab.json # Inference config
├── negative_prompt.json
└── outputs/

Extract a clip from your SO-101 dataset (the overhead external stream gives the best structure for augmentation):

ffmpeg -i ~/rebot-arm-dli-isaacsim/datasets/so101_vials_real/videos/chunk-000/observation.images.external/episode_000000.mp4 \
-c copy your_project/controls/episode_000/vision.mp4

Generate the edge control signal:

python -c "
import cv2
cap = cv2.VideoCapture('your_project/controls/episode_000/vision.mp4')
fps = cap.get(cv2.CAP_PROP_FPS)
w, h = int(cap.get(3)), int(cap.get(4))
out = cv2.VideoWriter('your_project/controls/episode_000/control_edge.mp4',
cv2.VideoWriter_fourcc(*'avc1'), fps, (w, h), True)
while True:
ret, frame = cap.read()
if not ret: break
edges = cv2.Canny(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY), 50, 150)
out.write(cv2.cvtColor(edges, cv2.COLOR_GRAY2BGR))
cap.release(); out.release()
print('edge control done')
"

Generate the segmentation signal with SAM2 (a separate install):

pip install sam2
python your_sam2_script.py \
--input your_project/controls/episode_000/vision.mp4 \
--output your_project/controls/episode_000/control_seg.mp4

Write the scene prompt, for example prompts/lab_prompt.json:

{
"subjects": [{
"description": "A white SO-ARM101 robotic arm performing a tabletop pick-and-place task",
"appearance_details": "White 3D-printed links, small two-finger gripper, wrist-mounted camera",
"action": "pick up a vial and place it into a yellow rack",
"pose": "Arm reaches toward the vial, gripper closes, the vial lifts and moves to the rack",
"state_changes": "Gripper transitions from open to closed; the held vial is lifted and placed"
}],
"background_setting": "A modern laboratory with white lab benches, soft diffuse lighting, and a plain grey mat",
"lighting": {
"description": "Even, diffuse overhead lighting with no hard shadows"
}
}

13.5 Run Inference​

Generate

13.5 Run Inference

cd /path/to/cosmos/packages/cosmos3

export TRANSFER_ROOT="/path/to/your_project"

CUDA_VISIBLE_DEVICES=0 \
.venv/bin/python -m cosmos_framework.scripts.inference \
--parallelism-preset=latency \
-i "$TRANSFER_ROOT/specs/my_lab.json" \
-o "$TRANSFER_ROOT/outputs/" \
--checkpoint-path Cosmos-Nano \
--seed 2026

Guidance parameters (the weight ratio is what matters; the sum should be ≈ 1.0):

ParameterRecommendedEffect
guidance3.0Text CFG strength — how strongly the new scene follows the prompt
control_guidance1.5Control CFG strength — how strictly motion is preserved
edge.weight0.75–0.9Preserves subject structure
seg.weight0.1–0.25Auxiliary semantic consistency
If the robot deforms

Increase edge.weight (0.9 → 0.95), decrease seg.weight, and raise control_guidance. If the background barely changes, raise guidance and describe the target scene in more detail. On OOM: reduce resolution to "480", reduce num_video_frames_per_chunk to 33.

Batch generation: wrap the command in a shell loop over your episode directories to multiply the dataset across several target scenes.

13.6 Integration into the Training Pipeline​

Back to GR00T

13.6 Integration into the Training Pipeline

The augmented videos reuse the original episode's action and state labels — no re-labeling is needed. To fold them into training:

  1. Build augmented episodes as LeRobot datasets (same six joint columns, same front/external camera keys, same task sentence);
  2. Merge them with the real dataset like the sim data in Chapter 12 — lerobot-edit-dataset --operation.type=merge accepts every source with matching features;
  3. Convert the merged dataset to GR00T v2 and fine-tune exactly as in Chapter 10, adjusting EPISODE_SAMPLING_RATE / DS_WEIGHTS_ALPHA when sources are unequal.

Quality check before training on augmented data: scrub through frames and confirm the robot looks right and backgrounds are believable, then run a short policy-transfer test — train briefly on augmented data and measure the real-world success rate against a policy trained on real data only.

13.7 References​

Links

13.7 References

Continue to Stage 4: Chapter 14 — Setup Your Jetson Edge Device.

Loading Comments...