13. Datasets Augment
Real data is expensive and narrow. Use NVIDIA Cosmos Transfer to re-render recorded episodes in new scenes while keeping the robot's motion — then co-train on the widened dataset.
It is recommended to complete the real-world loop first (collect → train → evaluate, Chapters 9–11). The workflow applies to both the real dataset from Chapter 9 and the simulated dataset from Chapter 12.
13.1 Why Augment
13.1 Why Augment
Collecting data in real-world scenarios entails high costs and significant limitations:
- Limited environments: data is typically collected across only a few fixed scenes;
- Background overfitting: policy networks may learn background features irrelevant to the task (wall color, lighting, object placement), causing the policy to fail to generalize;
- High collection costs: each additional scene requires redeployment, recalibration, and re-collection.
We leverage the state-of-the-art Cosmos Transfer model to modify video scenes, expanding dataset diversity to enhance model robustness.
13.2 How Cosmos Transfer Works
13.2 How Cosmos Transfer Works
NVIDIA Cosmos is a foundational AI platform designed to accelerate the development of Physical AI (robotics and autonomous vehicles). Cosmos Transfer is a video-to-video generative model. It takes three inputs:
| Input | Purpose | Example |
|---|---|---|
| Source video (vision) | Provides temporal dynamic information | SO-101 external-camera manipulation video |
| Control signals (controls) | Constrains spatial structure, preserves robot morphology | Canny edge map + SAM2 segmentation map |
| Text prompt | Describes the target scene's appearance | "A modern laboratory with white lab benches..." |
The control signals (edges + segmentation) lock the robot's shape and motion in place; the text prompt drives what the new background looks like.
| Control Type | Purpose | Pre-computation Required? |
|---|---|---|
| Edge | Preserves object outlines, prevents subject deformation | Optional (framework can auto-compute Canny) |
| Segmentation | Preserves semantic regions (robot/object/background) | Required (depends on SAM2) |
| Blur | Preserves global layout, allows texture variation | Optional |
| Depth | Preserves spatial depth relationships | Required (depends on DepthAnything) |
Transfer keeps the robot's motion trajectory, so the original actions (joint angles, gripper state) work as-is for the augmented videos. High edge weight (0.9) is what makes this reuse safe.
13.3 Environment Setup
13.3 Environment Setup
Hardware requirements:
| Model | Min GPU VRAM | Recommended GPU |
|---|---|---|
| Cosmos-Nano | ~24 GB | Single RTX 4090 / A5000 / Jetson AGX Thor |
| Cosmos-Super | ~80 GB × 4 | 4× A100 / H100 |
Installation:
# Clone the Cosmos repository
git clone https://github.com/NVIDIA/Cosmos.git
cd Cosmos
# Navigate to the framework directory
cd packages/cosmos3
# Create a virtual environment
uv venv --python 3.13
source .venv/bin/activate
# Install dependencies
uv pip install -e ".[inference]"
# Log in to Hugging Face and download the model weights
huggingface-cli login
hf download nvidia/Cosmos-Nano --local-dir /path/to/models/Cosmos-Nano
13.4 Prepare Your Data
13.4 Prepare Your Data
Directory structure:
your_project/
├── controls/
│ └── episode_000/
│ ├── vision.mp4 # Source video (robot recording)
│ ├── control_edge.mp4 # Canny edge map
│ └── control_seg.mp4 # SAM2 segmentation map
├── prompts/
│ ├── lab_prompt.json # Target-scene prompt
│ └── workshop_prompt.json
├── specs/
│ └── my_lab.json # Inference config
├── negative_prompt.json
└── outputs/
Extract a clip from your SO-101 dataset (the overhead external stream
gives the best structure for augmentation):
ffmpeg -i ~/rebot-arm-dli-isaacsim/datasets/so101_vials_real/videos/chunk-000/observation.images.external/episode_000000.mp4 \
-c copy your_project/controls/episode_000/vision.mp4
Generate the edge control signal:
python -c "
import cv2
cap = cv2.VideoCapture('your_project/controls/episode_000/vision.mp4')
fps = cap.get(cv2.CAP_PROP_FPS)
w, h = int(cap.get(3)), int(cap.get(4))
out = cv2.VideoWriter('your_project/controls/episode_000/control_edge.mp4',
cv2.VideoWriter_fourcc(*'avc1'), fps, (w, h), True)
while True:
ret, frame = cap.read()
if not ret: break
edges = cv2.Canny(cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY), 50, 150)
out.write(cv2.cvtColor(edges, cv2.COLOR_GRAY2BGR))
cap.release(); out.release()
print('edge control done')
"
Generate the segmentation signal with SAM2 (a separate install):
pip install sam2
python your_sam2_script.py \
--input your_project/controls/episode_000/vision.mp4 \
--output your_project/controls/episode_000/control_seg.mp4
Write the scene prompt, for example prompts/lab_prompt.json:
{
"subjects": [{
"description": "A white SO-ARM101 robotic arm performing a tabletop pick-and-place task",
"appearance_details": "White 3D-printed links, small two-finger gripper, wrist-mounted camera",
"action": "pick up a vial and place it into a yellow rack",
"pose": "Arm reaches toward the vial, gripper closes, the vial lifts and moves to the rack",
"state_changes": "Gripper transitions from open to closed; the held vial is lifted and placed"
}],
"background_setting": "A modern laboratory with white lab benches, soft diffuse lighting, and a plain grey mat",
"lighting": {
"description": "Even, diffuse overhead lighting with no hard shadows"
}
}
13.5 Run Inference
13.5 Run Inference
cd /path/to/cosmos/packages/cosmos3
export TRANSFER_ROOT="/path/to/your_project"
CUDA_VISIBLE_DEVICES=0 \
.venv/bin/python -m cosmos_framework.scripts.inference \
--parallelism-preset=latency \
-i "$TRANSFER_ROOT/specs/my_lab.json" \
-o "$TRANSFER_ROOT/outputs/" \
--checkpoint-path Cosmos-Nano \
--seed 2026
Guidance parameters (the weight ratio is what matters; the sum should be ≈ 1.0):
| Parameter | Recommended | Effect |
|---|---|---|
guidance | 3.0 | Text CFG strength — how strongly the new scene follows the prompt |
control_guidance | 1.5 | Control CFG strength — how strictly motion is preserved |
edge.weight | 0.75–0.9 | Preserves subject structure |
seg.weight | 0.1–0.25 | Auxiliary semantic consistency |
Increase edge.weight (0.9 → 0.95), decrease seg.weight, and raise
control_guidance. If the background barely changes, raise guidance and
describe the target scene in more detail. On OOM: reduce resolution to
"480", reduce num_video_frames_per_chunk to 33.
Batch generation: wrap the command in a shell loop over your episode directories to multiply the dataset across several target scenes.
13.6 Integration into the Training Pipeline
13.6 Integration into the Training Pipeline
The augmented videos reuse the original episode's action and state labels — no re-labeling is needed. To fold them into training:
- Build augmented episodes as LeRobot datasets (same six joint columns,
same
front/externalcamera keys, same task sentence); - Merge them with the real dataset like the sim data in
Chapter 12 —
lerobot-edit-dataset --operation.type=mergeaccepts every source with matching features; - Convert the merged dataset to GR00T v2 and fine-tune exactly as in
Chapter 10, adjusting
EPISODE_SAMPLING_RATE/DS_WEIGHTS_ALPHAwhen sources are unequal.
Quality check before training on augmented data: scrub through frames and confirm the robot looks right and backgrounds are believable, then run a short policy-transfer test — train briefly on augmented data and measure the real-world success rate against a policy trained on real data only.
13.7 References
13.7 References
- Cosmos Documentation
- Cosmos Transfer Cookbook
- NVIDIA SO-101 Workshop — Strategy 3: Augmenting Datasets With Cosmos
Continue to Stage 4: Chapter 14 — Setup Your Jetson Edge Device.