15. ACT Model and Action Chunking
Chapter 15 of the Seeed Embodied Intelligence Beginner's Course — ACT input and output, the ResNet and Transformer structure, attention intuition, CVAE, action chunk vs action horizon, error accumulation defenses, and ACT's capability boundaries.
15.1 From Behavioral Cloning to ACT
15.1 From Behavioral Cloning to ACT: Just Need a "Good Model"
Chapter 9 covered behavioral cloning: record teleoperation demos as a dataset, let the model learn "do what you see." That left a question — what model deserves this data? A deployable policy model must cross at least three hurdles:
- Understand images.
- Think coherently.
- Resist accumulated error.
ACT (Action Chunking with Transformers) is the policy model built to cross these three hurdles. Proposed by a Stanford team in 2023, it first gained fame on the low-cost ALOHA dual-arm platform performing delicate tasks like opening cup lids and sealing bags; later, Mobile ALOHA demonstrated complex tasks like cooking shrimp. Today it's built into LeRobot as one of the standard policies — and it's the model you'll train and deploy hands-on in this stage.
ACT's name IS its entire design philosophy: Action Chunking + Transformer (sequence modeler). The rest of this chapter unpacks these two terms.
15.2 ACT Input and Output
15.2 ACT Input and Output: See Both Ends First
The most reliable way to understand any model is to look at its "interface" first — what it takes in, what it puts out.
Input: Current Observation
During inference, ACT receives one observation frame containing two types of information:
| Input | On the reBot Arm | Dimensionality |
|---|---|---|
| Images (Observation) | Overhead + wrist dual RGB feeds | 2 × images |
| Joint state (State) | 6 joint angles + gripper opening | 7-dim vector |
Note that ACT is frame-only: it doesn't remember past frames; each decision is based on "what it sees now + where the joints are now."
Output: Action Chunk for a Short Future Window
ACT's output is not the next action, but a whole action sequence block (Action Chunk):
Input: 2 image streams + 7-dim joint state (current frame)
Output: action sequence for next k steps, each step 7-dim (6 joints + gripper)
i.e., a k × 7 action matrix
In LeRobot's ACT default config, k (chunk size) is typically 100 — one inference gives a complete action plan for about 100 future timesteps. This answers the foreshadowing from Chapter 9: Action Chunk isn't an abstract optimization trick; it's ACT's natural output form.
15.3 ACT Internal Structure
15.3 ACT Internal Structure: Three Workshops on One Assembly Line
Workshop 1: Visual Backbone (ResNet) — Pixels to Features
Overhead and wrist camera images each go through a ResNet18 (a CNN pre-trained on ImageNet), compressed into a set of visual features. Think of ResNet as the model's "visual cortex": raw pixels are meaningless to it; it extracts structured information like "there's a red object on the left of the table" or "there's an opening directly below the gripper."
Workshop 2: Transformer Encoder — Understanding "the Present"
Visual features + joint state vector merge and enter the Transformer encoder. The encoder's job is to fuse multiple streams into a unified understanding of the current situation: "where's the target, where am I, how far along is the task."
Workshop 3: Transformer Decoder — Planning "the Future"
The decoder takes the encoder's understanding and generates the action sequence for the next k steps in one pass. It doesn't spit out actions one by one; like writing sheet music, it composes the entire "future movement" in one breath — this is the fundamental reason action chunks are internally so coherent.
15.4 What Is a Transformer
15.4 What Is a Transformer: Intuition Behind Attention
ACT's workshops 2 and 3 are both Transformers, but what IS a Transformer?
It was proposed by Google in 2017 for machine translation; the paper is titled "Attention Is All You Need," and it later became the foundational architecture of large language models. No formulas needed — three intuitions suffice:
- Token: cut information into "parts." Transformers don't process raw sentences or raw pixels directly; they first cut input into standardized parts (tokens) — a sentence into words, an image into patches, a joint state vector can also be one token. Once all information is unified into "a sequence of parts," the same mechanism processes them all.
- Self-Attention: every part can "see" all other parts. This is the core of Transformer. When processing each part, it calculates its relevance to every other part and focuses on absorbing information from the most relevant ones — "what to look at" isn't prescribed by humans; the model learns it.
- Encoder and Decoder: one understands, one generates. The Encoder fuses an input sequence of parts into "an understanding of the current situation"; the Decoder takes this understanding and generates a new sequence of output parts — in translation, it's the target-language sentence; in ACT, it's the future action sequence.
15.5 How Transformer Is Used in ACT
15.5 How Transformer Is Used in ACT
- In the encoder: fusing multiple observations. Visual feature patches from both images (via ResNet) plus the joint state vector all become tokens fed into the encoder. Self-attention aligns them — "the gripper's current position" and "that red block in the image" are linked into a unified understanding: where's the target, where am I, how far along is the task.
- In the decoder: planning the whole action sequence at once. The decoder uses k query vectors corresponding to k future action steps; these queries draw information from the encoder's understanding while also coordinating with each other via self-attention — the action at step 37 "knows" what step 36 intends to do. The whole action block is therefore a coherent whole, not 100 isolated decisions.
- In attention focusing: knowing what to "look at." When generating each action, the model automatically focuses on image regions most relevant to the current action — approaching the target, it focuses on gripper-block relative position; while moving, it focuses on the target direction — rather than treating all regions equally.
One-sentence summary: ResNet "sees clearly," the Transformer encoder "understands," the Transformer decoder "plans coherently" — all powered by attention.
15.6 What Is CVAE
15.6 What Is CVAE
CVAE (Conditional Variational Autoencoder). LeRobot's ACT actually trains with this, you just don't see it in the lerobot-train --policy.type=act command line.
What Pit It Prevents: The Average Misses
CVAE's job isn't to "reduce the human to a single correct answer" but to acknowledge: given the current observation, actions can have multiple styles; during training, it first identifies which style this is, then reproduces that action sequence.
The paper did comparisons (on simulated tasks):
- If demonstrations are scripted (only one way), removing CVAE barely affects success rate.
- With human data, removing it drops success from ~35% to 2%.
So CVAE isn't to make formulas look fancier; it's to let the model handle data where "humans switch approaches and have shaky hands."
"Conditional" means: generated actions must be grounded in what's currently seen — whether it's a crayfish or a block on the table, it can't make things up. CVAE is "given what you see, compose how to move next."
15.7 Action Chunk vs. Action Horizon
15.7 Action Chunk vs. Action Horizon
These are the two concepts most precisely distinguished in this chapter; they are two independently tunable parameters:
- Action Chunk: the length of the action sequence the model predicts in one forward pass, i.e., the number of rows k in the output matrix. In LeRobot, ACT defaults to chunk size 100.
- Action Horizon: after predicting these 100 steps, how many are actually executed open-loop, before re-observing and re-predicting.
The relationship is: The predicted chunk can be long, but each time only the first small segment is trusted.
Why not execute all 100 steps? Because predictions get less reliable further out — the environment changes, objects may be bumped, and by the second half the model's "assumed situation" has already drifted from reality. Executing open-loop for too long = driving with eyes closed. The smaller the Horizon, the more often the model "opens its eyes to re-check," making it more robust to disturbance; but too small loses the smoothness from chunking.
It's like using phone navigation while driving. Navigation (the model) computes the whole route (chunk) in one pass, but you don't lock the steering wheel — every few miles you glance at real-time traffic (re-observe) and navigation re-plans accordingly (re-predict). The distance you "trust the old route and keep driving" is the Horizon.
15.8 Action Continuity and Error Accumulation
15.8 Action Continuity and Error Accumulation: ACT's Two Lines of Defense
Chapter 9 left two threats to real-robot stability: action jitter and error accumulation. Now let's see how ACT addresses them with structured methods.
- Defense 1: intra-chunk coherence fixes jitter.
- Defense 2: temporal ensembling fixes discontinuities.
15.9 What Tasks Is ACT Suited For?
15.9 What Tasks Is ACT Suited For?
Given ACT's design characteristics, its "comfort zone" is quite clear:
| Suited For | Reason |
|---|---|
| Tabletop manipulation (grasp, place, organize, plug/unplug) | ACT originated on these tasks; data needs and model size match |
| Single or few tasks | Behavioral cloning learns task-specific mappings; more tasks = more data needed |
| Short tasks (seconds to ~1 minute) | Error accumulates over time; shorter tasks are more stable |
| Tasks with sufficient visual info | Scenes where overhead + wrist dual cameras cover key info |
| Resource-limited hardware | ACT has relatively few parameters; consumer GPUs can train and infer; CPU can also infer |
15.10 ACT's Capability Boundaries
15.10 ACT's Capability Boundaries
Equally important is knowing what it can't do:
