Vision-language-action models, from the inside out

What a VLA actually is

A vision-language-action model is a conditional policy

$$\pi_\theta : (\,o_{1:t},\; \ell\,) \;\longmapsto\; \Delta(\mathcal{A}),$$

where $o_{1:t}$ is a history of camera frames (often plus proprioception) and $\ell$ is a natural-language instruction. The architectural claim is narrow and worth stating plainly: the visual and semantic backbone is a pretrained VLM, and only the output head is new. This is the move RT-2 made explicit [2] and that OpenVLA turned into a reproducible recipe [3] — everything interesting in a VLA is about how you attach $\mathcal{A}$ to a transformer that was trained to emit text.

That framing makes the design space small enough to reason about. Three decisions dominate real-world performance.

Decision 1: how to represent an action

Robot actions are continuous — typically an end-effector delta in $SE(3)$ plus a gripper command:

$$a_t = \bigl(\Delta x, \Delta y, \Delta z,\; \Delta r_x, \Delta r_y, \Delta r_z,\; g \bigr) \in \mathbb{R}^{7}.$$

There are three families of head, and they trade off differently.

Discretized tokens. Bin each dimension into $B$ bins and reuse the language head — the approach taken by RT-1 [1], RT-2 [2] and OpenVLA [3], all of which use $B = 256$. With per-dimension quantile binning, dimension $i$ maps to

$$\mathrm{tok}(a_t^{(i)}) = \Bigl\lfloor B \cdot F_i\bigl(a_t^{(i)}\bigr) \Bigr\rfloor,$$

where $F_i$ is the empirical CDF of that dimension over the training corpus. Quantile bins matter more than people expect: with uniform bins over a teleoperation dataset, the vast majority of samples land in the two bins nearest zero, and the model spends its capacity learning to predict "hold still". OpenVLA's use of 1st–99th-percentile clipping is the same observation applied at the tails [3].

The cost of discretization is a hard resolution floor. Quantization error is $O(1/B)$ per dimension, and $B$ is bounded by how many tokens you can afford to decode — see Decision 3. FAST attacks this directly by applying a discrete cosine transform to the action chunk and BPE-compressing the coefficients, buying high-frequency fidelity at far fewer tokens [10].

Regression. Predict $a_t$ directly under an $L_1$ or Huber loss. Cheap and low-latency, but the naive per-step version collapses multimodality: when two grasps are both valid, the conditional mean is the average of them, which is usually a collision — the observation that motivated Diffusion Policy [7]. The honest counterpoint is OpenVLA-OFT [11], which shows that $L_1$ regression becomes competitive once you add parallel decoding and predict a whole chunk at once, since the chunk itself carries much of the structure a per-step mean destroys.

Flow / diffusion heads. Train a small conditional velocity field $v_\phi$ on top of the frozen VLM embedding $z_t$, and integrate from noise:

$$\frac{\mathrm{d}a^{(\tau)}}{\mathrm{d}\tau} = v_\phi\bigl(a^{(\tau)},\, \tau,\, z_t\bigr), \qquad \tau: 0 \to 1 .$$

This is flow matching [4] in the action space, and it keeps continuous precision and multimodality at the cost of $K$ integration steps per inference. In practice $K \in [4, 10]$ is enough when the head is small, which is why flow-matching and diffusion action experts — $\pi_0$ [5], $\pi_{0.5}$ [6], RDT-1B [8], SmolVLA [9] — have largely displaced discretization in new systems.

Decision 2: chunking, and why single-step VLAs jitter

Predicting one action per forward pass produces visibly shaky trajectories. The reason is not model error but temporal independence: consecutive predictions are drawn from unlinked conditionals, so high-frequency disagreement between them shows up directly in the joint commands.

Action chunking, introduced with ACT [12], predicts a block of $H$ future actions from one observation:

$$\pi_\theta\bigl(a_{t:t+H} \mid o_{1:t}, \ell\bigr).$$

The chunk is internally consistent by construction. To avoid a discontinuity at the seam, chunks are overlapped and blended with an exponentially-weighted temporal ensemble over all predictions that cover time $t$ [12]:

$$\hat{a}_t = \frac{\sum_{k=0}^{H-1} w_k\, a_t^{(t-k)}}{\sum_{k=0}^{H-1} w_k}, \qquad w_k = e^{-\alpha k}.$$

Here $a_t^{(t-k)}$ is the prediction of $a_t$ made from the observation at time $t-k$. Small $\alpha$ smooths aggressively and adds lag; large $\alpha$ tracks new observations and reintroduces jitter.

There is a real cost. Within a chunk the policy is open-loop for $H/f$ seconds, so reactive behaviors — a slipping object, a human hand entering the workspace — cannot be handled faster than the chunk boundary. Chunk length is a closed-loop-bandwidth decision, not a modeling detail. Real-time chunking [13] is the current best answer: rather than shortening $H$, it starts generating the next chunk while the current one is still executing and inpaints the overlap, so the seam disappears without giving up horizon.

The honest statement of the tradeoff: chunking buys trajectory smoothness by spending reaction time. If your task needs both, you need a fast reactive layer underneath the VLA, not a shorter chunk.

Decision 3: the latency budget

This is where most VLA deployments actually fail. A $7\mathrm{B}$-parameter backbone at $\sim\!20$ tokens of visual prefix plus a decoded action chunk costs tens to low hundreds of milliseconds per call on a single accelerator. The control loop needs commands at $30$–$100\,\mathrm{Hz}$. Those numbers do not meet.

The reconciliation is a two-rate architecture:

$$\underbrace{\pi_{\mathrm{VLA}}}_{\approx 3\text{--}10\,\mathrm{Hz}} \;\longrightarrow\; \underbrace{\text{chunk } a_{t:t+H}}_{\text{interpolated}} \;\longrightarrow\; \underbrace{\text{impedance / joint controller}}_{500\text{--}1000\,\mathrm{Hz}} .$$

The VLA sets intent at a few hertz; a classical controller handles contact dynamics at kilohertz. This split is now explicit in the architectures themselves: GR00T N1 names its VLM the slow "System 2" and its diffusion transformer the fast "System 1" [15], and Gemini Robotics splits into a cloud-hosted backbone and an on-robot action decoder for the same reason [14]. Anyone reporting a VLA "running at 50 Hz" is almost always describing the controller rate, not the model rate — and the distinction is exactly what determines whether the system survives contact.

Two further constraints fall out of the budget:

• Camera history is expensive. Each additional frame adds a full visual prefix. Most systems keep $|o_{1:t}| \le 2$ and push temporal information into the chunk instead.
• Quantization is a control decision. Moving the backbone to 8-bit roughly doubles the model rate, which halves the open-loop window. That is often worth more than the accuracy it costs. SmolVLA makes the same argument at the scale axis — a $450\mathrm{M}$ backbone with asynchronous inference, rather than a $7\mathrm{B}$ one waiting on itself [9].

None of this works without cross-embodiment data at scale; Open X-Embodiment [16] and the Octo recipe trained on it [17] are what make a single backbone transferable across arms in the first place.

Where VLAs are weak

Three limitations are structural rather than incidental, and each is a live research direction.

No usable notion of consequence. The model maps observations to actions that resemble the demonstrations. It has no forward model, so it cannot evaluate whether an action leads somewhere recoverable. It will confidently execute the first step of a plan that has no second step.

Language grounds objects, not dynamics. Instructions reliably select what to act on. They do not reliably convey force, compliance, or timing — "gently" has no consistent referent in an action distribution learned from position targets. $\pi_{0.5}$'s hierarchical decomposition, where a high-level subtask is predicted in language before the low-level chunk [6], helps with what next but not with how hard.

Failure is silent. A conditional action distribution has no calibrated signal for "this state is outside my training support". Entropy of the chunk is a weak proxy at best; it is high on genuinely multimodal states and low on confidently wrong ones.

The first of those is the one that motivates learning an explicit model of the world, and it is the subject of the next issue.

References

1. Brohan et al. RT-1: Robotics Transformer for Real-World Control at Scale. 2022. arXiv:2212.06817
2. Brohan et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. 2023. arXiv:2307.15818
3. Kim et al. OpenVLA: An Open-Source Vision-Language-Action Model. 2024. arXiv:2406.09246
4. Lipman et al. Flow Matching for Generative Modeling. 2022. arXiv:2210.02747
5. Black et al. $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control. 2024. arXiv:2410.24164
6. Physical Intelligence et al. $\pi_{0.5}$: A Vision-Language-Action Model with Open-World Generalization. 2025. arXiv:2504.16054
7. Chi et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. 2023. arXiv:2303.04137
8. Liu et al. RDT-1B: A Diffusion Foundation Model for Bimanual Manipulation. 2024. arXiv:2410.07864
9. Shukor et al. SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics. 2025. arXiv:2506.01844
10. Pertsch et al. FAST: Efficient Action Tokenization for Vision-Language-Action Models. 2025. arXiv:2501.09747
11. Kim et al. Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (OpenVLA-OFT). 2025. arXiv:2502.19645
12. Zhao et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (ACT). 2023. arXiv:2304.13705
13. Black et al. Real-Time Execution of Action Chunking Flow Policies. 2025. arXiv:2506.07339
14. Gemini Robotics Team. Gemini Robotics: Bringing AI into the Physical World. 2025. arXiv:2503.20020
15. NVIDIA et al. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. 2025. arXiv:2503.14734
16. Open X-Embodiment Collaboration. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. 2023. arXiv:2310.08864
17. Octo Model Team. Octo: An Open-Source Generalist Robot Policy. 2024. arXiv:2405.12213