Oct 2025 – Jun 2026 · First author

LangGap: VLA Language Understanding Benchmark

Accepted at IROS 2026. Designed a systematic semantic perturbation evaluation framework revealing that state-of-the-art VLA models ignore language instructions despite high benchmark scores. Proposed multi-task same-scene training approach and constructed augmented dataset for fine-tuning.

IROS 2026 arXiv: 2603.00592 Poster (IROS 2026)

Demo

Overview

Vision-Language-Action (VLA) models achieve over 95% success on standard benchmarks. However, through systematic experiments, we find that current state-of-the-art VLA models largely ignore language instructions. Prior work lacks: (1) systematic semantic perturbation diagnostics, (2) a benchmark that forces language understanding by design, and (3) linguistically diverse training data. This paper constructs the LangGap benchmark, based on a four-dimensional semantic perturbation method – varying instruction semantics while keeping the tabletop layout fixed – revealing language understanding deficits in π0.5. Existing benchmarks like LIBERO assign only one task per layout, underutilizing available objects and target locations; LangGap fully diversifies pick-and-place tasks under identical layouts, forcing models to truly understand language. Experiments show that targeted data augmentation can partially close the language gap – success rate improves from 0% to 90% with single-task training, and 0% to 28% with multi-task training. However, as semantic diversity of extended tasks increases, model learning capacity proves severely insufficient; even trained tasks perform poorly. This reveals a fundamental challenge for VLA models in understanding diverse language instructions – precisely the long-term value of LangGap.

Details

1. Benchmark & Dataset

LangGap benchmark: 99 tasks total — 40 original LIBERO tasks + 59 extended semantic perturbation tasks. We provide a training dataset of 56 tasks: 16 self-collected extended tasks (150 demos each) + 40 original tasks (50 demos each), totaling ~4,100 trajectories.

2. Problem Discovery

When changing the instruction from “put bowl on plate” → “put bowl on stove” in the same visual scene, the model still executes the original action (goes to plate), achieving 0% success. This reveals that VLAs perform vision-to-action pattern matching rather than genuine language understanding.

We design a four-dimensional semantic perturbation diagnostic — Change Object, Change Target, Spatial Description, and Drawer Action — keeping the visual scene identical and only modifying the instruction:

π0.5 SUCCESS RATE (%) · SAME SCENES, ONLY THE INSTRUCTION CHANGES Original LIBERO 40 tasks · 800 episodes 93.8 Extended (LangGap) 59 tasks · 1,180 episodes 21.4 Change Object 38 tasks · 760 ep. 29.3 Change Target 13 tasks · 260 ep. 0.0 — all 260 episodes fail Spatial Description 5 tasks · 100 ep. 11.0 Drawer Action 3 tasks · 60 ep. 31.7 −72.4 pts 0 25 50 75 100
π0.5 diagnosis over 99 tasks and 1,980 episodes. Our re-run of the 40 original tasks matches the official score; the 59 extended tasks drop by 72.4 points. On Change Target π0.5 fails all 260 episodes and no tested model exceeds 2.3%, so the failure is not a phrasing artifact — and the aggregate score hides it.
LIBERO evaluation suites with perturbation annotations

3. Design Principles

  1. Same-Scene Multi-Task — Multiple tasks share identical initial visual states, eliminating visual shortcuts. A model ignoring language achieves at most 1/k success rate (k = tasks per scene).
  2. Instruction-Level Train/Eval Split — Training tasks do not include all test tasks; held-out evaluation contains unseen language instructions to test compositional generalization.
  3. Physical Feasibility Validation — All extended tasks verified in the LIBERO simulator to ensure graspability, reachability, and detectability.

4. Data Collection Pipeline

  • Scalable & Diverse Generation: We employ a scripted, waypoint-based collection pipeline to efficiently and stably gather 150 successful episodes per task. While the waypoints are hard-coded for each specific task, the simulator introduces slight natural variations in the initial tabletop layouts. This ensures the collected trajectories are visually and dynamically diverse, preventing models from merely memorizing rigid, identical paths.
  • Hierarchical Control Architecture: Each task utilizes a custom script that decomposes the pick-and-place process into multiple sequential waypoints. At the high level, we apply pure Proportional (P) control to calculate positional errors and output continuous action commands. These commands are then executed by the simulator’s low-level OSC (Operational Space Control) PD controller, achieving seamless, highly precise continuous control.
Waypoint sequence of a scripted demonstration: home, grasp, transport, place

Waypoint timing of a scripted demonstration (illustrative): home → above object → grasp → lift → above target → place. OSC_POSE controller with per-step pose error normalized to [−1, 1]; an episode is kept only if it succeeds within the step budget.

  • Why scripts instead of teleoperation: early attempts with mouse teleoperation were jittery, slow and inconsistent. Scripts written from the object coordinates in each task’s BDDL file were hand-tuned in simulation until every task was graspable, reachable and detectable — the resulting actions are clean and the collection is fully automatic: ~150 demos per task (the official LIBERO tasks have 50), 16 tasks ≈ 2,400 demos.
Sample of scripted demonstration rollouts

A sample of the 2,400 scripted demonstrations.

Fine-tuning: single task from 0% to 90%

Left: before training — the instruction says "stove", but the policy still places the bowl on the plate. Right: after fine-tuning on the extended data — the bowl goes onto the stove.

  • π0.5 with LoRA (r = 64) on a single RTX 4090.
  • Checkpoints are selected by rollouts, not by loss: the loss keeps decreasing while success rate is non-monotonic.
  • Training and test instructions are verbatim identical except for the 43 held-out tasks, whose instructions are new.

5. Results

Spatial Goal Object Long Total Extended by type orig. ext. orig. ext. orig. ext. orig. orig. ext. Ch.Obj Ch.Tgt π0.5 97.0 5.9 97.0 30.0 100 37.7 81.0 93.8 21.4 29.3 0.0 π0 47.0 3.6 63.0 0.0 43.0 18.6 40.0 48.3 8.6 10.8 0.0 π0-FAST 65.0 1.5 37.8 1.2 61.0 5.0 26.0 47.5 2.7 3.1 2.3 SmolVLA 17.0 3.2 44.0 0.0 50.0 13.2 41.0 38.0 6.4 7.6 0.0 π0.5 + ours (45) 95.0 10.2 85.0 27.2 100 37.0 78.0 89.5 22.8 28.4 6.2 π0.5 + ours (56) 97.0 7.1 77.0 26.1 98.0 35.0 70.0 85.5 20.4 27.5 5.0 Success rate (%). orig. = original LIBERO tasks; ext. = LangGap extended tasks; Ch.Obj / Ch.Tgt = extended tasks that change the object / the target. Darker = higher.
Full benchmark: recent VLA models on all four LIBERO suites, original versus extended tasks. Every model shows a large language gap; our 45-task fine-tuned π0.5 improves the extended total (22.8) and Change Target (0 → 6.2) while keeping 89.5 on the original tasks. The 43 held-out tasks were never trained on.
SUCCESS RATE (%) ON EXTENDED TASKS · π0.5 LoRA baseline π0.5 fine-tuned on LangGap data Single task 1 extended · eval 1 task 3.75 90.0 6 tasks 1 + 5 extended · eval 5 ext. 0.0 28.0 45 tasks 40 orig. + 5 ext. · eval 5 ext. 0.0 4.0 16 tasks 16 extended · eval 16 ext. 26.2 6.2 56 tasks 40 orig. + 16 ext. · eval 16 ext. 26.2 27.5 0 25 50 75 100
Progressive validation with the same extended data as the number of training tasks grows from 1 to 56. Single-task memorization reaches 90%, but gains collapse as semantically different tasks are learned together; mixing in the 40 official tasks (56 tasks) restores 27.5% while the language gap remains.

6. Long-Term Value

As semantic diversity of tasks increases, model learning capacity proves severely insufficient — even trained tasks perform poorly. This reveals a fundamental challenge that is architecture-agnostic: all tested models (π0.5, π0, π0-FAST, SmolVLA) exhibit the same language gap. LangGap provides a systematic diagnostic tool that remains valuable as new VLA architectures emerge, precisely because the language gap is a persistent problem that current training paradigms have yet to solve.