Jun 2025 – Feb 2026 · Habitat · Aerial VLN dataset pipeline

Vision-Language Navigation on Autonomous Drone

Built a robust pipeline to generate various 3D paths in the Habitat simulator. Overcame challenges of the simulator initially designed only for ground robots by designing a robust 3D navigation algorithm and obstacle detection method. Trained a strong and general policy for drone navigation.

Code: github.com/YC11Hou/habitat-aerial-nav

Demo

Ego-centric view during trajectory generation
3-stage trajectory: Takeoff, Cruise, Landing

Overview

Built a robust pipeline to generate diverse 3D navigation trajectories in the Habitat simulator for training vision-language navigation (VLN) policies on aerial robots.

Motivation: existing VLN datasets are for ground robots on discrete navigation graphs; drones need continuous takeoff → cruise → landing trajectories in 3D, and no such dataset existed, so we built one from scratch.

Simulator: Habitat with 90 indoor scenes.

Scale: 90 scenes × 200–300 start-goal pairs each → 10,000+ three-stage 3D trajectories with language instructions, open-sourced on GitHub.

Pipeline

① Start–goal pairs 200–300 per scene navigable, filtered ② 2D cruise path Lattice A* on (x, y, θ) H = ‖p − p_g‖/L + λ|Δθ| ③ 3D trajectory takeoff + cruise + landing RGB-D along the path ④ Instruction video → text model one per trajectory GRID A* · RIGHT-ANGLE TURNS start goal LATTICE A* · TURNS WHILE MOVING start goal 90 indoor scenes 200–300 start–goal pairs per scene 10,000+ three-stage 3D trajectories open source code and data on GitHub
Four-step generation pipeline. Lattice A* plans over motion primitives in (x, y, θ), so the drone turns while moving forward instead of making the right-angle turns of Grid A* (sketch below, illustrative).

  1. Start-Goal Pair Generation — For each of the 90 scenes, generate 200–300 random 2D start-goal pairs as navigation endpoints.

  2. 2D Cruise Path Planning — Determine a suitable constant cruising altitude for each scene, then plan a natural 2D path at that altitude using Lattice A*. Unlike standard Grid A* which produces rigid right-angle turns, Lattice A* plans over motion primitives in continuous state space \((x, y, \theta)\), producing smooth paths where the agent turns while moving forward. Heuristic:

    \[H(s) = \frac{\|p - p_{goal}\|}{L} + \lambda \cdot |\Delta\theta|\]

    This penalizes sharp turns to ensure smooth, realistic flight trajectories.

  3. 3D Trajectory Assembly — Prepend a takeoff segment and append a landing segment to each cruise path, forming a complete 3D trajectory. Collect RGB-D observations along the full path as video.

  4. Instruction Generation — Use a video-to-text model to generate natural language navigation instructions from the collected observation videos, producing a complete VLN dataset.

The design is deliberately simplified — the drone does not change altitude frequently mid-flight — yet takeoff and landing combined with instructions are already hard for current models. Policy training on the dataset is carried on by labmates.