TR2026-127

LEAP-VLA: Latent-Enhanced Action Prototyping via Continuous Residual Latent Spaces for Vision-Language-Action Models


    •  Yu, B.-Y., Peng, K.-C., Hsieh, J.-W., "LEAP-VLA: Latent-Enhanced Action Prototyping via Continuous Residual Latent Spaces for Vision-Language-Action Models", European Conference on Computer Vision (ECCV), Favaro, P. and Kukelova, Z. and Maki, A. and Rohrbach, A. and Schindler, K. and Tombari, F., Eds., DOI: 10.1007/​978-3-032-37450-9_20, September 2026, vol. 17034, pp. 367–385.
      BibTeX TR2026-127 PDF
      • @inproceedings{Yu2026sep,
      • author = {Yu, Bo-Yun and Peng, Kuan-Chuan and Hsieh, Jun-Wei},
      • title = {{LEAP-VLA: Latent-Enhanced Action Prototyping via Continuous Residual Latent Spaces for Vision-Language-Action Models}},
      • booktitle = {European Conference on Computer Vision (ECCV)},
      • year = 2026,
      • editor = {Favaro, P. and Kukelova, Z. and Maki, A. and Rohrbach, A. and Schindler, K. and Tombari, F.},
      • volume = 17034,
      • pages = {367–385},
      • month = sep,
      • publisher = {Springer},
      • doi = {10.1007/978-3-032-37450-9_20},
      • issn = {0302-9743},
      • isbn = {978-3-032-37449-3},
      • url = {https://www.merl.com/publications/TR2026-127}
      • }
  • MERL Contact:
  • Research Areas:

    Artificial Intelligence, Computer Vision, Machine Learning

Abstract:

Vision-Language-Action (VLA) models have advanced rapidly, yet most gains still come from larger backbones, larger embodied datasets, or expensive iterative decoders, while action representation remains under-explored. We argue that action space design is a primary bottleneck and present LEAP (Latent-Enhanced Action Prototyping)-VLA, a two-stage framework that learns a structured action latent space from demonstrations via Multi-level Soft Residual Quantization (MSRQ), then trains a lightweight VLM-external aligner to predict actions by prototype-aware soft selection in that space. This design replaces discrete code assignment and multi-step denoising with fully differentiable single-pass latent prediction, preserving pretrained visionlanguage alignment without modifying the VLM. Empirically, LEAPVLA outperforms the state-of-the-art methods with substantially fewer trainable parameters and no embodied pretraining, showing that a wellstructured continuous action latent space can offset model scale and data requirements.