TR2024-052
Interactive Planning Using Large Language Models for Partially Observable Robotic Tasks
-
- "Interactive Planning Using Large Language Models for Partially Observable Robotic Tasks", IEEE International Conference on Robotics and Automation (ICRA), DOI: 10.1109/ICRA57147.2024.10610981, May 2024, pp. 14054-14061.BibTeX TR2024-052 PDF Video
- @inproceedings{Sun2024may,
- author = {Sun, Lingfeng and Jha, Devesh K. and Hori, Chiori and Jain, Siddarth and Corcodel, Radu and Zhu, Xinghao and Tomizuka, Masayoshi and Romeres, Diego}},
- title = {Interactive Planning Using Large Language Models for Partially Observable Robotic Tasks},
- booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
- year = 2024,
- pages = {14054--14061},
- month = may,
- publisher = {IEEE},
- doi = {10.1109/ICRA57147.2024.10610981},
- isbn = {979-8-3503-8457-4},
- url = {https://www.merl.com/publications/TR2024-052}
- }
,
- "Interactive Planning Using Large Language Models for Partially Observable Robotic Tasks", IEEE International Conference on Robotics and Automation (ICRA), DOI: 10.1109/ICRA57147.2024.10610981, May 2024, pp. 14054-14061.
-
MERL Contacts:
-
Research Area:
Abstract:
Designing robotic agents to perform open vocabulary tasks has been the long-standing goal in robotics and AI. Recently, Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. However, planning for these tasks in the presence of uncertainties is challenging as it requires “chain-of-thought” reasoning, aggregating information from the environment, updating state estimates, and generating actions based on the updated state estimates. In this paper, we present an interactive planning technique for partially observable tasks using LLMs. In the proposed method, an LLM is used to collect missing information from the environment using a robot, and infer the state of the underlying problem from collected observations while guiding the robot to perform the required actions. We also use a fine-tuned Llama 2 model via self-instruct and compare its performance against a pre-trained LLM like GPT-4. Results are demonstrated on several tasks in simulation as well as real-world environments.
Related News & Events
-
NEWS MERL at the International Conference on Robotics and Automation (ICRA) 2024 Date: May 13, 2024 - May 17, 2024
Where: Yokohama, Japan
MERL Contacts: Anoop Cherian; Radu Corcodel; Stefano Di Cairano; Chiori Hori; Siddarth Jain; Devesh K. Jha; Jonathan Le Roux; Diego Romeres; William S. Yerazunis
Research Areas: Artificial Intelligence, Machine Learning, Optimization, Robotics, Speech & AudioBrief- MERL made significant contributions to both the organization and the technical program of the International Conference on Robotics and Automation (ICRA) 2024, which was held in Yokohama, Japan from May 13th to May 17th.
MERL was a Bronze sponsor of the conference, and exhibited a live robotic demonstration, which attracted a large audience. The demonstration showcased an Autonomous Robotic Assembly technology executed on MELCO's Assista robot arm and was the collaborative effort of the Optimization and Robotics Team together with the Advanced Technology department at Mitsubishi Electric.
MERL researchers from the Optimization and Robotics, Speech & Audio, and Control for Autonomy teams also presented 8 papers and 2 invited talks covering topics on robotic assembly, applications of LLMs to robotics, human robot interaction, safe and robust path planning for autonomous drones, transfer learning, perception and tactile sensing.
- MERL made significant contributions to both the organization and the technical program of the International Conference on Robotics and Automation (ICRA) 2024, which was held in Yokohama, Japan from May 13th to May 17th.
Related Video
Related Publications
- @inproceedings{Sun2023dec,
- author = {Sun, Lingfeng and Jha, Devesh K. and Hori, Chiori and Jain, Siddarth and Corcodel, Radu and Zhu, Xinghao and Tomizuka, Masayoshi and Romeres, Diego},
- title = {Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks},
- booktitle = {Advances in Neural Information Processing Systems (NeurIPS) Workshop on Instruction Tuning and Instruction Following},
- year = 2023,
- month = dec,
- url = {https://www.merl.com/publications/TR2023-148}
- }
- @article{Sun2023dec2,
- author = {Sun, Lingfeng and Jha, Devesh K. and Hori, Chiori and Jain, Siddarth and Corcodel, Radu and Zhu, Xinghao and Tomizuka, Masayoshi and Romeres, Diego},
- title = {Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks},
- journal = {arXiv},
- year = 2023,
- month = dec,
- url = {https://arxiv.org/abs/2312.06876}
- }