Abstract:

Integrating humanoid robots into daily life demands effective collaboration, where robots must accurately perceive human actions and context to work toward common objectives. Previous studies generate robot action sequences for human instructional videos, but most rely on offline processing with pre-segmented clips. In real-world interaction, however, robots must handle unsegmented multimodal inputs in a streaming manner. In this paper, we extend a Q-Former-based robot action generation framework to streaming processing. Our method extracts audiovisual features from short temporal chunks, incorporates leftcontext information into query embeddings, and incrementally generates action sequences and confirmation messages using a large language model. By carefully designing attention masks, the model can be trained efficiently in parallel, similar to offline methods. Experiments show that our streaming approach achieves low latency with less than 10% accuracy degradation compared to offline processing.