Abstract:

Large vision-language models (LVLMs), such as the Generative Pre-trained Transformer 4-omni (GPT-4o), are emerging multi-modal foundation models which have great potential as powerful artificial-intelligence (AI) assistance tools for a myriad of applications, including healthcare, industrial, and academic sectors. Although such foundation models perform well in a wide range of general tasks, their capability without fine-tuning is often limited in specialized tasks. However, full fine-tuning of large foundation models is challenging due to enormous computation/memory/dataset requirements. Ultrasound data from the forearm has been shown to be used for hand gesture estimation. However, this typically requires training deep learning models with a large quantity of labeled data. We show that GPT-4o can decode hand gestures from forearm ultrasound data even with no fine-tuning, and improves with few-shot, retrieval augmented in-context learning. In our experiments, the average classification accuracy improved from 19.3% (0-shot) to 74.0% (2-shot) for within-session testing, and from 20.0% (0-shot) to 61.3% (3-shot) for cross-session testing. This demonstrates the potential of LVLMs for ultrasound-based gesture recognition by enabling an alternative to prior ultrasound gesture pipelines that require dedicated model training and large labeled datasets. Additionally, we show that few-shot in-context learning and retrieval-augmented selection can substantially improve performance without any model fine-tuning.