We construct a benchmark of 15 representative tasks to map the capabilities and limitations of a general-purpose VLM in video annotation. Our results show VLM-alone annotation approaches human accuracy on average (HNS=97.0), while human verification of VLM results achieves the highest accuracy (HNS=121.5) while reducing annotation time by 48.9%.
We use a multimodal sensing system—encompassing pupil diameter, electrodermal activity, touchscreen telemetry, and driving kinematics—to examine how cognitive load shapes driver performance. The results show that increasing cognitive load degrades touchscreen interactions, disrupts motor–visual coordination, and reveals a characteristic hand-before-eye pattern under constrained attentional states.
We find that finger-motion patterns and the inherent structure of joints provide beneficial physical knowledge, which lead us to enhance motion perception accuracy by integrating physical priors into ML models. Thus, we propose MouseRing, a novel ring-shaped IMU device that enables continuous finger-sliding on unmodified physical surfaces like a touchpad.
We propose to explore teeth-clenching-based target selection in Augmented Reality, as the subtlety in the interaction can be beneficial to applications occupying the user hand or that are sensitive to social norms. To support the investigation, we implemented a novel EMG-based teeth-clenching detection system (ClenchClick) to support hand-free interactions in AR.
We propose a data and expert-informed framework for designing adaptive touchscreens that respond to a driver’s usage profile and cognitive state, maximizing usability while mitigating safety risks.
WritingRing is a single-IMU ring system that enables natural, real-time 2D handwriting input.The system achieves 1.63 mm trajectory accuracy along with 88.7% letter and up to 84.36% word recognition accuracy. The approach is lightweight and can be integrated into existing ring devices, supporting a range of hands-free input applications.
We propose TextOnly, a unified text-based function portal that uses a single text box to trigger actions across apps. The system integrates an LLM with a continually adapting BERT model to interpret user intent from short inputs with improving accuracy and speed. User studies show strong usability, with 71.35% top-1 accuracy and clear preference over manual actions and voice assistants.