Northwestern University engineers have introduced a new system for full-body motion capture that operates without the need for specialized rooms or expensive equipment. Named MobilePoser, this innovative technology utilizes sensors already present in consumer mobile devices such as smartphones, smartwatches, and wireless earbuds.
Karan Ahuja, who led the study, explained that MobilePoser achieves "state-of-the-art accuracy through advanced machine learning and physics-based optimization," making it possible to use this technology in gaming, fitness, and indoor navigation without requiring specialized equipment. The app was unveiled at the 2024 ACM Symposium on User Interface Software and Technology in Pittsburgh.
Ahuja is an expert in human-computer interaction and serves as the Lisa Wissner-Slivka and Benjamin Slivka Assistant Professor of Computer Science at Northwestern's McCormick School of Engineering. He directs the Sensing, Perception, Interactive Computing and Experience (SPICE) Lab.
Traditional motion-capture techniques often involve actors wearing suits covered with sensors in specialized rooms. These setups can cost upwards of $100,000. Ahuja's team aimed to create a more accessible version using existing mobile device equipment.
To address limitations of current systems like Microsoft Kinect, which rely on stationary cameras, Ahuja’s team utilized inertial measurement units (IMUs). These include accelerometers, gyroscopes, and magnetometers found within smartphones but typically offer low fidelity for accurate motion capture. By incorporating a custom-built AI algorithm trained with synthesized IMU measurements from high-quality data, MobilePoser enhances sensor performance.
The app estimates joint positions and rotations by processing sensor data through its AI algorithm while using a physics-based optimizer to ensure realistic body movements. It boasts a tracking error of 8 to 10 centimeters compared to Microsoft Kinect's 4 to 5 centimeters within camera view.
MobilePoser's accuracy improves when users wear multiple devices like a smartwatch alongside their smartphone. However, it remains adaptive even if only one device is available.
Beyond gaming applications offering immersive experiences for users; MobilePoser holds potential benefits for health professionals analyzing patient mobility or activity levels by providing detailed posture information rather than just step counts—a common limitation faced today according to Ahuja: "Our phones can calculate temperature in Rome but know less about our own bodies."
The development team has released pre-trained models along with data pre-processing scripts as open-source software encouraging further research contributions from others interested parties while planning future availability across Apple products including iPhone AirPods & Watch soon enough too!