Summary: The researchers are developing an AI system that can recreate fine hand muscle functions using only standard video footage. Traditionally, this type of measurement required incomparable electrodes attached to the skin, but the new method eliminates the need completely.
The system was trained with high-precision recordings of expert hand movements, allowing it to deal with hidden muscle signals with surprising accuracy. This breakthrough opens the door to cost-effective, remote analysis of fine motor control for health care, rehabilitation and performance training.
Key facts:
- Neuromuscular Tracking – AI measures hidden hand activity using video alone, without EMG electrodes.
- High accuracy in operations: A reliable system predicts the time and force of muscle activation, even in invisible players.
- Wide international applications: Possible applications include regenerative medicine, sports science, robotics and work-based environments.
Source: Institute of Science Tokyo
The movement of the hands during piano performance depends on the direct communication between the small muscles hidden under the skin.
Tracking these signals has traditionally required electromyography (EMG) sensors, which are expensive, cumbersome, and technically complex.
A research group led by Professor Fieki Koike at the Department of Computer Science, Science Tokyo Institute (Science Tokyo), Japan, and Dr. Shinichi Fuya from Sony Computer Science Laboratories, Japan, is now addressing this challenge using artificial intelligence.
Their new framework, the piano keystroke-pose-mister network (Pianokpm Net), estimates hand miniature activity using only video recordings. Their findings were published online on September 19, 2025, and will be presented 39th Conference on Neural Information Processing Systems (Neuraups 2025)held in San Diego, USA, on December 2, 2025.
The system is built on a new database, pianokpm, which captures how the technician moves, presses and controls their hands with exceptional precision. It includes 12.6 hours of synchronized data from 20 professional pianists performing seven different musical works.
Each performance was recorded with multiple video frames at 60 frames per second, 3D hand data, 1 KHZ Keystroke data, Audio, and 2 Khz EMG Signals from six small muscles.
The dataset contains more than five million frames of poses and 28 million EMG samples, creating the first detailed map linking visual movement with internal muscle activity.
“Incorporating this data, we propose Pianokpm Net to provide the highest possible EMG from the pose data,” shouts Koike.
Using this foundation, Pianokpm Net learns the bearing of muscle behavior from video data. By combining cues from Hands and Keystrokes, the model reconstructs the signal's mislal timing and strength.
In a comparative test against advanced frameworks, such as Neuropose and Codetalker, Pianokpm Net achieved high accuracy in predicting both timing and muscle activity. Even with spoilers and pieces of music not included in the training, the model maintains a powerful performance, ensuring its flexibility and generality.
This method turns a simple camera into a non-invasive tool for studying muscle coordination. This allows visualization of how skilled Piania control subtle muscle movements to achieve speed, control and precision. This enables a detailed physical examination without attaching any sensors to the body, reducing both cost and discomfort.
Technology has more power than the piano. In sports science, it can track muscle exertion to improve training accuracy and prevent overuse injuries. In rehabilitation, it can monitor the progress of recovery, providing doctors with continuous feedback without physical attachment. It can also improve human interaction systems – where understanding the user's muscle effort helps refine robotic assistance and displacement-based environments.
“Together, the pataset of pianokpm and pianokpm form the basis for cost-effective access to the signal of the internal functions of muscles and tendons, supporting the development of human and human interaction with a high-quality machine,” explained Koike.
The research group plans to make the dataset and model publicly available. This open release will allow scientists and developers to develop studies in machine learning, artificial intelligence and assistive robots. Broad access will help to establish benchmarks for measuring and measuring muscle mass, speeding up the development of many fields.
By linking perception and physiology, pianokpm net offers a new way to learn fine motor control. It replaces the EMG complex setup with video-based analysis, creating opportunities for functional research, clinical trials, and human technology design.
The program marked a clear step towards inexpensive, AI-driven analysis of skilled movements, where invisible muscle patterns can be stored and understood only visually.
In the future, this technology can contribute to the Distance Learning Skill by enabling the use of low communication networks, even in areas that do not have access to expensive biological measurements.
Important Questions Answered:
A: It accurately captures small hand movements using standard video recordings, without physical sensors.
A: The model surpasses existing deep learning methods for predicting the time and energy of muscle activation.
A: It replaces expensive, limited muscle sensors with a more expensive, non-invasive alternative.
Editing notes:
- This article was edited by the editor of neuroscience news.
- The journal is fully reviewed.
- Additional context added by our staff.
About This Ai and Neurotech News News
Author: Miki Yamaoka
Source: Institute of Science Tokyo
Contact: Miki Yamaoka – Institute of Science Tokyo
Image: This photo is posted in Neuroscience News
Actual research: The findings will be presented at 39th Conference on Neural Information Processing Systems (Neuraups 2025)
