Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Transcription of Multi-Instrumental Music to MIDI
Automatic Music Transcription (AMT) converts audio into symbolic notes, usually MIDI. One tool script is already working fine. However, recording a full multi-instrumental mix is always difficult. The Kyutai and Mirelo groups are now released MuScriptor to fill that void. It's an open-weight model trained on real, multi-instrument recordings in many genres.
This article explains how MuScriptor works, what the benchmarks show, and how to use it.
What is MuScriptor?
At its core, MuScriptor is a Transformer only decoder for music transcription. First, it reads the mel-spectrogram of a short audio segment. It then automatically predicts MIDI tokens such as pitch, time, and instrument. Basically, writing becomes a language matching function, which follows the MT3 token scheme.
Release moves three types of weight to the Hugging Face. Their sizes small (103M), medium (307M, default), and large (1.4B). The inference code uses the MIT license. Weights use CC BY-NC 4.0, so commercial use is restricted.
How the Three-Phase Pipeline Works
The main idea of MuScriptor is data, not structures. Accordingly, the training goes through three phases, and each one builds on the last.
- Pre-training use DThe Synthabout 1.45M MIDI files. A walking pipe joins them during training. Additions include pitch changes, tempo changes, speed adjustments, and instrument randomization. More than 250 audio fonts and random output bring endless audio fulfillment.
- The use of fine tuning DOf courseinternal set of 170,000 recordings. In total they include over 11,000 hours of aligned note annotations. Most alignment comes from audio-to-signal synchronization using interpolation and dynamic time warping. Poor pairs are sorted by twist distance and maximum time extension factor.
- Use of reinforcement learning after training DRL300 manually verified tracks. The team uses a GRPO-like approach that combines REINFORCE with the normalization of group-related gains. The award includes three F points: start, frame, and offset. As a result, the model learns to love clean texts.
