Utilizing Qwen2-Audio to transcribe music into sheet music
Computerized music transcription is the method of changing audio information like MP3 and WAV into sheet music, guitar tablature, and any format a musician might wish to study a track on their instrument.
We’ll go over the perfect present instruments for doing this, which occur to be deep learning-based, and a novel method for it.
The present state-of-the-art for this activity comes from Magenta, an open-source analysis venture developed by the now defunct (as of April 2023) Google Mind Crew.
They launched a paper Sequence-to-Sequence Piano Transcription with Transformers in 2021 which used a T5-inspired transformer mannequin (much like “t5-small”) with 54 million parameters and the Maestro dataset, reaching nice outcomes. The issue is approached as a sequence-to-sequence activity utilizing an encoder-decoder Transformer structure. The encoder processes mel spectrogram frames as enter and produces embeddings, whereas the decoder makes use of these embeddings through cross-attention to autoregressively generate a sequence of MIDI-like tokens. Their vocabulary consisted of 4 forms of tokens:
- Notice tokens (128 values for MIDI pitches)
- Velocity tokens (128 values together with zero for note-off)
- Time tokens (6,000 values in 10ms bins for absolute timing)
- EOS token (to mark sequence finish)
See the picture under for a visualisation of the structure and an instance sequence of their customized MIDI tokens:
Our mannequin is a generic encoder-decoder Transformer structure the place every enter place accommodates a single spectrogram body and every output place accommodates an occasion from our MIDI-like vocabulary. Outputs tokens are autoregressively sampled from the decoder, at every step taking the token with most chance.
In 2022, they launched a paper, MT3: Multi-Activity Multitrack Music Transcription. This experiment used the identical method because the final one however added further instrument tokens to symbolize the completely different devices. Once more, they used an analogous T5 mannequin and achieved nice efficiency towards lots of the datasets educated on, notably Slakh, Maestro and MusicNet.
MR-MT3 was launched the next yr as a slight enchancment to MT3.
Compute/GPU sources
Large sources have been wanted to coach this from scratch, regardless of being a lot smaller in dimension in comparison with even the smallest language fashions. The 2021 paper famous:
“We educated all fashions on 32 TPUv3 cores, leading to a per-core batch dimension of 8. Primarily based on validation set outcomes, overfitting didn’t appear to be an issue, so we allowed coaching to progress for 400K steps, which took about 2.5 days for our baseline fashions.”
The MT3 paper doesn’t present as particular particulars on coaching, stating they prepare for 1 million steps.
Different limitations
These fashions have some inherent limitations of their output flexibility. Whereas language fashions usually have massive vocabularies (usually 30,000+ tokens) which are extensively pre-trained on various pure language information, MT3 and comparable music transcription fashions use a a lot smaller, specialised token vocabulary (just a few thousand tokens) centered solely on musical occasions. This specialisation signifies that including new tokens, resembling for brand new devices or enjoying strategies like palm muting on guitars or pizzicato on violins, is probably going not simple — it requires vital retraining to combine these new tokens successfully with the prevailing vocabulary, and sometimes requires substantial coaching information demonstrating these strategies. This differs from massive language fashions which may usually describe such musical nuances in pure language with out modification, as they’ve encountered these ideas throughout their broad pre-training.
Switch studying and zero-shot
We will leverage switch studying from massive open-source pre-trained audio and language fashions. Examples of music era fashions embrace OpenAI’s Jukebox and Meta’s MusicGen.
GPT-4o is designed to deal with textual content, audio and pictures “natively”. Though OpenAI has not launched the technical particulars on this, it’s assumed that some weights within the community will course of all modalities. It’s attainable that the mannequin makes use of a decoder-only structure like language solely GPT fashions with out the necessity for encoder parts to transform completely different modalities to a dense illustration first. This design permits the mannequin to seamlessly course of and interpret inputs like textual content and pictures collectively, probably providing efficiency advantages each computationally and by way of mannequin understanding.
Many multi-modal fashions take an easier method harking back to the encoder-decoder structure: they mix two pre-trained fashions — an encoder for the particular enter modality (like ViT for imaginative and prescient or an audio encoder for sound) and a Giant Language Mannequin (resembling LLaMA, Gemma, or Qwen). These fashions are related by projection layers that align their representations in a shared latent house, usually utilizing only a single linear layer. These projection layers study to transform the encoder’s output right into a format that matches the LLM’s anticipated enter dimensions and traits. The projection creates new embeddings/tokens from the enter modality that may then be injected into the LLM’s enter sequence. LLaVA is a chief instance of this structure for vision-language duties, whereas Spotify’s Llark and Qwen-Audio apply the identical precept utilizing audio encoders as an alternative of imaginative and prescient encoders.
Right here’s some pseudocode on how the fashions are stitched collectively:
# Extract options from remaining layer of audio encoder
# Form: [batch_size, audio_seq_len, encoder_dim=1024]
audio_features = audio_model(audio_input)# Undertaking audio options to match LLM's embedding dimension
# Form: [batch_size, audio_seq_len, llm_embed_dim=4096]
audio_embeddings = projection_layer(audio_features)
# Get textual content embeddings from LLM's embedding layer
# Form: [batch_size, text_seq_len, llm_embed_dim=4096]
text_embeddings = llm.embed_text(text_input)
# Concatenate alongside sequence size dimension
# Form: [batch_size, audio_seq_len + text_seq_len, llm_embed_dim=4096]
combined_input = concatenate([audio_embeddings, text_embeddings], dim=1)
# Feed them into the LLM as regular for era
output = llm(combined_input)
Overview of structure
Llark makes use of OpenAI’s Jukebox and Qwen2-Audio makes use of OpenAI’s Whisper for the audio towers. Jukebox is a music era mannequin however it may well additionally soak up audio clips as enter and outputs a continuation of the audio clip. Whisper is used for transcribing voice to textual content.
Given their function, the selection of audio module is evident: Llark specialises in music evaluation, whereas Qwen2Audio primarily focuses on responding to voice directions with some primary audio and music evaluation capabilities.
Figuring out the optimum supply for extracting embeddings from massive pre-trained fashions entails analysis and experimentation. Moreover, deciding whether or not to fine-tune the complete module or freeze elements of it’s a essential design selection. For example, LlaVa’s coaching technique entails freezing the imaginative and prescient tower and specializing in fine-tuning the projection layer and language mannequin. We’ll go over this side of every mannequin under.
Llark: why Jukebox? Are these embeddings the perfect as of September 2024?
Figuring out the optimum location to extract embeddings from massive fashions usually requires in depth probing. This entails testing numerous activations or extracted layers of the mannequin on completely different classification duties by a technique of trial and error. For music era fashions, this might embrace duties like style recognition, instrument detection, emotion detection, in addition to evaluation of harmonic buildings and temporal patterns. Many business embedding fashions (like OpenAI’s embedding fashions) are educated particularly for embedding era with specialised architectures and coaching targets, moderately than being fine-tuned variations of present language fashions.
The 2 largest publicly accessible music era and music continuation (i.e.: ready to absorb audio as enter) fashions are Jukebox and MusicGen. MusicGen is newer and quicker, and due to this fact appeared like it might be the apparent option to me. Nevertheless, in line with this paper on probing MusicGen, embeddings extracted from Jukebox seem to outperform MusicGen on common in classification duties. The findings from this paper led to the authors of Llark utilizing the next method for extracting embeddings:
- Embeddings are derived from the output of the thirty sixth layer of the Jukebox encoder following the method described in Castellon et al. (2021)
- Unique Jukebox encoding:
* 4800-dimensional vectors at 345Hz
* For a 25s clip: over 4.14 * 10⁷ floating-point values - The authors use a downsampling method: Imply-pooling inside 100ms frames, leading to:
* Downsampled frequency: 10Hz
* Embedding dimension: 1.2 × 10⁶ for a 25s audio clip. Meaning a 2D array with form [240, 4800].
* Retains temporal data (not like Castellon et al. who common over the time dimension)
(The downsampled embedding dimension is roughly 6x bigger than CLIP ViT-L14 fashions utilized in many multimodal imaginative and prescient fashions)
Qwen2Audio: Whisper
The embedding extraction for Qwen2Audio isn’t talked about intimately within the paper. Whisper is an encoder-decoder structure the place the encoder generates deeply discovered representations of the audio and the decoder decodes the representations to textual content (the transcription). In Qwen2Audio, it seems they extract embeddings from the ultimate layer of Whisper’s encoder, though they don’t point out whether or not they freeze it throughout coaching.
Pre-trained weights, coaching information and datasets
Sadly Spotify has not supplied any datasets or their educated mannequin weights to the general public, noting:
“With respect to inputs: the inputs to our mannequin are public, open-source, Artistic Commons-licensed audio and related annotations. Nevertheless, every particular person audio file can have its personal, probably extra restrictive license. Lots of the audio information embrace “no derivatives” licenses. We encourage customers of the datasets to familiarize themselves with the restrictions of those licenses; in an effort to honor such licenses, we don’t launch any derivatives from the coaching information on this paper (together with query- response pairs or educated mannequin weights).”
They used the next datasets:
- MusicCaps (Agostinelli et al., 2023)
- YouTube8M-MusicTextClips (McKee et al., 2023)
- MusicNet (Thickstun et al., 2017)
- FMA (Defferrard et al., 2017)
- MTG-Jamendo (Bogdanov et al., 2019)
- MagnaTagATune (Legislation et al., 2009)
Llark particulars it’s coaching information era course of within the following extract:
“We use variants of ChatGPT to extract the instruction- tuning information for all experiments. Nevertheless, the precise language mannequin used varies by dataset. We choose the OpenAI mannequin as follows: We use GPT-4 for all reasoning duties. We discovered that GPT-4 was rather more adept at following the complicated directions within the Reasoning activity household. For datasets with greater than 25k samples, we restrict Reasoning information to a random subsample of 25k tracks.”
This leads to Q&A knowledge like this:
The datasets used for coaching Qwen2Audio should not shared both, however the educated mannequin is extensively accessible and in addition is carried out within the transformers library:
For this venture, fine-tuning off a pre-trained Llark mannequin would have been optimum, given it’s reportedly good efficiency towards the analysis benchmarks Spotify acknowledged within the paper.
Nevertheless, given they didn’t launch the weights for it, it’s unfeasible to begin coaching a mannequin like this from scratch and not using a good bit of experience and cash. Spotify educated it on:
Our mannequin is educated on 4 80GB NVIDIA A100 GPUs. Coaching takes roughly 54 hours.
This is able to price round $700 utilizing a supplier like LambdaLabs.
Due to the above, I went with Qwen. Nevertheless, Qwen2-Audio doesn’t carry out that properly throughout primary music duties like tempo and instrument detection. I element this under within the analysis part. Which means the mannequin might be not massive sufficient or pre-trained sufficient to realize this activity, however my hope is I may no less than set a place to begin and framework for fine-tuning on this activity sooner or later. As Alibaba state of their Qwen2-Audio weblog publish:
We additionally plan to construct bigger Qwen2-Audio fashions to discover the scaling legal guidelines of audio language fashions.
For my very own studying although, I did have a go at re-creating the mannequin utilizing torch and pre-trained fashions with the transformers library.
I additionally created datasets for Q&A knowledge and embeddings. I generated brief kind Q&A knowledge for the URMP dataset, e.g.: “What’s the tempo of this observe”, “What devices are enjoying on this audio”.
Right here’s a pocket book for working Jukebox in a Colab atmosphere to make the most of a budget T4 GPU’s. I uploaded each Q&A and embeddings datasets to HuggingFace right here.
Right here’s a pocket book with Llark replicated.
Transcription format
I selected ABC music notation because the output format that the language mannequin is predicted to transcribe the music in. Right here’s an instance of it:
X:1
M:4/4
L:1/16
Okay:none
Q:67V:1 title="Electrical Bass (finger)"
%%octave-default C4
GAA^2E3A2<A^2 | D^D^2E2A2A^4 A^2E2 | A2A^4A^2E2 A2A^4 | A^2E2A2A^4A^2E2A2 |
A^4 A^2E2 A2A^4A^2 E2 | A2A^4 |
V:2 title="Vibrant Acoustic Piano"
%%octave-default C5
[E3C3][E3C3][E3C3] [E3C3][A^,2E2A^2] | [E3A^3][E3A^3][E3A^3][E3A^3][E3A^3] |
[E3A^3][E3A^3][E3A^3] [E3A^3][E3A^3] | [E3A^3][E3A^3][E3A^3][E3A^3][E3A^3] |
[E3A^3][E3A^3][E3A^3] [E3A^3][E3A^3] | [E3A^3] |
V:3 title="Electrical Guitar (jazz)"
%%octave-default C5
E'3C'3A^4E'3C'3 | A^4E'3 C'3A^4E'3C'3 | A^4 E'3C'3A^4 E'3C'3 | A^4E'3C'3A^4E'3C'3 |
A^4E'3C'3 A^4E'3C'3 | A^4 |
On this notation we have now the time signature and tempo outlined on the prime denoted by ‘M’ and ‘Q’. The ‘L’ signifies the default notice size of the notation, on this case a sixteenth notice, which is the norm. We then outline every instrument and the default octave they need to adhere to when writing the notes for every of them. Right here’s a abstract of the important thing syntactical factors for writing notes in ABC music notation:
- Notes are represented by letters A-G, with lowercase letters indicating larger octaves
- Sharps are denoted by ^ earlier than the notice, flats by _
- Pure indicators are represented by =
- Notice size is indicated by numbers after the notice (C2 is twice so long as C)
- Dotted notes use a . after the notice (C. is a dotted quarter notice)
- Rests are represented by z, with numbers for period (z2 is a half relaxation)
- Chords are enclosed in sq. brackets [CEG]
- Ties are proven with a hyphen –
- Bar strains are represented by |
- Damaged rhythms use > or < between notes (C>D means dotted-C eighth notice adopted by D sixteenth notice)
Why ABC?
The explanations for selecting this notation are:
- It’s a minimalist format for writing music
- It’s extensively used and standard; language fashions have already got good comprehension of ABC notation as a result of in depth pre-training on it.
- It’s versatile and might simply be prolonged to incorporate tempo modifications, time signature modifications, further enjoying kinds like talked about above, and so forth…
I transformed the MIDI information supplied by the datasets to ABC notation utilizing this library. A pocket book for creating the datasets is right here.
To judge each the unique mannequin and every stage of fine-tuning I carried out thereafter, I randomly chosen 30 samples of various complexity from the URMP dataset and ran the mannequin 3 times on every pattern, manually analyzing all responses.
By way of handbook testing, I discovered the optimum decoding parameters to be a temperature of 0.7 and a top_p of 1.2. The utmost variety of tokens to return was capped at 2048. Adjusting the max appeared to have little distinction on efficiency.
The unique mannequin carried out poorly on this analysis set. Whereas it often predicted the tempo and devices accurately, it largely failed to take action. A textual content file with the analysis outcomes is on the market right here.
Given this start line, it’s unlikely that we’ll see sturdy outcomes from this experiment and not using a sturdy pre-trained mannequin. Nevertheless, the objective is to develop methods that may be utilized sooner or later as extra superior pre-trained fashions grow to be accessible.
I first tried fine-tuning with primary cross-entropy loss. Supervised fine-tuning with cross-entropy loss is a fast strategy to begin educating the mannequin however a primary loss operate like this has limitations as we are going to see under. The instinct behind this stage of coaching is that it might nudge the mannequin in the best course and it might choose up any patterns or any customised ABC notation the dataset might have which the mannequin might not have seen earlier than.
Cross-entropy loss with trainer forcing
First, we educated it in a typical supervised fine-tuning method for language fashions. I used the SFTtrainer from the trl library for this, which makes use of cross-entropy loss with trainer forcing outlined step-by-step under:
- The mannequin predicts the following token within the sequence.
- The loss is calculated primarily based on the distinction between the expected possibilities (logits) and the precise subsequent token.
- For the following prediction, the mannequin is given the precise right token (floor fact), moderately than its personal prediction. This is named trainer forcing, it helps stabilise coaching and considerably velocity it up, particularly within the early phases.
The outcomes from this coaching section have been poor. It degraded the efficiency of the unique mannequin. The mannequin, which beforehand dealt with tempo and instrument recognition properly, now largely acquired these mistaken. It additionally started producing garbled textual content output with countless repetition. This occurred even when setting a low studying price, making use of gradient clipping, and utilizing low LoRA ranks to mitigate massive modifications to the mannequin. General, it appeared the mannequin was very delicate to the coaching utilized.
Nevertheless, whereas this coaching section might provide some enhancements, it received’t result in optimum efficiency because of the limitations of our primary loss operate. This operate struggles to completely seize the mannequin’s efficiency nuances. For instance, when utilizing trainer forcing, instrument predictions can yield deceptively low loss throughout sure token sections. If an instrument title begins with “V”, the mannequin may confidently predict “Violin” or “Viola” primarily based on our dataset, no matter accuracy. Moreover, the loss operate might not precisely mirror near-misses, resembling predicting a tempo of 195 as an alternative of 200 — a small distinction that’s fairly correct however probably penalised closely depending on the distribution of possibilities amongst logits. It’s attainable that neighbouring numbers even have excessive possibilities.
RLHF with PPO
Due to these limitations, we are able to create our personal customized loss operate that may extra precisely rating the response from the mannequin. That’s, given a predicted sequence from the mannequin, the loss operate may give it a rating between 0 and 1 on how good it’s.
Nevertheless, integrating this tradition loss operate into supervised fine-tuning presents a big problem. The problem stems from the non-linearity launched by the customized loss operate, which prevents the direct calculation of gradients. Let’s break this down:
In conventional SFT with cross-entropy loss:
- The mannequin outputs logits (uncooked scores) for every token in its vocabulary
- These logits straight symbolize the mannequin’s prediction possibilities
- The loss operate compares these possibilities to the bottom fact
- Gradients could be computed straight by this comparability
- The chain rule of calculus permits us to propagate these gradients again by the mannequin
With our customized loss operate:
- The mannequin should first generate full textual content output
- This era course of entails sampling from chance distributions
- Our loss operate then analyses this textual content output (checking tempo, notes, and so forth.)
- This creates a non-differentiable step between the mannequin’s logits and our loss calculation
- The sampling and textual content evaluation steps break the gradient chain wanted for backpropagation
To beat this, reinforcement studying strategies like Proximal Coverage Optimisation (PPO) could be employed. PPO is particularly designed to deal with non-differentiable loss features and might optimise the mannequin by contemplating the complete coverage (the mannequin’s output distribution), moderately than counting on gradient data from logits.
Notice, there’s a lot of nice articles on right here explaining PPO!
The important thing perception of PPO is that as an alternative of attempting to straight backpropagate by the non-differentiable steps, it:
- Treats the mannequin’s outputs as actions in a reinforcement studying framework
- Makes use of the customized loss operate as a reward sign
- Updates the mannequin’s coverage (its chance distributions over tokens) to maximise anticipated reward
- Does this whereas making certain the up to date coverage doesn’t deviate too removed from the present one
This method permits us to successfully prepare the mannequin with the customized loss operate, making certain efficiency enhancements with out disrupting the core coaching dynamics. The PPO algorithm’s conservative replace technique helps keep stability throughout coaching, which is especially necessary when working with massive language fashions.
Often, this scoring operate could be carried out as a separate LLM within the type of a “reward mannequin” generally used when fine-tuning fashions through RLHF, which was a breakthrough first launched when ChatGPT got here out. As a result of nature of this activity, we are able to manually write code to attain the responses, which makes use of fewer sources and is faster.
For time signature and tempo recognition that is simple to calculate. We extract all predicted gadgets with regex, for instance extracting the metre:
def extract_metre(self, abc_string):
return re.search(r'M:(S+)', abc_string).group(1)
The mannequin ought to study the syntax and construction we would like it to output within the SFT stage. If it outputs one thing that can trigger our regex to not discover something or error, we are able to simply skip that pattern, assuming it’s a small minority of the dataset.
We extract the expected tempo and write a operate that’s extra forgiving for small errors however penalises bigger errors extra closely:
- For small variations (≤10 BPM), it makes use of linear scaling.
- For bigger variations, it switches to exponential scaling.
- The ultimate loss is capped between 0 and 1.
Let’s break down the important thing parts of this tradition loss:
Code for the customized loss is right here
1. Metre Loss
The metre loss focuses on the time signature of the piece. It compares the expected metre with the bottom fact, contemplating each the numerator and denominator individually, in addition to their ratio. This method permits for a nuanced analysis that may deal with numerous time signatures precisely.
The metre loss makes use of a mixture of linear and exponential scaling to penalise variations. Small discrepancies end in a linear enhance in loss, whereas bigger variations result in an exponential enhance, capped at a most worth of 1.
2. Tempo Loss
Tempo loss evaluates the accuracy of the expected beats per minute (BPM). Much like the metre loss, it makes use of a mixture of linear and exponential scaling.
For small tempo variations (≤10 BPM), the operate applies linear scaling. Bigger variations set off exponential scaling, making certain that vital tempo mismatches are penalised extra closely.
3. Pitch Loss
The pitch loss is probably essentially the most essential element, because it assesses the accuracy of the transcribed notes. This operate makes use of the Levenshtein distance to check the sequence of notes in every voice.
The pitch loss calculation accounts for a number of voices, matching every predicted voice to the closest floor fact voice. This method permits for flexibility in voice ordering whereas nonetheless sustaining accuracy within the total pitch content material.
4. Instrument Loss
The instrument loss evaluates the accuracy of instrument choice for every voice.
This operate considers actual matches, devices from the identical household, and makes use of string similarity for extra nuanced comparisons. It gives a complete evaluation of how properly the mannequin identifies and assigns devices to every voice.
5. Combining the Losses
The ultimate loss is a weighted mixture of those particular person parts:
total_loss = (0.5 * pitch_loss +
0.15 * metre_loss +
0.15 * tempo_loss +
0.2 * instrument_loss)
This weighting scheme prioritises pitch accuracy whereas nonetheless contemplating different necessary elements of music transcription.
PPO coaching typically requires much more reminiscence than SFT for a couple of causes:
- A number of coverage evaluations — PPO wants to take care of each the present coverage (mannequin weights) and an “previous” coverage to compute the chance ratio between them. This successfully doubles the mannequin parameters in reminiscence.
- Expertise buffer — PPO shops a buffer of experiences (states, actions, rewards, and so forth.) to carry out updates in mini-batches. This buffer could be fairly massive and takes vital reminiscence.
- Benefit estimation — Computing benefits requires preserving observe of worth estimates and returns throughout trajectories, including one other layer of reminiscence overhead.
- Extra optimisation targets — PPO tracks a number of loss parts (coverage loss, worth loss, entropy bonus) and their gradients, whereas SFT has a single loss.
Due to the above, we’re extra restricted than SFT within the dimension of the fashions we are able to prepare and the way a lot it prices. Whereas the above coaching I may do on an A100 40GB in Colab, for the PPO coaching I wanted extra reminiscence. I educated on an H100 80GB, which may prepare a LoRA with a rank of 128 and a batch dimension of 8.
My hyperparameter sweep was slim, I went with what appeared most intuitive utilizing batch sizes starting from 1 to 16 and studying charges from 2e-5 to 2e-4.
The mannequin made no enhancements to the duty. The textual content file with the outcomes is right here.
I tracked numerous coaching metrics utilizing Weights & Biases (WandB). Key metrics included the coverage loss, worth loss, complete loss, KL divergence, and the reward mannequin’s rating.
For all hyperparameter runs, the logs no enchancment within the rewards and loss calculated over time. The KL divergence remained throughout the pre-defined threshold.