Mel-Band RoFormer Duality student

Musetric no longer runs this model: since musetric/musetric#1157 its separation model is musetric/mel-band-roformer-duality-kara2-student. This repository stays for the builds that pin it.

A vocals / instrumental separation model distilled for Musetric from Aname's Mel-Band RoFormer Duality. It keeps the teacher's band split, its six pairs of time and band transformers and its mask estimator, in a shape that runs 3.4 to 3.5 times faster on phone GPUs, and separates vocals 0.80 dB below the teacher on MUSDB18-HQ.

These are the PyTorch weights in fp32 and their config: the original model, from which the ONNX core musetric/mel-band-roformer-duality-student-onnx is exported. The class that reads them is PooledMelBandRoformer of musetric/lab.

Intended uses & limitations

Intended:

  • Vocals / instrumental separation of songs, 44.1 kHz stereo, on 11 s windows as the teacher.
  • A source for exports to ONNX and for further distillation.

Limitations:

  • 0.80 dB below the teacher in the median SDR of the vocals on MUSDB18-HQ, 1.13 dB in the mean; on 13 of the 50 test tracks a 10 s window is more than 3 dB below the teacher. In earlier students of this shape such windows lost more of the loud voice below 4 kHz than the teacher.
  • Where the teacher lets a faint piece of an instrument into the vocals, the student can make it louder. Training against it made it rarer, not gone.
  • Trained and checked on 11 s windows (1101 STFT frames, hop 441).

How to use

With the lab's separator (reads the file, separates in 11 s windows every 8 s, writes both stems):

from pathlib import Path

from musetric_toolkit.separate_audio.mel_band_roformer_separator import MelBandRoformerSeparator

separator = MelBandRoformerSeparator(Path("model.safetensors"), Path("config.yaml"), sample_rate=44100)
separator.separate_audio("song.flac", "vocals.wav", "instrumental.wav")

Or the model alone:

import yaml
from safetensors.torch import load_file

from musetric_toolkit.separate_audio.roformer.pooled_mel_band_roformer import PooledMelBandRoformer

config = yaml.safe_load(open("config.yaml"))
model = PooledMelBandRoformer(**config["model"]).eval()
model.load_state_dict(load_file("model.safetensors"))
vocals = model(audio)  # audio: float32 [batch, 2, samples] at 44.1 kHz, an 11 s window

model.net_forward(stft_repr) is the neural network core alone, the same boundary as MelBandRoformer.net_forward: a real STFT [batch, 2050, frames, 2] in, per-bin complex masks [batch, 1, 2050, frames, 2] out.

Files

File Size SHA256
model.safetensors 420,581,272 B 1cf7f6892980477363d34ffd1319cd209a009c2e4f352705af2122ce977536c3
config.yaml 1,109 B 14f932600e40ec4fd5b221e6f9f72111e1bbb4dac7b7ec3376d91b246116bd14
training-data.csv 650,647 B fee1af7da3bf51c70115695d828a1d8970e0fff33a9597d984d5e5d94a1bc59c

The shape

105.1 M parameters against the teacher's 228.2 M. Against the teacher:

  • Pooled pairs. Pairs 0-1 run at full rate, pairs 2-3 on frames averaged by 4, pairs 4-5 by 2. A pooled pair adds its change back at full rate, x + up(F(down x) - down x), and its time positions are stretched by the same factor.
  • Merged high bands. In every pair the bands from 44 up run merged in neighbouring pairs, 52 band tokens instead of 60, each merged band at the mean position of the two.
  • Pairs 0-1. Time attention sees 32 frames on each side with 4 of the 8 heads (its output projection refit with a bias), the feed-forward is 768 wide instead of 1536, and the band transformer is a residual linear map.
  • Detail paths. Pairs 2-3 add a linear map of the detail their pooled view misses: the stream minus that view, and the view.
  • Mask estimator. Per band the teacher's first and last layers; the 1536 x 1536 middle layer becomes rank 128, and the estimator runs on pairs of frames, its masks repeated.

The time per 11 s window of this shape, exported with the teacher's recipe and timed with stand-in weights of the same shapes: 3.51x faster than the teacher on a Galaxy S24 Ultra (Adreno 750) and 3.39x on a OnePlus 9RT (Adreno 660), in Chrome on WebGPU.

Training

  • Start. The teacher's weights; each narrowed part from a fit to the teacher on the training data (the kept feed-forward units, the pruned heads, the low-rank middle of the mask estimator, the detail paths and the band maps).
  • Distillation, about 15 hours of training in all, counting the runs that were averaged. The target is the teacher's vocals on the training songs (multi-resolution STFT L1, plus the teacher's masks), with a term against vocal bleed in pauses. Half of the remixes are voiced by true a cappellas over instrumentals of five genres of the corpus, with the true voice as the target and a term against output louder than the voice. In the last run the teacher's target is set to zero in the 50 ms frames where it is 30 dB or more under its own sung level and under the mix: there it holds only the teacher's faint leaks. The final weights are the average of three runs from one start, trained 2 hours further.
  • Data. 2,829 openly licensed tracks, 48.8 h: 2,155 30 s clips of the FMA dataset (CC BY, CC0, Public Domain Mark) as songs and instrumentals, 656 a cappellas from ccMixter (CC BY 2.5, 3.0 and 4.0, CC0), and 18 songs from archive.org, ccMixter, Jamendo (Jam-ALT) and joshwoodward.com (CC BY). No NonCommercial, NoDerivatives or ShareAlike material. Every track, with its artist, license and source, is in training-data.csv.
  • Not trained on: MUSDB18-HQ, which is used for evaluation only.

Evaluation

MUSDB18-HQ test, 50 tracks, both models with the same code: fp32, 11 s windows every 8 s, Hamming-weighted overlap-add. SDR of the vocals against the true stems over whole tracks; bleed is the level of the vocals in the pauses of the true voice against its loud parts, the median and the 95th percentile over tracks.

Teacher Student
SDR, median 12.09 dB 11.29 dB
SDR, mean 11.92 dB 10.79 dB
Bleed in pauses -76.5 / -59.0 dB -85.6 / -66.5 dB

Per track the student is 1.13 dB below the teacher on average (95 % interval 0.95 to 1.30 dB) and better than it on 2 of the 50 tracks.

The work is described in musetric/musetric#1081.

Source & lineage

License & citation

Apache-2.0, as the teacher's weights it starts from. We do not claim authorship of the teacher; the compression and the training are ours.

@article{wang2023melbandroformer,
  title={Mel-Band RoFormer for Music Source Separation},
  author={Wang, Ju-Chiang and Lu, Wei-Tsung and Won, Minz},
  journal={arXiv preprint arXiv:2310.01809},
  year={2023}
}
Downloads last month
46
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for musetric/mel-band-roformer-duality-student

Finetuned
(2)
this model
Quantizations
1 model

Paper for musetric/mel-band-roformer-duality-student