laion/whisper-tiny-base-emotion-voice-burst

Whisper Tiny and Base: emotion, voice, quality, speaker and vocal-burst regression

LAION · encoder-only Whisper Tiny and Base · 192 scalar outputs, two speaker-vector heads and a 20 ms vocal-burst head. These checkpoints are trained models, not an ASR decoder. The interactive 30-clip audio gallery explains raw scores and training-reference z-scores and plays the example audio directly. See also the independent evaluation report.

Files and quick start

File	Purpose
tiny/model.safetensors, base/model.safetensors	Fine-tuned encoder plus all heads, in safe tensor format.
tiny/config.json, base/config.json	Original OpenAI Whisper architecture configuration.
tiny/preprocessor_config.json, base/preprocessor_config.json	Whisper 16 kHz, 80-bin log-mel feature extraction.
training_normalization.json	Exact mean and standard deviation used inside the trained score head.
display_normalization.json	Training-reference median and standard deviation for interpretable z-scores.
model.py, inference.py, fast_inference.py	Architecture, reference inference and optimized persistent batch inference.
training/	Original data adapters, model and S1→S10 training source. Cluster paths require adaptation to another environment.
training/curriculum_mix_schedule.json	Exact stage sizes and P3 exposure schedule.
evaluation/	Held-out P3 test/validation results and detailed English HTML report.

pip install -r requirements.txt
python inference.py example.wav --model-dir tiny > tiny_prediction.json
python inference.py example.wav --model-dir base > base_prediction.json

# One GPU with four CPU decode/resample workers; JSON Lines output:
python fast_inference.py clips/*.wav --model-dir base --device cuda \
  --decode-workers 4 --batch-size 8 > base_predictions.jsonl

# One CPU core, persistent model and one-thread feature extraction:
OMP_NUM_THREADS=1 python fast_inference.py clips/*.wav --model-dir tiny \
  --device cpu --cpu-threads 1 --decode-workers 0 --batch-size 1 \
  > tiny_cpu_predictions.jsonl

# Large corpus: one path per line in clips.txt, streamed without loading the
# full manifest into RAM. Store repeated field names once in schema.json:
python fast_inference.py --manifest clips.txt --model-dir base --device cuda \
  --decode-workers 4 --batch-size 8 --compact --schema-out schema.json \
  > base_predictions_compact.jsonl

# Throughput-oriented 15-second PCM16 WAV batch on one GH200 / four CPU cores:
python fast_inference.py --manifest wav_paths.txt --model-dir base --device cuda \
  --decode-workers 4 --batch-size 32 --compact --schema-out wav_schema.json \
  > wav_predictions.jsonl

inference.py accepts a 0.1–30 second audio file supported by soundfile, converts it to mono 16 kHz, and returns all raw scalar predictions, display z-scores, Orange vectors, frame probabilities and thresholded vocal-burst intervals. Use --device cpu if needed. The Python function predict(audio_path, model_dir) returns the same dictionary. tiny/ and base/ must stay beside the two normalization JSON files.

fast_inference.py loads the model once, accepts many files or a streamed --manifest, buckets nearby durations to reduce padding and emits one JSON object per line in input order. On GPU it computes the Whisper STFT and mel filters on the GPU, keeping the features device-resident; four CPU threads can decode and resample the next bounded chunk while the current chunk runs. On CPU it keeps one model and feature extractor resident and uses the requested CPU thread count. The program bounds audio memory by processing chunks of four batches. --compact omits the same 192 score names and z-formula from every row and requires a separate --schema-out JSON. For already decoded 16-kHz mono NumPy arrays, call FastWhisperSession(model_dir, device='cuda').predict_waves(waves) directly to skip file decoding. The GPU frontend is checked against the reference output before release.

The script uses trained normalization to turn network outputs back into the teachers' raw units, then display normalization to provide context:

raw_score = network_score × training_std + training_mean
display_z = (raw_score − training_teacher_median) / training_teacher_std

For example, Interest has a high raw baseline in the training corpus. A z-score near zero means the prediction is ordinary for Interest, even if its raw number exceeds most other emotion scores. Compare emotion ranks by z; keep the raw numbers when you need the original teacher scales. A z-score is not a probability, confidence interval, or human preference. High noise or anger z is context dependent. The gallery and the JSON include both numbers for every dimension.

characters per second (CPS) is not a direct model output. Where a transcript exists, the gallery computes transcript characters divided by audio seconds; for P3 examples it shows a stored metadata teacher estimate. It reports those values as raw and z using the P3 training teacher CPS distribution, clearly separated from the learned vn_TEMP_reg VoiceNet tempo score and voiceclap_attribute:talking_speed. CPS is script dependent across languages. The model does not transcribe audio: it has no Whisper decoder or ASR text head.

Output heads and architecture

Only the pretrained encoder from openai/whisper-tiny or openai/whisper-base initializes the model. The decoder is absent. The entire encoder is fine-tuned. Masked time pooling concatenates mean, minimum, maximum and standard deviation over valid encoder frames. Separate layer-normalization → linear(4d, 512) → GELU → linear heads predict 192 scalar targets, an Orange timbre vector (128D) and an Orange identity vector (250D). A separate convolutional frame head predicts vocal-burst logits at a 20 ms hop. Both Orange vectors are unit normalized.

Scalar indices	Target family	Count	Target provenance
0–39	Empathic Insight emotion intensities	40	Ladder, also P3 sidecars
40–96	VoiceNet delivery, style and voice dimensions	57	Ladder, also P3 sidecars
97–99	Genuineness, vocal-burst blend, legacy R_quality	3	Ladder; first two also P3
100–118	Empathic Insight Plus extra dimensions and quality scores	19	P3 sidecars
119–122	Meta AudioBox CE, CU, PC and PQ	4	P3 sidecars
123–129	DNSMOS raw and calibrated SIG, BAK, OVRL plus P.808 MOS	7	P3 sidecars
130	log(1 + vocal-burst count)	1	Ladder where present, and P3
131–191	VoiceCLAP attribute teacher scores	61	P3 sidecars

The score ordering is fully specified in training_normalization.json. R_quality was supervised on Ladder but lacks valid P3 held-out test labels; therefore its P3 test metric is unavailable. The raw VoiceCLAP 768D and Empathic Insight/BUD-E 3072D embeddings were stored as teacher sidecars but are not model outputs. Orange timbre and identity vectors are model outputs.

Data and optimization

The main speech corpus is LAION TTS Scaling Ladder DE/EN. Training used the complete local nested S1→S10 indexing, including the internally available larger tiers, with one curriculum pass per stage in order S1, S2, …, S10. A frozen 1,120,000-clip P3 synthetic vocal-burst collection supplied multi-teacher sidecars. Its source-disjoint split contained 1,031,454 train, 10,000 validation, 9,693 test and 68,853 excluded-overlap clips. The P3 training split was mixed at 10% of every stage; total P3 exposures were 7,193,474 out of 71,934,743 sample exposures. P3 examples may be reused between stages. The detailed schedule is included as JSON.

Stage	Ladder exposures	P3 exposures	Total
S1	34,112,433	3,790,270	37,902,703
S2	17,135,901	1,903,989	19,039,890
S3	6,666,998	740,778	7,407,776
S4	3,425,099	380,567	3,805,666
S5	1,764,990	196,110	1,961,100
S6	912,789	101,421	1,014,210
S7	382,768	42,530	425,298
S8	197,662	21,962	219,624
S9	100,858	11,206	112,064
S10	41,771	4,641	46,412

Each model was trained separately from its original OpenAI initialization on four JUPITER Booster nodes / 16 GH200 GPUs, with global microbatch 128 (8/GPU) and gradient accumulation 2 (effective batch 256). AdamW weight decay was 0.01. The peak encoder learning rate was 2 × 10⁻⁵, and the head learning rate 3 × 10⁻⁴. There were 5,620 warmup updates (2%) followed by one continuous cosine decay across 281,002 total optimizer updates, without restarts, to 10% of each peak learning rate. BFloat16 autocast and gradient norm clipping at 1.0 were used. Each training completed in about 11 h 52 min of wall time on 16 GPUs.

Scalar targets were trained with per-family masked Huber loss (delta=1) after the exact training_normalization.json mean/SD normalization. Missing teacher targets were masked, never filled with zeros as labels. The frame head used masked binary cross entropy (positive weight 5) plus soft Dice loss. Orange speaker heads used cosine distance plus a small Huber term on normalized teacher vectors; single-speaker validity controlled the mask. VoiceCLAP and BUD-E sidecars remained available for later use but were not direct embedding heads.

The original run scripts in training/ document the reader, loss weights, sampler, stage schedule, validation and checkpointing. The training code assumes JUPITER local data paths; update those paths to reproduce training elsewhere. The included .safetensors files contain all learned weights required for inference; optimizer state is not included.

Evaluation

The quantitative report uses 9,693 P3 test clips whose source components were separated from the P3 training and validation splits. The same deterministic protocol was used for both checkpoints. Values below are test results; the 30-clip gallery is selected for interpretation and is not used to estimate performance.

Measure on P3 test	Tiny	Base
Emotion mean normalized MAE, 40 dimensions ↓	0.265	0.247
VoiceNet mean normalized MAE, 57 dimensions ↓	0.241	0.225
AudioBox mean normalized MAE, 4 dimensions ↓	0.219	0.207
DNSMOS mean normalized MAE, 7 dimensions ↓	0.245	0.237
Frame vocal-burst F1 at threshold 0.5 ↑	0.950	0.960
Vocal-burst event F1, IoU ≥ 0.5 ↑	0.902	0.936
Burst count MAE ↓	0.189	0.166
Orange timbre cosine, 6,809 valid clips ↑	0.868	0.892
Orange identity cosine, 6,809 valid clips ↑	0.829	0.850

The HTML report gives per-dimension n, teacher dispersion, MAE, normalized MAE, Pearson r and R², plus frame and event metrics, onset/offset errors, and speaker cosine distributions. The source-disjoint test is still synthetic P3 audio with teacher-generated labels. These numbers do not establish performance on unrelated real-world corpora or agreement with human judgments.

Inference throughput: one GH200 with four CPU cores, or one CPU core

The optimized fast_inference.py keeps a model loaded between clips, computes the 80-bin Whisper mel spectrogram on the GPU, groups similar durations into batches and can decode the next bounded chunk with four CPU threads. The CPU-only path uses one thread and the reference Whisper frontend. Both are encoder-only inference; no ASR decoding runs. The same trained weights and heads are used in every measurement.

The following 15-second PCM rates include mel extraction and model forward, but exclude compressed-file decoding, output conversion and model loading. Measurements used one GH200, four allocated CPU cores, BF16 GPU autocast, one PyTorch CPU thread, and seven timed repeats after warmup. The PCM waveform was already in memory. "Reference" used CPU mel extraction; "fast" used GPU mel extraction. CPU rates also use the resident model.

Model	1 CPU core, batch 1	1 CPU core, batch 2	GH200 reference, batch 8	GH200 fast, batch 1	GH200 fast, batch 8	GH200 fast, batch 16	GH200 fast, batch 32
Tiny	4.38/s	4.46/s	228.5/s	207.6/s	1,496.6/s	2,770.7/s	3,661.4/s
Base	1.92/s	1.95/s	225.4/s	166.2/s	1,205.5/s	2,082.0/s	2,326.5/s

GPU mel extraction raises batch-eight PCM throughput by 6.5× for Tiny and 5.3× for Base versus the prior CPU-mel reference. Larger batches improve saturated-GPU throughput, subject to GPU memory and latency requirements. On one CPU core, batching yields only about 1–2% more throughput; the encoder forward pass dominates. Use the persistent session to avoid reloading weights for each clip.

Compressed-file throughput is much lower because the CPU must read, decode and resample audio. A separate end-to-end measurement reused 64 cached copies of one 12-second Opus clip, batch eight, and included file decoding, mel extraction, model forward and conversion of all outputs to CPU/JSON-compatible values. It excluded JSON serialization, disk writing and model loading. Results are therefore a controlled decoder benchmark, not a prediction for a heterogeneous production corpus.

Model	GH200 + serial CPU decode	GH200 + 4 CPU decode workers	1 CPU core, batch 1	1 CPU core, batch 4
Tiny	24.8/s	94.9/s	4.70/s	4.84/s
Base	24.7/s	94.5/s	2.30/s	2.35/s

With four decode workers, compressed-audio decoding is the measured bottleneck, so the two model sizes have similar end-to-end rates on this clip. Faster source decoding, larger preprocessing parallelism on a separate host, or predecoded PCM would allow more of the GPU throughput above to be used. Keep four workers per GPU when constrained to four CPU cores. Exact results and reproduction code are in evaluation/fast_inference_benchmark.json and training/benchmark_fast_inference.py.

For a more realistic WAV-to-annotation measurement, we used 1,024 distinct, cached 15-second PCM16 mono WAV files on the GPU node's local /tmp, with four CPU decode/read workers. Every prediction was serialized to compact JSONL and written to /e/scratch; model loading and WAV creation were excluded. The source waveform content was repeated across files. Rates are medians of three timed passes.

15-second WAV → compact JSONL	Batch 8	Batch 16	Batch 32
Tiny	628/s	768/s	846/s
Base	567/s	724/s	770/s

Thus more than 1,000 annotations/s was measured only for already decoded PCM in the mel+forward test, not for full 15-second WAV-to-JSONL annotation with four CPU cores. Shorter clips, a different storage layout or a leaner output format can change the rate; these values should not be extrapolated to 30-second WAVs or millions of separate files on shared storage. The individual WAV benchmark and its exact scope are in evaluation/wav_file_benchmark.json and training/benchmark_wav_files.py.

The optimized CPU path matched the reference predictions within 1.1 × 10⁻⁵ raw score units and reproduced burst intervals and vectors in the verification clip. GPU mel extraction changed predictions slightly because of numerical frontend and BF16 differences: maximum raw-score difference was 0.014 across the tested Tiny and Base clips, and burst intervals were identical. The mixed-duration batch check is recorded in evaluation/mixed_batch_verification.json. Dynamic INT8 CPU quantization improved raw speed in a probe but changed some scores substantially, so the released CPU path stays FP32; see evaluation/cpu_quantization_probe.json. The earlier single-clip reference measurements and forward-only timings remain in evaluation/inference_benchmark.json.

Display normalization and gallery construction

display_normalization.json records per-target name, index, n, median, mean, std and reference population. Indices 0–99 use all 100,858 S9 Ladder t0+t1 training examples. Indices 100–191 use a seeded, 100,000-example subset sampled without replacement from the P3 train split only. Its auxiliary_statistics also records the P3 train teacher CPS distribution for the gallery's metadata-derived CPS display. Missing values are ignored per target. The JSON provides the exact formula and sampling seed. This is an interpretation layer, not a checkpoint retrain or a replacement for the original mean/SD normalization.

The public gallery embeds 12 generated-study clips, 12 seeded Emolia/Emilia clips in German, English, French, Japanese, Korean and Chinese, and six teacher-selected P3 test positives with known inserted burst intervals. Amusement, Anger, Surprise, Fear, Sadness and Triumph guided selection of the six examples. Selection by teacher label makes those six curated, even though their P3 source components are test-disjoint. The full source manifest and model predictions are in gallery_data/.

License, provenance and attribution

The LAION fine-tuned weights, original release code, new documentation, and gallery presentation are released under Creative Commons Attribution 4.0. Attribute LAION and link this repository when using them. The starting OpenAI Whisper code and weights are MIT-licensed; their copyright notice and terms continue to apply to the original components. The Hugging Face Transformers and other dependencies retain their own licenses.

Training data and embedded gallery audio do not all acquire CC BY 4.0 through this model release. The Scaling Ladder card documents a mix of CC BY 4.0, CC0 and source-specific Eurospeech terms. The synthetic P3 mixtures include material from Emolia, Common Voice, Multilingual LibriSpeech, LAION Vocal Bursts Clean, TTS-AGI Vocal Bursts and, for some clips, AudioSet with Captions. Source audio retains its applicable original terms and attribution. gallery_data/provenance.json gives source identifiers for the six P3 examples. The generated-study audio was supplied as part of this LAION experiment; Emolia examples come from the cited corpus. Audio is included in the gallery for inspection, not relicensed wholesale.

Citation

@misc{laion_whisper_multitask_2026,
  title = {Whisper Tiny and Base for Emotion, Voice, Quality and Vocal-Burst Regression},
  author = {{LAION}},
  year = {2026},
  url = {https://huggingface.co/laion/whisper-tiny-base-emotion-voice-burst}
}