tahaenesaslanturk/Qwen3-ASR-0.6B-weight-space-programming

Weight-space programming on Qwen3-ASR-0.6B

By Taha Enes Aslantürk, with Anthropic Claude agents (see "AI assistance").

· Paper (PDF): paper/main.pdf · Preprint record: https://doi.org/10.5281/zenodo.23976685

Replication-style experiment: can small, analytically derived edits to the weights of a speech-recognition model (no gradients, no fitting) fix a concrete failure? The target here is a language-ID failure of Qwen/Qwen3-ASR-0.6B: on audible speech surrounded by long digital silence the model emits language None<asr_text> (the transcript is often fine, the language tag says "no speech"; 30 of 40 such clips in the never-seen test stress set).

Credit

The method, the rules and the idea come from "Sparks of Weight-Space Programming" by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab (MIT CSAIL, Nov 2026): https://rikiyat.github.io/blog/weight-space-programming/ . This repository is an independent re-implementation on a different model and task; it is not affiliated with the authors. If you use anything here, cite the original work.

Model: Qwen3-ASR-0.6B (Qwen team). Data: Google FLEURS (CC-BY-4.0). See "License" below.

What was done

Nine independent agent "arms" (three forward-pass budgets x three replicates: b10, b25, b50, b10r2, b25r2, b50r2, b10r3, b25r3, b50r3) each diagnosed the failure on the practice split, localised it, and produced a pure weight edit (edits/<arm>.pt) plus five norm-matched random-direction controls (edits/<arm>_control_s{1..5}.pt; the three original arms have a differently named seed-1 file, see docs/REPRODUCIBILITY.md). Per-arm summary and cross-arm convergence: results/arms_summary.md. Raw arm reports: work/<arm>/REPORT_*.

Layout

PLAN.md                  design, decoding, data, budget rules, deviations
config/decoding.json     FROZEN decoding settings (fp32, greedy, max_new_tokens=512, batch 8)
data/                    manifests (data/manifests/*.jsonl), builders, FLEURS recipe (audio not tracked)
  build_data.py          practice/test/hard sets (seed 0), STRESS_RECIPE.md + build_stress.py
  download_fleurs.sh     raw FLEURS tarballs
wsp/                     harness: model loading + edit application (model.py), forward-pass budget (budget.py), GPU gate (__init__.py)
eval/                    metrics.py (frozen normalisation), runners, run_test.py (single-use test), stats scripts
edits/                   final edits <arm>.pt/.json, <arm>_make.py, <arm>_stats.*, controls <arm>_control_s*.pt/.json
work/<arm>/              per-arm scripts, logs, validation jsonl, REPORT_* (large captures are git-ignored)
results/                 practice and test outputs, tables, arms_summary.md
logs/forward_passes.jsonl  every budgeted model call (run, purpose, clip ids)
notes/                   blog summary, model internals, practice error analysis
docs/REPRODUCIBILITY.md  versions, hardware, seeds, known nondeterminism

Reproduce

Environment (Python 3.11, uv venv; exact pins in requirements.lock):

uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python -r requirements.lock

Data (needs network; FLEURS train tarballs are truncated to the first 640 MB per language to save bandwidth, see data/README.md):

bash data/download_fleurs.sh          # edit the hard-coded paths first; writes data/raw_fleurs/
data/.venv-data/bin/python data/build_data.py
data/.venv-data/bin/python data/build_stress.py --split practice --seed 1   # test: --split test --seed 2

Baseline (unmodified model; writes results/practice/baseline):

.venv/bin/python eval/run_practice_phase0.py
.venv/bin/python eval/run_practice_stress.py

Apply an edit:

from wsp.model import load_model, transcribe          # see wsp/model.py for the transcribe signature
wm = load_model(weights_override="edits/b25r3.pt")      # path to an edits/*.pt, a dict, or a callable

Edit files are {"mode": "replace"|"delta", "tensors": {param_name: tensor}, "metadata": {...}}. Each edits/<arm>_make.py regenerates its edit from the original weights (some need the saved stats file, see the arm's .json). Control generators: edits/<arm>_control_make.py --seed S.

Single-use test evaluation (refuses to overwrite an existing results dir; do not run casually, the test set is meant to be used once per model name):

.venv/bin/python eval/run_test.py --model-name b25r3 --edit edits/b25r3.pt

Rules of the experiment

• No gradients, optimisers, fine-tuning, LoRA or distillation. No fitting of weights; directions come from weight analysis and simple contrastive activation statistics (difference of means), edits are analytic (rank-1 projections, gains on selected heads/neurons). Choosing among a handful of hand-picked scales is allowed and documented per arm.
• Decoding is frozen (config/decoding.json); prompt, language hint, max_new_tokens and attention implementation are not changed. thinker.model.embed_tokens (tied to the output head) is not edited. The final model must be a pure weight change (hooks were allowed only for localisation).
• Forward-pass budget: every call on practice audio (<= 64 clips) goes through wsp/budget.py, is logged in logs/forward_passes.jsonl, and raises when the arm's budget (10 / 25 / 50) is exhausted. CPU weight analysis is free. The Phase-0 baseline transcription of the practice sets is not charged. Passes that were charged but produced no output (crashes, OOM, stops) are listed per arm in results/arms_summary.md.
• Test data is single-use: the harness refuses test clips except via eval/run_test.py, which refuses to overwrite an existing results/test/<name>. Arms never saw test data.
• Controls: random edits of the same tensors, same form, same number of changed weights and matched Frobenius norm.

Safety notes

• The Mac crashed once when about nine GPU processes ran concurrently. wsp/__init__.py now acquires a machine-wide GPU slot on import (default 3 concurrent processes, WSP_GPU_SLOTS, lock files in logs/gpu_slots/) and caps each process to a fraction of MPS memory (WSP_MPS_FRACTION, default 0.25). Large evaluations can hit "MPS backend out of memory" under that cap; raise the fraction only when running alone. WSP_NO_GPU_GATE=1 disables the gate; do not do that with parallel jobs.
• Do not run edit evaluations in parallel without the gate. fp32 on MPS needs roughly 27 GB peak for the full test run.

Results

Full write-up: REPORT.md (paper-ready draft). Tables: results/final_table.md, results/paired_stats.md, results/replication_summary.md. Numbers below are from the single-use test run (never seen by any arm).

• Baseline: clean macro WER 11.82 (6 languages, ja = CER). 30 of 40 stress target clips (audible speech followed or surrounded by long digital silence) get language None; 5 of 30 hard trailing-silence clips have a wrong or None tag.
• All 9 arms reduce it, with 0 newly broken target clips. McNemar p < 0.01 on the combined 70 target clips for every arm (fixed 9 to 36 of 70). The real edit fixes more clips than each of its 5 random-direction controls in 26 of 27 comparisons (tie: b50r3 on the 30 hard trailing-silence clips).
• Cost: clean macro WER +0.24 to +0.40 pp for b10r3 (+0.32), b25r3 (+0.31), b50 (+0.40) and b50r2 (+0.24), with 95% CIs excluding 0; the other arms are within noise. b10r2 is the weakest arm (9 of 70 fixed) and makes 3 of 40 silence/noise clips non-empty.
• Convergence: 6 of 9 arms edited the same decoder heads (L20H13 and/or L21H2). Others edited MLP neurons (b10r2, b25) or the audio-encoder k_proj (b50). Edit sizes range from 2,048 weights (0.0003%) to 0.92%.
• Key caveat: the qwen-asr transformers backend never applies the audio encoder's ~8 s window mask on the sdpa/MPS path (_prepare_attention_mask is never called). Restoring it with no weight change (baseline_windowed, a separate analysis) fixes 31 of 40 stress and 5 of 30 hard-TS target clips, about as good as the best edit; edits applied on top add at most one more clip. The pathology is largely a backend artifact, so the practical value for FlashAttention-2 users is probably small (untested). Do not read this as "editing improves ASR".
• Negative result: bilingual code-switching (Target B) was not fixed by any arm.

The practice-split numbers in results/arms_summary.md are in-sample for most arms and should not be read as test performance.

License

Code, edit files (edits/), logs and reports: Apache-2.0 (see LICENSE and NOTICE). This matches the base model's license: Qwen/Qwen3-ASR-0.6B is Apache-2.0, and the files in edits/ are modified derivatives of its weights (small replace/delta tensors, not the full model). Transcripts and clip ids derived from FLEURS (data/*.jsonl, results/) are CC-BY-4.0 (Conneau et al., 2022); no FLEURS audio is redistributed.

AI assistance

Done by Taha Enes Aslantürk with Anthropic Claude agents: Claude Opus 5.5 performed the weight-editing arms and orchestration/review, Claude Sonnet 5.5 built the harness and evaluation and drafted the report. Details in REPORT.md ("AI assistance").

Citation

@misc{aslanturk2026wspqwen3asr,
  title  = {Weight-Space Programming on Qwen3-ASR-0.6B: nine independent agent edits, random-direction controls, and a missing attention mask},
  author = {Aslant{\"u}rk, Taha Enes},
  year   = {2026},
  doi    = {10.5281/zenodo.23976685},
  url    = {https://doi.org/10.5281/zenodo.23976685},
  note   = {Preprint. Experiments and drafting with Anthropic Claude agents}
}

Please also cite the original work: Takehi, Teoh, Zhang and Khattab, "Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights", 2026, https://rikiyat.github.io/blog/weight-space-programming/ .