Spectral Rewiring for Exploration, Purification, and Model Merging
Abstract
Reinforcement learning has become a standard post-training recipe for large language models, but dense full-parameter updates create two deployment-relevant bottlenecks: suppressed reasoning performance, often reflected by premature saturation of test-time scaling, and interference when consolidating multiple capabilities through multi-domain training or model merging. We show that the reasoning-effective component of these updates is largely concentrated in the base model's spectral space, motivating Subspace-Aligned Rewiring (SAR), a post-hoc editing method that retains this spectral core while removing orthogonal components. SAR therefore preserves reasoning gains and filters residual update directions that suppress performance or amplify cross-domain interference. Across several model families and scales, SAR extracts compact reasoning cores using as little as approximately 0.58% of total parameters: it preserves over 99% of post-training performance and improves high-k exploration in mathematical reasoning, and generalizes to agentic coding by improving six of seven open benchmarks on an in-house model. SAR also purifies mixed-domain training updates by releasing suppressed coding capability while maintaining math reasoning and instruction following. It further enables model merging across experts, yielding cross-domain generalization that surpasses previous merging baselines and even the best single-domain experts. Overall, SAR shows that extracting reasoning-effective updates from parameter geometry can serve as a training-free mechanism to improve reasoning and multi-domain performance.
Community
Reinforcement learning has become a standard post-training recipe for large language models, but dense full-parameter updates create two deployment-relevant bottlenecks: suppressed reasoning performance, often reflected by premature saturation of test-time scaling, and interference when consolidating multiple capabilities through multi-domain training or model merging. We show that the reasoning-effective component of these updates is largely concentrated in the base model's spectral space, motivating Subspace-Aligned Rewiring (SAR), a post-hoc editing method that retains this spectral core while removing orthogonal components. SAR therefore preserves reasoning gains and filters residual update directions that suppress performance or amplify cross-domain interference. Across several model families and scales, SAR extracts compact reasoning cores using as little as approximately 0.58% of total parameters: it preserves over 99% of post-training performance and improves high-k exploration in mathematical reasoning, and generalizes to agentic coding by improving six of seven open benchmarks on an in-house model. SAR also purifies mixed-domain training updates by releasing suppressed coding capability while maintaining math reasoning and instruction following. It further enables model merging across experts, yielding cross-domain generalization that surpasses previous merging baselines and even the best single-domain experts. Overall, SAR shows that extracting reasoning-effective updates from parameter geometry can serve as a training-free mechanism to improve reasoning and multi-domain performance.
RL post-training works incredibly well, but its parameter update is still a black box.
🧐Our question: can we understand and locate the effective part of an RL update, remove the noisy directions, and even improve the model after training?
We find the answer is yes.
📌The reasoning-effective part of an RL update can be represented as a compact rewiring matrix in the base model’s spectral space.
🤖 This leads to SAR: a training-free post-hoc method that projects the raw RL update onto this compact reasoning core, enabling us to understand, purify, and merge RL-trained models.
Key results:
1️⃣ The reasoning core of RL is highly compact:
With less than 1% spectral parameters, SAR can recover or improve full-RL gains.
2️⃣ Dropping noisy directions helps:
For math, SAR solves the exploration degradation often observed after RL. It also improves agentic coding on large-scale in-house models.
3️⃣ SAR purifies mixed-domain RL:
On a 32B model jointly trained for math, code, instruction following, and chat, SAR improves coding and math exploration while keeping instruction following stable.
4️⃣ SAR makes model merging stronger:
After SAR purification, merged models can surpass the best single-domain experts. We observe the same trend on production-scale in-house models.
Overall, SAR is training-free, broadly useful, and gives us a new geometric lens on what RL is really changing inside reasoning models.
Awesome paper by Zhilong Zhang et al.! Thanks for sharing! 🚀
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- ResMerge: Residual-based Spectral Merging of Large Language Models (2026)
- CARE-RL: Capability-Aware Reinforcement Learning for Mitigating Cross-Domain Conflicts (2026)
- Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging (2026)
- A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL (2026)
- Essential Subspace Merging for Multi-Task Learning (2026)
- On the Geometry of On-Policy Distillation (2026)
- You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
If you're running multiple fine-tuned LoRAs or adapters on the same base model, you've felt the interference problem — merge two good things and get something worse than either. This paper's claim is that the useful part of any update lives in a narrow spectral band of the base model's weights, so you can surgically apply edits without the cross-talk. I'd want to see how this holds up when the fine-tunes are on very different domains (code vs. creative writing, say) where the spectral overlap might be thinner. The real win would be shipping one model that can swap capabilities without reloading.
Get this paper in your agent:
hf papers read 2607.03065 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper