---
base_model: Qwen/Qwen3.6-27B
tags:
- text-generation
- lora
- peft
- svd
- weight-diff
- qwen3.5
- qwen3.6
- uncensored
- abliterated
license: apache-2.0
language:
- en
- zh
pipeline_tag: text-generation
---

# ThinkingCap Qwen3.6-27B LoRA (SVD Weight-Diff Extraction)

LoRA adapter extracted via **weight-diff SVD** from:

| Role | Model |
|------|-------|
| Source (fine) | [`huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated`](https://huggingface.co/huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated) |
| Base | [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B) |

The delta `W_fine - W_base` is compressed with **randomized SVD** (rank 16),
producing a standard PEFT LoRA (`lora_A`/`lora_B`, BF16). Applying it to the
base model reproduces the ThinkingCap + abliteration behavior change without
any training data.

## Target Modules

Qwen3.6-27B is a **hybrid `qwen3_5`** model (64 layers, `Qwen3_5ForConditionalGeneration`):

- `self_attn.q_proj/k_proj/v_proj/o_proj` — 16 full-attention layers (3, 7, 11, ..., 63)
- `mlp.gate_proj/up_proj/down_proj` — all 64 layers
- Linear-attention tensors (`linear_attn.*`) are intentionally **not** targeted
  (not compatible with standard PEFT/llama.cpp LoRA)

## Quality (rank 16)

| Family | count | avg \|Δ\|/base | energy captured |
|--------|-------|---------------|-----------------|
| down_proj | 64 | 1.8e-2 | 99.2% |
| o_proj | 16 | 1.9e-2 | 99.3% |
| gate_proj/up_proj | 128 | 6e-4 | ~28% |
| q/k/v_proj | 48 | 4e-4 | ~30-47% |

Delta-energy-weighted: **99.1% captured**, reconstruction error 0.087.

`lora_alpha == r == 16`, so the PEFT/GGUF scale is **1.0** (exact delta).
Use `--lora-scaled 0.5` (llama.cpp) or a merge ratio to weaken the effect.

## Usage

### PEFT (transformers)

> ⚠️ Must load with **`AutoModelForImageTextToText`** (→ `Qwen3_5ForConditionalGeneration`).
> `AutoModelForCausalLM` resolves to the text-only `Qwen3_5ForCausalLM` whose
> parameter names (`model.layers.*`) do **not** match this adapter
> (`model.language_model.layers.*`).

```python
import torch
from transformers import AutoModelForImageTextToText, AutoTokenizer
from peft import PeftModel

model = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3.6-27B", device_map="auto", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "hotdogs/thinkingcap-qwen36-r16-lora")

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-27B")
prompt = "Explain the theory of relativity"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

### llama.cpp (GGUF)

```bash
llama-cli -m Qwen3.6-27B.Q6_K.gguf \
    --lora thinkingcap_qwen36_r16.gguf \
    --lora-scaled 0.5 \
    -p "Explain the theory of relativity"
```

Requires a base GGUF of the **same architecture** (Qwen3.6-27B).

## Files

- `adapter_config.json` — PEFT config (r=16, alpha=16, target modules)
- `adapter_model.safetensors` — LoRA weights (BF16, 159.5 MB)
- `thinkingcap_qwen36_r16.gguf` — llama.cpp LoRA (159.4 MB)
- `tokenizer.*` — tokenizer files copied from the base model

## Notes

- Extracted with `weight-diff-extraction` (randomized SVD, CPU, ~11 min for 256 tensors).
- The delta includes both the **ThinkingCap fine-tune** and the **abliteration**
  applied by huihui-ai, relative to the Qwen3.6-27B base.
- Source model license: Qwen (Apache-2.0) + huihui-ai abliterated variant.