DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling
Paper
• 2406.11617 • Published
• 10
⚠️ Warning: This model can produce narratives and RP that contain violent and graphic erotic content. Adjust your system prompt accordingly, and use Mistral Tekken chat template.
A highly creative merge. Some refusals but you can use jailbreaks or ablate the model. A normtrue version was tested. Normfalse did better overall, it was slightly less censored, more detailed and creative.
Scores 14152 at Q0 Bench (Pass Q0G).
This model was merged using the following merge method: DELLA
architecture: MistralForCausalLM
models:
- model: B:\24B\!models--mistralai--Magistral-Small-2509\textonly
- model: B:\24B\!models--Gryphe--Tiamat-24B-Magistral\textonly
parameters:
density: 0.9
weight: 0.4
epsilon: 0.099
- model: B:\24B\!models--TheDrummer--Magidonia-24B-v4.3
parameters:
density: 0.9
weight: 0.4
epsilon: 0.099
- model: B:\24B\!models--TheDrummer--Precog-24B-v1
parameters:
density: 0.9
weight: 0.4
epsilon: 0.099
- model: B:\24B\!models--zerofata--MS3.2-PaintedFantasy-v3-24B
parameters:
density: 0.9
weight: 0.4
epsilon: 0.099
- model: B:\24B\!models--zerofata--MS3.2-PaintedFantasy-v4.1-24B
parameters:
density: 0.9
weight: 0.4
epsilon: 0.099
# Seed: 420
merge_method: della
base_model: B:\24B\!models--mistralai--Magistral-Small-2509\textonly
parameters:
lambda: 1.0
normalize: false
int8_mask: false
dtype: float32
out_dtype: bfloat16
tokenizer:
source: B:\24B\!models--TheDrummer--Magidonia-24B-v4.3
# chat_template: auto
name: 🌌 Magistaroth-24B-v1