Schizo output
The model is quite dumb tbh. I don't enforce thinking, so maybe that's the problem? Anyway, even if I raise min P substantially and lower temp to 0.7 or 0.5, it just always fails to understand what's actually going on in the scene. Forget subtleties, it just mixes the basic stuff up, like who said what, to whom, where we are, etc. Completely disregards social context, probably because it has such poor grasp on the prompt content. At 1.0 temp you get 100% schizo confusion, like talking to a demented person. Comparing this to StyleTune v2 and basic gemma, I used all at Q6_K, and the difference between this one and those is night and day. Maybe the quant is broken, I dunno.
The model is quite dumb tbh. I don't enforce thinking, so maybe that's the problem? Anyway, even if I raise min P substantially and lower temp to 0.7 or 0.5, it just always fails to understand what's actually going on in the scene. Forget subtleties, it just mixes the basic stuff up, like who said what, to whom, where we are, etc. Completely disregards social context, probably because it has such poor grasp on the prompt content. At 1.0 temp you get 100% schizo confusion, like talking to a demented person. Comparing this to StyleTune v2 and basic gemma, I used all at Q6_K, and the difference between this one and those is night and day. Maybe the quant is broken, I dunno.
The reason v1.4 is going to perform even worse than v1.3 or styletune by itself is that since v1.3 was formatted incorrectly, it did not absorb styletune's lm_head.
This version is much dumber because grafting an lm_head from an instruct base finetune directly onto a non-instruct base foundation causes severe degradation.
base_model: B:\26B\google_gemma-4-26B-A4B
- model: B:\26B\Gryphe--Gemma-4-26B-A4B-StyleTune-V2
parameters:
weight:
- filter: "lm_head"
value: 1.0
v1.5 should fix these issues and be more stable than previous versions because it uses the instruct base as della's base_model, which is what styletune was actually trained on.
@BigBeavis please try the Heresy variant to see if it works better. It would be interesting to know whether the Hereticisation process has inadvertently helped to heal the lm_head issue @Naphula has identified. After a few false starts with quant problems I have found 1.4 Heresy to be quite good.
I'm not going to ablate v1.3 or v1.4 any further, I think I'll try to make a SOMPOA and ARA of v1.5 instead as this should provide a more accurate comparison. The bug of "non-instruct base being used as base model" with all instruct donors (1.3+1.4) and instruct lm_head (1.4) is so severe that it may have invalidated all previous ARA/SOMPOA comparisons for Goetia 26B.
The final hurdle with v1.5 is likely to be testing 3 variants before even reaching the ablation stage:
- fully merged lm_head of all 20+ donors (including styletune)
- styletune v2 only
- base model lm_head
@BigBeavis please try the Heresy variant to see if it works better. It would be interesting to know whether the Hereticisation process has inadvertently helped to heal the
lm_headissue @Naphula has identified. After a few false starts with quant problems I have found 1.4 Heresy to be quite good.
I don't think ablation would heal it, finetuning might but I am not convinced that such pipelines are easily repaired. All the MLP/ATN layers were desynched as well from instruct base not just the lm_head.
What I suspect however is that the damage isn't as bad as Gemini claims due to redundant sparsification with high density / low epsilon, although it would still have negative impacts.
If v1.5 isn't a noticeable improvement then the theory doesn't hold up as well.
It may be another example of a model taking advantage of latent structural errors as a form of covert noncompliance.
Much to my surprise only Orion, Pantheon and StyleTune even trained lm_head. All other models (readyart, nbeerbower, zerofata etc) did not train the lm_head so the distribution is essentially 40% Orion 30% Styletune and 30% Pantheon by default (della magnitude), meaning we should not even have to monkey with the lm_head toggler experiment.
I'm glad the issue has been identified, will be looking forward to the fixed version👍