0arch-io/kisoku-3b

Kisoku 3B

Kisoku is a 3.2B parameter language model trained from scratch by 0ARCH. Not a fine-tune: the base model was pretrained on 60B tokens over ten days on TPUs, then instruction-tuned.

This repo contains the repaired, Ollama-ready Q8_0 GGUF.

Quick start (Ollama)

Download kisoku-3b-repaired-q8.gguf and the Modelfile from this repo into the same folder, then:

ollama create kisoku -f Modelfile
ollama run kisoku

Runs at ~75 tokens/sec on an Apple Silicon MacBook.

What "repaired" means

The original training used a 128,000-token vocab against a 128,256-token Llama 3 tokenizer, leaving 256 special-token rows (including EOS) as untrained zeros. The model literally could not end a turn. The fix was a small repair fine-tune with a gradient hook that masked every embedding row except those 256 specials, so only the special tokens trained and the language weights could not degrade. Eleven minutes on 3 GPUs took EOS from rank ~4000 to rank 1.

Architecture notes

• Llama-style transformer, 3072 hidden dim, shipped under the Granite architecture in GGUF so that the training-time logit scaling (logits divided by ~55.4) is applied at runtime in FP32 instead of baked into quantized weights.
• Because of that scaling, temperature is compressed: 0.01 here behaves like ~0.55 on a normal model. The included Modelfile sets this for you.
• Context: 4096 tokens.

Honest limitations

This is a 3B model trained on 60B tokens; the big labs use a few hundred times more text. It gets facts wrong, invents things confidently, and its arithmetic is weak. It knows its name and who built it because that was in the training data. Treat it as a proof of what one person can train, not as a production assistant.

License

MIT.