vivekkopthsd/vivek0-chat

VIVEK-0 Chat — tiny from-scratch LM (books + chat format)

A 1.3MB transformer trained from scratch on reliability + ML/probability books (SRE, Think Bayes, Nielsen NN&DL, MML), then fine-tuned on User/Assistant chat-formatted book content. Chat eval ppl 6.20; SRE held-out improved 7.43→6.41 after chat FT (no catastrophic forgetting).

Model Details

Property	Value
Parameters	328,054 (1.3MB fp32)
Architecture	Transformer with causal self-attention (encoder-style blocks), d=128, 2 layers, 4 heads, FFN 256
Tokenizer	Character-level (vocab from training corpus; <unk>=3, <pad>=0, <cls>=1, <sep>=2)
Context	256 characters
Optimizer	AdamW (lr 5e-4, wd 0.1) → AdamW chat FT (lr 2e-4)
Training data	SRE + ML corpus (1.87M chars) then chat pairs (3.2M chars)
Evaluation	chat held-out 6.20; SRE held-out 6.41

Evaluation

Held-out perplexity on chapters never seen during training (ch. 31-34 + appendices):

• This model: chat held-out 6.20; SRE held-out 6.41
• Baselines: bigram 12.09, interpolated trigram 6.91
• For reference, a DeBERTa-large-class model is 435M parameters (1.7GB fp32); these models are 328K parameters (1.3MB) — a ~1,300x parameter reduction. Training time: minutes on GPU, approximately one hour on CPU.

Generation Samples (honest)

Perplexity measures next-character predictability, not coherence. Actual output (temperature 0.8, prompt "An SLO is a "):

An SLO is a service be Back in process gate a possible SLO With Triguration SRE Mattion it a passed spicies of Cach SLI2 is has SLOs a SREs in the fully because secondition Entible capacity time correnting or a consensus of the are disapply of the server...

This model produces domain-flavored word salad, not grammatical prose. It memorizes the corpus's vocabulary and rough statistics; syntax is not coherent. Use as a research artifact for studying tiny-model training dynamics — not for text generation.

Capabilities

• Predicts the next character within its training distribution (perplexity well below n-gram baselines)
• Useful as a training-dynamics testbed (optimizer, data, scaling studies)
• NOT usable as a text generator, chat assistant, or knowledge source

Limitations

• Character-level: generates one character per step; context limited to 256 characters
• Trained on a narrow book corpus — domain-limited, not a general assistant
• No alignment/instruction tuning (VIVEK-0 Chat is minimally chat-formatted only)
• Tiny models reflect their training distribution; verify outputs before use

Training Data Licenses

• Google SRE Book: free for non-commercial use (Google's terms for the public SRE book)
• Think Bayes / Think Stats (Allen Downey): CC BY-NC
• MML Book (Deisenroth et al.): CC BY-NC-SA
• Nielsen NN&DL: free online book (author-permitted non-commercial use) Model weights are released under MIT; please respect the underlying corpus licenses for any redistribution of derivative training data.

Usage

import torch, json
from model import TinyLM  # architecture definition (in this repo)

vocab = json.load(open("vocab.json"))
cfg = json.load(open("config.json"))
model = TinyLM(vocab_size=len(vocab), d_model=cfg["d_model"], n_layers=cfg["n_layers"],
               n_heads=cfg["n_heads"], ff=cfg["ff"], max_len=cfg["max_len"])
model.load_state_dict(torch.load("model.safetensors", weights_only=True))
model.eval()

Training Details

• All models trained on a single P100 GPU or CPU (Kaggle), seeds fixed (42), TF32 off, max_grad_norm 1.0
• Language-model data is fed as fixed-length 256-char chunks (no padding); classifier experiments used right-padding after CLS
• Muon: lr 2e-2 warmup→cosine, weight decay 0.1 applied to the parameter (reference-exact p.mul_(1 - lr*wd)), NS5 orthogonalization with update scale max(1, r/c)^0.5, AdamW (lr 1e-3) for embeddings/head/1D params
• AdamW: lr 5e-4 (continue-training 2e-4), weight decay 0.1

Reproducibility

Training scripts and raw metrics (per-epoch logs, split definitions, seeds) are available on request from the author's repository; the Muon implementation follows arXiv:2502.16982 and the KellerJordan/muon reference, the AdamW recipe follows Raschka's Build a Large Language Model (From Scratch) (O'Reilly).