File size: 4,771 Bytes
74956a7 406cb5e 74956a7 406cb5e 74956a7 406cb5e 74956a7 406cb5e 74956a7 406cb5e 5ca9ba9 406cb5e 74956a7 5ca9ba9 74956a7 406cb5e 74956a7 406cb5e 74956a7 406cb5e 74956a7 406cb5e 74956a7 406cb5e 74956a7 406cb5e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | ---
language:
- en
license: mit
tags:
- from-scratch
- tiny-transformer
- character-level
- language-model
- reliability
- pytorch
metrics:
- perplexity
model-index:
- name: VIVEK-0 Chat
results:
- task:
type: language-modeling
dataset:
type: held-out-chapters
name: SRE Book held-out chapters (ch.31-34 + appendices)
metrics:
- name: Perplexity
type: perplexity
value: 6.20
---
# VIVEK-0 Chat — tiny from-scratch LM (books + chat format)
A 1.3MB transformer trained from scratch on reliability + ML/probability books (SRE, Think Bayes, Nielsen NN&DL, MML), then fine-tuned on User/Assistant chat-formatted book content. Chat eval ppl 6.20; SRE held-out improved 7.43→6.41 after chat FT (no catastrophic forgetting).
## Model Details
| Property | Value |
|---|---|
| Parameters | 328,054 (1.3MB fp32) |
| Architecture | Transformer with causal self-attention (encoder-style blocks), d=128, 2 layers, 4 heads, FFN 256 |
| Tokenizer | Character-level (vocab from training corpus; `<unk>`=3, `<pad>`=0, `<cls>`=1, `<sep>`=2) |
| Context | 256 characters |
| Optimizer | AdamW (lr 5e-4, wd 0.1) → AdamW chat FT (lr 2e-4) |
| Training data | SRE + ML corpus (1.87M chars) then chat pairs (3.2M chars) |
| Evaluation | chat held-out 6.20; SRE held-out 6.41 |
## Evaluation
Held-out perplexity on chapters **never seen during training** (ch. 31-34 + appendices):
- **This model**: chat held-out 6.20; SRE held-out 6.41
- **Baselines**: bigram 12.09, interpolated trigram 6.91
- For reference, a DeBERTa-large-class model is ~435M parameters (~1.7GB fp32); these models are 328K parameters (1.3MB) — a ~1,300x parameter reduction. Training time: minutes on GPU, approximately one hour on CPU.
## Generation Samples (honest)
Perplexity measures next-character predictability, not coherence. Actual output
(temperature 0.8, prompt "An SLO is a "):
> An SLO is a service be Back in process gate a possible SLO With Triguration SRE Mattion it a passed spicies of Cach SLI2 is has SLOs a SREs in the fully because secondition Entible capacity time correnting or a consensus of the are disapply of the server...
**This model produces domain-flavored word salad, not grammatical prose.** It
memorizes the corpus's vocabulary and rough statistics; syntax is not coherent.
Use as a research artifact for studying tiny-model training dynamics — **not**
for text generation.
## Capabilities
- Predicts the next character within its training distribution (perplexity well
below n-gram baselines)
- Useful as a training-dynamics testbed (optimizer, data, scaling studies)
- NOT usable as a text generator, chat assistant, or knowledge source
## Limitations
- Character-level: generates one character per step; context limited to 256 characters
- Trained on a narrow book corpus — domain-limited, not a general assistant
- No alignment/instruction tuning (VIVEK-0 Chat is minimally chat-formatted only)
- Tiny models reflect their training distribution; verify outputs before use
## Training Data Licenses
- Google SRE Book: free for non-commercial use (Google's terms for the public SRE book)
- Think Bayes / Think Stats (Allen Downey): CC BY-NC
- MML Book (Deisenroth et al.): CC BY-NC-SA
- Nielsen NN&DL: free online book (author-permitted non-commercial use)
Model weights are released under MIT; please respect the underlying corpus licenses for any redistribution of derivative training data.
## Usage
```python
import torch, json
from model import TinyLM # architecture definition (in this repo)
vocab = json.load(open("vocab.json"))
cfg = json.load(open("config.json"))
model = TinyLM(vocab_size=len(vocab), d_model=cfg["d_model"], n_layers=cfg["n_layers"],
n_heads=cfg["n_heads"], ff=cfg["ff"], max_len=cfg["max_len"])
model.load_state_dict(torch.load("model.safetensors", weights_only=True))
model.eval()
```
## Training Details
- All models trained on a single P100 GPU or CPU (Kaggle), seeds fixed (42), TF32 off, max_grad_norm 1.0
- Language-model data is fed as fixed-length 256-char chunks (no padding); classifier experiments used right-padding after CLS
- Muon: lr 2e-2 warmup→cosine, weight decay 0.1 applied **to the parameter** (reference-exact
`p.mul_(1 - lr*wd)`), NS5 orthogonalization with update scale `max(1, r/c)^0.5`,
AdamW (lr 1e-3) for embeddings/head/1D params
- AdamW: lr 5e-4 (continue-training 2e-4), weight decay 0.1
## Reproducibility
Training scripts and raw metrics (per-epoch logs, split definitions, seeds) are available on request from the author's repository; the Muon implementation follows arXiv:2502.16982 and the KellerJordan/muon reference, the AdamW recipe follows Raschka's *Build a Large Language Model (From Scratch)* (O'Reilly).
|