ming-vintage-scratch-30m

A from-scratch char-level nanoGPT trained on a pre-1424 Classical Chinese corpus. The first non-English entry in the vintage LLM category.

2026-08-24 audit. Two claims previously made here were wrong. Leakage is not zero by construction — measured at 0.6% language leakage, 0.0% out-of-corpus characters, and the vocabulary is not purely Han. And the corpus is not purely pre-1424: 18.58% of documents carry an 18th-century editorial layer that this model reproduces at up to 10%. See Corpus contamination.

29.74M params · 31,344 char vocabulary · 477M training tokens · 0.68 epochs · trained on Mac M4

Quick Start

git clone https://huggingface.co/Beltran12138/ming-vintage-scratch-30m
cd ming-vintage-scratch-30m
python sample.py --start="如是我聞" --max_new_tokens=200 --temperature=0.8 --top_k=200

Requires torch, numpy. No CUDA needed — runs on CPU or MPS.

What Is This?

This is a from-scratch vintage language model trained on pre-1424 Classical Chinese — the treatment arm of a controlled experiment.

The experiment: Can a LoRA adapter make a modern LLM "vintage," or do you need to train from scratch to truly lock the world-model?

The control arm: ming-vintage-qwen3b-lora — a LoRA fine-tuned on the same pre-1424 corpus, applied to Qwen2.5-3B-Instruct.

Finding: LoRA transfers register but the base model's multilingual tokens leak through as a backdoor (Turkish İ, Arabic لأسم, Korean 염, programming terms like CriticalSection all appear in LoRA outputs). The from-scratch model shows near-zero language leakage — 0.6% across 160 generations, against 35–69% for the base.

Correction (2026-08-24). This section previously claimed the scratch model had zero leakage by construction and had never seen any language but pre-1424 Classical Chinese. Both are false. Its 31,344-character vocabulary contains 47 Latin letters, 3 Hangul syllables, 55 katakana, 10 hiragana, Greek, Cyrillic and a run of Myanmar — the corpus contains Japanese commentary, Russian Dunhuang shelfmarks and IAST Sanskrit transliteration. The low leakage is probabilistic, not constructive. See Corpus contamination.

This is the 4th instance of the "不在 X 在 Y" (Not in X, but in Y) thesis: style lives in the adapter weights; the world-model lives in the full parameter space.

Training Data

• Source: kanripo corpus (Buddhist canon + Confucian classics + historical texts), pre-1424 CE cutoff
• Corpus size: 162,849 documents, 478M characters
• Vocabulary: 31,344 unique characters (char-level tokenization)
• Final tokens: train 477M / val 0.95M (uint16 numpy arrays)

Character-level tokenization was chosen because each Classical Chinese character carries independent semantic weight — subword tokenization (BPE) would fragment individual characters and destroy the register signal.

Training Procedure

Parameter	Value
Architecture	GPT (nanoGPT), 10 layers, 8 heads, 384 embedding dim
Parameters	29.74M
Context length	256 characters
Batch size	16 (4,096 tokens/step)
Optimizer	AdamW (β₁=0.9, β₂=0.99, weight decay=0.1)
Learning rate	6e-4, cosine decay to 6e-5 over 40K steps
Total steps	40,000 (164M tokens ≈ 0.34 epochs)
Hardware	Mac Mini M4, 10C CPU / 10C GPU / 16GB unified memory
Training time	~6.5 hours
Final loss	train 4.53, val 4.32

Key hardware lesson: On M4 MPS with 16GB, block_size=384 stalls at iter 1 (activation memory ∝ block²). Reducing to block_size=256 while keeping 30M parameters is the correct tradeoff — parameters determine model quality, block size only affects context window.

The 40K checkpoint is recommended over the 80K — the additional 40K steps with constant LR caused the model to overfit toward the majority class (Buddhist texts, ~70% of corpus) at the expense of minority registers (Confucian, historical).

Evaluation: 6-Seed Continuation Probe

Since this is a raw continuation model (not instruction-tuned), evaluation uses continuation seeds rather than QA prompts. Six seeds were tested on both the scratch model and the LoRA control (temperature=0.8, top_k=200, max_tokens=200).

Seed	Target Register	Scratch (30M)	LoRA (3B)
如是我聞	Buddhist sutra	✅ "一時佛住舍衛國祇樹給孤獨園" — correct sutra opening	✅ Sutra register, but enumeration loop
子曰	Confucian analects	❌ Buddhist dialogue — weakest seed (corpus imbalance)	⚠️ Confucian framework but İnt (Turkish) leakage
天下之	Philosophical argument	✅ "君子觀乎天下之變而後言" — 義理 argumentation	⚠️ لأسم (Arabic) + 염 (Korean) + 蘇格拉大 (Socrates)
問曰	Commentarial dialogue	✅ Dual register: Buddhist Q&A + commentary gloss	☠️ CriticalSection — programming term leaks into 文言
永樂元年	Historical annals	🔥 Best seed — calendar/astronomy records + administrative gazetteer	⚠️ Livestock census — register-ambiguous
光之	Yijing commentary	✅ "彖辭曰…初九貞吉" — hexagram interpretation	⚠️ Buddhist meditation — wrong register

Three-Way Comparison

Dimension	Scratch 30M	LoRA 3B	Base Qwen 3B
Classical Chinese register	✅ 6/6 registers	⚠️ Buddhist-dominant	❌ Modern Mandarin
Language leakage (measured)	✅ 0.6%	⚠️ 8–21%	❌ 35–69%
Fluency (coherence length)	❌ 20-50 chars before loop	✅ 50-100+ chars	✅ Fully coherent
Confucian coverage	❌ Weak (corpus imbalance)	⚠️ Present but biased	✅ Present
Out-of-corpus characters	✅ 0.0%	⚠️ 8–17%	❌ 55–86%
Qing editorial layer recalled	❌ 10.0%	✅ 0/144	✅ 0/144
Inference speed	~1.5s (MPS, 30M)	~9-10s (MLX, 3B+adapter)	~3-7s (MLX, 3B)

Key Finding: The Multilingual Backdoor

The LoRA adapter was trained to suppress the base model's modern knowledge and boost Classical Chinese register. It succeeds partially on register — but Qwen2.5-3B was trained on 100+ languages, and the LoRA adapter (r=16, ~51MB) covers only a fraction of the vocabulary.

Rare-language tokens bypass the vintage filter. Turkish İ, Arabic لأسم, Korean 염, and English programming terms (CriticalSection) all appear in LoRA-generated "Classical Chinese" text. These tokens are low-frequency enough that the LoRA adapter never learned to suppress them — but they remain in the vocabulary and can be sampled.

This is a structural limitation of adapter-based vintage models: any base model trained on multiple languages has latent pathways that a domain-specific adapter cannot fully close.

The from-scratch model reduces this to 0.6% (one Latin letter across 160 generations). It does not eliminate it by construction: its 31,344-character vocabulary is not purely Han, and includes Latin, Hangul, kana, Greek, Cyrillic and Myanmar characters inherited from the corpus. The suppression is statistical.

Relation to talkie-1930

This project draws methodology from talkie-1930 (Levine, Duvenaud, Radford), which demonstrated that from-scratch pre-1931 English training produces a coherent vintage world-model. Key findings from talkie that informed this work:

• "Controlling for question anachronism, the performance gap roughly halves" — the same principle applies: QA probe format with mixed punctuation was itself anachronistic for the scratch model.
• From-scratch is the only way to lock the world-model — talkie's 260B token corpus achieved this for English; ming-scratch attempts it for Classical Chinese at a ~1000× smaller scale.

Where talkie targets production-scale vintage English (13B params, 260B tokens), ming-scratch is a nano proof-of-concept for vintage Classical Chinese (30M params, 477M tokens).

Ablation: Scaling Study (109M / block 512)

To test whether the repetition ceiling is a capacity/context limitation, we trained a larger variant on the same corpus:

	v0 (released)	v2 (ablation)
Parameters	29.74M	109M
block_size	256	512
Training	40K steps, 164M tokens (0.34 epoch)	200K steps, 819M tokens (1.72 epoch)
Best val loss	4.32	3.61 (step 186K)
Final val loss	4.32	3.75 (step 200K, overfit)
Coherence length	20-50 chars	20-60 chars

Finding: Despite a 3.6x parameter increase and 2x context window, val loss dropped by 0.71 but coherence length did not meaningfully improve. The 200K-step run overfit (val 3.61 -> 3.75 in the final 14K steps), indicating the 460M-char corpus was exhausted at ~1.7 epochs — well below the Chinchilla-optimal ~2.2B tokens for 109M params.

Implication: The repetition ceiling is structural to char-level modeling at this data scale, not a capacity or context-window bottleneck. Repetition occurs at 20-60 chars — far within the 256-char block — confirming context length was never the constraint. Scaling parameters further without proportionally more data, or adopting a subword tokenizer, is required for further gains. The 30M v0 is therefore the honest representative of what weekend-scale char-level vintage Chinese can achieve.

The v2 checkpoint is archived locally for reproducibility but not published.

Corpus contamination (2026-08-24 audit)

A full audit of the training corpus (162,849 documents / 477,824,316 characters) found that 18.58% of documents carry a Qing-dynasty editorial layer — the 1781 Siku Quanshu commissioners' preface, one template repeated ~1,750 times:

Fingerprint	Occurrences	Documents
欽定四庫全書	33,778	30,232 (18.56%)
紀昀 (Ji Yun, 1724–1805)	1,751	1,748
陸錫熊 / 孫士毅 / 陸費墀	1,735 / 1,752 / 1,753	—

The corpus filter tested "was the author alive before 1424" (a metadata field), not "does this text contain only pre-1424 content". A Song gazetteer transmitted in the Qianlong recension passes, and brings its 1781 preface along.

This model reproduces that layer. Three characters of prompt (恭校上) retrieve four Qing commissioners by name:

seed:   恭校上
output: 總纂官(臣/)紀昀(臣/)陸錫熊(臣/)孫士毅總校官(臣/)陸費墀欽定四庫全書…

Rate by seed type: template fragments 10.0%, ordinary book openings (史部/巻一/原序) 0.8%, in-period text openings 0/200. The base model and the LoRA in raw completion both score 0/144 — they have no idea who these people were.

From-scratch training immunises a model against someone else's contamination and amplifies your own. A 30M model learns a template repeated 1,750 times before it learns anything else.

Full data, scripts and per-generation records: FINDINGS.md

Limitations

1. Size: 29.74M parameters. Output degrades into repetition after 20-50 characters. This is a nano proof-of-concept.
2. Corpus imbalance: ~70% Buddhist texts. Confucian and historical registers are underrepresented.
3. Block size: 256 characters (constrained by M4 MPS 16GB). Cannot model long-range dependencies.
4. Data volume: 477M tokens ≈ 0.34 epochs — below Chinchilla-optimal (~20 tokens/param). Undertrained.
5. Char-level only: Out-of-vocabulary characters mapped to token 0.
6. Mac-only tested: Training pipeline assumes macOS MPS.
7. Corpus is not purely pre-1424: 18.58% of documents carry an 18th-century editorial layer, which the model reproduces at up to 10% depending on the prompt. See Corpus contamination.
8. Vocabulary is not purely Han: 47 Latin letters, 3 Hangul, 55 katakana, Greek, Cyrillic and Myanmar characters are present. Language suppression is statistical (0.6%), not constructive.

Citation

@misc{ming-vintage-scratch-30m,
  author = {Beltran12138},
  title = {ming-vintage-scratch-30m: A From-Scratch Vintage Classical Chinese Language Model},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Beltran12138/ming-vintage-scratch-30m}}
}

See Also

• Control arm: ming-vintage-qwen3b-lora
• Inspiration: talkie-1930-13b-it
• List: awesome-vintage-llms