stunmuffin/tinyturk-v2.7d-tinystories-en

TinyTurk v2.7d - TinyStories EN

1.1M-parameter English GPT trained from scratch on 200,000 TinyStories.

Model Description

TinyTurk is a family of extremely small language models (~1M parameters). This is the English variant, trained on Microsoft's TinyStories dataset. It uses the same architecture as the Turkish k_hikaye models, but with an English-specific 512 BPE tokenizer and 26x more training data.

• Developed by: stunmuffin
• Model type: Decoder-only Transformer (GPT-style)
• Language: English
• License: Apache 2.0
• Variant: Baseline (no distillation)

Architecture

• Tokenizer: 512 BPE (SentencePiece, English)
• Context length: 128
• Embedding: 128
• Layers: 4
• Heads: 4
• FFN: 128 -> 512 -> 128
• Micro-FFN: 128 -> 256 -> 128 (gated, gate=0.75)
• Position encoding: RoPE
• Normalization: RMSNorm (pre-norm)
• Weight tying: yes (embedding <-> LM head)
• Activation: GELU
• Dropout: 0.10

Training

• Data: roneneldan/TinyStories - 200,000 stories (train), 10,000 (val), 10,000 (test)
• Optimizer: AdamW (wd=0.01)
• Learning rate: 3e-4, cosine decay, 5% warmup
• Batch size: 32
• Epochs: 30 (stopped early; best at epoch 29)
• Hardware: NVIDIA GTX 1080 Ti
• Training time: ~1.5 hours
• Best val loss: 1.3349
• Perplexity: 3.80

Evaluation

Language modeling

Split	Loss	Perplexity
train	1.3016	3.68
val	1.3349	3.80
test	1.2849	3.61

Test loss is lower than val loss - no overfitting.

BLiMP (English, 67 phenomena, 50 items each = 3,350 items, chance=50%)

• Accuracy: 55.61%
• Gap over chance: +5.61 points

Best phenomena

Phenomenon	Score
principle_A_case_1	100.00%
wh_questions_subject_gap_long_distance	100.00%
wh_vs_that_no_gap	100.00%
principle_A_domain_1	98.00%
wh_vs_that_no_gap_long_distance	96.00%
sentential_negation_npi_licensor_present	92.00%
wh_questions_subject_gap	90.00%

Worst phenomena (below chance)

Phenomenon	Score
only_npi_licensor_present	0.00%
matrix_question_npi_licensor_present	2.00%
wh_vs_that_with_gap	8.00%
wh_vs_that_with_gap_long_distance	8.00%
only_npi_scope	20.00%
npi_present_2	22.00%
left_branch_island_simple_question	24.00%
npi_present_1	26.00%

Layer-wise Analysis (BLiMP)

We extract hidden states after each Transformer block, apply the final RMSNorm + LM head, and run BLiMP at each level. This reveals where in the network English grammatical capability emerges.

BLiMP accuracy by layer

Level	Accuracy	Gap over chance
emb	52.36%	+2.36
L1	51.31%	+1.31
L2	52.51%	+2.51
L3	53.73%	+3.73
L4	55.61%	+5.61

Visual

  emb ########################## 52.36%
   L1 #########################  51.31%  <- dip
   L2 ########################## 52.51%
   L3 ########################## 53.73%
   L4 ########################### 55.61%

Key findings

1. Embedding already scores 52.36% - well above chance. Many BLiMP phenomena are solvable from surface cues (e.g., determiner-noun agreement).
2. L1 causes a dip (52.36% -> 51.31%). The first Transformer layer degrades the useful embedding signal before other layers recover it.
3. Gradual, monotonic improvement from L1 to L4. Each layer adds ~1-2 percentage points.
4. L4 is still improving. The model has not saturated at 4 layers.

Comparison: TR vs EN layer-wise profile

Same architecture, different languages. Layer-wise diagnostics reveal fundamentally different learning profiles:

Property	Turkish (k_hikaye)	English (TinyStories)
Embedding accuracy	0.00%	52.36%
L1	0.00%	51.31% (dip)
L2	0.00%	52.51%
Critical layer	L3 (threshold jump)	none (gradual)
L4 accuracy	25.83%	55.61%
Learning shape	Threshold	Gradual

Interpretation: Turkish exhibits a threshold phenomenon - layers 1-2 do nothing grammatical, grammar begins at L3. English shows gradual development starting from the embedding layer. This is likely because Turkish's agglutinative morphology requires deeper transformations, while English syntax is more surface-level.

Note: The Turkish and English benchmarks are different (TurBLiMP vs BLiMP, cloze vs minimal-pair). Direct numerical comparison is not valid; the comparison here is about the shape of layer-wise development.

Generation Samples

Sampling: temperature=0.8, top_k=40, seed=42

Prompt: 'Once upon a time'

Once upon a time, in a big, green forest, there lived a jolly monkey. He liked to play with his friends and play with his friends. One day, he was walking in the forest when he saw a big, scary snake. The snake was very mean and kept flying around.

Prompt: 'In a small village'

In a small village, there was a little girl named Lily. She loved to play with her teddy bear. One day, Lily went for a walk in the park with her mom. The sun was shining, and the birds flew high in the sky. Lily saw a lot of dust. She wanted to see the dust too.

Greedy (temperature=0.01, top_k=1)

Prompt: 'Once upon a time'

Once upon a time, there was a little girl named Lily. She loved to play outside in the sunshine. One day, she saw a big, red ball in the grass. She wanted to play with it, but it was too high for her to reach. Lily's mom saw her and said, "Lily, you need to reach the ball. It's too high up in the sky." Lily was

Usage

import torch
import sentencepiece as spm
from model import TinyTurkGPTV27, TinyTurkV27Config

tok = spm.SentencePieceProcessor()
tok.load('tokenizer_en_bpe_512.model')

config = TinyTurkV27Config(vocab_size=512, context_length=128)
model = TinyTurkGPTV27(config)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()

prompt = 'Once upon a time'
ids = tok.encode(prompt, out_type=int)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    for _ in range(80):
        logits, _ = model(x[:, -128:])
        next_id = torch.argmax(logits[0, -1, :]).item()
        x = torch.cat([x, torch.tensor([[next_id]])], dim=1)

print(tok.decode(x[0].tolist()))

Known Limitations

• Extremely small (1.1M parameters) - limited generation quality
• Trained on simple children's stories - narrow domain
• Context window: 128 tokens
• English only
• BLiMP 55.61% - close to chance level, fails on abstract syntax (NPI, island, gaps)
• Occasional semantic drift and repetitions

Scientific Context

This model is part of a broader comparison: same architecture, different language and data scale. It shows that for tiny models, data quantity matters more than architectural tweaks - but only up to a point.

Model	Language	Train stories	Val loss
k_hikaye baseline	TR	7,693	1.5562
TinyStories EN	EN	200,000	1.3349

Ongoing/future experiments:

• Turkish TinyStories (same data scale, TR)
• English k_hikaye-sized (7.7K, EN)
• Teacher-Augmented (KD) on EN
• Depth scaling: 4 -> 6 -> 8 layers

Citation

@misc{tinyturk-v27d-tinystories-en,
  title = {TinyTurk v2.7d - TinyStories EN},
  author = {stunmuffin},
  year = {2026},
  howpublished = {https://huggingface.co/stunmuffin/tinyturk-v2.7d-tinystories-en}
}

Related

• Turkish baseline (k_hikaye): stunmuffin/tinyturk-v2.7d-k-hikaye-baseline
• Turkish TAT variant: stunmuffin/tinyturk-v2.7d-k-hikaye-tat
• Dataset: roneneldan/TinyStories