TinyTurk v2.7d — k_hikaye Baseline

1.1M-parameter Turkish GPT trained from scratch on 7,693 mini-stories.

Model Description

TinyTurk is a family of extremely small Turkish language models (~1M parameters) designed to test how far a tiny transformer can go on Turkish morphology and syntax. This version is trained on 7,693 short stories (k_hikaye dataset).

  • Developed by: stunmuffin
  • Model type: Decoder-only Transformer (GPT-style)
  • Language: Turkish
  • License: Apache 2.0
  • Variant: Baseline (no distillation)

Architecture

  • Tokenizer: 512 BPE (SentencePiece)
  • Context length: 128
  • Embedding: 128
  • Layers: 4
  • Heads: 4
  • FFN: 128 -> 512 -> 128
  • Micro-FFN: 128 -> 256 -> 128 (gated, gate=0.75)
  • Position encoding: RoPE
  • Normalization: RMSNorm (pre-norm)
  • Weight tying: yes (embedding <-> LM head)
  • Activation: GELU
  • Dropout: 0.10

Training

  • Data: k_hikaye - 9,617 mini-stories (cleaned, deduplicated)
  • Split: 7,693 train / 961 val / 963 test (80/10/10, seed=42)
  • Optimizer: AdamW (wd=0.01)
  • Learning rate: 3e-4, cosine decay, 5% warmup
  • Batch size: 32
  • Epochs: 100
  • Hardware: NVIDIA GTX 1080 Ti
  • Training time: 676 seconds (~11 min)
  • Best val loss: 1.5562
  • Perplexity: 4.74

Evaluation

TurBLiMP (16,000 items, chance=50%)

  • Accuracy: 66.86%
  • Gap over chance: +16.86 points

By phenomenon

Phenomenon Score
passives 100.00%
irregular_forms 98.10%
quantifiers 96.60%
npi_licensing 91.30%
scrambling 89.00%
island_effects 84.30%
binding 79.50%
nominalization 68.90%
argument_structure_ditransitive 67.60%
determiners 59.10%
relative_clauses 53.00%
argument_structure_transitive 50.70%
anaphor_agreement 50.50%
subject_agreement 35.80%
suspended_affixation 32.20%
ellipsis 13.10%

Language modeling

  • Val loss: 1.5562
  • Perplexity: 4.74

Grammar diagnostic (456 items, cloze)

Metric Score
Top-1 12.06%
Top-5 28.95%

By category (Top-5)

Category Score
Local syntax 41.25%
Long distance 23.13%
Morphology 41.67%
Word order 12.50%

Layer-wise Analysis

We extract hidden states after each Transformer block, apply the final RMSNorm + LM head, and run the 456-item diagnostic at each level. This reveals where in the network each linguistic capability emerges.

Top-1 accuracy by layer

Category emb L1 L2 L3 L4
local_syntax 0.00% 0.00% 0.00% 10.00% 13.75%
long_distance 0.00% 0.00% 1.25% 1.88% 6.88%
morphology 0.00% 0.00% 0.00% 15.00% 25.83%
word_order 0.00% 0.00% 0.00% 0.00% 2.08%

Top-5 accuracy by layer

Category emb L1 L2 L3 L4
local_syntax 0.00% 0.00% 1.25% 36.25% 41.25%
long_distance 0.00% 0.00% 5.00% 19.38% 23.12%
morphology 0.00% 0.00% 1.67% 40.83% 41.67%
word_order 0.00% 0.00% 0.00% 10.42% 12.50%

Key findings

  1. Embedding, L1, L2 produce zero linguistic accuracy (Top-1). The first two layers do not perform grammatical tasks - only surface feature extraction.
  2. L3 is the critical layer. All four categories jump from near-zero to meaningful accuracy at L3.
  3. L4 continues to develop all categories. The model is still learning at the last layer.
  4. Long-distance and word-order remain weak at L4 (6.88% and 2.08%), suggesting these may require more depth.

Interpretation: TinyTurk exhibits a threshold phenomenon, not gradual specialization. Layers 1-2 are essentially feature extractors; grammar begins at L3 and develops at L4. This provides a scientific rationale for future depth scaling experiments (4 -> 6 -> 8 layers).

Usage

import torch
import sentencepiece as spm
from model import TinyTurkGPTV27, TinyTurkV27Config

tok = spm.SentencePieceProcessor()
tok.load('tokenizer_v11_bpe_512.model')

config = TinyTurkV27Config(vocab_size=512, context_length=128)
model = TinyTurkGPTV27(config)
model.load_state_dict(torch.load('pytorch_model.bin', map_location='cpu'))
model.eval()

prompt = 'Ali bahçeye'
ids = tok.encode(prompt, out_type=int)
x = torch.tensor([ids], dtype=torch.long)

with torch.no_grad():
    for _ in range(30):
        logits, _ = model(x)
        next_id = torch.argmax(logits[0, -1, :]).item()
        x = torch.cat([x, torch.tensor([[next_id]])], dim=1)

print(tok.decode(x[0].tolist()))

Depth Scaling

We trained TinyTurk v2.7d with 4, 6, and 8 layers (same architecture, data, hyperparameters; only depth varies).

Val loss

Model Parameters best_val PPL Overfit gap
4L 1,121,920 1.5562 4.74 0.215
6L 1,650,048 1.5525 4.72 0.384
8L 2,178,176 1.5604 4.76 0.536

Val loss follows a U-shaped curve: 6L is the sweet spot, 8L is worse than 4L.

Layer-wise Top-1 diagnostic (final layer)

Category 4L 6L 8L Winner
local_syntax 13.75% 11.25% 8.75% 4L
long_distance 6.88% 2.50% 3.75% 4L
morphology 25.83% 29.17% 30.00% 8L
word_order 2.08% 7.29% 3.12% 6L

Key findings

  1. No single winner. Each linguistic category has its own optimal depth:
    • local_syntax, long_distance -> 4L (shallow)
    • word_order -> 6L (medium)
    • morphology -> 8L (deep)
  2. Depth scaling is not monotonic. Val loss U-curve.
  3. Late-layer forgetting. In 8L, local_syntax peaks at L6 (10%) and drops at L8 (8.75%).
  4. Morphology is the only category that monotonically scales with depth.

For a general-purpose TinyTurk at this scale, 6L is the best compromise. Full report: DEPTH_SCALING_REPORT.md

Known Limitations

  • Extremely small (1.1M parameters) - limited generation quality
  • Trained on simple children's stories - narrow domain
  • Context window: 128 tokens
  • Turkish only
  • Weak on ellipsis (13.10%), suspended affixation (32.20%), and subject agreement (35.80%) - far below or near chance level
  • Long-distance dependencies and word order are still underdeveloped at L4

Scientific Context

This model is the baseline for the Teacher-Augmented TinyTurk (TAT) experiment, which tests whether output-level KD helps extremely small LMs.

Result of the TAT experiment: mixed-to-negative. KD did not improve the baseline.

Metric Baseline TAT Winner
Val loss 1.5562 1.5566 Baseline
Top-1 diagnostic 12.06% 9.21% Baseline
Top-5 diagnostic 28.95% 29.39% TAT (marginal)
TurBLiMP (16k) 66.86% 65.27% Baseline

Interpretation: For 1M-parameter models, output-level KD is net harmful - likely due to broken tokenizer mapping, insufficient capacity, and single-position teacher signals.

Citation

@misc{tinyturk-v27d-k-hikaye-baseline,
  title = {TinyTurk v2.7d - k_hikaye Baseline},
  author = {stunmuffin},
  year = {2026},
  howpublished = {https://huggingface.co/stunmuffin/tinyturk-v2.7d-k-hikaye-baseline}
}

Related

Downloads last month
22
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support