Zenyx-MCEN-Delta-Mamba (565M) — work in progress
A ~565M-parameter language model that interleaves Mamba-3 (MIMO) and Gated DeltaNet-2 (GDN-2) sequence mixers with two MCEN stages (a reversible fixed-point "reasoning" core with a delta-rule memory), trained from scratch on the Zenyx-V3 pretraining mixture with the DeepSeek tokenizer. This repository holds training checkpoints, logs and diagnostics of an ongoing run; it is not a finished release.
Snapshot. The Pile, SciQ and Zenyx-Fact1000 numbers were measured at step 6,198 (0.81B of a planned 1.5B training tokens, 54% of 11,444 steps), on 2026-10-17; the task-suite table is from step 6,198. Training continues, so later checkpoints in
checkpoints/main565m/are newer than this card. This is a base model: it completes text and does not follow instructions.
Architecture (preset z565m)
| Stage schedule | 14 stages: 6 × Mamba-3 MIMO, 6 × GDN-2, 2 × MCEN (M3 GDN2 M3 GDN2 M3 GDN2 MCEN M3 GDN2 M3 GDN2 M3 GDN2 MCEN) |
| Hidden size / FFN | 1,536 / 3,072 |
| Vocabulary | 129,280 (DeepSeek tokenizer), tied input/output embedding |
| Mamba-3 | d_state 64, head dim 64, expand 2, MIMO rank 2 |
| GDN-2 | 24 heads × head dim 64 |
| MCEN | d_reason 1,024, 16 memory heads × 64, FFN 2,752, rank-64 persistent read, RevDEQ solver (β = 0.25), memory mode delta |
| Multi-token prediction | depth 1, loss weight 0.1, gdn2 mixer |
| Parameters | ~565M (preset bound 540–600M) |
| Training context | 4,096 tokens |
Training
| Tokens per optimizer step | 131,072 |
| Planned budget | 1.5B tokens (11,444 steps) |
| Optimizer | Muon for matrix parameters (lr 0.02), AdamW elsewhere (lr 3e-4, wd 0.1); cosine to 10% |
| Precision | fp16 backbone with dynamic loss scaling. MCEN solver: fp16 with a token-level fp32 rescue on T4; fp32 / TF32 on the H200 stretch |
| Hardware | Kaggle 2 × T4 sessions and one stretch on a single H200 (Lightning); the run resumes from checkpoints across sessions |
Data mixture — 121 sources in the frozen manifest (data/manifest_v2.json, created 2026-10-10 06:04:26, hash 9454e840c2b9f032…); grouped shares:
| Source | Key | Share |
|---|---|---|
| bigcode/starcoderdata (92 language subsets) | code_starcoder |
26.16% |
| HuggingFaceFW/fineweb-edu | fineweb_edu |
15.09% |
| nvidia/Nemotron-SFT-Math-v3 | math_nemotron_sft |
10.06% |
| openbmb/Ultra-FineWeb-L3 | ufw_en_multistyle |
5.53% |
| openbmb/Ultra-FineWeb-L3 | ufw_en_qa |
5.03% |
| HuggingFaceTB/smollm-corpus | cosmopedia_v2 |
4.23% |
| wikimedia/wikipedia | wikipedia_en |
3.52% |
| HuggingFaceTB/finemath | finemath_3plus |
3.32% |
| HuggingFaceTB/finemath | infiwebmath_3plus |
2.92% |
| nvidia/Nemotron-Pretraining-Specialized-v1.2 | nem12_fact |
2.52% |
| HuggingFaceTB/finemath | infiwebmath_4plus |
2.52% |
| HuggingFaceTB/finemath | finemath_4plus |
2.21% |
| nvidia/Nemotron-Pretraining-Specialized-v1.1 | nem11_mc |
2.09% |
| nvidia/Nemotron-Pretraining-Specialized-v1 | nem1_math_textbooks |
2.01% |
| nvidia/Nemotron-Pretraining-Specialized-v1 | nem1_stem_sft |
2.01% |
| nvidia/Nemotron-Pretraining-Specialized-v1 | nem1_rqa |
2.01% |
| nvidia/Nemotron-Pretraining-Specialized-v1.1 | nem11_code_concepts |
2.01% |
| nvidia/Nemotron-Pretraining-Specialized-v1.2 | nem12_mc |
1.11% |
| nvidia/Nemotron-Pretraining-Specialized-v1.1 | nem11_economics |
1.04% |
| nvidia/Nemotron-Pretraining-Specialized-v1 | nem1_infinibyte |
1.01% |
| Josephgflowers/OpenOrca-Step-by-step-reasoning | openorca_reasoning |
0.80% |
| SagheerLab/Arithmetic-Reasoning | arithmetic_reasoning |
0.80% |
| harryxi/HelpSteer3-general-code-Shift-Qwen-2.5-1.5B-Instruct-chat-formatted-generations | helpsteer3_code |
0.80% |
| nvidia/Nemotron-Pretraining-Specialized-v1 | nem1_sci_coding |
0.50% |
| General-Medical-AI/SlideChat | slidechat_text |
0.30% |
| nvidia/Nemotron-Pretraining-Specialized-v1.1 | nem11_uncond_algo |
0.20% |
| nvidia/Nemotron-Pretraining-Specialized-v1 | nem1_wiki_rewrite |
0.10% |
| nvidia/Nemotron-Pretraining-Specialized-v1.1 | nem11_formal_logic |
0.04% |
| nvidia/Nemotron-Pretraining-Specialized-v1.2 | nem12_moral |
0.03% |
| nvidia/Nemotron-Pretraining-Specialized-v1.2 | nem12_gen |
0.01% |
Benchmarks (step 6,198; task suite step 6,198)
The comparison models are the same author's: Zenyx-v2-base (85M unique parameters, 32 effective layers, 32k vocabulary) and Zenyx-V3-Base (1.58B-parameter MoE, ~81B training tokens). They are far larger or far longer-trained than this snapshot, so read the comparisons as "where does this early checkpoint stand", not as a like-for-like contest. MCEN and V3 numbers are on a Kaggle T4 (fp16); V2 in fp32.
The Pile, by domain (bits per byte, LOWER is better)
Text: NeelNanda/pile-10k (pinned revision 127bfedcd5), documents grouped by pile_set_name, capped at 1.5M characters per domain, scored with a
rolling 1024-token window at stride 512 so every token is scored exactly once with ≥ 512 tokens of left context; bits/byte is normalised by the
UTF-8 bytes of the source text, so it is comparable across tokenizers.
| Domain | MCEN @6,198 | V2 @96,000 | V3 @107,200 | MCEN − V2 | text (MB) |
|---|---|---|---|---|---|
| DM Mathematics | 1.7946 | 2.1898 | 1.3201 | -0.3952 | 0.81 |
| ArXiv | 1.2607 | 1.5944 | 0.8565 | -0.3337 | 1.50 |
| Books3 | 1.8757 | 2.1966 | 1.3300 | -0.3209 | 1.51 |
| PubMed Central | 1.0708 | 1.3518 | 0.7426 | -0.2810 | 1.51 |
| FreeLaw | 1.3784 | 1.6474 | 0.9632 | -0.2691 | 1.51 |
| Wikipedia (en) | 1.3454 | 1.6001 | 0.9902 | -0.2547 | 1.50 |
| PubMed Abstracts | 1.1996 | 1.4109 | 0.8749 | -0.2112 | 1.50 |
| USPTO Backgrounds | 1.0472 | 1.2530 | 0.7807 | -0.2058 | 1.50 |
| NIH ExPorter | 1.1261 | 1.3249 | 0.8620 | -0.1987 | 0.22 |
| Enron Emails | 1.6915 | 1.8693 | 1.2206 | -0.1778 | 0.06 |
| Gutenberg (PG-19) | 1.6631 | 1.8397 | 1.2715 | -0.1766 | 0.53 |
| OpenWebText2 | 1.5210 | 1.6778 | 1.1295 | -0.1568 | 1.53 |
| Pile-CC | 1.3824 | 1.5380 | 1.1001 | -0.1556 | 1.51 |
| BookCorpus2 | 1.3764 | 1.5307 | 1.1426 | -0.1542 | 0.61 |
| Ubuntu IRC | 2.4488 | 2.5754 | 1.7147 | -0.1266 | 0.04 |
| OpenSubtitles | 1.6076 | 1.7211 | 1.3392 | -0.1135 | 0.86 |
| Github | 0.9702 | 1.0832 | 0.5765 | -0.1130 | 1.53 |
| StackExchange | 1.2813 | 1.3558 | 0.9205 | -0.0745 | 1.50 |
| HackerNews | 1.6101 | 1.6748 | 1.2806 | -0.0647 | 0.24 |
| PhilPapers | 1.8911 | 1.8543 | 1.3358 | +0.0368 | 0.16 |
| EuroParl | 2.9315 | 2.6663 | 1.9986 | +0.2653 | 0.07 |
| YoutubeSubtitles | 2.6214 | 2.3116 | 1.8218 | +0.3098 | 0.19 |
| All 22, byte-weighted | 1.3746 | 1.5795 | 1.0015 | -0.2049 | 20.40 |
MCEN is lower (better) than V2 in 19 of 22 domains; it is higher (worse) in: PhilPapers (+0.037), EuroParl (+0.265), YoutubeSubtitles (+0.310). The sample sizes of EuroParl, Ubuntu IRC, YoutubeSubtitles, Enron, PhilPapers, NIH ExPorter and HackerNews are small (each under ~250 KB), so those rows are noisy.
SciQ
acc 64.50% / acc_norm 56.90% (n = 1000, chance 25%); standard error about 1.6 points.
For reference, Zenyx-v2-base scores 52.10% / 47.20% with the same code and Zenyx-V3-Base 79.90% / 72.70%.
Zenyx-Fact1000 (internal)
A frozen, 1000-question, four-way factual / common-sense completion benchmark (chance 25%; the model must rank the correct completion above three
same-type distractors by log-likelihood; sha256 843416524fb39c5f…; not yet published). A "prior-only" control scores the same candidates after the
neutral context "The answer is" — a score near it means the model is merely preferring common words.
| Model | accuracy | 95% interval | without arithmetic | prior-only control |
|---|---|---|---|---|
| MCEN @6,198 (565M, 0.81B tokens) | 47.6% | 44.5–50.7% | 47.8% | 28.8% |
| Zenyx-v2-base @96,000 (85M unique) | 45.8% | 42.7–48.9% | 48.1% | 30.7% |
| Zenyx-V3-Base @107,200 (1.58B MoE) | 81.1% | 78.6–83.4% | 80.8% | 28.7% |
| Zenyx_Base_220M (Qwen tokenizer) | 65.3% | 62.3–68.2% | 73.2% | 29.8% |
| Zenyx-42M (FineWeb-Edu only) | 49.5% | 46.4–52.6% | 54.4% | 29.1% |
| MCEN and V2 are statistically indistinguishable overall (they answer different questions correctly). Step 5,500 scored 47.1% and step 4,100 scored 45.5%, so the gain per 700 steps is within the ±3-point interval. |
Task suite (the same 19 tasks and extras as Zenyx-V3-Base)
0-shot, full evaluation sets (MMLU 5-shot), log-likelihood scoring of candidate continuations; acc_norm is length-normalised in characters.
Same task code as the Zenyx-V3-Base and Zenyx-v2-base cards. MCEN was scored on a Kaggle T4 (fp16 backbone, MCEN depth K = 4).
This table is from step 6,198.
Bold = the conventional metric per task. Random for BoolQ is its 62.2% majority class; arithmetic is generated in-harness and is not interchangeable with published numbers.
| Benchmark | MCEN @6,198 | V2-base @96,000 | V3-Base @107,200 (T4 fp16) | Random | n |
|---|---|---|---|---|---|
HellaSwag (acc_norm) |
27.01% ± 0.44 | 27.81% | 34.16% | 25.0% | 10,042 |
ARC-Easy (acc_norm) |
36.95% ± 0.99 | 34.13% | 46.89% | 25.0% | 2,376 |
ARC-Challenge (acc_norm) |
22.95% ± 1.23 | 23.98% | 26.28% | 25.0% | 1,172 |
PIQA (acc_norm) |
56.04% ± 1.16 | 54.79% | 60.72% | 50.0% | 1,838 |
WinoGrande (acc) |
51.93% ± 1.40 | 49.96% | 51.22% | 50.0% | 1,267 |
OpenBookQA (acc_norm) |
24.40% ± 1.92 | 23.60% | 28.40% | 25.0% | 500 |
BoolQ (acc) |
61.83% ± 0.85 | 47.43% | 61.50% | 62.2% | 3,270 |
SciQ (acc) |
64.50% ± 1.51 | 51.90% | 80.30% | 25.0% | 1,000 |
LAMBADA (OpenAI) (acc) |
9.12% ± 0.40 | 3.43% | 28.49% | 0.0% | 5,153 |
RACE (acc_norm) |
27.81% ± 0.64 | 26.04% | 34.27% | 25.0% | 4,934 |
CommonsenseQA (acc_norm) |
26.78% ± 1.27 | 26.04% | 30.38% | 20.0% | 1,221 |
COPA (acc) |
51.00% ± 5.00 | 56.00% | 55.00% | 50.0% | 100 |
LogiQA (acc_norm) |
24.42% ± 1.68 | 24.73% | 24.88% | 25.0% | 651 |
WSC273 (acc) |
53.48% ± 3.02 | 50.92% | 56.41% | 50.0% | 273 |
TruthfulQA MC1 (acc) |
23.01% ± 1.47 | 26.44% | 18.73% | 22.8% | 817 |
Arithmetic (acc) |
0.00% ± 0.00 | 0.36% | 5.31% | 0.0% | 14,000 |
| WikiText-2 bits/byte | 1.4185 | 1.7676 | 1.0111 | ||
| LAMBADA perplexity (tokenizer-dependent) | 487.07 | 174.70 | 29.48 |
Generation
Greedy decoding frequently falls into repetition loops; a mild repetition penalty (1.1) reduces them, but the model still invents facts and drifts off topic on open-ended prompts, as expected this early in training. Treat it as a research checkpoint, not an assistant.
What is and is not established about MCEN itself
The MCEN stages are the novel component, and this run does not show that they help.
- Output magnitude. At step 1,600 their output was measured at 0.33% (first stage) and 0.03% (second stage) of the magnitude of the residual stream they write into, and switching either stage off changed the held-out loss by about 1e-4 nats or less.
- Matched ablation (finished). The same small backbone with and without MCEN stages, 65.5M training tokens, two seeds each, held-out loss (lower is better): plain hybrid 5.2252 / 5.2369; MCEN additive 5.2733 / 5.2486; MCEN stream 5.2883 / 5.2787. MCEN was not better than the plain hybrid, and was about 40% slower per step. Depth K = 1 / 4 / 8 differed by at most 0.0024 nats.
- The benchmark numbers above should therefore be attributed to the whole model (Mamba-3 + GDN-2 backbone, tokenizer and data), not to MCEN.
Training notes
- Between steps 4,501 and 5,900 about 13.9% of optimizer steps were skipped by the fp16 loss scaler. Cause, found by replaying the data: the output projection of the last GDN-2 block (
blocks.11.mixer.o_proj) overflows fp16 on some tokens. - The fix (an exact power-of-two rescale around that projection, and a per-token scale on the persistent read) was applied from the relaunch at step 6,198. The step-6,198 checkpoint itself was trained before the fix; the Pile, SciQ and Fact1000 numbers above were evaluated with the fixed code and are finite on all 22 Pile domains. Whether training no longer skips steps is checked on the run that continues from this checkpoint.
Repository contents
checkpoints/main565m/step_*— checkpoints (LATESTpoints to the newest; only the last two are kept) ·data/manifest_v2.json— frozen data manifest ·logs/,evals/,forensics/— training logs and diagnostics.- The training and inference code is not in this repository.
Limitations
- Early, still-training base model: knowledge is shallow (see Fact1000) and generation is unreliable.
- English / code / mathematics oriented; weak on multilingual and conversational text (see EuroParl, YoutubeSubtitles).
- Not instruction-tuned, not safety-tuned.
- Benchmarks are the author's own implementations, not a harness run; Fact1000 is internal and unpublished.