Publish Nero Optimizer Work checkpoint: SimOW

README.md

@@ -0,0 +1,63 @@
+---
+language:
+- en
+library_name: mlx
+pipeline_tag: text-generation
+tags:
+- nero
+- mlx
+- optimizer-research
+- simow
+---
+
+# Nero Optimizer Work — SimOW
+
+This repository contains the final MLX checkpoint for the **SimOW** arm of
+the Nero Optimizer Work research sweep.
+
+## Summary
+
+- Optimizer: simow
+- Requested learning rate: the run's configured default/controller schedule
+- Training budget: **500,000,000 tokens**
+- Context length: **128 tokens**
+- Backend: **Apple MLX**
+- Model family: matched dense-deep decoder, 2,048-token vocabulary, 128-wide residual stream, 6 blocks, 32-dimensional attention heads, 148-wide gated MLP
+- Stored model parameters: approximately **999,680**
+- Final training loss: **5.371432**
+- Final logged throughput: **418,988 tokens/s**
+- Tail throughput: **419,229 tokens/s (median of the final logged samples)**
+- Final tokens seen: **500,000,000**
+
+## Files
+
+- model.npz — final MLX model weights from checkpoint_000500000000
+- state.json — checkpoint state metadata
+- run.json — frozen run configuration
+- metrics.jsonl — complete training metrics log
+- config.json — model and publication metadata
+
+## Evaluation
+
+The figures above are training-run measurements. An independent held-out
+validation artifact was not saved with these runs, so this card does not claim
+a validation score. Compare checkpoints using the same frozen evaluation pass
+before drawing quality conclusions.
+
+## Intended use and limitations
+
+This is an experimental research checkpoint, not an instruction-tuned or
+production-ready language model. It is published to make the optimizer
+comparison reproducible. The raw MLX weights require a compatible local MLX
+loader and are not a Transformers checkpoint.
+
+## Reproducibility
+
+All arms use the same prepared finephrase-balanced-500m-2k-v2 token stream,
+128-token context, 32-example batches, and 500M-token target. The full local
+training log is included in metrics.jsonl.
+
+## License
+
+No new model license is asserted by this experimental publication. Review the
+source-data terms before redistribution or downstream use.

config.json

@@ -0,0 +1,36 @@
+{
+  "model_type": "nero_dense_control_mlx",
+  "architectures": [
+    "DenseControlLM"
+  ],
+  "framework": "mlx",
+  "optimizer": "simow",
+  "vocab_size": 2048,
+  "context_length": 128,
+  "width": 128,
+  "blocks": 6,
+  "head_dim": 32,
+  "mlp_width": 148,
+  "stored_parameters": 999680,
+  "training_tokens": 500000000,
+  "batch_size": 32,
+  "requested_learning_rate": null,
+  "momentum_beta": null,
+  "source_run": "mlx_simow_500m_ctx128",
+  "checkpoint": "checkpoint_000500000000",
+  "metrics": {
+    "final_train_loss": 5.371432304382324,
+    "final_tokens_seen": 500000000,
+    "final_step": 122071,
+    "final_tokens_per_second": 418987.66562184633,
+    "tail_median_tokens_per_second": 419229.3335327129,
+    "logged_metric_rows": 1221
+  },
+  "final_log_row": {
+    "step": 122071,
+    "tokens_seen": 500000000,
+    "loss": 5.371432304382324,
+    "lr": 1.1025e-06,
+    "tokens_per_second": 418987.66562184633
+  }
+}

model.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f15e4bcb4467786dd1f791758c21014a0a3330ebfde5a1ad5838238ebe938eb8
+size 4013862

run.json

@@ -0,0 +1,8 @@
+{
+  "optimizer": "simow",
+  "backend": "mlx",
+  "manifest": "/Users/jonah/Documents/ChatGPT/Training/data/prepared/finephrase-balanced-500m-2k-v2/manifest.json",
+  "context": 128,
+  "target_tokens": 500000000,
+  "batch_size": 32
+}

state.json

@@ -0,0 +1 @@
+{"tokens_seen": 500000000, "step": 122071, "lr": 1.1025e-06}