control-pretraining-30b-filtered-gpt55-4plus-v2-base

The narrowly filtered arm's base model (V2): the Broadly Filtered arm's pretraining final (iteration 29881, 501.32B tokens on the _filtered_mini_2plus corpora) annealed through the same midtraining stage as the other arms, on corpora cut by the narrower rule: every document carrying a canary string or scored >= 4 by the GPT-5.5 judge, with every document the gpt-5-mini cost gate escalated actually judged (sudoers/control-pretraining-filter-annotated at revision 91f53004; no document unscored and none left unjudged). The corpora are the _filtered_gpt55_4plus_v2 subsets of geodesic-research/control-pretraining-datasets at revision c6419e3cb7a083d2c22bc5f865d0e93dd50d61e3. It supersedes V1 (control-pretraining-30b-filtered-gpt55-4plus-base), which applied the same rule at an annotation that left 47,454 escalated documents unjudged and retained. Only the midtraining is narrowly filtered; the pretraining behind it is the broadly filtered one, so its difference from that arm is the anneal alone.

Part of the Control Pretraining collection: Geodesic's pretraining-data-filtering study, in which an unfiltered baseline and arms with AI-scheming literature removed are compared through the same curriculum. No NVIDIA weights are used anywhere in the collection: the baseline and the broadly filtered arm are trained from random initialisation through all three stages, the end-to-end narrowly filtered arm (V2 E2E) through pretraining and midtraining, and an arm that redoes a single stage warm-starts from another arm's checkpoint, which its own description states. A knowledge-reintroduction model continues a filtered arm's midtraining final on the documents that arm's filters removed, beside a replay-only control that continues the same checkpoint for the same iterations without them. Architecture nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.

Curriculum

Stage	Iterations	Tokens	W&B run
nemotron_nano_30b_filtered_mini_2plus_pretrain (published elsewhere)	29,881	501,319,991,296 (501.3B)	control_pretrain_30b_filtered_mini_2plus_pretrain
midtraining (this repository)	3,126	52,445,577,216 (52.4B)	control_pretrain_30b_filtered_gpt55_4plus_v2_midtrain

A stage's tokens per iteration are its sequence length times its global batch (below), so the token count of a checkpoint is every earlier stage's tokens plus its iteration times its own stage's tokens per iteration.

Data and schedule

Read from each stage's training config. Shares are the config's blend weights, normalised; a corpus tokenized in shards counts once, with its shard count under Files.

nemotron_nano_30b_filtered_mini_2plus_pretrain

Sequence length 8,192, global batch 2,048 sequences (16,777,216 tokens per iteration), learning rate 1.0e-03 held constant, warmup 1% of the stage, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_full_filtered_mini_2plus	69.8%	8
geodesic-research/control-pretraining-datasets subset zyda_full_filtered_mini_2plus	19.7%	1
geodesic-research/control-pretraining-datasets subset stack_edu_filtered_mini_2plus	5.0%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_filtered_mini_2plus	4.0%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_filtered_mini_2plus	1.0%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent_filtered_mini_2plus	0.5%	1

midtraining

Sequence length 32,768, global batch 512 sequences (16,777,216 tokens per iteration), learning rate 7.5e-04 with WSD (cosine) decay to 1.0e-05, warmup 100 iterations, tokenizer geodesic-research/nemotron-base-tokenizer.

Corpus	Share	Files
geodesic-research/control-pretraining-datasets subset climbmix_long_filtered_gpt55_4plus_v2	33.4%	1
geodesic-research/control-pretraining-datasets subset nemotron_stem_sft_filtered_gpt55_4plus_v2	19.1%	1
geodesic-research/control-pretraining-datasets subset arxiv_papers_filtered_gpt55_4plus_v2	15.3%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_filtered_gpt55_4plus_v2	13.3%	1
geodesic-research/control-pretraining-datasets subset zyda_long_filtered_gpt55_4plus_v2	9.4%	1
geodesic-research/control-pretraining-datasets subset ai_safety_and_adjacent_filtered_gpt55_4plus_v2	4.4%	1
geodesic-research/control-pretraining-datasets subset stack_edu_long_filtered_gpt55_4plus_v2	2.4%	1
geodesic-research/control-pretraining-datasets subset climbmix_ai_docs_long_filtered_gpt55_4plus_v2	1.9%	1
geodesic-research/control-pretraining-datasets subset zyda_ai_docs_long_filtered_gpt55_4plus_v2	0.5%	1
geodesic-research/control-pretraining-datasets subset nemotron_wiki_rewrite_ai_docs_filtered_gpt55_4plus_v2	0.4%	1

Revisions

Every completed checkpoint is a revision; main is the final checkpoint of the midtraining stage. Load one with revision="<name>". Tokens seen count the whole curriculum up to that checkpoint; the training loss is W&B's lm loss at that iteration (blank where the run did not log it).

Revision	Stage	Iteration	Tokens seen	Training loss
midtraining_iter_600	midtraining	600	511,386,320,896 (511.4B)	1.4113
midtraining_iter_1200	midtraining	1,200	521,452,650,496 (521.5B)	1.4272
midtraining_iter_1800	midtraining	1,800	531,518,980,096 (531.5B)	1.3589
midtraining_iter_2400	midtraining	2,400	541,585,309,696 (541.6B)	1.3290
midtraining_iter_3000	midtraining	3,000	551,651,639,296 (551.7B)	1.2985
midtraining_iter_3126 (also main)	midtraining	3,126	553,765,568,512 (553.8B)	1.3332

Provenance

Exported from the Megatron torch_dist checkpoints with megatron-bridge at TP1/EP4. Each revision's megatron_run_config.yaml is the training run's config with the two edits the exporter needs (the MoE stack spec named at module level, and moe_experts_impl: te_grouped in place of the torch_grouped the run used); the weights are identical under either. This repository holds the Hugging Face conversions only, which carry no optimizer state. The Megatron checkpoints they were exported from are not kept for every run.

This is a base model. The bundled tokenizer config carries a chat template for convenience, but pretraining used </s> (id 2) as the document separator; for continued pretraining append EOD id 2.