Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+data/raw/merged_text/corpus.txt filter=lfs diff=lfs merge=lfs -text

data/README.md

@@ -0,0 +1,213 @@
+# Data Module
+
+This module handles all data preprocessing, tokenization, and preparation for training.
+
+## Overview
+
+The data pipeline converts raw text into binary token files optimized for training:
+- **Raw text collection** from multiple sources
+- **Tokenization** using BPE tokenizer
+- **Binary serialization** for efficient loading
+- **Train/validation splitting**
+
+## Directory Structure
+
+```
+data/
+├── raw/                    # Raw text sources
+│   ├── books/             # Book corpus
+│   ├── wikipedia/         # Wikipedia dumps
+│   ├── fineweb/           # Web crawl data
+│   └── merged_text/
+│       └── corpus.txt     # Combined corpus
+├── bin/                   # Tokenized binary files
+│   ├── train.bin         # Training data (uint16)
+│   └── val.bin           # Validation data (uint16)
+└── prepare_data.py       # Tokenization script
+```
+
+## Data Processing Pipeline
+
+```
+┌─────────────────────────────────────────────┐
+│ 1. Raw Text Sources                         │
+│    - Books: 15 files                        │
+│    - Wikipedia: 3 dumps                     │
+│    - FineWeb: 1 crawl                       │
+└──────────────────┬──────────────────────────┘
+                   │
+                   ▼
+┌─────────────────────────────────────────────┐
+│ 2. Merge & Clean                            │
+│    → corpus.txt (all text combined)         │
+└──────────────────┬──────────────────────────┘
+                   │
+                   ▼
+┌─────────────────────────────────────────────┐
+│ 3. Tokenize (prepare_data.py)              │
+│    - Load BPE tokenizer                     │
+│    - Process line-by-line                   │
+│    - Append EOS tokens                      │
+└──────────────────┬──────────────────────────┘
+                   │
+                   ▼
+┌─────────────────────────────────────────────┐
+│ 4. Convert to NumPy (uint16)               │
+│    - Vocab size: 32,000 fits in uint16     │
+│    - Memory efficient (2 bytes/token)       │
+└──────────────────┬──────────────────────────┘
+                   │
+                   ▼
+┌─────────────────────────────────────────────┐
+│ 5. Train/Val Split (90/10)                 │
+│    - train.bin: 325M tokens                 │
+│    - val.bin: 36M tokens                    │
+└─────────────────────────────────────────────┘
+```
+
+## Data Preparation Script
+
+**File**: `prepare_data.py`
+
+```python
+import numpy as np
+from transformers import AutoTokenizer
+from tqdm import tqdm
+
+# 1. Load tokenizer
+tokenizer = AutoTokenizer.from_pretrained("Tokenizer/BPE")
+eos_id = tokenizer.eos_token_id
+
+# 2. Read corpus
+with open("data/raw/merged_text/corpus.txt") as f:
+    lines = f.readlines()
+
+# 3. Tokenize
+all_tokens = []
+for line in tqdm(lines):
+    tokens = tokenizer.encode(line.strip())
+    tokens.append(eos_id)  # Mark end of line
+    all_tokens.extend(tokens)
+
+# 4. Convert to uint16
+ids = np.array(all_tokens, dtype=np.uint16)
+
+# 5. Split
+val_count = int(len(ids) * 0.1)
+train_ids = ids[:-val_count]
+val_ids = ids[-val_count:]
+
+# 6. Save
+train_ids.tofile("data/bin/train.bin")
+val_ids.tofile("data/bin/val.bin")
+```
+
+## Example: Text → Tokens
+
+**Input Text** (`corpus.txt`):
+```
+The quick brown fox jumps over the lazy dog.
+Machine learning is transforming the world.
+```
+
+**Tokenization Process**:
+
+```
+Line 1: "The quick brown fox jumps over the lazy dog."
+  Tokens: [1, 334, 3855, 288, 267, 2959, 354, 267, 12397, 8885, 2]
+          [<s>, The, quick, brown, fox, jumps, over, the, lazy, dog, </s>]
+
+Line 2: "Machine learning is transforming the world."
+  Tokens: [1, 5234, 1234, 456, 7890, 267, 9876, 2]
+          [<s>, Machine, learning, is, transforming, the, world, </s>]
+
+Combined: [1, 334, 3855, ..., 2, 1, 5234, ..., 2]
+```
+
+**Binary Format**:
+
+```
+train.bin structure:
+  Byte 0-1:   Token 0 (uint16)
+  Byte 2-3:   Token 1 (uint16)
+  Byte 4-5:   Token 2 (uint16)
+  ...
+  Byte N-2:N  Token N/2 (uint16)
+
+Total size: 325,004,796 tokens × 2 bytes = ~650 MB
+```
+
+## Dataset Statistics
+
+### Corpus Size
+
+```
+Raw Text:
+  - Total files: 19
+  - Total size: ~1.4 GB
+  - Total lines: ~5.2M
+
+Tokenized:
+  - Total tokens: 361,116,440
+  - Train tokens: 325,004,796 (90%)
+  - Val tokens: 36,111,644 (10%)
+```
+
+## Usage
+
+### Prepare Data
+
+```bash
+# Tokenize corpus
+python data/prepare_data.py
+```
+
+**Output:**
+```
+Loading tokenizer from Tokenizer/BPE...
+Vocab size: 32000
+EOS ID: 2
+Reading data/raw/merged_text/corpus.txt...
+Total lines: 5,234,567
+Tokenizing...
+100%|████████████| 5.2M/5.2M [02:34<00:00]
+Total tokens: 361,116,440
+Train tokens: 325,004,796
+Val tokens:   36,111,644
+✅ Saved binary files to data/bin/
+```
+
+### Load in Training
+
+```python
+from train.dataloader import DataLoader
+
+loader = DataLoader("data/bin", batch_size=16, block_size=512, split="train")
+x, y = loader.get_batch(device="cuda")
+
+# x: [16, 512] input tokens
+# y: [16, 512] target tokens (shifted by 1)
+```
+
+## Memory-Mapped Loading
+
+The binary files are loaded using `np.memmap` for efficiency:
+
+```python
+# Traditional loading (BAD)
+data = np.fromfile("train.bin", dtype=np.uint16)  # Loads 650MB into RAM!
+
+# Memory-mapped loading (GOOD)
+data = np.memmap("train.bin", dtype=np.uint16, mode='r')  # OS handles paging
+```
+
+**Benefits:**
+- **No RAM overhead**: File stays on disk
+- **Fast random access**: OS caches hot pages
+- **Scalable**: Works with TB-scale datasets
+
+## References
+
+- [The Pile: An 800GB Dataset](https://arxiv.org/abs/2101.00027)
+- [Data Quality for Language Models](https://arxiv.org/abs/2201.06009)
+- [Efficient Data Loading](https://pytorch.org/docs/stable/data.html)

data/bin/train.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de480e746786af5675ce42681e009835772c7688567c16ceb429239dfb8eb38b
+size 650009592

data/bin/val.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0715cff0afb66a0f922639521ec5aaa7e75803134b4283df10501f981a20954
+size 72223288

data/prepare_data.py

@@ -0,0 +1,72 @@
+import os
+import numpy as np
+from transformers import AutoTokenizer
+from tqdm import tqdm
+
+def process_data():
+    # 1. Config
+    input_file_path = "data/raw/merged_text/corpus.txt"  # PATH TO YOUR DATA
+    tokenizer_path = "Tokenizer/BPE"                     # PATH TO YOUR NEW TOKENIZER
+    output_dir = "data/bin"
+    val_split_ratio = 0.1  # 10% for validation
+
+    os.makedirs(output_dir, exist_ok=True)
+
+    # 2. Load Tokenizer
+    print(f"Loading tokenizer from {tokenizer_path}...")
+    tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
+    
+    # Ensure eos_token is present (usually ID 2)
+    eos_id = tokenizer.eos_token_id
+    print(f"Vocab size: {tokenizer.vocab_size}")
+    print(f"EOS ID: {eos_id}")
+
+    # 3. Read Data
+    print(f"Reading {input_file_path}...")
+    with open(input_file_path, 'r', encoding='utf-8') as f:
+        # Read all lines
+        lines = f.readlines()
+        
+    print(f"Total lines: {len(lines):,}")
+
+    # 4. Tokenize
+    # We use a simple list comprehension for the 80M scale. 
+    # For 100B scale, we would use parallel processing (multiprocessing).
+    print("Tokenizing...")
+    all_tokens = []
+    
+    # Using tqdm for progress bar
+    for line in tqdm(lines):
+        text = line.strip()
+        if not text:
+            continue
+            
+        # Encode text and append EOS token
+        # This tells the model where one sentence ends and the next begins
+        tokens = tokenizer.encode(text)
+        tokens.append(eos_id) 
+        all_tokens.extend(tokens)
+
+    token_count = len(all_tokens)
+    print(f"Total tokens: {token_count:,}")
+
+    # 5. Convert to Numpy (uint16 saves 50% RAM)
+    # 32,000 fits easily in uint16 (max 65,535)
+    ids = np.array(all_tokens, dtype=np.uint16)
+
+    # 6. Split Train/Val
+    val_count = int(token_count * val_split_ratio)
+    train_ids = ids[:-val_count]
+    val_ids = ids[-val_count:]
+
+    print(f"Train tokens: {len(train_ids):,}")
+    print(f"Val tokens:   {len(val_ids):,}")
+
+    # 7. Save to disk (Memory Mapped friendly)
+    train_ids.tofile(os.path.join(output_dir, "train.bin"))
+    val_ids.tofile(os.path.join(output_dir, "val.bin"))
+    
+    print(f"✅ Saved binary files to {output_dir}/")
+
+if __name__ == "__main__":
+    process_data()

data/raw/books/.gitattributes

@@ -0,0 +1,27 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

data/raw/books/README.md

@@ -0,0 +1,344 @@
+---
+annotations_creators:
+- no-annotation
+language_creators:
+- crowdsourced
+language:
+- en
+license:
+- cc-by-sa-3.0
+- gfdl
+multilinguality:
+- monolingual
+size_categories:
+- 1M<n<10M
+source_datasets:
+- original
+task_categories:
+- text-generation
+- fill-mask
+task_ids:
+- language-modeling
+- masked-language-modeling
+paperswithcode_id: wikitext-2
+pretty_name: WikiText
+dataset_info:
+- config_name: wikitext-103-raw-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1305088
+    num_examples: 4358
+  - name: train
+    num_bytes: 546500949
+    num_examples: 1801350
+  - name: validation
+    num_bytes: 1159288
+    num_examples: 3760
+  download_size: 315466397
+  dataset_size: 548965325
+- config_name: wikitext-103-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1295575
+    num_examples: 4358
+  - name: train
+    num_bytes: 545141915
+    num_examples: 1801350
+  - name: validation
+    num_bytes: 1154751
+    num_examples: 3760
+  download_size: 313093838
+  dataset_size: 547592241
+- config_name: wikitext-2-raw-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1305088
+    num_examples: 4358
+  - name: train
+    num_bytes: 11061717
+    num_examples: 36718
+  - name: validation
+    num_bytes: 1159288
+    num_examples: 3760
+  download_size: 7747362
+  dataset_size: 13526093
+- config_name: wikitext-2-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1270947
+    num_examples: 4358
+  - name: train
+    num_bytes: 10918118
+    num_examples: 36718
+  - name: validation
+    num_bytes: 1134123
+    num_examples: 3760
+  download_size: 7371282
+  dataset_size: 13323188
+configs:
+- config_name: wikitext-103-raw-v1
+  data_files:
+  - split: test
+    path: wikitext-103-raw-v1/test-*
+  - split: train
+    path: wikitext-103-raw-v1/train-*
+  - split: validation
+    path: wikitext-103-raw-v1/validation-*
+- config_name: wikitext-103-v1
+  data_files:
+  - split: test
+    path: wikitext-103-v1/test-*
+  - split: train
+    path: wikitext-103-v1/train-*
+  - split: validation
+    path: wikitext-103-v1/validation-*
+- config_name: wikitext-2-raw-v1
+  data_files:
+  - split: test
+    path: wikitext-2-raw-v1/test-*
+  - split: train
+    path: wikitext-2-raw-v1/train-*
+  - split: validation
+    path: wikitext-2-raw-v1/validation-*
+- config_name: wikitext-2-v1
+  data_files:
+  - split: test
+    path: wikitext-2-v1/test-*
+  - split: train
+    path: wikitext-2-v1/train-*
+  - split: validation
+    path: wikitext-2-v1/validation-*
+---
+
+# Dataset Card for "wikitext"
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:** [https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/)
+- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Paper:** [Pointer Sentinel Mixture Models](https://arxiv.org/abs/1609.07843)
+- **Point of Contact:** [Stephen Merity](mailto:[email protected])
+- **Size of downloaded dataset files:** 391.41 MB
+- **Size of the generated dataset:** 1.12 GB
+- **Total amount of disk used:** 1.52 GB
+
+### Dataset Summary
+
+ The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified
+ Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License.
+
+Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over
+110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation
+and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models
+that can take advantage of long term dependencies.
+
+Each subset comes in two different variants:
+- Raw (for character level work) contain the raw tokens, before the addition of the <unk> (unknown) tokens.
+- Non-raw (for word level work) contain only the tokens in their vocabulary (wiki.train.tokens, wiki.valid.tokens, and wiki.test.tokens).
+  The out-of-vocabulary tokens have been replaced with the the <unk> token.
+
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### wikitext-103-raw-v1
+
+- **Size of downloaded dataset files:** 191.98 MB
+- **Size of the generated dataset:** 549.42 MB
+- **Total amount of disk used:** 741.41 MB
+
+An example of 'validation' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" The gold dollar or gold one @-@ dollar piece was a coin struck as a regular issue by the United States Bureau of the Mint from..."
+}
+```
+
+#### wikitext-103-v1
+
+- **Size of downloaded dataset files:** 190.23 MB
+- **Size of the generated dataset:** 548.05 MB
+- **Total amount of disk used:** 738.27 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to..."
+}
+```
+
+#### wikitext-2-raw-v1
+
+- **Size of downloaded dataset files:** 4.72 MB
+- **Size of the generated dataset:** 13.54 MB
+- **Total amount of disk used:** 18.26 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" The Sinclair Scientific Programmable was introduced in 1975 , with the same case as the Sinclair Oxford . It was larger than t..."
+}
+```
+
+#### wikitext-2-v1
+
+- **Size of downloaded dataset files:** 4.48 MB
+- **Size of the generated dataset:** 13.34 MB
+- **Total amount of disk used:** 17.82 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to..."
+}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### wikitext-103-raw-v1
+- `text`: a `string` feature.
+
+#### wikitext-103-v1
+- `text`: a `string` feature.
+
+#### wikitext-2-raw-v1
+- `text`: a `string` feature.
+
+#### wikitext-2-v1
+- `text`: a `string` feature.
+
+### Data Splits
+
+|       name        | train |validation|test|
+|-------------------|------:|---------:|---:|
+|wikitext-103-raw-v1|1801350|      3760|4358|
+|wikitext-103-v1    |1801350|      3760|4358|
+|wikitext-2-raw-v1  |  36718|      3760|4358|
+|wikitext-2-v1      |  36718|      3760|4358|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+The dataset is available under the [Creative Commons Attribution-ShareAlike License (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/).
+
+### Citation Information
+
+```
+@misc{merity2016pointer,
+      title={Pointer Sentinel Mixture Models},
+      author={Stephen Merity and Caiming Xiong and James Bradbury and Richard Socher},
+      year={2016},
+      eprint={1609.07843},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+
+### Contributions
+
+Thanks to [@thomwolf](https://github.com/thomwolf), [@lewtun](https://github.com/lewtun), [@patrickvonplaten](https://github.com/patrickvonplaten), [@mariamabarham](https://github.com/mariamabarham) for adding this dataset.

data/raw/books/wikitext-103-raw-v1/test-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f1bea067869d04849c0f975a2b29c4ff47d867f484f5010ea5e861eab246d91
+size 732610

data/raw/books/wikitext-103-raw-v1/train-00000-of-00002.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74da360f23826045b3e6ac6375411fdb15f003030aa74f2596ed08b857cb9212
+size 156987808

data/raw/books/wikitext-103-raw-v1/train-00001-of-00002.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba090ac30dbf5461e8dcbdd1a1b8e6f3cf9c2c756d64f0c1220450acd514f720
+size 157088770

data/raw/books/wikitext-103-raw-v1/validation-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:204929b7ff9d6184953f867dedb860e40aa69c078fc1e54b3baaa8fb28511c4c
+size 657209

data/raw/books/wikitext-103-v1/test-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abdfc9f83b1103b502924072460d4c92f277c9b49c313cef3e48cfcf7428e125
+size 721735

data/raw/books/wikitext-103-v1/train-00000-of-00002.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2ecca8c3250e79518e45d125f3a9a757d8014f6b2d8435c602be87c1f79ec3b
+size 155788327

data/raw/books/wikitext-103-v1/train-00001-of-00002.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:720f2503551f33c25bb822aad74d699fee4d5331a7373d0c262f1bfb01354fcf
+size 155928670

data/raw/books/wikitext-103-v1/validation-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a586125adab06f115018c43507ac267ea70850ce6218cbb96e08bb3b4db0899b
+size 655106

data/raw/books/wikitext-2-raw-v1/test-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f1bea067869d04849c0f975a2b29c4ff47d867f484f5010ea5e861eab246d91
+size 732610

data/raw/books/wikitext-2-raw-v1/train-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e83889baabc497075506f91975be5fac0d45c5290b6b20582c8cd1e853d0c9f7
+size 6357543

data/raw/books/wikitext-2-raw-v1/validation-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:204929b7ff9d6184953f867dedb860e40aa69c078fc1e54b3baaa8fb28511c4c
+size 657209

data/raw/books/wikitext-2-v1/test-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6b3913da714b63a60a571698b20ff15441fb015783ea1b5285f707d4f2f00a9
+size 685430

data/raw/books/wikitext-2-v1/train-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dfc27e4360c639dc1fba1e403bfffd53af4a5c75d5363b5724d49bf12d07cce6
+size 6068114

data/raw/books/wikitext-2-v1/validation-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:717de9a0c1c0b0b1dfdd8f1e6ad8a30ece618bbde81f5da8207277547d324215
+size 617738

data/raw/extract_all.py

@@ -0,0 +1,53 @@
+import os
+import pyarrow.parquet as pq
+from glob import glob
+from tqdm import tqdm
+
+INPUT_DIRS = [
+    "books",
+    "fineweb",
+    "wikipedia",
+]
+
+OUTPUT_DIR = "merged_text"
+os.makedirs(OUTPUT_DIR, exist_ok=True)
+OUT_FILE = os.path.join(OUTPUT_DIR, "corpus.txt")
+
+def extract_text_from_parquet(path):
+    try:
+        table = pq.read_table(path)
+        df = table.to_pandas()
+
+        # Look for likely text column
+        for col in ["text", "content", "document", "article", "source"]:
+            if col in df.columns:
+                return df[col].astype(str).tolist()
+
+        # Fallback: take the first string-like column
+        for col in df.columns:
+            if df[col].dtype == object:
+                return df[col].astype(str).tolist()
+
+        return []
+    except Exception as e:
+        print(f"Error reading {path}: {e}")
+        return []
+
+all_parquet_files = []
+for d in INPUT_DIRS:
+    all_parquet_files.extend(glob(f"{d}/**/*.parquet", recursive=True))
+
+print("Total parquet files found:", len(all_parquet_files))
+
+with open(OUT_FILE, "w", encoding="utf-8") as fout:
+    for file in tqdm(all_parquet_files, desc="Extracting text"):
+        texts = extract_text_from_parquet(file)
+        for t in texts:
+            t = t.strip()
+            if len(t) < 50:
+                continue
+            if not any(c.isalpha() for c in t):
+                continue
+            fout.write(t + "\n\n")
+
+print("DONE! Saved merged corpus →", OUT_FILE)

data/raw/fineweb/.gitattributes

@@ -0,0 +1,59 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.lz4 filter=lfs diff=lfs merge=lfs -text
+*.mds filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+# Audio files - uncompressed
+*.pcm filter=lfs diff=lfs merge=lfs -text
+*.sam filter=lfs diff=lfs merge=lfs -text
+*.raw filter=lfs diff=lfs merge=lfs -text
+# Audio files - compressed
+*.aac filter=lfs diff=lfs merge=lfs -text
+*.flac filter=lfs diff=lfs merge=lfs -text
+*.mp3 filter=lfs diff=lfs merge=lfs -text
+*.ogg filter=lfs diff=lfs merge=lfs -text
+*.wav filter=lfs diff=lfs merge=lfs -text
+# Image files - uncompressed
+*.bmp filter=lfs diff=lfs merge=lfs -text
+*.gif filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.tiff filter=lfs diff=lfs merge=lfs -text
+# Image files - compressed
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text
+*.webp filter=lfs diff=lfs merge=lfs -text
+# Video files - compressed
+*.mp4 filter=lfs diff=lfs merge=lfs -text
+*.webm filter=lfs diff=lfs merge=lfs -text

data/raw/fineweb/train-00000-of-00099.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7c386575467e252ff81316a193bb1e07ebe067aec34cbbd5076ee7dd2ffe42f
+size 289110403

data/raw/merged_text/corpus.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ad42d10157bf9f296b7752bbabc47b936de7af220927c9be54ceeb2ecada01d
+size 1599143862

data/raw/verify_compression_ratio.py

@@ -0,0 +1,15 @@
+from transformers import PreTrainedTokenizerFast
+
+tok = PreTrainedTokenizerFast(tokenizer_file="tokenizer/hf/tokenizer.json")
+
+import os
+
+with open("tokenizer/corpus.txt","r") as f:
+    text = f.read()
+
+num_bytes = len(text.encode("utf-8"))
+num_tokens = len(tok.encode(text))
+
+ratio = num_bytes / num_tokens
+print("Compression ratio:", ratio)
+# Expected ratio is around 3.5 to 4.5 for a good tokenizer

data/raw/wikipedia/.gitattributes

@@ -0,0 +1,59 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.lz4 filter=lfs diff=lfs merge=lfs -text
+*.mds filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+# Audio files - uncompressed
+*.pcm filter=lfs diff=lfs merge=lfs -text
+*.sam filter=lfs diff=lfs merge=lfs -text
+*.raw filter=lfs diff=lfs merge=lfs -text
+# Audio files - compressed
+*.aac filter=lfs diff=lfs merge=lfs -text
+*.flac filter=lfs diff=lfs merge=lfs -text
+*.mp3 filter=lfs diff=lfs merge=lfs -text
+*.ogg filter=lfs diff=lfs merge=lfs -text
+*.wav filter=lfs diff=lfs merge=lfs -text
+# Image files - uncompressed
+*.bmp filter=lfs diff=lfs merge=lfs -text
+*.gif filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.tiff filter=lfs diff=lfs merge=lfs -text
+# Image files - compressed
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text
+*.webp filter=lfs diff=lfs merge=lfs -text
+# Video files - compressed
+*.mp4 filter=lfs diff=lfs merge=lfs -text
+*.webm filter=lfs diff=lfs merge=lfs -text

data/raw/wikipedia/README.md

@@ -0,0 +1,26 @@
+---
+dataset_info:
+  features:
+  - name: text
+    dtype: string
+  - name: tokens
+    sequence: int64
+  - name: token_count
+    dtype: int64
+  splits:
+  - name: train
+    num_bytes: 167968257.38066393
+    num_examples: 69445
+  - name: test
+    num_bytes: 1726968.6193360796
+    num_examples: 714
+  download_size: 49543706
+  dataset_size: 169695226.0
+configs:
+- config_name: default
+  data_files:
+  - split: train
+    path: data/train-*
+  - split: test
+    path: data/test-*
+---

data/raw/wikipedia/data/test-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:062222ddf69aa636b56a2c48299ece565eb85fbee8d9efbce0a1f47b436617ac
+size 511192

data/raw/wikipedia/data/train-00000-of-00001.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4785fbc0ae815936f34a7923af854ab7752a456d64f5fc497ed7f234330afd94
+size 49032514