Commit History
-
Master Release: Full unicode Indic alphabet & whitespace support (0% unk rate across all 22 languages)
Verified signature76eca8a -
Fix regex pre-tokenizer pattern and expand master conjunct codebook for zero unk tokens
Verified signature76a5d23 -
Fix tokenizer.json specification for Hugging Face PreTrainedTokenizerFast compatibility
Verified signatureecc9179 -
Initial release: TimeMeshin-OTM-Tokenizer (15,917 Aksharas across 22 Indic Languages)
Verified signature5d71c37 -
initial commit
Verified signatureb4ef016