Commit History

  1. Master Release: Full unicode Indic alphabet & whitespace support (0% unk rate across all 22 languages)

    Verified signature
    76eca8a
  2. Fix regex pre-tokenizer pattern and expand master conjunct codebook for zero unk tokens

    Verified signature
    76a5d23
  3. Fix tokenizer.json specification for Hugging Face PreTrainedTokenizerFast compatibility

    Verified signature
    ecc9179
  4. Initial release: TimeMeshin-OTM-Tokenizer (15,917 Aksharas across 22 Indic Languages)

    Verified signature
    5d71c37
  5. initial commit

    Verified signature
    b4ef016