ace-1
/

mgpt2-tokenizer

Model card Files Files and versions

mgpt2-tokenizer / tokenization_mgpt2.py

ace-1's picture

Publish mgpt2 tokenizer (GPT-2 exact merges) + eval metrics

24cf5c6 verified 3 months ago

history blame contribute delete

80 Bytes

	from tokenizer.hf_tokenizer import MGPT2Tokenizer

	__all__ = ['MGPT2Tokenizer']