Namuun123/mn_sentencepiece_tokenizer

SentencePiece Tokenizer

This repository contains a fine-tuned SentencePiece tokenizer on Mongolian text.

Files

• tokenizer_config.json: The tokenizer configuration file
• mn_tokenizer.model: The SentencePiece model file
• mn_tokenizer.vocab: The SentencePiece vocabulary file

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Namuun123/mn_sentencepiece_tokenizer")