| ---
|
| license: mit
|
| language:
|
| - en
|
| - zh
|
| base_model:
|
| - CosyVoice3
|
| pipeline_tag: text-to-speech
|
| library_name: transformers
|
| tags:
|
| - CosyVoice3
|
| - Speech
|
| ---
|
|
|
| # CosyVoice3
|
| This version of CosyVoice3 has been converted to run on the Axera NPU using **w8a16** quantization.
|
| Compatible with Pulsar2 version: 4.2
|
|
|
| ## Convert tools links:
|
| For those who are interested in model conversion, you can try to export axmodel through the original repo :
|
| [Cosyvoice](https://github.com/FunAudioLLM/CosyVoice)
|
|
|
| [Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html)
|
|
|
| [AXera NPU HOST LLM Runtime](https://github.com/AXERA-TECH/CosyVoice3.Axera)
|
|
|
| ## Support Platform
|
|
|
| - AX650
|
| - AX650N DEMO Board
|
| - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
|
| - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)
|
|
|
| **Speech Generation**
|
| | Stage | Time |
|
| |------|------|
|
| | llm prefill ( input_token_num + prompt_token_num 在 [0,128 ] ) | 104 ms |
|
| | llm prefill ( input_token_num + prompt_token_num 在 [128,256 ] ) | 160 ms |
|
| | Decode | 14 token/s |
|
|
|
| ## How to use
|
|
|
| Download all files from this repository to the device
|
|
|
| ### 1. PrePare
|
|
|
| #### 1.1 Copy this project to AX650 Board
|
|
|
| #### 1.2 Prepare Dependencies
|
|
|
| **Running HTTP Tokenizer Server** and **Processing Prompt Speech** require these Python packages. If you run these two step on a PC, install them on the PC.
|
| ```
|
| pip3 install -r scripts/requirements.txt
|
| ```
|
|
|
| ### 2. Start HTTP Tokenizer Server
|
| ```
|
| cd scripts
|
| python scripts/cosyvoice3_tokenizer.py --host {your host} --port {your port}
|
| ```
|
|
|
|
|
| ### 3. Run on Axera Device
|
| There are 2 kinds of device, AX650 Board , AXCL aarch64 Board and AXCL x86 Board.
|
|
|
| #### 3.1 Run on AX650 Board
|
| 1) Moidfy the HTTP host in `run_ax650.sh`.
|
|
|
| 2) Run `run_ax650.sh`
|
| ```shell
|
| root@ax650 ~/CosyVoice3 # bash run_ax650.sh
|
| rm: cannot remove 'output*.wav': No such file or directory
|
| [I][ Init][ 108]: LLM init start
|
| [I][ Init][ 34]: connect http://10.122.86.184:12345 ok
|
| bos_id: 0, eos_id: 1773
|
| 7% | ███ | 2 / 27 [3.11s<42.04s, 0.64 count/s] embed_selector init ok[I][ Init][ 138]: attr.axmodel_num:24
|
| 100% | ████████████████████████████████ | 27 / 27 [10.32s<10.32s, 2.62 count/s] init post axmodel ok,remain_cmm(7178 MB)
|
| [I][ Init][ 216]: max_token_len : 1023
|
| [I][ Init][ 221]: kv_cache_size : 128, kv_cache_num: 1023
|
| [I][ Init][ 229]: prefill_token_num : 128
|
| [I][ Init][ 233]: grp: 1, prefill_max_token_num : 1
|
| [I][ Init][ 233]: grp: 2, prefill_max_token_num : 128
|
| [I][ Init][ 233]: grp: 3, prefill_max_token_num : 256
|
| [I][ Init][ 233]: grp: 4, prefill_max_token_num : 384
|
| [I][ Init][ 233]: grp: 5, prefill_max_token_num : 512
|
| [I][ Init][ 237]: prefill_max_token_num : 512
|
| [I][ Init][ 249]: LLM init ok
|
| [I][ Init][ 154]: Token2Wav init ok
|
| [I][ main][ 273]:
|
| [I][ Run][ 388]: input token num : 142, prefill_split_num : 2
|
| [I][ Run][ 422]: input_num_token:128
|
| [I][ Run][ 422]: input_num_token:14
|
| [I][ Run][ 607]: ttft: 236.90 ms
|
| [Main/Token2Wav Thread] Processing batch of 28 tokens...
|
| Successfully saved audio to output_0.wav (32-bit Float PCM).
|
| [Main/Token2Wav Thread] Processing batch of 53 tokens...
|
| Successfully saved audio to output_1.wav (32-bit Float PCM).
|
| [Main/Token2Wav Thread] Processing batch of 78 tokens...
|
| Successfully saved audio to output_2.wav (32-bit Float PCM).
|
| [Main/Token2Wav Thread] Processing batch of 78 tokens...
|
| Successfully saved audio to output_3.wav (32-bit Float PCM).
|
| [Main/Token2Wav Thread] Processing batch of 78 tokens...
|
| Successfully saved audio to output_4.wav (32-bit Float PCM).
|
| [Main/Token2Wav Thread] Processing batch of 78 tokens...
|
| Successfully saved audio to output_5.wav (32-bit Float PCM).
|
| [Main/Token2Wav Thread] Processing batch of 78 tokens...
|
| Successfully saved audio to output_6.wav (32-bit Float PCM).
|
| [Main/Token2Wav Thread] Processing batch of 78 tokens...
|
| Successfully saved audio to output_7.wav (32-bit Float PCM).
|
| [Main/Token2Wav Thread] Processing batch of 78 tokens...
|
| Successfully saved audio to output_8.wav (32-bit Float PCM).
|
| [Main/Token2Wav Thread] Processing batch of 78 tokens...
|
| Successfully saved audio to output_9.wav (32-bit Float PCM).
|
| [I][ Run][ 723]: hit eos, llm finished
|
| [I][ Run][ 753]: llm finished
|
| [Main/Token2Wav Thread] Buffer is empty and LLM finished. Exiting.
|
|
|
|
|
| [I][ Run][ 758]: total decode tokens:271
|
| [N][ Run][ 759]: hit eos,avg 21.47 token/s
|
|
|
| Successfully saved audio to output_10.wav (32-bit Float PCM).
|
| Successfully saved audio to output.wav (32-bit Float PCM).
|
|
|
| Voice generation pipeline completed.
|
| Type "q" to exit, Ctrl+c to stop current running
|
| text >>
|
| ```
|
|
|
| Output Speech：
|
| [output.wav](asset/output.wav)
|
|
|
|
|
| #### Or run on AX650 Board with Gradio GUI
|
| 1) Start server
|
| ```
|
| bash run_api_ax650.sh
|
| ```
|
| 2) Start Gradio GUI
|
| ```
|
| python scripts/gradio_demo.py
|
| ```
|
|
|
| #### 3.2 Run on AXCL aarch64 Board
|
| ```
|
| bash run_axcl_aarch64.sh
|
| ```
|
| #### Or run on AXCL aarch64 Board with Gradio GUI
|
| 1) Start server
|
| ```
|
| bash run_api_axcl_aarch64.sh
|
| ```
|
| 2) Start Gradio GUI
|
| ```
|
| python scripts/gradio_demo.py
|
| ```
|
| 3) Open the page from a browser
|
| The page url is : `https://{your device ip}:7860`
|
|
|
| Note that you need to run these two commands in the project root directory.
|
|
|
| #### 3.3 Run on AXCL x86 Board
|
| ```
|
| bash run_axcl_x86.sh
|
| ```
|
| #### Or run on AXCL aarch64 Board with Gradio GUI
|
| 1) Start server
|
| ```
|
| bash run_api_axcl_x86.sh
|
| ```
|
| 2) Start Gradio GUI
|
| ```
|
| python scripts/gradio_demo.py
|
| ```
|
| 3) Open the page from a browser
|
| The page url is : `https://{your device ip}:7860`
|
|
|
| Note that you need to run these two commands in the project root directory.
|
|
|
| 
|
|
|
| ### Optional. Process Prompt Speech
|
| If you want to replicate a specific sound, do this step.
|
| You can use audio in asset/ .
|
|
|
| #### (1). Downlaod wetext
|
| ```
|
| pip3 install modelscope
|
| modelscope download --model pengzhendong/wetext --local_dir pengzhendong/wetext
|
| ```
|
|
|
| #### (2). Process Prompt Speech
|
| Example:
|
| ```
|
| python3 scripts/process_prompt.py --prompt_text asset/zh_man1.txt --prompt_speech asset/zh_man1.wav --output zh_man1
|
| ```
|
|
|
| Pass parameters according to the actual situation.
|
| ```
|
| python3 scripts/process_prompt.py -h
|
|
|
| usage: process_prompt.py [-h] [--model_dir MODEL_DIR] [--wetext_dir WETEXT_DIR] [--sample_rate SAMPLE_RATE] [--prompt_text PROMPT_TEXT] [--prompt_speech PROMPT_SPEECH]
|
| [--output OUTPUT]
|
|
|
| options:
|
| -h, --help show this help message and exit
|
| --model_dir MODEL_DIR
|
| tokenizer configuration directionary
|
| --wetext_dir WETEXT_DIR
|
| path to wetext
|
| --sample_rate SAMPLE_RATE
|
| Sampling rate for prompt audio
|
| --prompt_text PROMPT_TEXT
|
| The text content of the prompt(reference) audio. Text or file path.
|
| --prompt_speech PROMPT_SPEECH
|
| The path to prompt(reference) audio.
|
| --output OUTPUT Output data storage directory
|
| ```
|
|
|
| After executing the above command, files like the following will be generated:
|
| ```
|
| flow_embedding.txt
|
| flow_prompt_speech_token.txt
|
| llm_embedding.txt
|
| llm_prompt_speech_token.txt
|
| prompt_speech_feat.txt
|
| prompt_text.txt
|
| ```
|
|
|
| When you run run_ax650.sh, pass the output path here to the prompt_files parameter of the run_ax650.sh script.
|
| |
