Instructions to use nextai-team/Moe-2x7b-QA-Code with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nextai-team/Moe-2x7b-QA-Code with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nextai-team/Moe-2x7b-QA-Code")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nextai-team/Moe-2x7b-QA-Code")
model = AutoModelForCausalLM.from_pretrained("nextai-team/Moe-2x7b-QA-Code", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nextai-team/Moe-2x7b-QA-Code with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nextai-team/Moe-2x7b-QA-Code"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nextai-team/Moe-2x7b-QA-Code",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nextai-team/Moe-2x7b-QA-Code

SGLang

How to use nextai-team/Moe-2x7b-QA-Code with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nextai-team/Moe-2x7b-QA-Code" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nextai-team/Moe-2x7b-QA-Code",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nextai-team/Moe-2x7b-QA-Code" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nextai-team/Moe-2x7b-QA-Code",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nextai-team/Moe-2x7b-QA-Code with Docker Model Runner:
```
docker model run hf.co/nextai-team/Moe-2x7b-QA-Code
```

nextai-team commited on Feb 9, 2024

Commit

fc5ad3e

verified ·

1 Parent(s): 3a0e6d4

Update README.md

Browse files

Files changed (1) hide show

README.md +2 -1

README.md CHANGED Viewed

@@ -16,6 +16,7 @@ metrics:
 **Model Description**
 Moe-2x7b-QA-Code is a state-of-the-art language model specialized in Question Answering (QA) and code-related queries. Leveraging the Mixture of Experts (MoE) architecture, this model has been trained on a diverse dataset encompassing technical documentation, forums, and code repositories to provide accurate and context-aware responses to both technical and general questions.
 ***How to Use***
@@ -58,7 +59,7 @@ Performance demonstrates significant improvements in accuracy and relevance over
 This model, like any other, has its limitations. It may exhibit biases inherent in the training data or struggle with questions outside its training scope. Users should critically assess the model's outputs, especially for sensitive or critical applications.
-Training Data:
 The Moe-2x7b-QA-Code model was trained on a curated dataset comprising technical documentation, Stack Overflow posts, GitHub repositories, and other code-related content. This extensive training set ensures the model's proficiency in understanding and generating code-related content alongside general language understanding.

 **Model Description**
 Moe-2x7b-QA-Code is a state-of-the-art language model specialized in Question Answering (QA) and code-related queries. Leveraging the Mixture of Experts (MoE) architecture, this model has been trained on a diverse dataset encompassing technical documentation, forums, and code repositories to provide accurate and context-aware responses to both technical and general questions.
 ***How to Use***
 This model, like any other, has its limitations. It may exhibit biases inherent in the training data or struggle with questions outside its training scope. Users should critically assess the model's outputs, especially for sensitive or critical applications.
+**Training Data**
 The Moe-2x7b-QA-Code model was trained on a curated dataset comprising technical documentation, Stack Overflow posts, GitHub repositories, and other code-related content. This extensive training set ensures the model's proficiency in understanding and generating code-related content alongside general language understanding.