Instructions to use Severian/Jamba-Hercules with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Severian/Jamba-Hercules with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Severian/Jamba-Hercules", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Severian/Jamba-Hercules", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Severian/Jamba-Hercules", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Severian/Jamba-Hercules with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Severian/Jamba-Hercules"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Severian/Jamba-Hercules",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Severian/Jamba-Hercules

SGLang

How to use Severian/Jamba-Hercules with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Severian/Jamba-Hercules" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Severian/Jamba-Hercules",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Severian/Jamba-Hercules" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Severian/Jamba-Hercules",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Severian/Jamba-Hercules with Docker Model Runner:
```
docker model run hf.co/Severian/Jamba-Hercules
```

Training Jamba (LR etc.)

by ptrdvn - opened Apr 1, 2024

Discussion

ptrdvn

Apr 1, 2024

Hey, great model! I'm Peter from Lightblue, and we have a Jamba finetune too (lightblue/Jamba-v0.1-chat-multilingual).

Just looking at your LR, you might be an order of magnitude too low potentially, as we trained with LR= 0.0002 and it worked pretty well.

The whole training setup is here if youre interested

https://huggingface.co/lightblue/Jamba-v0.1-chat-multilingual#training

Let me know if I can help at all :)

Severian

Owner Apr 1, 2024

Hey @ptrdvn ! Thank you so much for the input and sharing your training! This is incredibly promising and makes me feel hopeful about the power of Jamba. Your model also has great outputs! Can't wait to see more iterations and/or new models you cook up

After seeing your results with the Lr and hyperparameters, I am going to definitely use your advice on my next training. Hopefully, I can get a new one trained and pushed over the next few days if the resources are there.

I appreciate the insight and willingness to help! I'll let you know you know how it goes and reach out for sure

ptrdvn

Apr 1, 2024

No worries, good luck!

Severian changed discussion status to closed Apr 2, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment