Instructions to use mattshumer/ref_70_e3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mattshumer/ref_70_e3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mattshumer/ref_70_e3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mattshumer/ref_70_e3")
model = AutoModelForCausalLM.from_pretrained("mattshumer/ref_70_e3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use mattshumer/ref_70_e3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "mattshumer/ref_70_e3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/ref_70_e3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/mattshumer/ref_70_e3

SGLang

How to use mattshumer/ref_70_e3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mattshumer/ref_70_e3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/ref_70_e3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mattshumer/ref_70_e3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mattshumer/ref_70_e3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use mattshumer/ref_70_e3 with Docker Model Runner:
```
docker model run hf.co/mattshumer/ref_70_e3
```

Third party evaluation on this so-called "LLM".

by Enigrand - opened Sep 9, 2024

Discussion

Enigrand

Sep 9, 2024

•

edited Sep 9, 2024

Another followup from Artificial Analysis:

Reflection 70B update: Quick note on timeline and outstanding questions from our perspective

Timeline:

We tested the initial Reflection 70B release and saw worse performance than Llama 3.1 70B.
We were given access to a private API which we tested and saw impressive performance but not to the level of the initial claims. As this testing was performed on a private API, we were not able to independently verify exactly what we were testing.
Since then, there have been additional HF releases which some providers have hosted. The latest version appears to be: huggingface.co/mattshumer/re…. We are seeing significantly worse results when benchmarking ref_70_e3 than what we saw via the private API.

Outstanding questions:

We are not clear on why a version would be published which is not the version we tested via Reflection’s private API.
We are not clear why the model weights of the version we tested would not be released yet.

As soon as the weights are released on Hugging Face, we plan to re-test and compare to our evaluation of the private endpoint.

P.S. Just for the record, t****.ai stackunseen hyped up this "LLM". If you believe the hype, just trust their results.

nisten

Sep 9, 2024

Why are you posting articles and not your own inference for a local model?

Enigrand

Sep 9, 2024

•

edited Sep 9, 2024

@nisten , why don't you post your evaluation results here so everyone can see clearly.

Or if you want to claim their results are wrong or something, show your evidences here or my post collecting third-party results.

Enigrand

Sep 9, 2024

This comment has been hidden

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment