Instructions to use FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B")
model = AutoModelForCausalLM.from_pretrained("FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B

SGLang

How to use FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B with Docker Model Runner:
```
docker model run hf.co/FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B
```

Adding Evaluation Results

by leaderboard-pr-bot - opened Mar 4, 2024

base: refs/heads/main

←

from: refs/pr/2

Discussion Files changed

+118

-2

Files changed (1) hide show

README.md +118 -2

README.md CHANGED Viewed

@@ -1,10 +1,113 @@
 ---
 tags:
 - llm
 - llama
 - spellcheck
 - grammar
-license: llama2
 ---
 <!-- header start -->
@@ -136,4 +239,17 @@ After probably 10 different versions with subsequent changes, I can now say that
 The goal was to create a model that wouldn't change the style of the text. Often, LLM models, when asked to edit text, will attempt to rewrite the text even if the text is already fine. This proved to be quite challenging for such a small model where the main task was to determine the right balance between fixing the text (and not changing its style) and copying it verbatim.
-The strict model assumes that you're already a good writer that doesn't need hand-holding and that every word you've written you've meant.

 ---
+license: llama2
 tags:
 - llm
 - llama
 - spellcheck
 - grammar
+model-index:
+- name: Karen_TheEditor_V2_STRICT_Mistral_7B
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: AI2 Reasoning Challenge (25-Shot)
+      type: ai2_arc
+      config: ARC-Challenge
+      split: test
+      args:
+        num_few_shot: 25
+    metrics:
+    - type: acc_norm
+      value: 59.56
+      name: normalized accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: HellaSwag (10-Shot)
+      type: hellaswag
+      split: validation
+      args:
+        num_few_shot: 10
+    metrics:
+    - type: acc_norm
+      value: 81.79
+      name: normalized accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: MMLU (5-Shot)
+      type: cais/mmlu
+      config: all
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 59.56
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: TruthfulQA (0-shot)
+      type: truthful_qa
+      config: multiple_choice
+      split: validation
+      args:
+        num_few_shot: 0
+    metrics:
+    - type: mc2
+      value: 49.36
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: Winogrande (5-shot)
+      type: winogrande
+      config: winogrande_xl
+      split: validation
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 74.35
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B
+      name: Open LLM Leaderboard
+  - task:
+      type: text-generation
+      name: Text Generation
+    dataset:
+      name: GSM8k (5-shot)
+      type: gsm8k
+      config: main
+      split: test
+      args:
+        num_few_shot: 5
+    metrics:
+    - type: acc
+      value: 30.17
+      name: accuracy
+    source:
+      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=FPHam/Karen_TheEditor_V2_STRICT_Mistral_7B
+      name: Open LLM Leaderboard
 ---
 <!-- header start -->
 The goal was to create a model that wouldn't change the style of the text. Often, LLM models, when asked to edit text, will attempt to rewrite the text even if the text is already fine. This proved to be quite challenging for such a small model where the main task was to determine the right balance between fixing the text (and not changing its style) and copying it verbatim.
+The strict model assumes that you're already a good writer that doesn't need hand-holding and that every word you've written you've meant.
+# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
+Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_FPHam__Karen_TheEditor_V2_STRICT_Mistral_7B)
+|             Metric              |Value|
+|---------------------------------|----:|
+|Avg.                             |59.13|
+|AI2 Reasoning Challenge (25-Shot)|59.56|
+|HellaSwag (10-Shot)              |81.79|
+|MMLU (5-Shot)                    |59.56|
+|TruthfulQA (0-shot)              |49.36|
+|Winogrande (5-shot)              |74.35|
+|GSM8k (5-shot)                   |30.17|