Instructions to use Athekunal/ultrabert-jev-380m-softmax-yesno with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Athekunal/ultrabert-jev-380m-softmax-yesno with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="Athekunal/ultrabert-jev-380m-softmax-yesno")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("Athekunal/ultrabert-jev-380m-softmax-yesno") model = AutoModelForMaskedLM.from_pretrained("Athekunal/ultrabert-jev-380m-softmax-yesno", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Athekunal/ultrabert-jev-380m-softmax-yesno
A decision model: it reads a state, a question and a list of options, and picks one option in a single forward pass, without generating text. It is UltraBERT-380M fully fine-tuned with one mask token in front of each option. Option k is scored from the masked-token logits at its mask.
Recipe: softmax over " yes" minus " no". Trained for one epoch (batch 32, learning rate 1.5e-5) on the 13-dataset Reason+Long mixture of LLM-as-Jev, with Nimble added.
| This checkpoint | Accuracy (mean over 13 datasets) |
|---|---|
| Dev split | 72.8% |
| Test split | 72.6% |
Usage
Load the weights in fp32. The masked-token logits are large (around 200), and bf16 weights changed accuracy by up to 8 points on long documents (MuSiQue) in our tests. All reported numbers use fp32 weights.
Use the prompt below exactly. The model was trained on this format with the tokens
" yes"and" no"; plain fill-mask use will not give these results.
Tested with transformers 5.20.2 and torch 2.9, which include the UltraBERT model class. Do not pass trust_remote_code=True: the base repo's remote code names the weights differently and would leave them unloaded.
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
REPO = "Athekunal/ultrabert-jev-380m-softmax-yesno"
tok = AutoTokenizer.from_pretrained(REPO)
# Load the weights in fp32. bf16 weights change the results (up to 8 points on MuSiQue).
model = AutoModelForMaskedLM.from_pretrained(REPO, dtype=torch.float32)
model = model.to("cuda").eval()
YES = tok(" yes", add_special_tokens=False)["input_ids"][0]
NO = tok(" no", add_special_tokens=False)["input_ids"][0]
def build_prompt(state: list[str], question: str, options: list[str]) -> str:
"""The exact prompt the model was trained on: one mask in front of each option."""
text = "State: " + "".join(part + "\n\n" for part in state) + question + "\n"
text += "Mark the correct option with yes and every other option with no.\nOptions:\n"
text += "\n".join(f"[{k}] {tok.mask_token} {o}" for k, o in enumerate(options, 1))
return text
@torch.no_grad()
def decide(state: list[str], question: str, options: list[str], score: str = "yes-no"):
"""(index of the chosen option, scores s_k). This checkpoint uses score="yes-no"."""
enc = tok(build_prompt(state, question, options), return_tensors="pt").to(model.device)
logits = model(**enc).logits[0]
masks = (enc["input_ids"][0] == tok.mask_token_id).nonzero()[:, 0]
s = logits[masks, YES] if score == "yes" else logits[masks, YES] - logits[masks, NO]
return int(s.argmax()), s
choice, s = decide(
state=["The parcel left the depot on Monday and was signed for on Wednesday."],
question="Was the parcel delivered?",
options=["Yes", "No"],
)
print(choice, s)
The decision is the option with the highest score s, for every recipe. state is a list of passages (for example a contract, or a context and a set of facts); each is followed by a blank line.
Related checkpoints
All four recipes share the data, the settings and the prompt; they differ in the score and the loss.
| Repo | Score | Loss |
|---|---|---|
| ultrabert-jev-380m-softmax-yes | " yes" logit |
softmax over the options |
| ultrabert-jev-380m-softmax-yesno | " yes" β " no" |
softmax over the options |
| ultrabert-jev-380m-sigmoid-norm | " yes" β " no" |
sigmoid of each score, divided by the sum |
| ultrabert-jev-380m-sqrt-softplus | " yes" β " no" |
square-root softplus of each score, divided by the sum |
Limitations
- Trained and evaluated in English, on one training mixture; accuracy outside it is lower (about 58β60% on six external benchmarks such as MMLU and BoolQ).
- Inputs were at most 10,240 tokens in training.
- The model gives no reasons for its choice.
training_config.yaml holds the full training configuration.
- Downloads last month
- 60
Model tree for Athekunal/ultrabert-jev-380m-softmax-yesno
Base model
dmis-lab/UltraBERT-380M