Athekunal/ultrabert-jev-380m-softmax-yesno

A decision model: it reads a state, a question and a list of options, and picks one option in a single forward pass, without generating text. It is UltraBERT-380M fully fine-tuned with one mask token in front of each option. Option k is scored from the masked-token logits at its mask.

Recipe: softmax over " yes" minus " no". Trained for one epoch (batch 32, learning rate 1.5e-5) on the 13-dataset Reason+Long mixture of LLM-as-Jev, with Nimble added.

This checkpoint Accuracy (mean over 13 datasets)
Dev split 72.8%
Test split 72.6%

Usage

Load the weights in fp32. The masked-token logits are large (around 200), and bf16 weights changed accuracy by up to 8 points on long documents (MuSiQue) in our tests. All reported numbers use fp32 weights.

Use the prompt below exactly. The model was trained on this format with the tokens " yes" and " no"; plain fill-mask use will not give these results.

Tested with transformers 5.20.2 and torch 2.9, which include the UltraBERT model class. Do not pass trust_remote_code=True: the base repo's remote code names the weights differently and would leave them unloaded.

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

REPO = "Athekunal/ultrabert-jev-380m-softmax-yesno"

tok = AutoTokenizer.from_pretrained(REPO)
# Load the weights in fp32. bf16 weights change the results (up to 8 points on MuSiQue).
model = AutoModelForMaskedLM.from_pretrained(REPO, dtype=torch.float32)
model = model.to("cuda").eval()

YES = tok(" yes", add_special_tokens=False)["input_ids"][0]
NO = tok(" no", add_special_tokens=False)["input_ids"][0]


def build_prompt(state: list[str], question: str, options: list[str]) -> str:
    """The exact prompt the model was trained on: one mask in front of each option."""
    text = "State: " + "".join(part + "\n\n" for part in state) + question + "\n"
    text += "Mark the correct option with yes and every other option with no.\nOptions:\n"
    text += "\n".join(f"[{k}] {tok.mask_token} {o}" for k, o in enumerate(options, 1))
    return text


@torch.no_grad()
def decide(state: list[str], question: str, options: list[str], score: str = "yes-no"):
    """(index of the chosen option, scores s_k). This checkpoint uses score="yes-no"."""
    enc = tok(build_prompt(state, question, options), return_tensors="pt").to(model.device)
    logits = model(**enc).logits[0]
    masks = (enc["input_ids"][0] == tok.mask_token_id).nonzero()[:, 0]
    s = logits[masks, YES] if score == "yes" else logits[masks, YES] - logits[masks, NO]
    return int(s.argmax()), s


choice, s = decide(
    state=["The parcel left the depot on Monday and was signed for on Wednesday."],
    question="Was the parcel delivered?",
    options=["Yes", "No"],
)
print(choice, s)

The decision is the option with the highest score s, for every recipe. state is a list of passages (for example a contract, or a context and a set of facts); each is followed by a blank line.

Related checkpoints

All four recipes share the data, the settings and the prompt; they differ in the score and the loss.

Repo Score Loss
ultrabert-jev-380m-softmax-yes " yes" logit softmax over the options
ultrabert-jev-380m-softmax-yesno " yes" βˆ’ " no" softmax over the options
ultrabert-jev-380m-sigmoid-norm " yes" βˆ’ " no" sigmoid of each score, divided by the sum
ultrabert-jev-380m-sqrt-softplus " yes" βˆ’ " no" square-root softplus of each score, divided by the sum

Limitations

  • Trained and evaluated in English, on one training mixture; accuracy outside it is lower (about 58–60% on six external benchmarks such as MMLU and BoolQ).
  • Inputs were at most 10,240 tokens in training.
  • The model gives no reasons for its choice.

training_config.yaml holds the full training configuration.

Downloads last month
60
Safetensors
Model size
0.4B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Athekunal/ultrabert-jev-380m-softmax-yesno

Finetuned
(28)
this model

Paper for Athekunal/ultrabert-jev-380m-softmax-yesno