syedmohaiminulhoque/spotlight-attention-steering

SpotLight: Dynamic Attention Steering for Instruction Following

Implementation of "Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering" (EACL 2026)

Paper: Praveen Venkateswaran, Danish Contractor (IBM Research) | ArXiv: 2505.12025

What is SpotLight?

SpotLight is a training-free, inference-time method that dynamically steers attention toward instruction tokens in decoder-only transformers. It works by adding a log-ratio bias to pre-softmax attention logits:

For each query position i:
  1. Compute ψ_current = Σ_{j∈S} softmax(logits_i)[j]  (current attention on instructions)
  2. If ψ_current < ψ_target:
      bias = log(ψ_target / ψ_current)
      logits_i[j] += bias  for all j ∈ S  (instruction tokens)
  3. Mathematical guarantee: ψ_new ∈ [ψ_target/(1+ψ_target), ψ_target]

Key properties:

• 🚀 No training, no fine-tuning, no profiling — plug and play
• 🎯 Dynamic: only steers when attention is insufficient (no over-steering)
• 🔧 Applied to ALL heads and ALL layers simultaneously
• 📊 Validated across 7 model families (Qwen2.5, Llama 3.1, Mistral, Granite)

Quick Start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from spotlight.steering import SpotLightSteering
from spotlight.utils import find_instruction_span

# Load model with eager attention (REQUIRED for SpotLight)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-3B-Instruct",
    attn_implementation="eager",
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")

# Initialize SpotLight (ψ_target=0.1 is the paper default)
spotlight = SpotLightSteering(model, psi_target=0.1)

# Your prompt with instructions
prompt = """Write a poem about the ocean.

Your response should follow the instructions below:
- Do not use any commas
- Write at least 100 words
- Use all lowercase letters"""

# Find instruction token indices
formatted = tokenizer.apply_chat_template([{"role": "user", "content": prompt}],
    tokenize=False, add_generation_prompt=True)
delim = "Your response should follow the instructions below:"
instr_text = prompt[prompt.find(delim):]
start, end = find_instruction_span(tokenizer, formatted, instr_text)

# Generate WITH SpotLight steering
inputs = tokenizer(formatted, return_tensors="pt").to(model.device)
with spotlight.steer(slice(start, end)):
    output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
response = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# Clean up
spotlight.remove_hooks()

Demo Results (Qwen2.5-0.5B-Instruct, 10 samples)

Method	Prompt Acc (Strict)	Instruction Acc (Strict)
Baseline	0.0000	0.1667
SpotLight (ψ=0.1)	0.1000	0.2222

SpotLight improved instruction-level accuracy by 33% even on this tiny model. The paper reports +26% prompt-level accuracy averaged across 7 models (3B-72B) on the full IFEval benchmark.

Paper Results (from original paper)

Model	Baseline (P/I)	SpotLight (P/I)
Qwen2.5-3B	0.42 / 0.53	0.53 / 0.62
Mistral-7B	0.35 / 0.47	0.40 / 0.53
Qwen2.5-7B	0.47 / 0.59	0.54 / 0.66
Llama 3.1-8B	0.42 / 0.55	0.51 / 0.62
Granite 3.1-8B	0.41 / 0.54	0.48 / 0.60
Llama 3.1-70B	0.45 / 0.57	0.54 / 0.64
Qwen2.5-72B	0.49 / 0.61	0.55 / 0.67

Supported Architectures

• ✅ Llama 3.x (LlamaAttention)
• ✅ Qwen 2.x (Qwen2Attention)
• ✅ Mistral (MistralAttention)
• ✅ Granite (GraniteAttention)
• ✅ GPT-2 (GPT2Attention)
• ✅ Other models using q_proj/k_proj/v_proj/o_proj pattern

Repository Structure

spotlight/
├── __init__.py          # Package init
├── steering.py          # Core SpotLight algorithm
└── utils.py             # Token span detection utilities
ifeval_checker.py        # IFEval instruction checker (25 types)
spotlight_experiment.py  # Full IFEval evaluation script
results/                 # Evaluation results

How It Works

1. Monkey-patch each attention layer's forward() to intercept pre-softmax logits
2. Before softmax, compute current attention proportion ψ_current on instruction tokens
3. If ψ_current < ψ_target (model isn't attending enough):
   • Add log(ψ_target / ψ_current) to instruction token logits
   • This multiplicatively boosts instruction attention in probability space
4. The dynamic correction ensures ψ_new ∈ [ψ_target/(1+ψ_target), ψ_target]

Dataset

Uses ibm-research/Split-IFEval — a preprocessed version of IFEval where task descriptions are separated from formatting instructions, enabling precise instruction span identification.

Citation

@inproceedings{venkateswaran2026spotlight,
  title={Spotlight Your Instructions: Instruction-following with Dynamic Attention Steering},
  author={Venkateswaran, Praveen and Contractor, Danish},
  booktitle={Proceedings of the 2026 Conference of the European Chapter of the Association for Computational Linguistics},
  year={2026}
}