Update README.md

README.md

@@ -1,3 +1,60 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+---
+# LLaDA-Prometheus
+
+## Model Description
+
+This model is a fine-tuned version of the LLaDA 8B Base model, obtained through a specialized Supervised Fine-Tuning (SFT) process. It innovatively discards the complex attention mask design typically associated with block diffusion, while preserving full attention mechanisms. This allows the model to achieve block diffusion-style inference efficiently—leveraging KV cache for streamlined generation, outputting an EOS token upon completion of the response to seamlessly exit the generation process.
+
+Key innovations:
+- **Full Attention Preservation**: Maintains standard full attention without the overhead of intricate masking.
+- **Block Diffusion Inference**: Enables iterative block-wise generation via KV cache management, ensuring coherent and controlled outputs.
+- **EOS Handling**: Trained to naturally emit EOS tokens at response boundaries.
+
+This approach balances computational efficiency with high-quality generation, making it suitable for tasks requiring structured, multi-step reasoning.
+
+## Usage
+
+To load and use this model with Hugging Face Transformers:
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_name = "maomaocun/LLaDA-Prometheus-no-template"
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, trust_remote_code=True).to("cuda")
+
+prompt = "Can you tell me an engaging short story about a brave young astronaut who discovers an ancient alien civilization on a distant planet? Make it adventurous and heartwarming, with a twist at the end."
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+input_ids = inputs['input_ids']
+attention_mask = inputs.get('attention_mask', torch.ones_like(input_ids))
+for chunk in model.generate(
+    input_ids=input_ids,
+    attention_mask=attention_mask,
+    max_gen_length=1024,
+    block_length=64,
+    threshold=0.9,
+    streaming=True,
+    eos_token_id=tokenizer.eos_token,
+):
+    all_generated_ids = torch.cat([input_ids, chunk], dim=-1)
+    text = tokenizer.batch_decode(all_generated_ids, skip_special_tokens=False)[0].split(tokenizer.eos_token)[0]
+    print(text, end='', flush=True)
+```
+
+For block diffusion-style inference, customize the generation loop to manage KV cache and block outputs as needed.
+
+## Benchmarks
+
+The following table compares performance across key evaluation benchmarks. Results are reported as accuracy percentages where applicable.
+
+| Model                          | GSM8K | GPQA  | BBH   | MATH  | HumanEval | MBPP  | MMLU-Pro | MMLU-Generate |
+|--------------------------------|-------|-------|-------|-------|-----------|-------|----------|---------------|
+| LLaDA 8B Base in Pure Diffusion  | 69.06 | 31.91 | 44.77 | 30.84 | 32.92     | 40.8  | 24.26    | 65.9          |
+| LLaDA 8B Instruct in Pure Diffusion | 77.48 | 29.01 | 51.49 | 22.32 | 38.71     | 39.2  | 36.41    | 65.5          |
+| LLaDA-Prometheus in Block Diffusion       | 77.4  | 33.03 | 48.74 | 31.94 | 40.24     | 42    | 33.45    | 65.53         |
+
+These results demonstrate competitive performance, particularly in code generation (HumanEval, MBPP) and reasoning tasks (BBH, MATH), with gains over the base instruct variant in several areas.