File size: 2,625 Bytes
256b90e ae3ec61 5ec17d5 ae3ec61 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 |
---
license: apache-2.0
---
<div align="center">
<h1>UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretation</h1>
<a href="https://arxiv.org/abs/2504.21336"><img src='https://img.shields.io/badge/arXiv-Preprint-red' alt='Paper PDF'></a>
<a href='https://huggingface.co/Luffy503/UniBiomed'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue'></a>
<a href='https://huggingface.co/datasets/Luffy503/UniBiomed'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-green' alt='Dataset'></a>
</div>
We introduce **UniBiomed**, the first universal foundation model for grounded biomedical image interpretation, which is capable of generating accurate diagnostic findings and simultaneously segmenting the corresponding biomedical targets. UniBiomed is based on a novel integration of Multi-modal Large Language Model (MLLM) and Segment Anything Model (SAM), which can effectively unify diverse biomedical tasks in universal training for advancing grounded interpretation.
Github link: https://github.com/Luffy03/UniBiomed
We will consistently update more powerful versions of models in this repo.
## Usage
```python
import argparse
import torch
from transformers import (AutoModel, AutoTokenizer,
BitsAndBytesConfig, CLIPImageProcessor,
GenerationConfig)
def parse_args():
parser = argparse.ArgumentParser(description='UniBiomed')
parser.add_argument('--model_path', default='Luffy503/UniBiomed')
return args
args = parse_args()
# load model
model = AutoModel.from_pretrained(
args.model_path,
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
use_flash_attn=True,
trust_remote_code=True,
).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(
args.model_path,
trust_remote_code=True,
)
# define data input, image and text instruction
data_dict = {}
image, text = None, None
data_dict['image'] = image
data_dict['text'] = text
# output
pred_dict = model.predict_forward(**data_dict, tokenizer=tokenizer)
# text description
prediction = pred_dict['prediction']
# segmentation mask
mask = pred_dict['prediction_masks'][0][0]
```
## Citation
If you find this repo useful for your research, please consider citing the paper as follows:
```bibtex
@article{wu2025unibiomed,
title={UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretation},
author={Wu, Linshan and Nie, Yuxiang and He, Sunan and Zhuang, Jiaxin and Chen, Hao},
journal={arXiv preprint arXiv:2504.21336},
year={2025}
}
``` |