Instructions to use google/paligemma2-3b-pt-224 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/paligemma2-3b-pt-224 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="google/paligemma2-3b-pt-224")

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("google/paligemma2-3b-pt-224")
model = AutoModelForMultimodalLM.from_pretrained("google/paligemma2-3b-pt-224", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use google/paligemma2-3b-pt-224 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/paligemma2-3b-pt-224"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/paligemma2-3b-pt-224",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/google/paligemma2-3b-pt-224

SGLang

How to use google/paligemma2-3b-pt-224 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/paligemma2-3b-pt-224" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/paligemma2-3b-pt-224",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/paligemma2-3b-pt-224" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/paligemma2-3b-pt-224",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use google/paligemma2-3b-pt-224 with Docker Model Runner:
```
docker model run hf.co/google/paligemma2-3b-pt-224
```

Error when deploying model in inference or amazon sagemaker endpoint

by iuf26 - opened Dec 9, 2024

Discussion

iuf26

Dec 9, 2024

The code as it is suggested for deploying the model in Amazon SageMaker or on the inference endpoint is not working.
There are errors about shards or tokenizer file.

GopiUppari

Google org Dec 12, 2024

Hi @iuf26 ,

Below are the possible reasons for an above issue:

Model Sharding Issue which means the model weights are split across multiple shards because of its large size. If the environment where the model is being deployed doesn't properly load all shards, we willl encounter errors.
To avoid this problem, make sure that all shard files (model-00001-of-00002.bin, model-00002-of-00002.bin, etc.) are uploaded to the same directory in the storage location (S3 for SageMaker).
Tokenizer File Issue which means the tokenizer configuration file (tokenizer.json, tokenizer_config.json) may be missing or incorrectly referenced in your code or deployment setup.
To avoid it, make sure that the tokenizer files are included in the model directory.
And also make sure that use the device_map parameter (e.g., "auto") to manage large models efficiently during loading. Ensure sufficient memory is allocated on the endpoint instance.

If the issue still persists, could you please share screenshots of error message then will help in an better way.

Thank you.

iuf26

Dec 24, 2024

•

edited Dec 24, 2024

Hi @GopiUppari ,
Thank you for your answer above

The problem is that Paligemma exists only in newer Transformer versions, and SageMaker uses an image that is not updated (it only recognizes Transformers 4.37 as the latest version). When you deploy the model using HuggingfaceModel at inference time, you encounter an error stating that the model type 'paligemma' is not recognized.

To deploy this model, you would need a custom ECR image for inference that uses a newer version of Transformers (I used transformers==4.47.0).

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment