Misc improvements

README.md

@@ -1,14 +1,16 @@
 ---
 library_name: transformers
-tags: []
+license: apache-2.0
+datasets:
+- HuggingFaceM4/the_cauldron
+- HuggingFaceM4/Docmatix
+pipeline_tag: image-text-to-text
 ---
 
 # Model Card for Model ID
 
 SmolVLM is a compact open multimodal model that accepts arbitrary sequences of image and text inputs to produce text outputs. Designed for efficiency, SmolVLM can answer questions about images, describe visual content, create stories grounded on multiple images, or function as a pure language model without visual inputs. Its lightweight architecture makes it suitable for on-device applications while maintaining strong performance on multimodal tasks.
 
-We release the checkpoints under the Apache 2.0 license.
-
 ## Model Details
 
 ### Model Description
@@ -28,7 +30,7 @@ This is the model card of a 🤗 transformers model that has been pushed on the
 
 - **Repository:** [More Information Needed]
 - **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** https://huggingface.co/spaces/HuggingFaceTB/SmolVLM
+- **Demo:** [SmolVLM Demo](https://huggingface.co/spaces/HuggingFaceTB/SmolVLM)
 
 ## Uses
 
@@ -40,14 +42,16 @@ To fine-tune SmolVLM on a specific task, you can follow the fine-tuning tutorial
 
 SmolVLM leverages the lightweight SmolLM2 language model to provide a compact yet powerful multimodal experience. It introduces several changes compared to previous models:
 
-- Image compression: We introduce a more radical image compression compared to Idefics3 to enable the model to infer faster and use less RAM.
-- Visual Token Encoding: It uses 81 visual tokens to encode image patches of size 384*384. Larger images are divided into patches, each encoded separately, enhancing efficiency without compromising performance.
+- **Image compression:** We introduce a more radical image compression compared to Idefics3 to enable the model to infer faster and use less RAM.
+- **Visual Token Encoding:** It uses 81 visual tokens to encode image patches of size 384*384. Larger images are divided into patches, each encoded separately, enhancing efficiency without compromising performance.
 
 More details about the training and architecture are available in our technical report.
 
 
 ### How to get started
 
+You can use transformers to load, infer and fine-tune SmolVLM.
+
 ```python
 import torch
 from PIL import Image
@@ -108,13 +112,29 @@ print(generated_texts[0])
 **Precision**: For better performance, load and run the model in half-precision (`torch.float16` or `torch.bfloat16`) if your hardware supports it.
 
 ```python
+from transformers import AutoModelForVision2Seq
+import torch
+
 model = AutoModelForVision2Seq.from_pretrained(
     "HuggingFaceTB/SmolVLM-Instruct",
     torch_dtype=torch.bfloat16
-).to(DEVICE)
+).to("cuda")
+```
+
+You can also load SmolVLM with 4/8-bit quantization using bitsandbytes, torchao or Quanto. Refer to [this page](https://huggingface.co/docs/transformers/en/main_classes/quantization) for other options.
+
+```python
+from transformers import AutoModelForVision2Seq, BitsAndBytesConfig
+import torch
+
+quantization_config = BitsAndBytesConfig(load_in_8bit=True)
+model = AutoModelForVision2Seq.from_pretrained(
+    "HuggingFaceTB/SmolVLM-Instruct",
+    quantization_config=quantization_config
+)
 ```
 
-**Vision Encoder Efficiency**: Adjust the image resolution by setting size={"longest_edge": N*384} when initializing the processor, where N is your desired value. The default N=4 works well, but for documents, N=5 might be beneficial. Decreasing N can save GPU memory for lower-resolution images. This is also useful if you want to fine-tune on videos.
+**Vision Encoder Efficiency**: Adjust the image resolution by setting `size={"longest_edge": N*384}` when initializing the processor, where N is your desired value. The default `N=4` works well, but for documents, `N=5` might be beneficial. Decreasing N can save GPU memory for lower-resolution images. This is also useful if you want to fine-tune on videos.
 
 
 ## Misuse and Out-of-scope Use
@@ -133,10 +153,7 @@ SmolVLM is not intended for high-stakes scenarios or critical decision-making pr
 
 ### License
 
-SmolVLM is built upon the following pre-trained models:
-
-https://huggingface.co/google/siglip-so400m-patch14-384
-https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct
+SmolVLM is built upon [the shape-optimized SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384) as image encoder and [SmolLM2](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct) for text decoder part.
 
 We release the SmolVLM checkpoints under the Apache 2.0 license.
 
@@ -155,5 +172,4 @@ TODO
 
 ## Evaluation
 
-TODO
-
+TODO