Large Language Models
Collection
Precompiled language models for on-device text generation. • 31 items • Updated • 1
Gemma-4-E2B-it-TextOnly-GPTQ-a16w4: Optimized for SiMa.ai Modalix
Overview
This repository contains the Gemma-4-E2B-it-TextOnly-GPTQ-a16w4 model, optimized and compiled for the SiMa.ai Modalix platform.
- Model architecture: Gemma 4 E2B (text-only)
- Quantization: A16W4 (BF16 activations, INT4 language weights), with mixed-precision exceptions retained from the prequantized checkpoint.
- Pre-quantization: The source checkpoint was pre-quantized with GPTQ before compilation.
- Maximum context length: 4096 tokens
- Source model: google/gemma-4-E2B-it
Performance
The following measurements characterize text generation on Modalix.
Measured with MoLE using batch size 1, four samples per input length, and max_new_tokens=128. Values are arithmetic means. TTFT includes language-model prefill and the first generated token; generation rate is measured after the first token.
| Input tokens | Mean TTFT (seconds) | Mean generation rate (tokens/second) |
|---|---|---|
| 128 | 0.11 | 36.41 |
| 256 | 0.16 | 36.26 |
| 512 | 0.29 | 35.56 |
| 1024 | 0.60 | 34.96 |
| 2048 | 1.23 | 33.34 |
| 3072 | 1.92 | 32.59 |
Prerequisites
To run this model, you need a SiMa.ai Modalix device with the SiMa.ai Neat Runtime installed.
Installation and Deployment
Download the precompiled model directly on Modalix:
hf download simaai/Gemma-4-E2B-it-TextOnly-GPTQ-a16w4 --local-dir /media/nvme/llima/models/Gemma-4-E2B-it-TextOnly-GPTQ-a16w4
The model is installed under:
/media/nvme/llima/models/Gemma-4-E2B-it-TextOnly-GPTQ-a16w4
Alternatively, download it on a host and copy it to Modalix:
hf download simaai/Gemma-4-E2B-it-TextOnly-GPTQ-a16w4 --local-dir Gemma-4-E2B-it-TextOnly-GPTQ-a16w4
scp -r Gemma-4-E2B-it-TextOnly-GPTQ-a16w4 sima@<modalix-ip>:/media/nvme/llima/models/
Usage
Validate the model directly on Modalix:
llima run Gemma-4-E2B-it-TextOnly-GPTQ-a16w4
For application integration, see GenAI with LLiMa.
Recompilation
To build for an 8,192-token context, recompile the matching source checkpoint with this repository's compile.sh in a compatible SiMa Model Compiler and LLiMa compilation environment. Change --max_num_tokens 4096 to --max_num_tokens 8192 in the script, then follow its source-checkpoint and output-directory instructions.
The context budget includes input and generated tokens. A larger context increases KV-cache memory requirements and must fit the target device. The downloadable artifacts use a 4,096-token context; any performance measurements in this card refer to that build.
Limitations
- This repository contains compiled Modalix artifacts and cannot be loaded with
transformers.AutoModel. - Quantization can cause minor differences from the full-precision source model.