Text Generation
GGUF
English
ik_llama.cpp
gemma-4
gemma-4-26b-a4b
google-deepmind
Mixture of Experts
mixture-of-experts
26b
a4b
ik-llama.cpp
iqk
trellis-quant
maxoptima
quantized
imatrix
v3-calibration
austriani
conversational
Instructions to use Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp # Run inference directly in the terminal: llama cli -hf Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp # Run inference directly in the terminal: llama cli -hf Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp # Run inference directly in the terminal: ./build/bin/llama cli -hf Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
Use Docker
docker model run hf.co/Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
- Llama new
- LM Studio
- Jan
- vLLM
How to use Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
- Ollama
How to use Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp with Ollama:
ollama run hf.co/Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
- Unsloth Desktop
- Pi
How to use Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp with Docker Model Runner:
docker model run hf.co/Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
- Lemonade
How to use Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
Run and chat with the model
lemonade run user.Gemma-4-26B-A4B-it-ik_llama.cpp-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "Austriani/Gemma-4-26B-A4B-it-ik_llama.cpp" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Austriani / ik_llama.cpp
Gemma-4-26B-A4B-it
Architectural Quantization Archive
26B
Total Params
4B
Active Params
1024
Imatrix Ctx
500
Cycles
Base Architecture
This repository hosts ik_llama.cpp-compatible GGUF quantizations of google/gemma-4-26B-A4B-it. This is Google DeepMind's 26B-parameter Mixture-of-Experts model, activating only 3.8B parameters per token through 128 fine-grained experts and top-8 routing. It is built for cost-efficient inference, native function calling, and multimodal input, supporting a 256K-token context window.
The architecture uses 30 layers with a hybrid attention scheme: interleaved local sliding-window attention (1024 tokens) and global attention. It has 16 attention heads over 8 KV heads. The model is released under the Apache 2.0 license.
MaxOptima Custom Quant
This release includes MaxOptima, a bespoke, surgical quantization mix designed specifically for the 128-expert MoE architecture of Gemma 4. The goal is to protect the expert routing and critical attention layers while aggressively compressing the bulk of the expert FFNs, producing a file significantly smaller than a standard Q4_K_M.
MaxOptima Strategy: Embeddings and output layers are locked at Q6_K. Router inputs, norms, scales, and biases are kept at full precision (F32) to preserve routing integrity. Key attention layers are kept at IQ5_K. The bulk of the expert FFNs are compressed to IQ4_KSS and IQ4_NL, with boundary blocks protected at higher precision.
View Full Custom Quant Configuration
^output\.weight=Q6_K,^token_embd\.weight=Q6_K,ffn_gate_inp\.weight=F32,rope_freqs\.weight=F32,.*_norm\.weight=F32,.*\.scale=F32,.*_scale\.weight=F32,.*\.bias=F32,blk\.(0|1|2|3|4|5|6|11|17|18|23|24|25|26|27|28|29)\.attn_k\.weight=IQ5_K,attn_k\.weight=IQ4_KSS,attn_v\.weight=IQ5_K,blk\.(0|1|2|3|4|5|11|17|23|25|26|27|28|29)\.attn_output\.weight=IQ5_K,attn_output\.weight=IQ4_KS,attn_q\.weight=IQ5_K,blk\.(0|1|2|3|4|25|26|27|28|29)\.ffn_gate_up_exps\.weight=IQ5_K,ffn_gate_up_exps\.weight=IQ4_KSS,blk\.(0|1|2|3|4|25|26|27|28|29)\.ffn_down_exps\.weight=Q5_0,ffn_down_exps\.weight=IQ4_NL,ffn_down\.weight=Q5_0,ffn_(gate|up)\.weight=IQ5_K,.*=IQ4_KS
Available Quants
MaxOptima // Custom MoE Mix
MaxOptima
Custom mix. F32 routing/scales, IQ5_K attention, IQ4_KSS/IQ4_NL expert FFNs. Best quality-to-size ratio for MoE inference.
Standard IQK // High & Mid Fidelity
IQ5_K
Higher fidelity 5-bit. Best quality-to-size for systems with sufficient memory.
IQ4_K
Standard 4-bit IQK. Uses 256-value super-blocks for scaling.
IQ4_KS
"Small" 4-bit variant with tighter packing. Lower memory footprint.
IQ4_KS_R4
Row-interleaved variant of IQ4_KS, optimized for ik_llama.cpp's matmul kernels.
Trellis // Alternative 4-Bit
IQ4_KT
Dynamic programming trellis quant. Uses a fundamentally different error-minimization strategy than IQK.
Imatrix Specification
This release utilizes the exact same V3 Calibration Dataset used across previous Austriani releases. No datasets were changed. The calibration was executed at 500 cycles per chunk with a 1024-token context window, processing approximately 1000 chunks.
Compatibility
These GGUF files are built exclusively for ik_llama.cpp. IQK, Trellis, and MaxOptima quants will not load in mainline llama.cpp. Please refer to the official base repository for mainline GGUF compatibility.
- Downloads last month
- 521
Hardware compatibility
Log In to add your hardware
We're not able to determine the quantization variants.