AILLM
2 min read
Taming a 26B MoE Model on 8GB VRAM with TurboQuant
Forget Docker images and pre-built binaries. Here's how I compiled a custom llama.cpp fork with TurboQuant, ran a 26B Gemma 4 MoE model on a consumer RTX 3070, and squeezed 262K context into less than 4GB of VRAM.