Quantize Bonsai 2 27B to 4-bit with AutoGPTQ
🤖 Researched and drafted automatically from the official docs, and reviewed before publishing. Commands are taken from the source projects — but always sanity-check before running anything on your own hardware.
AutoGPTQ applies GPTQ weight-only quantization to compress large language models down to 4-bit precision. This cuts VRAM requirements dramatically—a 27B model that normally needs 54GB can run in 6GB or less. The tradeoff is minimal: perplexity stays close to the original, and inference actually speeds up on compatible hardware. We’ll walk through installing AutoGPTQ, quantizing a 27B model, and running it locally.
Important: Keep this quantization pipeline and inference entirely on your LAN or behind a VPN. Never expose the model server to the public internet.
Prerequisites
You need:
- A Linux or Windows machine with CUDA 11.8+ or ROCm 5.7+ (AutoGPTQ does not support Maxwell or older NVIDIA GPUs)
- At least 60GB free disk space for the original model weights and quantization artifacts
- A GPU with enough VRAM to load the model during quantization (RTX 3090, A100, or equivalent; you can use CPU offloading for smaller GPUs, but it’s slower)
- Python 3.10+
pipandgit
Step 1: Install AutoGPTQ
Clone the AutoGPTQ repository and install from source. This ensures you get the latest CUDA extensions and Triton support:
git clone https://github.com/AutoGPTQ/AutoGPTQ.git
cd AutoGPTQ
pip install numpy gekko pandas
pip install -vvv --no-build-isolation -e .
If you’re on CUDA 11.8, you can alternatively use the pre-built wheel (faster):
pip install auto-gptq --no-build-isolation --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
For CUDA 12.1, the default PyPI wheel works:
pip install auto-gptq --no-build-isolation
To enable Triton backend (Linux only, faster inference on supported GPUs):
pip install auto-gptq[triton] --no-build-isolation
Step 2: Prepare Calibration Data
Quantization needs representative examples to calibrate the weight compression. Download a small dataset (512–1024 samples) of text relevant to your use case. For general-purpose models, the Wikitext-2 or OpenWebText subsets work well.
Create a Python script to load and prepare the data:
from transformers import AutoTokenizer
import json
# Load tokenizer for your model
tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-3b-code-base")
# Load your calibration dataset (example: a text file with one sample per line)
with open("calibration_data.txt", "r") as f:
texts = [line.strip() for line in f if line.strip()]
# Tokenize examples
examples = [
tokenizer(text, return_tensors="pt", max_length=512, truncation=True)
for text in texts[:512] # Use first 512 samples
]
print(f"Prepared {len(examples)} calibration examples")
Save this as prepare_calibration.py and run it to verify the data loads correctly.
Step 3: Quantize the Model
Create a quantization script. The key parameters are:
bits=4: Quantize to 4-bit integers (reduces size by ~8x)group_size=128: Quantize in groups of 128 weights (recommended; balances quality and speed)desc_act=False: Disable activation ordering for faster inference (slight quality trade-off)
Create quantize_bonsai.py:
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import logging
logging.basicConfig(
format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
level=logging.INFO,
datefmt="%Y-%m-%d %H:%M:%S"
)
model_name = "ibm-granite/granite-3b-code-base" # Replace with your 27B model
output_dir = "./bonsai-27b-gptq-4bit"
# Load tokenizer and calibration data
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
with open("calibration_data.txt", "r") as f:
texts = [line.strip() for line in f if line.strip()][:512]
examples = [
tokenizer(text, return_tensors="pt", max_length=512, truncation=True)
for text in texts
]
# Define quantization config
quantize_config = BaseQuantizeConfig(
bits=4,
group_size=128,
desc_act=False,
)
# Load model (will be loaded to CPU by default; move to GPU as needed)
model = AutoGPTQForCausalLM.from_pretrained(
model_name,
quantize_config,
device_map="auto" # Automatically distribute across available GPUs
)
print("Starting quantization...")
model.quantize(examples)
print(f"Saving quantized model to {output_dir}")
model.save_quantized(output_dir, use_safetensors=True)
print("Done.")
Run the quantization (this takes 10–30 minutes depending on model size and GPU):
python quantize_bonsai.py
Step 4: Load and Inference
Once quantized, load the model for inference with minimal VRAM:
from transformers import AutoTokenizer, TextGenerationPipeline
from auto_gptq import AutoGPTQForCausalLM
quantized_model_dir = "./bonsai-27b-gptq-4bit"
# Load quantized model to GPU
model = AutoGPTQForCausalLM.from_quantized(
quantized_model_dir,
device="cuda:0",
use_triton=True # Enable Triton for faster inference (Linux only)
)
tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
# Generate text
prompt = "Write a Python function to sort a list:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0]))
# Or use pipeline for cleaner API
pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)
result = pipeline(prompt, max_new_tokens=256)
print(result[0]["generated_text"])
Step 5: Serve Behind a VPN
To use the quantized model as a local API, wrap it with a simple Flask or FastAPI server and bind it to localhost or a private interface only:
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM
app = FastAPI()
model = AutoGPTQForCausalLM.from_quantized(
"./bonsai-27b-gptq-4bit",
device="cuda:0"
)
tokenizer = AutoTokenizer.from_pretrained("./bonsai-27b-gptq-4bit")
class GenerateRequest(BaseModel):
prompt: str
max_tokens: int = 256
@app.post("/generate")
def generate(req: GenerateRequest):
inputs = tokenizer(req.prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=req.max_tokens)
return {"text": tokenizer.decode(outputs[0])}
if __name__ == "__main__":
import uvicorn
# Bind to localhost only; access via VPN or SSH tunnel
uvicorn.run(app, host="127.0.0.1", port=8000)
Run with uvicorn and access only from your LAN or through a VPN tunnel.
Tuning for Your Hardware
- Smaller GPUs (6–12GB): Use
device_map="auto"to offload layers to CPU. Inference slows down but remains viable. - Multi-GPU setups: AutoGPTQ automatically distributes the model across available GPUs with
device_map="auto". - Marlin kernel: If you have NVIDIA Ampere or newer (RTX 30-series, A100), enable the Marlin int4×fp16 kernel by passing
use_marlin=Truewhen loading. This speeds up inference significantly.
Is It Worth It?
Yes. A 27B model quantized to 4-bit fits on a single 12GB GPU instead of requiring 48–54GB. Perplexity loss is typically 1–3%, which is imperceptible for most tasks. Inference is actually faster because the model is smaller. The one cost is 10–30 minutes of one-time quantization. If you’re running a homelab and want to serve a capable language model without renting cloud GPUs, AutoGPTQ is the practical choice. Just keep it behind a firewall.
Related video
New self-hosted AI & homelab shorts, daily.
Subscribe on YouTubeRelated guides
Run Local AI on Old NVIDIA/AMD GPUs with Ollama
Dust off that graphics card and run Llama, Gemma, Qwen locally with Ollama. Pick a model that fits your VRAM, skip the hype.
Run AI Models on PS5 with llama.cpp GGUF
Turn your PlayStation 5 into a local AI inference engine using llama.cpp's native GGUF support. Run Llama, Qwen, and other quantized models directly on console hardware for private, offline inference.
Run MiniMax H3 in ComfyUI: Local Audio & 2K Video
Generate audio and 2K video together on your own hardware using MiniMax H3 in ComfyUI. No API costs, full control, offline workflows.