aihomelabprivacy

Quantize Bonsai 2 27B to 4-bit with AutoGPTQ

🤖 Researched and drafted automatically from the official docs, and reviewed before publishing. Commands are taken from the source projects — but always sanity-check before running anything on your own hardware.

AutoGPTQ applies GPTQ weight-only quantization to compress large language models down to 4-bit precision. This cuts VRAM requirements dramatically—a 27B model that normally needs 54GB can run in 6GB or less. The tradeoff is minimal: perplexity stays close to the original, and inference actually speeds up on compatible hardware. We’ll walk through installing AutoGPTQ, quantizing a 27B model, and running it locally.

Important: Keep this quantization pipeline and inference entirely on your LAN or behind a VPN. Never expose the model server to the public internet.

Prerequisites

You need:

  • A Linux or Windows machine with CUDA 11.8+ or ROCm 5.7+ (AutoGPTQ does not support Maxwell or older NVIDIA GPUs)
  • At least 60GB free disk space for the original model weights and quantization artifacts
  • A GPU with enough VRAM to load the model during quantization (RTX 3090, A100, or equivalent; you can use CPU offloading for smaller GPUs, but it’s slower)
  • Python 3.10+
  • pip and git

Step 1: Install AutoGPTQ

Clone the AutoGPTQ repository and install from source. This ensures you get the latest CUDA extensions and Triton support:

git clone https://github.com/AutoGPTQ/AutoGPTQ.git
cd AutoGPTQ
pip install numpy gekko pandas
pip install -vvv --no-build-isolation -e .

If you’re on CUDA 11.8, you can alternatively use the pre-built wheel (faster):

pip install auto-gptq --no-build-isolation --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/

For CUDA 12.1, the default PyPI wheel works:

pip install auto-gptq --no-build-isolation

To enable Triton backend (Linux only, faster inference on supported GPUs):

pip install auto-gptq[triton] --no-build-isolation

Step 2: Prepare Calibration Data

Quantization needs representative examples to calibrate the weight compression. Download a small dataset (512–1024 samples) of text relevant to your use case. For general-purpose models, the Wikitext-2 or OpenWebText subsets work well.

Create a Python script to load and prepare the data:

from transformers import AutoTokenizer
import json

# Load tokenizer for your model
tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-3b-code-base")

# Load your calibration dataset (example: a text file with one sample per line)
with open("calibration_data.txt", "r") as f:
    texts = [line.strip() for line in f if line.strip()]

# Tokenize examples
examples = [
    tokenizer(text, return_tensors="pt", max_length=512, truncation=True)
    for text in texts[:512]  # Use first 512 samples
]

print(f"Prepared {len(examples)} calibration examples")

Save this as prepare_calibration.py and run it to verify the data loads correctly.

Step 3: Quantize the Model

Create a quantization script. The key parameters are:

  • bits=4: Quantize to 4-bit integers (reduces size by ~8x)
  • group_size=128: Quantize in groups of 128 weights (recommended; balances quality and speed)
  • desc_act=False: Disable activation ordering for faster inference (slight quality trade-off)

Create quantize_bonsai.py:

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import logging

logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
    level=logging.INFO,
    datefmt="%Y-%m-%d %H:%M:%S"
)

model_name = "ibm-granite/granite-3b-code-base"  # Replace with your 27B model
output_dir = "./bonsai-27b-gptq-4bit"

# Load tokenizer and calibration data
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)

with open("calibration_data.txt", "r") as f:
    texts = [line.strip() for line in f if line.strip()][:512]

examples = [
    tokenizer(text, return_tensors="pt", max_length=512, truncation=True)
    for text in texts
]

# Define quantization config
quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False,
)

# Load model (will be loaded to CPU by default; move to GPU as needed)
model = AutoGPTQForCausalLM.from_pretrained(
    model_name,
    quantize_config,
    device_map="auto"  # Automatically distribute across available GPUs
)

print("Starting quantization...")
model.quantize(examples)

print(f"Saving quantized model to {output_dir}")
model.save_quantized(output_dir, use_safetensors=True)
print("Done.")

Run the quantization (this takes 10–30 minutes depending on model size and GPU):

python quantize_bonsai.py

Step 4: Load and Inference

Once quantized, load the model for inference with minimal VRAM:

from transformers import AutoTokenizer, TextGenerationPipeline
from auto_gptq import AutoGPTQForCausalLM

quantized_model_dir = "./bonsai-27b-gptq-4bit"

# Load quantized model to GPU
model = AutoGPTQForCausalLM.from_quantized(
    quantized_model_dir,
    device="cuda:0",
    use_triton=True  # Enable Triton for faster inference (Linux only)
)

tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)

# Generate text
prompt = "Write a Python function to sort a list:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0]))

# Or use pipeline for cleaner API
pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)
result = pipeline(prompt, max_new_tokens=256)
print(result[0]["generated_text"])

Step 5: Serve Behind a VPN

To use the quantized model as a local API, wrap it with a simple Flask or FastAPI server and bind it to localhost or a private interface only:

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM

app = FastAPI()

model = AutoGPTQForCausalLM.from_quantized(
    "./bonsai-27b-gptq-4bit",
    device="cuda:0"
)
tokenizer = AutoTokenizer.from_pretrained("./bonsai-27b-gptq-4bit")

class GenerateRequest(BaseModel):
    prompt: str
    max_tokens: int = 256

@app.post("/generate")
def generate(req: GenerateRequest):
    inputs = tokenizer(req.prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=req.max_tokens)
    return {"text": tokenizer.decode(outputs[0])}

if __name__ == "__main__":
    import uvicorn
    # Bind to localhost only; access via VPN or SSH tunnel
    uvicorn.run(app, host="127.0.0.1", port=8000)

Run with uvicorn and access only from your LAN or through a VPN tunnel.

Tuning for Your Hardware

  • Smaller GPUs (6–12GB): Use device_map="auto" to offload layers to CPU. Inference slows down but remains viable.
  • Multi-GPU setups: AutoGPTQ automatically distributes the model across available GPUs with device_map="auto".
  • Marlin kernel: If you have NVIDIA Ampere or newer (RTX 30-series, A100), enable the Marlin int4×fp16 kernel by passing use_marlin=True when loading. This speeds up inference significantly.

Is It Worth It?

Yes. A 27B model quantized to 4-bit fits on a single 12GB GPU instead of requiring 48–54GB. Perplexity loss is typically 1–3%, which is imperceptible for most tasks. Inference is actually faster because the model is smaller. The one cost is 10–30 minutes of one-time quantization. If you’re running a homelab and want to serve a capable language model without renting cloud GPUs, AutoGPTQ is the practical choice. Just keep it behind a firewall.

Related video

New self-hosted AI & homelab shorts, daily.

Subscribe on YouTube

Related guides

← All guides