Run 70B LLMs on 4GB GPUs with AirLLM
🤖 Researched and drafted automatically from the official docs, and reviewed before publishing. Commands are taken from the source projects — but always sanity-check before running anything on your own hardware.
AirLLM solves a hard problem: running massive language models on hardware that shouldn’t be able to fit them. Instead of loading all 70 billion parameters into VRAM at once, it streams weights from system RAM on-demand, keeping only the active layers in GPU memory. This trades a bit of latency for the ability to run models that would otherwise require $10k+ of hardware.
You’ll need a 4GB GPU (RTX 4060, RTX 3060, or equivalent), at least 32GB system RAM, and Python 3.8+. The model runs locally—no API calls, no cloud vendor lock-in. Inference is slower than on high-end GPUs, but for homelab work, local testing, or privacy-critical applications, it’s a solid tradeoff.
Install dependencies
- Clone the AirLLM repository and enter the directory:
git clone https://github.com/airtrain-ai/airllm.git
cd airllm
- Install Python dependencies. AirLLM requires PyTorch with CUDA support:
pip install -e .
- Verify your GPU is detected:
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"
If this returns False, check your NVIDIA drivers and CUDA installation.
Download a 70B model
- Install Hugging Face CLI tools:
pip install huggingface-hub
- Log in to Hugging Face (required for gated models like Llama 2):
huggingface-cli login
- Download a compatible 70B model. AirLLM works best with models available in GGUF or standard PyTorch format. For example, download a quantized Llama 2 70B variant:
huggingface-cli download meta-llama/Llama-2-70b-hf --local-dir ./models/llama2-70b
Quantized versions (4-bit, 8-bit) will fit more comfortably. Check the Hugging Face model card for VRAM requirements before downloading.
Load and run inference
- Create a Python script to load the model with AirLLM:
cat > run_inference.py << 'EOF'
import torch
from airllm import AirLLM
model_path = "./models/llama2-70b"
model = AirLLM(model_path, device="cuda")
prompt = "What is machine learning?"
inputs = model.tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_length=256)
result = model.tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)
EOF
python run_inference.py
- Adjust generation parameters as needed:
max_length: token limit for outputtemperature: randomness (0.7 is default)top_p: nucleus sampling threshold
Start with conservative settings; generation will be slower on 4GB VRAM.
Optimize for 4GB VRAM
- Enable 8-bit quantization to reduce memory footprint further:
cat > quantized_inference.py << 'EOF'
import torch
from airllm import AirLLM
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_8bit=True,
bnb_8bit_use_double_quant=True,
bnb_8bit_quant_type="nf8",
bnb_8bit_compute_dtype=torch.float16
)
model = AirLLM(
"./models/llama2-70b",
device="cuda",
quantization_config=bnb_config
)
EOF
- Reduce batch size and context length if OOM errors occur:
model.config.max_position_embeddings = 2048 # instead of 4096
- Monitor GPU and system RAM during inference:
watch -n 1 nvidia-smi
In another terminal, check system RAM:
free -h
AirLLM should keep GPU usage under 4GB and stream weights from system RAM as needed.
Run as a local API service
- AirLLM integrates with vLLM for serving. Install the vLLM integration:
pip install vllm
- Start a local OpenAI-compatible API:
python -m vllm.entrypoints.openai.api_server \
--model ./models/llama2-70b \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.9
- Query the API from another machine on your LAN:
curl http://192.168.1.100:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama2-70b",
"prompt": "Explain quantum computing",
"max_tokens": 256
}'
Replace 192.168.1.100 with your actual host IP.
Keep it private
Do not expose this service to the public internet. AirLLM has no authentication layer. Run it only on your LAN, behind a firewall, or through a VPN tunnel. If you need remote access, use WireGuard or SSH port forwarding:
ssh -L 8000:localhost:8000 user@homelab-host
Then connect to localhost:8000 from your local machine.
Is it worth it?
Yes, if you have 32GB+ RAM and a 4GB GPU, and you want to experiment with large models without renting cloud GPUs. Inference is 5–10× slower than on an RTX 4090, but for local testing, fine-tuning, or privacy-sensitive work, the cost-per-inference is effectively zero after hardware amortization. The tradeoff is latency and complexity; if you need fast inference, rent a cloud GPU. If you need privacy and can tolerate slower responses, AirLLM makes 70B models practical on consumer hardware.
Gear used in this build
* Affiliate links — I earn a small commission at no cost to you.It's gear I use and would genuinely recommend. As an Amazon Associate I earn from qualifying purchases. See the full disclosure.
Related video
New self-hosted AI & homelab shorts, daily.
Subscribe on YouTubeRelated guides
Run Local AI on Old NVIDIA/AMD GPUs with Ollama
Dust off that graphics card and run Llama, Gemma, Qwen locally with Ollama. Pick a model that fits your VRAM, skip the hype.
Run AI Models on PS5 with llama.cpp GGUF
Turn your PlayStation 5 into a local AI inference engine using llama.cpp's native GGUF support. Run Llama, Qwen, and other quantized models directly on console hardware for private, offline inference.
Run MiniMax H3 in ComfyUI: Local Audio & 2K Video
Generate audio and 2K video together on your own hardware using MiniMax H3 in ComfyUI. No API costs, full control, offline workflows.