aihomelabprivacy

Run Local AI on Old NVIDIA/AMD GPUs with Ollama

🤖 Researched and drafted automatically from the official docs, and reviewed before publishing. Commands are taken from the source projects — but always sanity-check before running anything on your own hardware.

You’ve got a graphics card sitting in a drawer. NVIDIA or AMD, doesn’t matter—Ollama runs open-source language models on real hardware you already own. No cloud bills, no model-size flexing, just inference on metal.

Ollama wraps llama.cpp and handles GPU acceleration for you. The catch: your card needs enough VRAM for the model you want to run. A 6GB card runs Gemma 2B comfortably. An 8GB card handles Mistral 7B. Check your card’s compute capability against Ollama’s GPU support list before you start.

Install Ollama

  1. Download and install Ollama for your OS:
curl -fsSL https://ollama.com/install.sh | sh

On Windows, use the PowerShell installer instead:

irm https://ollama.com/install.ps1 | iex

On Linux, the shell script handles systemd setup and GPU driver detection. On macOS and Windows, the installer sets up the daemon.

  1. Verify installation and GPU detection:
ollama --version

Run a quick test to see if Ollama finds your GPU. The first model pull will take a few minutes depending on your internet and model size.

Pick a Model That Fits Your VRAM

Model size and VRAM requirements are not negotiable. Check the Ollama library at ollama.com/library for quantization details.

NVIDIA GPUs:

  • RTX 2060 (6GB): Gemma 2B, Phi 2.7B
  • RTX 3060 (12GB): Mistral 7B, Llama 2 7B
  • RTX 4070 (12GB): Llama 2 13B, Qwen 14B
  • RTX 4090 (24GB): Llama 2 70B, Qwen 32B

AMD GPUs (ROCm):

  • RX 6600 (8GB): Gemma 2B, Phi 2.7B
  • RX 6700 XT (12GB): Mistral 7B, Llama 2 7B
  • RX 6900 XT (16GB): Llama 2 13B, Qwen 14B

Check the Ollama GitHub for the full compute capability and ROCm version support matrix. Don’t guess—a model that doesn’t fit will either crash or fall back to CPU, which defeats the purpose.

Run a Model

  1. Pull and run a model:
ollama run gemma:2b

The first run downloads the model (stored in ~/.ollama/models on Linux/macOS, %USERPROFILE%\.ollama\models on Windows). Subsequent runs load from disk.

  1. Chat with the model at the prompt:
>>> Why does my GPU run hot?

Type your question and press Enter. Press Ctrl+D to exit.

Expose Ollama Over the Network (Optional)

By default, Ollama listens on localhost:11434. If you want to send requests from another machine on your LAN:

On Linux, edit the systemd service:

sudo systemctl edit ollama

Add or modify the [Service] section:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

On macOS and Windows, set the environment variable before starting the daemon. On macOS, edit ~/.zprofile or ~/.bash_profile:

export OLLAMA_HOST=0.0.0.0:11434

On Windows, set it in System Properties → Environment Variables or via PowerShell before launching Ollama.

Security note: Never expose port 11434 to the internet. Keep it on your LAN or behind a VPN. Ollama has no built-in authentication.

Query Ollama via REST API

Once running, send requests to the API:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma:2b",
  "messages": [{
    "role": "user",
    "content": "What is the capital of France?"
  }],
  "stream": false
}'

Use this endpoint to integrate Ollama with scripts, applications, or web frontends. The response is JSON; set "stream": true for token-by-token output.

Use Ollama with Python or JavaScript

Python:

pip install ollama
from ollama import chat

response = chat(model='gemma:2b', messages=[
  {
    'role': 'user',
    'content': 'Explain quantum computing in one sentence.',
  },
])
print(response.message.content)

JavaScript/Node.js:

npm i ollama
import ollama from "ollama";

const response = await ollama.chat({
  model: "gemma:2b",
  messages: [{ role: "user", content: "What is 2+2?" }],
});
console.log(response.message.content);

Monitor GPU Usage

While a model runs, check GPU utilization:

NVIDIA:

nvidia-smi

Watch the memory usage and compute load. If it’s maxing out, try a smaller model.

AMD:

radeontop

Or check /sys/class/drm/card*/device/gpu_busy_percent on Linux.

Is It Worth It?

Yes, if your GPU sits idle and you want private, offline inference. Ollama handles the complexity; you just pick a model and run it. Response times are slower than cloud APIs but fast enough for development, local automation, and experimentation. You own your data and your hardware does the work. No subscriptions, no rate limits, no vendor lock-in. The only trade-off is VRAM—be honest about what your card can handle, or you’ll waste time debugging OOM crashes.

Related video

New self-hosted AI & homelab shorts, daily.

Subscribe on YouTube

Related guides

← All guides