aihomelabprivacy

Run 104GB Qwen on 48GB Mac with Ollama

🤖 Researched and drafted automatically from the official docs, and reviewed before publishing. Commands are taken from the source projects — but always sanity-check before running anything on your own hardware.

Ollama lets you run large language models locally on your Mac without cloud services. The trick to fitting a 104GB model on 48GB hardware is quantization—reducing precision to cut memory while keeping quality acceptable. You’ll pull the Qwen model, let Ollama handle the heavy lifting, and get usable inference speeds.

Install Ollama on macOS

  1. Download and install Ollama using the official installer:
curl -fsSL https://ollama.com/install.sh | sh

Or download the DMG manually from https://ollama.com/download/Ollama.dmg and drag it to Applications.

  1. Verify the installation:
ollama --version

Ollama will start as a background service and listen on localhost:11434.

Pull and Run a Quantized Qwen Model

  1. List available Qwen models from the Ollama library:
ollama list

Ollama hosts quantized versions of popular models. For a 48GB Mac, pull a Qwen variant that fits in memory—typically a Q4 or Q5 quantization of a 32B or 70B model.

  1. Pull the model (this downloads it to ~/.ollama/models/):
ollama pull qwen:latest

If you need a specific quantization, check ollama.com/library for exact tags. The pull command may take 10–30 minutes depending on model size and your network.

  1. Run the model interactively:
ollama run qwen:latest

You’ll enter a chat prompt. Type your question and press Enter. Ollama will stream tokens at ~12 tokens/sec on a Mac with unified memory.

Optimize for Speed and Memory

  1. Reduce context window if you don’t need long histories. Smaller context = faster inference. You can set this in a Modelfile (see docs.ollama.com/modelfile), but the default quantized models ship with sensible defaults.

  2. Monitor memory usage during inference:

vm_stat

Watch for page-ins/page-outs. If they spike, your model is swapping to disk and performance will tank. If that happens, try a smaller quantization (Q3 instead of Q5).

  1. Use the REST API for programmatic access instead of the CLI if you’re building integrations:
curl http://localhost:11434/api/chat -d '{
  "model": "qwen:latest",
  "messages": [{
    "role": "user",
    "content": "What is machine learning?"
  }],
  "stream": false
}'

Set "stream": true to get tokens as they generate (lower latency for user-facing apps).

Python and JavaScript Integration

For Python:

pip install ollama
from ollama import chat

response = chat(model='qwen:latest', messages=[
  {
    'role': 'user',
    'content': 'Explain quantization in machine learning.',
  },
])
print(response.message.content)

For JavaScript:

npm i ollama
import ollama from "ollama";

const response = await ollama.chat({
  model: "qwen:latest",
  messages: [{ role: "user", content: "What is quantization?" }],
});
console.log(response.message.content);

Keep It Private

Ollama runs on localhost by default. If you want to expose it to other machines on your LAN for testing, you can set the OLLAMA_HOST environment variable, but never expose Ollama to the public internet. It has no authentication. If you need remote access, run it behind a VPN or use a reverse proxy with authentication (nginx with basic auth, Cloudflare Tunnel, etc.).

Troubleshooting

  • Model won’t fit in memory: Pull a smaller quantization. Q3 uses ~40% less memory than Q5 but with slightly lower quality.
  • Slow inference: Check Activity Monitor for CPU/memory pressure. Unified memory on M-series Macs is fast, but if you’re hitting swap, inference will be slow. Reduce context length or model size.
  • Ollama crashes: Restart the service with killall ollama && ollama serve, or reboot.

Is It Worth It?

Yes, if you want a private, offline LLM that doesn’t phone home to OpenAI or Anthropic. 12 tokens/sec is slower than cloud APIs, but fast enough for most tasks—writing, coding, research, summarization. You own the model, the data stays local, and you avoid API costs. The tradeoff is that setup and optimization take time, and you’re limited by your hardware. For a 48GB Mac, you’re in the sweet spot: enough memory for solid models, fast enough for daily use, no GPU required.

Gear used in this build

* Affiliate links — I earn a small commission at no cost to you.It's gear I use and would genuinely recommend. As an Amazon Associate I earn from qualifying purchases. See the full disclosure.

Related video

New self-hosted AI & homelab shorts, daily.

Subscribe on YouTube

Related guides

← All guides