Run LLMs Locally with Ollama: Complete Self-Hosting Guide

Ollama lets you run large language models entirely on your own hardware. No API calls, no subscriptions, no vendor lock-in. You get the model weights, the inference engine, and a REST API you control. This guide walks you through installation, running your first model, and integrating it into your homelab.

Install Ollama

Ollama runs on macOS, Windows, Linux, and Docker. Pick your platform:

Linux

1. Download and run the installer:

curl -fsSL https://ollama.com/install.sh | sh

2. Verify the installation:

ollama --version

The service starts automatically. Check status with:

sudo systemctl status ollama

macOS

1. Run the installer:

curl -fsSL https://ollama.com/install.sh | sh

Or download the DMG manually from https://ollama.com/download/Ollama.dmg and drag to Applications.

2. Start Ollama from Applications or Launchpad.

Windows

1. Run the PowerShell installer:

irm https://ollama.com/install.ps1 | iex

Or download the installer from https://ollama.com/download/OllamaSetup.exe.

2. Launch Ollama from Start Menu.

Docker

1. Pull the official image:

docker pull ollama/ollama

2. Run a container with GPU access (adjust for your setup):

docker run --gpus all -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

The REST API listens on localhost:11434 inside the container and on your host.

Run Your First Model

Ollama comes with a library of open models. Start with a lightweight one:

1. Pull and run Gemma 4:

ollama run gemma4

This downloads the model weights (on first run) and drops you into an interactive chat. Type your questions and press Enter.

2. Exit the chat:

/bye

Query Models via REST API

Ollama exposes a REST API on port 11434. Use it to integrate models into scripts and applications.

1. Start Ollama if it isn’t already running. On Linux, it runs as a systemd service. On macOS and Windows, launch the app.

2. Make a chat request:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{
    "role": "user",
    "content": "Why is the sky blue?"
  }],
  "stream": false
}'

Set "stream": true for streaming responses (useful for long outputs).

3. Parse the JSON response to extract the assistant’s message.

Use Ollama with Python

1. Install the Python client:

pip install ollama

2. Write a script:

from ollama import chat

response = chat(model='gemma4', messages=[
  {
    'role': 'user',
    'content': 'Why is the sky blue?',
  },
])
print(response.message.content)

3. Run it:

python your_script.py

Use Ollama with JavaScript

1. Install the npm package:

npm i ollama

2. Write a script:

import ollama from "ollama";

const response = await ollama.chat({
  model: "gemma4",
  messages: [{ role: "user", content: "Why is the sky blue?" }],
});
console.log(response.message.content);

3. Run it:

node your_script.js

Explore Available Models

Ollama hosts a library of open models at https://ollama.com/library. Popular choices include:

• gemma4: Fast, efficient, good for general tasks.
• llama2: Meta’s flagship model, strong reasoning.
• mistral: Lightweight, fast inference.
• neural-chat: Optimized for conversation.

Run any model the same way:

ollama run mistral

Models are cached locally after download, so subsequent runs are instant.

Secure Your Setup

Ollama listens on localhost:11434 by default, which is safe on a single machine. If you run it on a server or homelab:

1. Never expose port 11434 to the public internet. Anyone with access can query your models and consume resources.

2. Keep Ollama behind a firewall or VPN. If you need remote access, use a VPN or SSH tunnel:

ssh -L 11434:localhost:11434 user@your-homelab

Then query localhost:11434 from your local machine.

3. Monitor resource usage. Large models consume significant CPU and RAM. Run top or htop to watch for runaway processes.

Integrate with Web Interfaces

Ollama works with several open-source chat UIs. Popular options include:

• Open WebUI: Self-hosted ChatGPT alternative with a polished interface.
• LibreChat: Multi-provider support, knowledge bases, plugins.
• Lobe Chat: Modern chat with plugin ecosystem.

All of these connect to Ollama’s REST API on port 11434. Refer to each project’s documentation for setup.

Is It Worth It?

Yes, if you value privacy, control, and no API costs. Ollama is lightweight, fast, and works offline. The trade-off: you’re responsible for hardware and model selection. Smaller models like Gemma 4 or Mistral run on modest machines (8GB RAM, no GPU required, though a GPU helps). Larger models demand more resources. Start small, benchmark on your hardware, and scale up if needed. For a homelab, Ollama is the foundation of a truly independent AI stack.

Sources

• https://github.com/ollama/ollama
• https://ollama.com/library