aihomelabprivacy

Self-host Milvus: ditch expensive vector DB SaaS

🤖 Researched and drafted automatically from the official docs, and reviewed before publishing. Commands are taken from the source projects — but always sanity-check before running anything on your own hardware.

Vector databases have become the default choice for AI applications—RAG systems, semantic search, recommendation engines—but paying per query or per vector stored adds up fast. Milvus is a distributed, open-source vector database written in Go and C++ that runs on your own hardware. You get the same search performance, metadata filtering, and hybrid search capabilities without the SaaS bill. For a homelab or small production workload, Milvus Standalone mode runs in Docker on a single machine and scales to billions of vectors when you need it.

Prerequisites

You’ll need Docker and Docker Compose installed on your host machine. A system with at least 4 CPU cores and 8 GB RAM is sufficient for development and small production workloads. For heavier indexing and search, bump to 8+ cores and 16+ GB RAM.

Step 1: Pull and run Milvus Standalone with Docker

Milvus provides an official Docker image. The simplest way to get started is to run it standalone:

docker pull milvusdb/milvus:latest

Then start the container. Milvus listens on port 19530 for gRPC by default:

docker run -d --name milvus \
  -p 19530:19530 \
  -p 9091:9091 \
  -v milvus_data:/var/lib/milvus \
  milvusdb/milvus:latest

This creates a named volume milvus_data to persist your vectors and metadata across container restarts. Port 9091 is the metrics port; you can drop it if you don’t need Prometheus scraping.

Step 2: Install the Python SDK

On your host or in a Python environment where you’ll build your application, install pymilvus:

pip install -U pymilvus

For a quick local prototype without running a server, you can also use Milvus Lite:

pip install pymilvus[milvus-lite]

Milvus Lite creates a local SQLite-backed vector store in a file, perfect for testing before moving to the server.

Step 3: Connect and create a collection

Once the container is running, connect via Python:

from pymilvus import MilvusClient

client = MilvusClient(
    uri="http://localhost:19530",
)

Create a collection to hold your vectors. A collection is like a table; it has a schema defining fields and their types:

client.create_collection(
    collection_name="documents",
    dimension=768,  # Match your embedding model output
)

The dimension must match the output size of your embedding model (e.g., OpenAI’s text-embedding-3-small is 1536, many open models are 768).

Step 4: Insert vectors and metadata

Prepare your data as a list of dictionaries. Each record should include the vector and any metadata you want to filter or retrieve later:

data = [
    {"id": 1, "vector": [0.1, 0.2, 0.3, ...], "text": "First document", "source": "blog"},
    {"id": 2, "vector": [0.4, 0.5, 0.6, ...], "text": "Second document", "source": "paper"},
]

res = client.insert(collection_name="documents", data=data)
print(res)

Milvus automatically creates an index on the vector field. For production workloads, you can tune the index type (HNSW, IVF, FLAT, SCANN, DiskANN) and parameters after inserting data.

Step 5: Search vectors

Perform a vector search by passing one or more query vectors:

query_vector = [0.15, 0.25, 0.35, ...]  # Your query embedding

res = client.search(
    collection_name="documents",
    data=[query_vector],
    limit=5,  # Return top 5 results
    output_fields=["text", "source"],
)

for hits in res:
    for hit in hits:
        print(f"Distance: {hit['distance']}, Text: {hit['entity']['text']}")

The distance metric (L2, cosine, IP) depends on your embedding model and index choice.

Milvus supports filtering on scalar fields (integers, strings, JSON) during search:

res = client.search(
    collection_name="documents",
    data=[query_vector],
    filter='source == "blog"',  # Only return results from blog
    limit=5,
    output_fields=["text", "source"],
)

For hybrid search combining dense vectors with full-text search (BM25), Milvus can store both sparse and dense embeddings in the same collection and rerank results.

Step 7: Persist and backup

Your data lives in the Docker volume milvus_data. To back it up, stop the container and copy the volume:

docker stop milvus
docker run --rm -v milvus_data:/data -v $(pwd):/backup \
  alpine tar czf /backup/milvus_backup.tar.gz -C /data .

To restore, reverse the process or mount the volume to a new container.

Step 8: Scale beyond Standalone (optional)

If you outgrow a single machine, Milvus distributes across Kubernetes. The official docs cover multi-node deployment with separate coordinators, query nodes, and data nodes. For now, Standalone handles billions of vectors on modest hardware with proper index tuning.

Monitoring and tuning

Milvus exposes Prometheus metrics on port 9091. Scrape them with Prometheus and visualize in Grafana to track query latency, memory usage, and index build time. Start with FLAT or HNSW indexing; switch to IVF or DiskANN if search latency becomes a bottleneck.

Important security note: Keep Milvus on your LAN or behind a VPN. The default configuration does not enforce authentication. If you need to expose it, enable authentication and TLS encryption as described in the official security guide.

Is it worth it?

Yes, if you’re running RAG, semantic search, or recommendation systems in production. A $50/month Pinecone or Weaviate subscription becomes a one-time hardware cost. Milvus is battle-tested at scale and open-source, so you own the code. The trade-off: you manage upgrades, backups, and scaling. For prototypes or low-traffic hobby projects, Milvus Lite (in-process, file-backed) costs nothing and runs anywhere Python runs. For anything serious, self-hosting Milvus saves money and keeps your vectors off someone else’s servers.

Related guides

← All guides