multi-tenant-rag-inference

A local-first, production-style starter for a Multi-Tenant Inference-as-a-Service platform with:

• FastAPI backend
• Ollama for local LLM inference
• Qdrant for tenant-scoped vector storage
• PostgreSQL for tenants, users, and auth metadata
• Prometheus + Grafana for basic observability
• Docker Compose for one-command local startup

This version is designed for learning first and CPU-only machines. You do not need a GPU to begin.

What You Should Download First

Start in this exact order:

1. Git
2. Python 3.11
3. Docker Desktop
4. Ollama
5. VS Code with Python extension

CPU-Friendly Model Recommendation

Because you currently do not have GPU support, start with one of these:

• phi3:mini
• gemma2:2b
• qwen2.5:1.5b

For the smoothest first run, use:

ollama pull qwen2.5:1.5b

Then try:

ollama run qwen2.5:1.5b

Learning Path

1. Bring up infrastructure with Docker Compose.
2. Run the FastAPI app locally.
3. Add tenant-aware request handling.
4. Add Ollama chat calls.
5. Add tenant-scoped Qdrant collections for RAG.
6. Add a simple ReAct-style agent.
7. Add metrics and dashboarding.
8. Add tests and demo data for two tenants.

Project Structure

RAG/
├── backend/
│   ├── app/
│   │   ├── api/
│   │   ├── core/
│   │   ├── models/
│   │   ├── services/
│   │   └── main.py
│   ├── Dockerfile
│   └── requirements.txt
├── docker/
│   └── prometheus.yml
├── docs/
│   ├── ARCHITECTURE.md
│   └── SETUP.md
├── .env.example
└── docker-compose.yml

Quick Start

Copy-Item .env.example .env
ollama pull qwen2.5:1.5b
docker compose up --build

Open:

• FastAPI docs: http://localhost:8000/docs
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000
• Qdrant: http://localhost:6333/dashboard

Usage steps:

1. log in as platform admin $adminLogin = Invoke-RestMethod -Method Post -Uri "http://localhost:8000/v1/auth/login" -ContentType "application/json" -Body '{"username":"admin","password":"admin123"}' $adminToken = $adminLogin.access_token

2. Create a tenant $createTenant = Invoke-RestMethod -Method Post -Uri "http://localhost:8000/v1/admin/tenants" -ContentType "application/json" -Headers @{ Authorization = "Bearer $adminToken" } -Body '{ "name": "Tenant C", "slug": "tenant-c", "owner_username": "tenant-c-user", "owner_password": "tenant-c123" }'

3. Login as a user $login = Invoke-RestMethod -Method Post -Uri "http://localhost:8000/v1/auth/login" -ContentType "application/json" -Body '{"username":"tenant-c-user","password":"tenant-c123"}' $tenantToken = $login.access_token

4. verify identity Invoke-RestMethod -Method Get -Uri "http://localhost:8000/v1/auth/me" -Headers @{ Authorization = "Bearer $tenantToken" }

5. Upload document Invoke-RestMethod -Method Post -Uri "http://localhost:8000/v1/documents/upload" -Headers @{ Authorization = "Bearer $tenantToken" } -Form @{ file = Get-Item ".\Kafka_Notes_Merged.pdf" }

6. Ask a question Invoke-RestMethod -Method Post -Uri "http://localhost:8000/v1/ask" -ContentType "application/json" -Headers @{ Authorization = "Bearer $tenantToken" } -Body '{"message":"What does the document say about Kafka?","top_k":3}'

Drawback list

1. A document of 500+ pages takes approx. 8-9 minutes to be ingested.