Open source · MIT · Python 3.10+

Turn any local LLM into a managed API

Real API keys, token accounting, and RAG memory that makes a small model remember far more than its context window holds. It sits between your app and Ollama, vLLM or llama.cpp, without touching how you serve models.

Get started View on GitHub
pip install localgate
localgate dashboard Overview in the Sunrise theme: requests, total tokens, prompt and completion tokens, active keys, agent tool-call success, backend health, tokens per day and usage by model
The built-in dashboard, Sunrise theme. Real data from a local install.

What Ollama doesn't do for you

Model servers solve serving. localgate is the management layer around it.

OpenAI-compatible

Works with any OpenAI SDK, LangChain or curl. Unknown fields are forwarded to the backend.

API key management

Create, revoke and rate-limit per key. Keys are hashed, never stored raw.

RAG memory

Send an X-Session-ID: turns are chunked, embedded, retrieved and folded into a rolling summary.

Token accounting

Prompt and completion tokens per key, per model, over time.

Dashboard

Chat, agent runs, memory, keys and usage at /dashboard/. No build step, works offline.

Built-in coding agent

localgate code reads, edits and tests your project, with diffs, sandboxed commands and a session ledger.

Share your GPU

Per-device keys with model, quota and expiry limits, a fair queue, and QR-code setup.

Any backend, any DB

Ollama, vLLM, llama.cpp or any OpenAI-compatible server. SQLite by default, Postgres with one env var.

Production-ready

JSON logs with correlation IDs, Prometheus metrics, liveness/readiness split, graceful shutdown.

How it works

Your app talks to localgate exactly as it would talk to OpenAI. localgate checks the key, adds memory, forwards to your inference server and records usage.

Your app

OpenAI SDK, LangChain, curl, or localgate code, pointed at localhost:8000/v1.

localgate

  • Auth & per-key rate limits
  • RAG memory + rolling summary
  • Model aliasing, optional prompt cache
  • Token accounting in SQLite or Postgres

Inference

Ollama, vLLM, llama.cpp or any OpenAI-compatible server, on this machine or another.

Agent runs page in the Sunrise theme: run count, tool success rate, reliability by model and by tool, and a recent run's tool calls step by step
Agent runs: every localgate code task as a timeline, with reliability per model and per tool.

Quick start

Straight from the README. Also available as uv tool install localgate, pipx install localgate or the image ghcr.io/anjalllll/localgate.

1. Run a backend and the gateway

pip install localgate
ollama serve                 # your inference backend
ollama pull llama3
ollama pull nomic-embed-text # enables RAG memory

localgate serve              # first run: create your dashboard login
localgate keys create --name my-app

2. Use it like OpenAI

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1",
                api_key="lg_9f3a...")

client.chat.completions.create(
    model="llama3",
    messages=[{"role": "user", "content": "Hello!"}],
)
Dashboard: once localgate serve is running, open http://localhost:8000 for the getting-started page and dashboard. It runs on your machine, so there is no hosted demo.