Two local models. One endpoint.
tinygw picks which one answers.

tinygw is a small OpenAI-compatible gateway that runs next to your model servers. For each request it decides: strong model or light model. It forwards the call, streams the answer back, and writes nothing down.

client ──► tinygw :3000 ─┬─► strong   Flash-Next      :8000
                         └─► light    Inkling 2 Mini  :8001

How a request is routed

Checked in this order. The first step that decides wins.

  1. 1

    Forced

    Header x-tinygw-route: strong, model tinygw/strong, or the exact upstream model id. You always have the last word.

  2. 2

    Hard rules

    Code fences, diffs and patches, words like audit, review, security, or a prompt over ~6,000 tokens go to the strong model. A message under 80 characters goes to the light one. No model is called.

  3. 3

    Decision model (optional)

    Only what the rules left open. A small decision model such as Liquid AI's d1-omni-600M (MLX, Apple Silicon) answers one question, strong or light, with a calibrated probability. One forward pass, no generated text.

  4. 4

    Confidence fallback

    If that probability is under 0.8, the request goes to the strong model. When unsure, spend compute instead of mis-routing.

Every response says what happened: x-tinygw-route, x-tinygw-stage, x-tinygw-reason, x-tinygw-confidence.

Quickstart

# install
git clone https://github.com/jackjohns839/tinygw && cd tinygw
uv venv && uv pip install -e .
cp tinygw.example.toml tinygw.toml

# see where a prompt would go, and why
tinygw route "review this auth middleware for token leaks"
route=strong  stage=rule  reason=keyword:review

# run it, then point your client at http://127.0.0.1:3000/v1
tinygw serve

# optional: the d1-omni decision model on Apple Silicon
uv pip install -e '.[d1omni]'
TINYGW_BACKEND_NAME=d1omni tinygw serve
# tinygw.toml
[strong]
base_url = "http://127.0.0.1:8000/v1"
[light]
base_url = "http://127.0.0.1:8001/v1"
[backend]
name = "rules"               # or "d1omni"
confidence_threshold = 0.8

Routing is hard. We say so.

Deciding which model should answer means guessing how hard a request is before anyone has answered it. Published evaluations show off-the-shelf decision models doing close to chance at exactly this. Small ones score modestly on general benchmarks too.

So tinygw doesn't put a model in charge. Rules handle the clear cases, the model only sees the fuzzy middle, and low confidence means the strong model. The default backend uses no model at all. Try a decision model on your own traffic with tinygw route before you trust it.

Why not a big gateway?

LiteLLM, Portkey and friends handle many providers, keys, budgets and dashboards. tinygw is for one machine with two local models: no database, no UI, nothing persisted, three dependencies. If you need provider fallbacks or spend tracking, use one of those.

Source

github.com/jackjohns839/tinygw · Apache-2.0 · The repository is private for now and will open up once it has had more real-world use.