Our focus: intent routing between a light and a strong local model. Smart & Security first.

Two local models. One endpoint.
tinygw picks which one answers.

tinygw is a small OpenAI-compatible gateway that runs next to your model servers. For each request it decides: strong model or light model. It forwards the call, streams the answer back, and writes nothing down unless you turn the log on.

agent ──► tinygw :3000 ─┬─► strong   Qwen3.8-Flash-Next  :8000
                        └─► light    Inkling 2 Mini      :8001

How a request is routed

Checked in this order. The first step that decides wins.

  1. 1

    Forced

    Header x-tinygw-route: strong, model tinygw/strong, or the exact upstream model id. You always have the last word.

  2. 2

    Hard rules

    Code fences, diffs and patches, words like audit, review, security, or a prompt over ~6,000 tokens go to the strong model. A message under 80 characters goes to the light one. No model is called.

  3. 3

    Decision model (optional)

    Only what the rules left open. A small decision model such as Liquid AI's d1-omni-600M (MLX, Apple Silicon) answers one question, strong or light, with a calibrated probability. One forward pass, no generated text. Its weights are downloaded once when you enable it; pre-fetch them and it runs offline.

  4. 4

    Confidence fallback

    If that probability is under 0.8, the request goes to the strong model. When unsure, spend compute instead of mis-routing.

Every response says what happened: x-tinygw-route, x-tinygw-stage, x-tinygw-reason, x-tinygw-confidence.

Quickstart

# install
git clone https://github.com/xxxx/tinygw && cd tinygw
uv venv && uv pip install -e .
cp tinygw.example.toml tinygw.toml

# see where a prompt would go, and why
tinygw route "review this auth middleware for token leaks"
route=strong  stage=rule  reason=keyword:review

# run it, then point your agent at http://127.0.0.1:3000/v1
tinygw serve

# optional: the d1-omni decision model on Apple Silicon
uv pip install -e '.[d1omni]'
TINYGW_BACKEND_NAME=d1omni tinygw serve
# tinygw.toml
[strong]
base_url = "http://127.0.0.1:8000/v1"
[light]
base_url = "http://127.0.0.1:8001/v1"
[backend]
name = "rules"               # or "d1omni"
confidence_threshold = 0.8

Optional log: close the loop on routing quality

A router is only as good as its mistakes are visible. Turn on the log and tinygw appends one JSON line per request to a local file: the routing decision (route, stage, rule or confidence, estimated tokens) together with the full request and the response text, reassembled from the stream.

Hand that file to another model and ask, request by request, whether strong or light was the right call. That is how you debug a rule, benchmark a decision backend and tune the thresholds.

# tinygw.toml
[log]
enabled = true                 # off by default
path = "tinygw.log.jsonl"      # local file, mode 0600
max_bytes = 0                  # rotate to .1 at this size; 0 = never

Off by default and local only. It is the one switch that writes prompts and responses to disk, and tinygw says so on startup when it is on. Leave it off unless you are debugging or benchmarking.

Stats as static files, not an endpoint

tinygw serves three OpenAI-compatible routes and nothing else: no /stats, no health check, no admin page. Turn stats on and it writes aggregate numbers to a directory you choose: requests and share per route, rule hits, error rate, latency p50/p95 and time to first token. Never content.

# tinygw.toml
[stats]
enabled = true                 # off by default
dir = "tinygw-stats"           # stats.json + stats.html, rewritten in place
interval_seconds = 30

tinygw never serves these files. To see them in a browser, point your own nginx or caddy at the directory. That server is yours to run and secure.

Routing is hard. We say so.

Deciding which model should answer means guessing how hard a request is before anyone has answered it. Published evaluations show off-the-shelf decision models doing close to chance at exactly this. Small ones score modestly on general benchmarks too.

So tinygw doesn't put a model in charge. Rules handle the clear cases, the model only sees the fuzzy middle, and low confidence means the strong model. The default backend uses no model at all. Try a decision model on your own traffic with tinygw route before you trust it.

Why not a big gateway?

LiteLLM, Portkey and friends handle many providers, keys, budgets and dashboards. tinygw is for one machine with two local models: no database, no UI, nothing persisted by default, three dependencies. If you need provider fallbacks or spend tracking, use one of those.

Source

github.com/xxxx/tinygw/ · Apache-2.0