CodeNex AI API Proxy
An open-source gateway that makes six different model backends answer to one API shape, with failover that lets recovered providers back in.
- provider backends reachable through one compatible API surface
- 6provider backends reachable through one compatible API surface
- independent implementations, one process-supervised and one single-binary
- 2independent implementations, one process-supervised and one single-binary
- source, so the routing and failover logic can be read directly
- Publicsource, so the routing and failover logic can be read directly
Every provider has its own request format, its own auth, and its own failure modes, so client code ends up knowing about all of them. This gateway absorbs that: one compatible surface in front of six backends, with format translation, health-aware routing, and worker supervision. It is public and inspectable, which is the point — the reasoning is readable in the code, not just described here.
The problem
Tools built against one provider's API cannot talk to another without rewriting their client, and juggling several providers means every consumer reimplements the same translation, retry, and fallback logic — inconsistently. Provider outages and rate limits then surface as failures in each client separately, with no shared notion of which backend is currently healthy.
Constraints
- Clients must not change: existing tooling expects a specific request and response shape, including streaming.
- Providers fail in different ways — rate limits, transient errors, expired credentials — and a single classification would be wrong for most of them.
- Streaming has to survive translation, so responses cannot be fully buffered before forwarding.
- Credentials for several accounts and providers live in one process, so isolation and configuration hygiene matter.
Decisions
What I chose, why, and what I turned down to get there.
Health-tracked provider pool with automatic recovery
Providers are selected least-recently-used from a pool that tracks health, and an unhealthy provider is taken out of rotation and then retried rather than blacklisted permanently. Most provider failures are transient — a rate limit is not a death sentence — so permanent removal steadily degrades capacity in exchange for nothing.
Considered and rejected
- Permanent blacklisting on failure — simpler bookkeeping, and it retires healthy capacity after a momentary rate limit
Route on model name rather than explicit configuration
The requested model name is enough to infer the right backend, so clients get sensible routing without per-provider setup. Configuration that could have been inference is configuration that will be wrong somewhere.
Two implementations, deliberately
A process-supervised runtime with a metrics store and cache for feature-rich deployments, and a single compiled binary for low-footprint ones. Keeping both honest forced the protocol translation to be specified rather than incidental to one language's conventions.
Supervise workers instead of exiting on crash
A parent process monitors workers and restarts them, so one bad request cannot take the gateway down. A gateway is a single point of failure by construction; the only acceptable answer is that it recovers without operator attention.
Why it exists at work as well as at home
The same problem shape recurred internally: several command-line coding tools, each expecting a different provider format, all needing to route through one governed path with fallback between accounts. Building this in the open first meant the internal version started from a design that had already been tested against real client tools rather than being reasoned about from scratch.
- Response caching and per-provider metrics make the cost and reliability picture visible per backend instead of aggregated into one number.
- Streaming is preserved through translation, which is the constraint that rules out the simplest possible implementation.
Stack
- Go
- Gin
- Node.js
- Redis
- PostgreSQL
- React
- OpenAI-compatible APIs
What was mine
Designed and built solo as an open-source project, published under my own account.