Code Intelligence Gateway
A code-search service for agents, taken from container proof of concept to serverless production — then rescued from a storage-throughput failure that no amount of application debugging would have found.
- product repositories searchable by agents through one gateway
- 8product repositories searchable by agents through one gateway
- copy time that identified storage throttling as the root cause
- 76 KB / 14 mincopy time that identified storage throttling as the root cause
- shared-volume reads left on the query hot path after the fix
- 0shared-volume reads left on the query hot path after the fix
- independent failure classes on this service; the protocol defect is the other
- 1 of 2independent failure classes on this service; the protocol defect is the other
Agents answering questions about eight product repositories need a code-search service that stays up. I designed that service, took it from a local container to a serverless deployment, and then root-caused the failure that kept killing it: not a bug in the code, but a shared network filesystem whose throughput credits were being drained by the service's own indexing schedule.
The problem
The service would not stabilize. Health checks failed, the orchestrator killed the task, and it restarted into the same failure — a crash loop that looked like an application defect. It was not. Every code query read a vector index off a shared network filesystem, and a separate scheduled job re-cloned and re-indexed eight repositories onto that same volume every thirty minutes. The two patterns together exhausted the volume's burst-throughput credits, after which all I/O throttled to baseline, startup reads stalled, and the health endpoint never answered in time.
Constraints
- Per-request session isolation was required, but spinning up a container per request was impractical — cold start and target registration cost more than the queries themselves.
- The failure presented as an application crash loop, so the evidence pointing at storage had to be found rather than reported.
- Only the question and the snippets the agent reads may leave the container; the code checkout itself cannot.
- The orchestrator's health grace period was zero, so anything slow during startup read as a dead task.
Architecture
Decisions
What I chose, why, and what I turned down to get there.
Move the index and checkout onto local ephemeral storage
The index is derived data — it can be rebuilt from source on boot, so it never needed durable shared storage in the first place. Moving it, and the code checkout with it, took the hot read path off the throttled volume entirely. The volume had been doing work that no longer had a reason to be shared.
Considered and rejected
- Paying for a higher provisioned-throughput storage tier — treats a design mistake as a billing line, and the read pattern would have kept growing into it
Give the gateway ownership of its own git sync
The external job that cloned and indexed on a schedule was writing into storage the service depended on, from outside the service's control. Folding that work into the gateway — shallow pulls on an interval it owns — removed the write storm and put the lifecycle in one place. The external job was retired rather than tuned.
Non-blocking startup with an explicit warmup gate
Health had been gated on slow I/O, which is what turned a throughput problem into a crash loop. Bootstrap now runs in the background so health answers immediately, and queries return an explicit warming-up notice until both the first index and the control-plane copy finish. Correctness and availability are decoupled: the service is honest about not being ready instead of being killed for it, and it never answers from a half-populated checkout.
Considered and rejected
- Reporting healthy as soon as the process starts — would have stopped the restarts while letting agents query an empty index, trading a loud failure for a silent one
Serialize git operations behind a mutex
Three things could trigger a sync — boot, the interval, and a manual trigger — and two passes over the same checkout at once corrupts it. A mutex is the cheapest correct answer to a race that would otherwise appear as random index corruption under load.
The measurement that identified the cause
The decisive evidence was a copy that had no business being slow. Moving six files totalling seventy-six kilobytes took fourteen minutes. Latency at that scale cannot be explained by data volume, and nothing in the application touches those files on a hot path — which rules out application-level causes and leaves infrastructure-level throttling as the only explanation that fits.
- Throughput uncorrelated with payload size is the signature of a credit-based storage tier running at baseline. Once that was the hypothesis, the burst-credit metrics confirmed it directly.
- The crash loop was downstream of the throttling, not the cause of it. Reading the symptom as an application defect is what made this expensive to find, and it is the general lesson: a health-check failure names the victim, not the culprit.
What the service actually is
A read-only code-search agent over eight product repositories, behind an authenticated gateway with request queueing and per-query session isolation, backed by a locally rebuilt vector index. Specialized agents and domain skills sit on top for documentation generation, feature-flag cataloguing, and configuration discovery. It removed most support escalations to engineering, because the questions that used to need an engineer with repository access can now be asked directly.
- Session isolation happens at the session layer rather than by isolating infrastructure per request — the compromise that made the cost model workable.
- Deployment targets a modern ARM architecture by default with fallback support retained, sized at two virtual CPUs and eight gigabytes, autoscaling on CPU and memory with one task always guaranteed.
Stack
- Go
- Docker
- AWS Fargate
- AWS ECS
- Application Load Balancer
- Model Context Protocol
- Neo4j
- ripgrep
What was mine
The service design, the deployment specification, and the root-cause analysis are mine. The infrastructure-as-code and cluster provisioning were executed by platform DevOps against that specification.