CTS Router Architecture & Latency

CTS Model Router is engineered as a low-overhead, memory-safe Go service (inference-router and api-gateway) capable of sustaining tens of thousands of concurrent streaming connections.

---

1. Request Lifecycle & Routing Engine

When a client application initiates an inference request, the payload traverses four deterministic stages:

1. Ingress & Auth        [ API Gateway: HMAC validation + Rate-limit check ]
                                │
2. Pre-Debit Hold        [ Billing Service: Atomic hold against wallet balance ]
                                │
3. Policy Guard          [ FaceGuard: PII regex check + injection mitigation ]
                                │
4. Inference Dispatch    [ Inference Router: Provider selection + failover ]
                                │
5. Stream Reconcile      [ Token counter calculates actual completion tokens ]
                                │
6. Final Ledger Settle   [ Wallet hold settled; unused hold credit restored ]

---

2. Dynamic Provider Failover

CTS Router groups upstream endpoints into prioritized routing cascades:

Cascade Tier	Target Providers	SLA Latency	Purpose
Tier 1 (Local / Private)	Dedicated GCE Spot / Host vLLM clusters	<120ms TTFT	Lowest cost, zero-data-egress dedicated workloads
Tier 2 (High-Throughput)	DeepSeek API, Gemini 2.5 Flash	<220ms TTFT	Bulk reasoning, cost-sensitive production traffic
Tier 3 (Frontier Cloud)	Anthropic Claude 3.7, OpenAI GPT-4o	<450ms TTFT	Complex agentic synthesis, mission-critical tasks

If an upstream endpoint returns HTTP 429 Too Many Requests, 502 Bad Gateway, or times out after 1,200ms, CTS Router transparently dispatches the request to the next available tier without breaking the downstream SSE connection.

---

3. Cryptographic Zero-Retention

Unlike common commercial aggregators that cache prompt text on disk for model analytics:

• Volatile Transit: Prompt buffers reside strictly in ephemeral RAM for the duration of the HTTP connection.
• No Disk Logging: Production logs record only token metadata (tenant_id, tokens_prompt, tokens_completion, duration_ms), never raw user prompts.
• Regulatory Alignment: Fully audited for NDPA 2023 (Nigeria Data Protection Act) and GDPR compliance.