What I Learned Building a FinOps Control Plane for an AI Gateway

2026-06-09T00:00:00+00:00

Most teams add a chatbot and discover two problems later: token spend has no owner, and retrieval changes are impossible to compare.

Cost visibility: every query hits the same model tier. Nobody can answer “what did last Tuesday’s support bot traffic cost?”</li>
Retrieval quality: you change BM25 weights or add a reranker and hope answers feel better. There is no before/after score.</li> </ol>
I wanted a gateway that treats those as engineering problems, not product guesses.

Classify query complexity (Groq)</li>
Retrieve with hybrid RAG (pgvector + BM25 + rerank)</li>
Route to cheap vs expensive tiers (local Ollama vs Groq)</li>
Export routing labels, latency breakdown, modeled cost, and eval scores to Grafana + LangSmith</li> </ol>
Stack: Python, PostgreSQL/pgvector, Prometheus, Pushgateway, Grafana, LangSmith, Docker Compose.
</a> Routing (load test, n=12)</h3>
Tier</th> Share</th></tr></thead>

Simple (local Ollama)</td> 7/12 (58%)</td></tr>
Medium</td> 3/12</td></tr>
Complex</td> 2/12</td></tr> </tbody></table>
</a> Latency</h3>
Component</th> p50 (approx.)</th></tr></thead>

End-to-end</td> ~276s (concurrency=2, local Ollama)</td></tr>
Retrieval</td> ~2.3s</td></tr>
Generation (simple tier)</td> ~356s (local Ollama dominates wall time)</td></tr>
Generation (medium/complex)</td> ~1–4s (Groq in demo mode)</td></tr> </tbody></table>
Retrieval is fast. Local generation is slow and free. That tradeoff is visible in the Grafana Latency Breakdown p50 panel.
</a> Cost (simulated FinOps model, not actual API spend)</h3>
Demo mode uses Groq + Ollama ($0 actual spend). Grafana applies Claude list rates × tokens for medium/complex tiers as a counterfactual model.
Metric</th> Value</th></tr></thead>

Simulated spend (tier routing)</td> $0.0143 (12 queries)</td></tr>
Simulated all-complex counterfactual</td> $0.0609</td></tr>
Modeled savings vs counterfactual</td> ~77% (simulation only)</td></tr>
Actual API spend (demo mode)</td> $0</td></tr> </tbody></table>
Label simulated numbers clearly. Do not present them as production FinOps proof.
</a> RAG eval (RAGAS pilot, n=8)</h3>
Eight Q&A pairs is enough to compare pipelines directionally. It is not statistically powered for strong faithfulness claims.
Metric</th> Baseline (cosine)</th> Hybrid+rerank</th> Delta</th></tr></thead>

context_precision</td> 0.6042</td> 0.7917</td> +0.1875</td></tr>
faithfulness</td> 0.7690</td> 1.0000</td> +0.2310 (noisy at n=8)</td></tr>
answer_relevancy</td> 0.7716</td> 0.7947</td> +0.0231</td></tr> </tbody></table>
Lead metric: context_precision +0.19 on the pilot set. Do not over-claim faithfulness at n=8.

Scale the eval set to n=50+ before claiming retrieval wins to stakeholders.</li>
Separate measured spend from modeled spend in every dashboard panel (this repo does; many demos do not).</li>
Add circuit breakers when cheap tiers fail open to expensive models.</li>
Wire real billing exports (AWS Cost Explorer, OpenAI usage API) instead of list-price simulation.</li> </ol>
Showing you can instrument an LLM gateway like any other backend service</li>
Comparing retrieval pipelines with structured eval, not vibes</li>
Explaining tiered routing tradeoffs with numbers attached</li> </ul>

A replacement for production PostgreSQL/API performance work (my day job at Careem)</li>
Proof that hybrid RAG is “production ready” at n=8 eval</li>
A $3,500 Upwork audit deliverable by itself</li> </ul>
`git clone https://github.com/muhammadahmed-01/FinOps-AI-Gateway copy .env.example .env # GROQ_API_KEY + LANGCHAIN_API_KEY .\scripts\bootstrap_demo.ps1</code></pre> Grafana: http://localhost:3001 → dashboard FinOps AI Gateway.`Muhammad Ahmed · Backend engineer @ Careem · GitHub</a> · LinkedIn</a>

Shift-Left FinOps: Adding SageMaker Support to Infracost

2026-05-10T00:00:00+00:00

GitHub PR: infracost/infracost#3567</a> (open, awaiting upstream review)
“Shift-Left FinOps” is about moving cost visibility as far forward in the development lifecycle as possible. This project involved contributing to Infracost, a tool that sits in the CI/CD pipeline to show engineers the “price tag” of their Terraform changes before they hit production.
`Inference Variants: Support for both provisioned instance-based hosting and serverless configurations.</li> Shadow Production Variants: Mapping costs for test models running in parallel with production traffic—a common source of “bill shock.”</li>`Async & Data Capture: Accounting for the overhead of asynchronous inference configurations and data capture fees for model monitoring.</li> </ul> </a> Implementation Details</h3> Working in the Go codebase of Infracost, I had to map complex Terraform resource attributes to the AWS Price List API. This required: Resource Mapping: Translating nested configuration blocks into cost components.</li> Logic for Serverless vs. Provisioned: Handling the conditional logic where pricing changes from “per hour” to “per GB/second” based on the serverless_config</code> block.</li> Unit Testing: Ensuring that the pricing engine accurately calculates the monthly estimate across various edge cases (e.g., provisioned concurrency).</li> </ol> </a> Key Learnings</h3> Contributing to a FinOps tool is a masterclass in the “financial underbelly” of the cloud. It forces you to realize that as a software engineer, you are effectively a “financial architect.” Every line of infrastructure code is a spending decision. By building visibility into the PR level, we empower engineers to be proactive rather than reactive.
💥 The "Money Vanishing" Story: Concurrency & Data Integrity! 2026-05-08T00:00:00+00:00 GitHub (reproduce locally): https://github.com/muhammadahmed-01/Go-Concurrency-Ledger-Integrity · Measured results: k6/K6_RESULTS.md</a> Maps directly to DDIA Chapter 7 (Transactions) and Chapter 8 (Distributed Systems). Hardware: Intel i5-12500H · 16GB RAM · Windows 11 (battery; absolute latencies would be lower on AC/server hardware; relative comparisons hold) Stack: Go · PostgreSQL · Prometheus · Grafana · k6 Load: 800 VUs ramped over 50s (200 → 500 → 800), starting balance: $1,000,000 </blockquote> 1. The Problem</h2> Two goroutines simultaneously deduct $10 from the same account. After both finish, only $10 was deducted — not $20. Money vanished. Account balance: 1,000,000 Goroutine A: READ balance → 1,000,000 Goroutine B: READ balance → 1,000,000 ← same stale value, before A writes Goroutine A: WRITE balance = 999,990 Goroutine B: WRITE balance = 999,990 ← overwrites A. One deduction lost. Final: 999,990 (should be 999,980)</code></pre> This is the Lost Update Problem — one of the most common bugs in concurrent systems, and the most dangerous because it produces no errors. 2. Root Cause</h2> Read-Modify-Write without synchronization. // ❌ BUGGY — classic lost update balance := SELECT balance FROM accounts WHERE id=1 // read // another goroutine reads the same value here UPDATE accounts SET balance = balance-10 WHERE id=1 // overwrite, not decrement</code></pre> The check and write are not atomic. Between reading and writing, another goroutine reads the same stale value. Both write back a balance reflecting only their deduction. Maps to DDIA §7.1 — Dirty Reads and Lost Updates. 3. Three Strategies Tested</h2> Fix #1 — Pessimistic Locking</h3> Principle: Assume conflict will happen. Lock the resource before reading. BEGIN; SELECT balance FROM accounts WHERE id = 1 FOR UPDATE; -- acquires exclusive row lock -- other goroutines trying FOR UPDATE BLOCK here UPDATE accounts SET balance = balance - 10 WHERE id = 1; COMMIT; -- lock released</code></pre> SELECT FOR UPDATE</code> acquires an exclusive row-level lock. Any other transaction attempting to read or write the same row blocks until the first commits or rolls back. Guarantees serialization — no two goroutines inside the critical section simultaneously. ✅ Pros</th> ❌ Cons</th></tr></thead> Correctness guaranteed</td> Serializes all deductions → lower throughput</td></tr> Simple mental model</td> Lock contention grows with concurrency</td></tr> Database enforces it</td> Deadlock risk with multiple rows</td></tr> </tbody></table> Fix #2 — Optimistic Concurrency Control (OCC)</h3> Principle: Assume conflict is unlikely. Don’t lock — detect conflicts on write and retry. -- Read without any lock SELECT balance, version FROM accounts WHERE id = 1; -- balance=999,990, version=42 -- Write only if version hasn't changed (Compare-And-Swap) UPDATE accounts SET balance = balance - 10, version = version + 1 WHERE id = 1 AND version = 42; -- If another goroutine already incremented version → rows_affected = 0 → RETRY</code></pre> The WHERE version = $current_version</code> guard is a CAS operation. If rows_affected == 0</code>, a conflict occurred — retry from the read. The database never blocks; goroutines spin instead of waiting. ✅ Pros</th> ❌ Cons</th></tr></thead> Higher throughput when contention is low</td> CPU burns on retries under high contention</td></tr> No deadlock risk</td> Retry logic adds code complexity</td></tr> Scales horizontally</td> Fairness not guaranteed — goroutines can starve</td></tr> </tbody></table> 4. Real Benchmark Results</h2> Key insight: Completed iterations differ significantly per mode — this is a finding, not noise. Buggy has no overhead so all requests complete. Pessimistic serializes. Optimistic hits retry limits. </blockquote> Metric</th> Buggy</th> Optimistic</th> Pessimistic</th></tr></thead> Completed Iterations</td> 160,207</td> 21,798</td> 51,914</td></tr> Effective Deductions</td> 964 ❌</td> 12,768 ✅</td> 51,293 ✅</td></tr> Final Balance</td> 990,360 ❌</td> 872,320 ✅</td> 487,070 ✅</td></tr> Money Lost</td> $9,640 vanished</td> $0</td> $0</td></tr> p95 Latency (peak)</td> ~600ms</td> ~4,250ms 🔴</td> ~2,200ms 🔴</td></tr> Error Rate</td> 0% ✅</td> ~5–8% (409/500)</td> ~15–28% (503)</td></tr> k6 SLA Thresholds</td> ✅ Passed</td> ❌ Failed</td> ✅ Passed</td></tr> DB Version at End</td> 1 (never incremented)</td> 12,769 ✅</td> N/A</td></tr> Data Integrity</td> ❌ CORRUPTED</td> ✅ Correct</td> ✅ Correct</td></tr> </tbody></table> 5. Detailed Findings</h2> 🚨 Finding 1 — The Invisible Failure (Buggy Mode)</h3> The most dangerous failure mode in production. </blockquote> ✅ HTTP 200 on every request, 0% error rate, lowest latency, highest throughput</li> ❌ Only 964 effective deductions out of 160,207 attempts. $9,640 vanished silently.</li> Version column stayed at 1</code> — every goroutine overwrote the same cycle</li> </ul> ⚡ The Key Insight: Your monitoring shows all green while silently destroying data. This is why data integrity bugs are more dangerous than crashes. Crashes are obvious. Silent corruption is not. 🚨 Finding 2 — The Retry Storm (Optimistic Mode)</h3> Postgres logs confirmed two distinct failure modes: 409</code> responses where the version CAS guard rejected the write after retries were exhausted, and 500</code> responses where the Go context timeout cancelled the in-flight Postgres query mid-execution — visible in DB logs as canceling statement due to user request</code>.</li> Only 21,798 iterations completed — 7× less useful work than buggy mode</li> p95 climbed to ~4,250ms — highest of all three</li> version=12,769</code> matched 12,768 effective deductions</code> exactly — correctness maintained, at enormous cost</li> Postgres hit 49.6% CPU just managing retry overhead</li> System continued processing in-flight retries after k6 stopped — retry storms have a latency tail that outlasts the load</li> </ul> The version column is the smoking gun: proves OCC was always correct — but correctness required ~580,000 retries. 🚨 Finding 3 — The Availability Cliff (Pessimistic Mode)</h3> ~155 failed requests/second at peak (503)</li> p95 reached ~2,200ms</li> Only 51,914 iterations — 3× fewer than buggy, but nearly all correct</li> Without the 2s context timeout, goroutines would pile up until OOM. 503 is the right production behavior.</li> </ul> 6. Broader Concepts This Experiment Demonstrates</h2> ACID Guarantees</h3> The buggy endpoint violates Isolation — two transactions observe each other’s in-progress state. Pessimistic locking restores it. Maps to DDIA §7.2. Isolation Levels</h3> Read Committed (PostgreSQL default): prevents dirty reads, but not lost updates</li> Repeatable Read / Serializable: prevents lost updates, higher overhead</li> SELECT FOR UPDATE</code> at Read Committed gives Serializable-like safety for specific rows without changing the whole transaction’s isolation level — surgical and efficient</li> </ul> Lost Updates — Six Prevention Strategies (DDIA §7.4)</h3> Atomic writes (UPDATE ... SET balance = balance - 10</code> — safe if no read in between)</li> Explicit locking (SELECT FOR UPDATE</code>) — what we tested</li> Compare-and-set (OCC version column) — what we tested</li> Conflict detection (database detects and aborts)</li> Application-level locking (Redis distributed lock)</li> Serializable isolation (nuclear option — correct but expensive)</li> </ol> Throughput vs Consistency</h3> Buggy: max throughput, zero consistency</li> Pessimistic: min throughput, max consistency</li> Optimistic: good throughput when contention is low, catastrophic under heavy conflict</li> </ul> This is the fundamental tension in distributed systems — the tradeoff DDIA explores across Chapters 7–9. Retries and Idempotency</h3> OCC requires retries. If your operation is not idempotent, retrying causes double-charges, double-sends, double-emails. Always pair retry logic with idempotency keys. Maps to DDIA §11.5. What This Looks Like at Stripe/Netflix Scale</h3> At millions of requests/second on the same row, even OCC breaks down. Escalation path: Shard the account — split one row into N shards, sum on read (reduces hot-row contention by N×)</li> Event sourcing — append-only log, no in-place updates, replay to get balance</li> CRDT-based balance — conflict-free replicated data types, mathematically merge concurrent updates without coordination</li> </ul> 7. The Contention Spectrum</h2> SINGLE HOT ROW (extreme contention) │ Buggy ─────────────────────┼────────── Pessimistic (fast, silent │ (correct, serialized, corruption) │ 503s under load) │ Optimistic (correct but retry storm at high contention, excellent at low contention) │ MANY INDEPENDENT ROWS (low contention) (OCC wins — conflicts rare, no blocking, horizontally scalable)</code></pre> 8. Real-World Architectural Solutions at Scale</h2> Option A — LMAX Sequencer (Right Answer for High-Frequency Deductions)</h3> Remove the database from the hot path entirely. Route all deductions through a single-threaded processor fed by an in-memory ring buffer. DB write is async — the processor never waits for disk. [800 VUs] → [Ring Buffer] → [Single Worker Thread] → [Async DB write] (sequential, zero contention)</code></pre> Why this beats pessimistic + semaphore: A semaphore reduces how many goroutines fight over the row — the fight still exists. LMAX removes the fight entirely. Processes at memory speed: no locks, no retries, no conflicts. LMAX-based systems handle millions of ops/sec per core. Cons: High implementation complexity, fully async I/O required, hard to scale horizontally. Use when: Payment processors, matching engines, HFT — anywhere hot row contention is unavoidable and throughput targets can’t be met with locking. Option B — Pessimistic Locking + Go Semaphore (Pragmatic Middle Ground)</h3> Cap concurrent DB writers so excess requests queue cheaply in Go, not expensively in Postgres: var sem = make(chan struct{}, 25) func deductHandler(w http.ResponseWriter, r *http.Request) { sem <- struct{}{} defer func() { <-sem }() // Now 25 goroutines compete, not 800 runPessimisticDeduction(r.Context(), db) }</code></pre> Still DB-row-bound, but eliminates the goroutine flood causing 503 storms. Right choice when simplicity matters more than maximum throughput. Option C — Sharding, Eventual Consistency, Event Sourcing</h3> Approach</th> Use When</th></tr></thead> Sharding</td> Many independent accounts, low per-account contention</td></tr> Async Queue (SQS/Kafka)</td> Can tolerate stale reads, non-critical aggregates</td></tr> Event Sourcing</td> Full audit trail required, append-only semantics</td></tr> </tbody></table> 9. Decision Framework</h2> Is contention on a single hot row unavoidable? ├── No (many independent rows) → OCC (Optimistic Locking) └── Yes ├── Maximum throughput required → LMAX Sequencer └── Simplicity > max throughput ├── Low-medium traffic → Pessimistic + Semaphore └── Can tolerate stale reads → Eventual Consistency</code></pre> The SAA Blueprint: Learning to 'Think' in AWS 2026-04-15T00:00:00+00:00 Passing the AWS Solutions Architect – Associate (SAA-C03) with a score of 861/1000 was less about the badge and more about a fundamental shift in how I approach backend engineering. In a high-scale environment like Careem, “making it work” is the bare minimum. The real challenge is making it resilient, cost-effective, and observable. Here are the core principles I carried away from the certification process. Depth over Trivia</h3> The SAA syllabus is vast, but the real engineering value lies in the Core Pillars: Resilience, Performance, Security, and Cost. I chose to ignore the “trivia” (maximum limits of obscure services) and focused on the integration logic of the core stack: RDS, SQS, S3, and VPC networking. The “Differential Learning” Framework</h3> When you can’t easily lab “Enterprise” features like Control Tower or AWS Organizations in a personal account, you have to use a different approach. I used LLMs as an investigative tool to stress-test my understanding of subtle differences between similar services. Instead of reading a syllabus, I asked: “If I have a multi-account strategy with strict compliance requirements, why would I use Control Tower over a manual Organizations setup?” This turned abstract concepts into concrete architectural decisions. From “How to Code” to “How it Fails”</h3> The most permanent change is the lens through which I view every system design decision. I no longer just see a Go service; I see a component that must handle: Exponential Backoff & Jitter: Ensuring a network blip doesn’t trigger a retry storm.</li> Decoupling: Using SQS not just for async work, but as a buffer to protect downstream databases.</li> Cost as a First-Class Citizen: Realizing that a sub-optimal S3 storage class is effectively a technical debt on the company’s bottom line.</li> </ul> If you are a backend engineer, don’t study for the paper. Study for the mental map that allows you to reason through tradeoffs when there is no single “right” answer.

Shift-Left FinOps: Adding SageMaker Support to Infracost

💥 The "Money Vanishing" Story: Concurrency & Data Integrity!

3. Three Strategies Tested</h2>

5. Detailed Findings</h2>

6. Broader Concepts This Experiment Demonstrates</h2>

8. Real-World Architectural Solutions at Scale</h2>

The SAA Blueprint: Learning to 'Think' in AWS

Systems & Stories

AWS Cost Hygiene: S3 Lifecycle Savings at Careem

What I Learned Building a FinOps Control Plane for an AI Gateway

Systems & Stories

AWS Cost Hygiene: S3 Lifecycle Savings at Careem

What I Learned Building a FinOps Control Plane for an AI Gateway

Shift-Left FinOps: Adding SageMaker Support to Infracost

💥 The "Money Vanishing" Story: Concurrency & Data Integrity!

3. Three Strategies Tested</h2>

5. Detailed Findings</h2>

6. Broader Concepts This Experiment Demonstrates</h2>

8. Real-World Architectural Solutions at Scale</h2>

The SAA Blueprint: Learning to 'Think' in AWS