· Rajat Pandit · AI Infrastructure
KV Cache Offloading in K8s: The Stateless Truce
Your beloved stateless Kubernetes architecture is fundamentally at war with the massive, stateful memory requirements of long-context LLM inference. We need a truce.
AI Infrastructure
· Rajat Pandit · AI InfrastructurevLLM Continuous Batching: How PagedAttention Optimizes GPU Throughput
vLLM continuous batching and PagedAttention explained: see how dynamic KV cache allocation eliminates memory fragmentation and boosts GPU throughput by 3x–5x.
Deploying Agentic AI as a Service (AaaS)
Deep dive into deploying agentic ai as a service (aaas).
Speculative Decoding Infrastructure: Squeezing Latency without Hardware Upgrades
The bottleneck for LLMs is memory bandwidth, not compute. Discover how to use speculative decoding on GCP to achieve 3x speedups by using small "draft" models to accelerate massive "oracle" models.
HBM-Aware Load Balancing with libtpu and GKE
CPU load is a trailing indicator for AI inference. Discover how to use libtpu metrics and the GKE Gateway API to build high-density, memory-aware traffic routing for TPUs.
Beyond Vibe-Checks: Trajectory Evaluation & Synthetic Adversaries
Is your agent actually reasoning, or just lucky? Discover why trajectory analysis and synthetic red-teaming are the only ways to build production-grade autonomous systems.