· Rajat Pandit · AI Infrastructure
KV Cache Offloading in K8s: The Stateless Truce
Your beloved stateless Kubernetes architecture is fundamentally at war with the massive, stateful memory requirements of long-context LLM inference. We need a truce.
Tag: GCP
Speculative Decoding Infrastructure: Squeezing Latency without Hardware Upgrades
The bottleneck for LLMs is memory bandwidth, not compute. Discover how to use speculative decoding on GCP to achieve 3x speedups by using small "draft" models to accelerate massive "oracle" models.
HBM-Aware Load Balancing with libtpu and GKE
CPU load is a trailing indicator for AI inference. Discover how to use libtpu metrics and the GKE Gateway API to build high-density, memory-aware traffic routing for TPUs.
Beyond Vibe-Checks: Trajectory Evaluation & Synthetic Adversaries
Is your agent actually reasoning, or just lucky? Discover why trajectory analysis and synthetic red-teaming are the only ways to build production-grade autonomous systems.
· StrategyThe Valuation of Open Weights: The Intelligence Supply Chain
Open source models are transforming AI from a variable SaaS cost into a strategic capital asset. Discover why owning the weights is the key to Sovereign AI and a 70% reduction in long-term TCO.
Writing Pallas Kernels for JAX: Stepping Outside the XLA Safety Net
When XLA's heuristics fail for custom attention mechanisms, you can't just hope for a compiler update. Here is how you write Triton-like kernels directly in Python using JAX Pallas.