Hierarchical KV Caching: Scaling Context Beyond VRAM Limits
As context windows scale to a million tokens, the KV cache becomes too large for GPU memory. The solution is a multi-tiered cache that offloads data to CPU and NVMe without killing latency.
AI Infrastructure
How xAI Built Grok: Training Data and Compute Infrastructure
How xAI built Grok from training data to compute infrastructure: the JAX and Rust stack, GPU cluster architecture, and why they moved beyond PyTorch.
Demystifying Google TPU SparseCore: Accelerating Recommendation Systems
How Google TPU SparseCore solves embedding lookup bottlenecks in recommender models. Learn the co-designed architecture of Trillium's SparseCores.
AI Training Chip Performance: Real Scaling Data vs Marketing Hype (Blackwell to Hopper)
AI training chip performance data: analyzing real scaling from Hopper to Blackwell. 3.2x training, 50x inference gains, and why memory bandwidth matters more than FLOPs.
The Battle for Memory: PagedAttention vs RingAttention on Kubernetes
Comparing raw memory management strategies for infinite-context enterprise agents.
· Rajat Pandit · AI InfrastructureKV Cache Offloading in K8s: The Stateless Truce
Your beloved stateless Kubernetes architecture is fundamentally at war with the massive, stateful memory requirements of long-context LLM inference. We need a truce.