TTFT (Time To First Token): Measuring Inference Correctly
TTFT reveals the real bottleneck in LLM inference. Learn why Time To First Token matters more than average latency, and how to separate prefill vs decode.
AI Infrastructure
Hierarchical KV Caching: Scaling Context Beyond VRAM Limits
As context windows scale to a million tokens, the KV cache becomes too large for GPU memory. The solution is a multi-tiered cache that offloads data to CPU and NVMe without killing latency.
How xAI Built Grok: Training Data and Compute Infrastructure
How xAI built Grok from training data to compute infrastructure: the JAX and Rust stack, GPU cluster architecture, and why they moved beyond PyTorch.
Demystifying Google TPU SparseCore: Accelerating Recommendation Systems
How Google TPU SparseCore solves embedding lookup bottlenecks in recommender models. Learn the co-designed architecture of Trillium's SparseCores.
AI Training Chip Performance: Real Scaling Data vs Marketing Hype (Blackwell to Hopper)
AI training chip performance data: analyzing real scaling from Hopper to Blackwell. 3.2x training, 50x inference gains, and why memory bandwidth matters more than FLOPs.
The Battle for Memory: PagedAttention vs RingAttention on Kubernetes
Comparing raw memory management strategies for infinite-context enterprise agents.