The Infinite Board Problem: Pruning State in Long-Running Reasoning Loops
How to manage the shared state size in complex reasoning loops to prevent context window overflow without losing critical history.
From the Front Lines of Tech
Sharing strategic insights and lessons from over 20 years of building scalable systems, leading high-performing teams, and navigating complex technology shifts.
Hierarchical KV Caching: Scaling Context Beyond VRAM Limits
As context windows scale to a million tokens, the KV cache becomes too large for GPU memory. The solution is a multi-tiered cache that offloads data to CPU and NVMe without killing latency.
How xAI Built Grok: JAX, Rust, and the GPU Training Infrastructure
How xAI built Grok without PyTorch: Discover the JAX and Rust training infrastructure, GPU cluster architecture, and hardware stack powering Grok.
· StrategyThe CAIO's First 100 Days: Beyond Pilot Purgatory
Moving from setting up the office to surviving the execution phase without failing ROI checks. A guide for the new Chief AI Officer.
Demystifying Google TPU SparseCore: Accelerating Recommendation Systems
How Google TPU SparseCore solves embedding lookup bottlenecks in recommender models. Learn the co-designed architecture of Trillium's SparseCores.
The Real Performance Improvement Rate of AI Training Chips
Analyze the actual performance improvement rate of training chips and GPUs vs marketing hype. Here is the data on real compute scaling for training and inference.