RadixAttention vs PagedAttention: The New Frontier in Context Management
A deep dive into the mechanics of SGLang's RadixAttention and why it represents a breakthrough for multi-turn agentic workflows compared to vLLM's PagedAttention.
From the Front Lines of Tech
Sharing strategic insights and lessons from over 20 years of building scalable systems, leading high-performing teams, and navigating complex technology shifts.
The Infinite Board Problem: Pruning State in Long-Running Reasoning Loops
How to manage the shared state size in complex reasoning loops to prevent context window overflow without losing critical history.
Hierarchical KV Caching: Scaling Context Beyond VRAM Limits
As context windows scale to a million tokens, the KV cache becomes too large for GPU memory. The solution is a multi-tiered cache that offloads data to CPU and NVMe without killing latency.
Dissecting the xAI Training Stack: Why Grok Chose JAX + Rust
A deep dive into the engineering choices behind xAI's massive compute cluster, exploring why JAX and Rust are replacing the standard PyTorch stack for extreme-scale training.
· StrategyThe CAIO's First 100 Days: Beyond Pilot Purgatory
Moving from setting up the office to surviving the execution phase without failing ROI checks. A guide for the new Chief AI Officer.
Demystifying the TPU SparseCore: The Hidden Workhorse of Recommendation Systems
While LLMs grab the headlines, recommendation models quietly run the global economy. We explore how Google’s TPU SparseCore architecture solves the massive memory bottleneck of embedding lookups.