How xAI Built Grok: Training Data and Compute Infrastructure
How xAI built Grok from training data to compute infrastructure: the JAX and Rust stack, GPU cluster architecture, and why they moved beyond PyTorch.
Tag: JAX
Speculative Decoding Infrastructure: Squeezing Latency without Hardware Upgrades
The bottleneck for LLMs is memory bandwidth, not compute. Discover how to use speculative decoding on GCP to achieve 3x speedups by using small "draft" models to accelerate massive "oracle" models.
JAX Pallas: Writing GPU Kernels for Maximum Performance
JAX Pallas is NVIDIA's GPU programming API for high-performance compute kernels. Write optimized kernels for matrix multiplication and memory access patterns.
Single-Batch Inference: Speculative Decoding on an A100
See how speculative decoding performs for single-batch requests on an NVIDIA A100. We analyze acceptance rates, latency, and the mechanics of the draft model gamble.
MoE Routing Collapse: When Your Specialists Stop Specializing
A model is only as smart as its router. We explore the physics of expert zones, the tax of token dropping, and how to keep your load balancer honest.
JAX XLA: Why Your GPU is Idle 40% of the Time
Recompilation is the silent killer of training throughput. If you see 'Jit' in your profiler, you are losing money. We dive into XLA internals.