The Real Performance Improvement Rate of AI Training Chips
Analyze the actual performance improvement rate of training chips and GPUs vs marketing hype. Here is the data on real compute scaling for training and inference.
Tag: Blackwell
Visualizing All-Reduce Bandwidth: The Physics of Distributed Training
When your model doesn't fit on one GPU, you're no longer just learning coding-you're learning physics. We dive deep into the primitives of NCCL, distributed collectives, and why the interconnect is the computer.
Nvidia Blackwell: Microscaling, FP4, and FP6 Formats
Nvidia Blackwell microscaling and the new FP4 formats double inference speeds. Dive into how the second-generation Transformer Engine uses scale factors and sparsity for AI workloads.