The Real Performance Improvement Rate of AI Training Chips
Analyze the actual performance improvement rate of training chips and GPUs vs marketing hype. Here is the data on real compute scaling for training and inference.
Tag: Blackwell
Visualizing All-Reduce Bandwidth: The Physics of Distributed Training
When your model doesn't fit on one GPU, you're no longer just learning coding-you're learning physics. We dive deep into the primitives of NCCL, distributed collectives, and why the interconnect is the computer.
Demystifying Nvidia Blackwell: FP4 and Microscaling Explained
Break down the new FP4 format and microscaling scale factors in the NVIDIA Blackwell architecture. Understand how it differs from FP8 and its impact on AI training.