TTFT (Time To First Token): Measuring Inference Correctly
TTFT reveals the real bottleneck in LLM inference. Learn why Time To First Token matters more than average latency, and how to separate prefill vs decode.
Tag: Performance
Speculative Decoding Infrastructure: Squeezing Latency without Hardware Upgrades
The bottleneck for LLMs is memory bandwidth, not compute. Discover how to use speculative decoding on GCP to achieve 3x speedups by using small "draft" models to accelerate massive "oracle" models.
HBM-Aware Load Balancing with libtpu and GKE
CPU load is a trailing indicator for AI inference. Discover how to use libtpu metrics and the GKE Gateway API to build high-density, memory-aware traffic routing for TPUs.
Dynamic LoRA Adapters: The Anti-Monolith Strategy
Stop training dozens of specialized foundation models. Discover how dynamic Low-Rank Adaptation hot-swapping fundamentally transforms multi-tenant inference infrastructure.
Benchmarking FP8 Stability: Where Gradients Go to Die
FP8 is the new frontier for training efficiency, but it breaks in the most sensitive layers. We dissect the E4M3/E5M2 split and how to spot divergence.
Performance over Portability? Running Local LLMs on the Asus ProArt 13
Can a thin-and-light PC handle production-level LLMs? We benchmark the Asus ProArt 13 with RTX 4060, the Ryzen AI 9 NPU, and the 8GB VRAM bottleneck.