· Rajat Pandit · AI Infrastructure
Continuous Batching in vLLM: Killing the Hardware Idle
If your GPUs are idling at 40% utilization during inference, you are burning capital on memory bottlenecks, not computation.
Tag: vLLM
Speculative Decoding: Cheating Physics for Latency
Using a 'Draft' model costs 10% more VRAM but saves 50% Latency. Here is the mechanics of the gamble.
The Efficiency Moat - Navigating the New Economics of AI Inference
As the AI industry moves from model training to large-scale deployment, the strategic bottleneck has shifted from parameter count to inference orchestration. This post explores how advanced techniques like RadixAttention, Chunked Prefills, and Deep Expert Parallelism are redefining the ROI of GPU clusters and creating a new standard for high-performance AI infrastructure.
Scaling Structural Bias - Pre-training Custom Qwen3 on TPU v6e
An end-to-end guide to orchestrating Custom Qwen3 pre-training on Google Cloud's Trillium TPUs. I dive into modifying the Qwen3 architecture for structured JSON outputs, leveraging XPK for orchestration, and serving the final artifacts with vLLM's high-performance openXLA backend.