Serverless Inference: Conquering the 5-Second Cold Start

A serverless inference endpoint charges you per request. You do not pay for idle capacity. You do not manage servers. You scale from zero to thousands of concurrent requests automatically. The economics sound perfect until the first cold start hits.

Five seconds. That is how long a user waits for the model to load into GPU memory, initialize the runtime, allocate the KV cache, and produce the first token. In interactive applications, this is unacceptable. Users abandon pages that take more than two seconds to respond. Five seconds is a complete failure.

And cold starts are not a one-time problem. They happen every time your request count dips below your provisioned capacity and the cloud provider reclaims the GPU. You might scale to two hundred concurrent requests during the morning rush, drop to zero overnight, then wake up to two hundred cold starts in the first thirty seconds of the next business day.

The fundamental assumption that a single GPU is the right unit of compute has been invalidated by rack-scale design. When you need the GPU to actually perform the work rather than just passively wait, the serverless paradigm needs rethinking entirely.

The Anatomy of a Cold Start

To solve a problem, you need to understand it. Here is what happens during a cold start, step by step.

The cloud provider receives your request. It finds no available GPU workers. It allocates a new container and schedules it on a GPU machine. The container pulls the model weights from a remote storage system. Loading a 7-billion parameter model from distributed storage into GPU memory takes roughly two seconds. Even fast cloud internal networks struggle to match the bandwidth of a direct memory-to-GPU transfer.

The runtime initializes. Python imports. CUDA context creation. Framework setup. This step typically takes one to two seconds.

The model weights are loaded. The KV cache is allocated. Depending on your context window size, this can take an additional half-second to two seconds. Large context windows mean larger KV cache allocations. A 128K context window consumes significantly more memory than an 8K window.

The model produces the first token. This is the actual compute phase. For a 7B model on a modern GPU, the first token takes approximately 30 to 50 milliseconds. This step is fast. It is not the bottleneck.

The total is a sum of these steps. Two to eight seconds depending on model size, context window, provider infrastructure, and warm cache state. The range is wide because every variable matters.

Persistent Workers: The Proven Fix

The simplest solution is keeping model workers alive between requests. This is technically not serverless because you are paying for idle capacity. But it solves the cold start problem completely.

Set up a pool of persistent GPU workers that stay loaded between requests. When a request arrives, it lands on an existing worker. The first token latency drops from five seconds to fifty milliseconds. The cost is paying for the GPU even when it is idle.

The economics depend on your concurrency profile. A single A100 running 24/7 costs approximately $2,500 per month. If your average request rate generates$3,000 in per-request revenue, paying $2,500 is clearly worthwhile. You eliminate cold starts and get interactive latency consistently.

If your request rate averages ten requests per hour, that same GPU costs twenty-seven dollars per request in dedicated capacity. Pure serverless at $0.60 per request would be cheaper despite the cold start latency.

The break-even point is roughly where the number of active hours per day multiplied by the average request rate exceeds a certain threshold. For a 7B model on an A100, that threshold is approximately two hundred requests per hour sustained. Below that, you are paying more for persistence than you save from eliminating cold starts. Above that, persistence pays for itself.

Speculative Warmers: The Middle Ground

A more sophisticated approach uses speculative warmers. When request volume drops below a threshold, you do not shut down all workers. You keep a single worker alive at a reduced capacity state. This worker stays loaded but operates at minimal throughput.

When traffic resumes, the warmed worker handles the first batch of requests immediately. The cold start only affects requests that exceed the warmed worker’s capacity. As the warm worker scales up or additional workers provision, the cold start rate decreases.

This approach reduces cold starts by approximately sixty to seventy percent compared to shutting everything down completely. The persistent cost is lower because you only keep one worker warm, not the full pool.

The implementation requires careful monitoring and threshold tuning. Too aggressive a scale-down and you still see cold starts. Too conservative and you waste GPU time keeping workers warm. The optimal threshold depends on your traffic pattern. Steady traffic with predictable daily patterns is easier to optimize. Sporadic traffic spikes are much harder.

Hybrid Architecture: The Production Pattern

The pattern that works best in production combines always-on capacity for baseline traffic with serverless capacity for bursts.

You maintain a smaller persistent model cluster that handles your minimum expected load. This cluster is sized for your normal usage. If you expect two hundred requests per hour, you provision enough workers to handle that without cold starts.

When traffic exceeds this baseline, additional requests route to serverless capacity. These requests might experience cold starts, but only the surge volume. The baseline traffic always gets sub-100ms latency.

This approach optimizes cost and performance simultaneously. You pay persistent only for the load you expect. You pay serverless only for the excess. The user experience for the majority of requests is excellent. A small fraction of surge requests experience cold starts, which is more acceptable than every request experiencing a cold start.

I worked on a system that used this hybrid pattern. Baseline twelve requests per minute handled by persistent workers. Peak traffic up to two thousand concurrent requests handled by serverless capacity. The cold start rate ranged from zero percent during normal hours to approximately fifteen percent during unexpected viral traffic spikes. The average latency was ninety milliseconds. The cost was forty percent lower than running persistent capacity for peak load.

The critical insight is this: cold starts are an acceptable cost when they are limited to edge cases. Every user experiencing a cold start is a product failure. A minority of users experiencing them during unexpected traffic spikes is an acceptable engineering tradeoff. And the choice of inference framework matters more than most realize — choosing JAX over PyTorch for your model stack introduces faster compilation and loading times that directly attack the cold start problem from a different angle.

Infrastructure Hacks That Reduce Cold Start Time

Several optimizations specifically target cold start latency.

Use quantized model weights. A 7B model in INT8 is half the size of the same model in FP16. Half the data means approximately one second of transfer time savings. The quality loss is minimal for most inference workloads. The cold start improvement is measurable.

Use model caching at the container level. Some inference providers cache frequently used models in local storage within the container infrastructure. When a new container needs the same model, it pulls from local cache instead of distributed storage. This can reduce loading time by thirty to fifty percent.

Pre-warm containers during predicted traffic spikes. If your traffic pattern is predictable, schedule container warm-up before the peak. The model loads into memory during a low-traffic window so it is ready when users arrive. This prevents cold starts during your most valuable traffic periods.

Choose inference frameworks optimized for fast loading. Some frameworks load models faster than others. vLLM’s PagedAttention integration includes optimizations that reduce memory allocation time. TGI has faster model loading pipelines. SGLang supports efficient tensor parallelism that speeds up initial GPU allocation.

Use smaller models where quality permits. A 3B model loads significantly faster than a 7B model. If your use case does not require the larger model’s capabilities, the smaller model gives you both quality and speed. The model size directly scales with cold start time. All of these optimizations compound into the broader economics of inference — see The Efficiency Moat for the full trade-off analysis.

What This Means for Serverless Economics

When people talk about serverless inference economics, they assume cold starts do not exist. The per-request pricing looks attractive. You pay nothing for idle time. You only pay for what you use.

But if your users experience five-second cold starts, your conversion rate drops. Your engagement metrics collapse. Your customer satisfaction scores hit rock bottom. The free idle capacity is worth nothing if nobody uses the service because the response time is terrible.

The hybrid architecture addresses this directly. You pay slightly more than pure serverless but dramatically less than pure persistent capacity. You get good latency for most requests and acceptable behavior for the minority. The economics work. The user experience works. Both objectives are satisfied.

This is the pattern that enterprises actually deploy in production. The pure serverless approach is academically appealing but practically limited. The pure persistent approach is operationally simple but economically wasteful for variable workloads. The hybrid approach captures the benefits of both and accepts the tradeoffs of each.

For companies evaluating serverless inference, the real question is not whether cold starts exist. Every infrastructure choice has tradeoffs. The question is whether your traffic pattern makes cold starts acceptable and which approach minimizes their impact while controlling costs. The hybrid architecture works for most companies. The specific baseline and burst sizing should be tuned to your actual traffic patterns.

Measuring Cold Start Effectiveness

Track the cold start rate as a production metric. Calculate what percentage of requests experience latency above a defined threshold, typically two seconds for serverless inference. This is your cold start rate.

Set up monitoring that distinguishes between warm requests and cold requests. The metric tells you whether your strategy is working. If the cold start rate is above ten percent, your scale-down thresholds are too aggressive or your persistent capacity is undersized. If the cold start rate is below five percent, you might be over-provisioning persistent capacity and wasting money.

The optimal range is two to five percent cold start rate. This keeps the user experience acceptable while maintaining cost efficiency. Adjust your persistent capacity and warmers until you land in this range.

FAQ

What is a reasonable cold start time for serverless inference?

Two seconds is acceptable. Five seconds destroys user experience. Eight seconds is a complete failure for interactive applications.

Can I eliminate cold starts entirely?

Not with true serverless architecture. You can minimize them through warmers and persistence. The hybrid approach reduces cold starts to an acceptable edge case rather than eliminating them.

How much does a persistent GPU cost per month?

Single A100 persistent capacity costs approximately $2,500 per month on major cloud providers. Smaller GPUs like L4 instances cost approximately$1,000 to $1,500 per month.

Does quantization reduce cold start time?

Yes. INT8 quantization reduces model size by approximately fifty percent, saving roughly one second of loading time on most hardware. Quality loss is minimal for inference.

Which framework loads models fastest?

vLLM generally has the fastest model loading times due to PagedAttention optimizations. TGI and SGLang are competitive but vary by model type and configuration.