APAC Inference Placement: Data Residency, Latency and Cost Across Mumbai, Singapore and Tokyo Regions

In my work with teams serving users across Asia Pacific, the most expensive inference decision is rarely the one on the model comparison slide. It is the one nobody put on a slide: which region the request lands in. I have watched a team spend two quarters shaving 15 percent off per-token cost through quantization and batching, then lose more than that in a single afternoon because their traffic was quietly hairpinning through a region two oceans from the user.

The pattern is easy to miss because APAC is not one place. Mumbai, Singapore and Tokyo sit thousands of kilometres apart, run under different data protection regimes, and have very different GPU supply. A model that feels fast in a Virginia demo can feel sluggish in Pune, and a deployment that is cheap in Singapore can be unlawful for a particular dataset in Mumbai.

So I cover something very narrow here. Where you run inference (region, GPU availability, network path) often moves latency and cost more than the choice between two comparable models. And residency rules can pick the region for you before cost ever gets a vote. Once you accept that, the useful question is which of three placement patterns fits your constraints: hosted regional endpoints, in-region cloud GPUs, or on-prem hardware.

Residency comes first, and it is not binary

Treat residency as a filter, not a preference. Before you compare anything, write down which data classes your inference path touches: prompts, retrieved documents, tool outputs, logs, embeddings and cached responses. Each can carry a different obligation.

A few anchors, with the usual caveat that I am an engineer and not your legal counsel (so please double fact check):

• India. The Digital Personal Data Protection Act, 2023 generally allows cross-border transfers except to countries the government restricts by notification (s.16), rather than imposing a blanket localization rule, but sector regulators layer on top. The Reserve Bank of India’s 2018 directive requires payment data to be stored only in India, while processing abroad is permitted under conditions (the data must be deleted abroad and brought back within a short window, per the RBI FAQs), so check the directive and its FAQs for your case. A bank or payments company in Mumbai often has a hard in-country requirement for some data and a softer one for the rest.
• Singapore. The Personal Data Protection Act focuses on accountability for transfers, meaning you must ensure comparable protection abroad. Financial and government workloads add their own expectations on top.
• Japan. The Act on the Protection of Personal Information requires specific consent or equivalent safeguards for transfers to third countries. Many Japanese enterprises add stricter contractual rules than the law itself.

The trap I see most is treating “the model call” as the only data flow. The prompt might stay in region while the observability pipeline ships full request logs to a US-hosted tracing vendor. Or the vector store is in Mumbai while the reranker is a hosted API in Singapore. An auditor reads the whole path, not your architecture diagram’s happy box.

Residency also has degrees. Some organizations need processing and storage in-country. Others only need storage in-country and accept transient processing elsewhere. Others need no cross-border transfer without a documented basis. These produce very different placement options, so pin down which one you actually have before you design.

Latency: physics sets the floor, the network path sets the rest

Light in fibre covers roughly 200 km per millisecond. The great-circle distances between the three cities give you a rough lower bound on round-trip time:

These are floors from distance alone. Real cable routes are longer and add switching, so measured round trips are typically meaningfully higher. Treat my numbers as estimates and measure your own path from your own users’ networks.

Why does this matter for inference, where generating a response takes seconds? Three reasons.

First, time to first token is what a user feels. If prefill takes 300 ms and you add an unnecessary 80 to 150 ms of network, you have made the product feel a third slower on the metric people notice.

Second, agentic and RAG pipelines are chatty. A single user request might trigger a retrieval call, a rerank, two model calls and a tool invocation. If any of those cross a region boundary, you pay the round trip once per hop, not once per request. Ten hops at 60 ms is 600 ms you never see in a model benchmark.

Third, streaming hides throughput but not setup. Token streaming makes a long generation feel smooth, which is good, but connection setup, TLS and queueing still land before the first byte.

Here is the picture I draw on whiteboards when a team says “we’ll just use the Singapore endpoint for everyone”:

The dotted lines are the cost of centralizing. They are sometimes worth paying, and sometimes not.

Cost: the per-token price is the smaller lever

Regional price differences on the same instance type exist and are real, but the bigger swings come from elsewhere:

• GPU availability. The cheapest list price is irrelevant if you cannot get the instance. Capacity of the newest accelerators is uneven across regions and across time, and quota approvals can take longer than engineering. Singapore has historically been a deep pool for regional workloads; Mumbai and Tokyo can be tighter for specific GPU types. Check current availability for the exact instance family in each region, because this changes and I will not pretend a table in an article stays accurate.
• Utilization. A GPU that sits at 25 percent utilization costs four times what the same GPU at 100 percent costs per useful token. Spreading small traffic across three regions can leave three half-empty fleets, where one pooled region would run hot.
• Egress and cross-region transfer. Moving embeddings, documents or long contexts between regions is billed, and for retrieval-heavy workloads it can rival compute.
• Idle reserve. Meeting a latency target in every region means holding warm capacity in every region.

This is where the cost architecture tradeoff gets real: pooling saves money and spends latency, while distributing spends money and buys latency. Neither is correct. The correct answer depends on how much the latency is worth in your product.

Three placement patterns and when each wins

Hosted regional endpoints

You call a managed model API that offers in-region processing, or a regional variant of a cloud provider’s model service. You do not manage GPUs.

It wins when: volume is low to moderate, your team has little capacity to run serving infrastructure, residency requires in-region processing but not dedicated hardware, and you want access to frontier-class models you cannot self-host.

Watch for: model availability varies by region and by model version, and the newest models often reach some regions later than others. Some “regional” offerings route requests across a group of regions for capacity, which may or may not satisfy your residency reading. Read the data processing terms, not the marketing page, and verify the current region list for the specific model before committing. You also accept the provider’s rate limits and pricing, and per-token costs stay linear as volume grows.

In-region cloud GPUs

You rent instances in Mumbai, Singapore or Tokyo and run an open-weights or licensed model with your own serving stack.

It wins when: sustained volume keeps GPUs highly utilized, you need control over latency tails, batching, quantization or fine-tuned weights, and residency demands that data never leave a named country but you are fine with a cloud tenant.

Watch for: you now own capacity planning, autoscaling, model upgrades, security patching and evaluation. Cold starts are painful with large weights, so scaling to zero is often a fantasy. Quotas and accelerator scarcity can leave you unable to scale exactly when demand spikes. And you are responsible for the safety and quality evaluation that a managed provider would otherwise bundle in.

On-prem or colocated hardware

You buy or lease GPUs and place them in your own facility or a colocation in-country.

It wins when: residency or sovereignty rules exclude public cloud tenancy for that data, volume is high and steady enough to amortize hardware, and you have staff who can run it. Some regulated finance, healthcare and public sector workloads land here honestly, not out of nostalgia.

Watch for: lead times on hardware, power and cooling constraints, refresh cycles as accelerators change, and a long period before you reach break-even. If your peak is three times your average, you bought for the peak and pay for it all year. It also locks the capability ceiling to what you can run on that hardware.

A worked example (illustrative numbers)

Take a fictional insurer with customers in India, Singapore and Japan, running a claims assistant. The numbers below are made up to show the reasoning, not measured.

• Indian claims documents must be processed in India. Singapore and Japanese data have no strict in-country processing rule but have contractual limits on moving data outside the region.
• Indian volume is high and steady, perhaps 70 percent of traffic. Singapore and Japan are each small and spiky.

A sensible placement: run Indian traffic on in-region Mumbai GPUs, because the residency filter requires it and volume keeps them busy. For Singapore, use a hosted regional endpoint, since volume is too low to justify dedicated capacity. For Japan, the same, provided the provider’s Tokyo processing terms match the contract; if they do not, small in-region Tokyo GPUs or a Japanese provider become the fallback.

The tempting alternative is one Singapore deployment for everyone. It has the best capacity and simplest operations. It also breaks the India rule, adds roughly 40 ms or more to every Mumbai round trip, and makes every retrieval hop in the claims pipeline more expensive. The simpler diagram is the costlier system.

How to decide without making this a hard and fast rule

I use a short sequence, in this order:

1. List data flows and classify them. Prompts, context, logs, embeddings, caches. Mark each as in-country processing, in-country storage, or transfer allowed with safeguards.
2. Strike regions that fail the filter. Do this before pricing anything.
3. Measure the network path from real user locations, including your retrieval and tool dependencies. Add the hops, not just the model call.
4. Verify GPU and model availability per region for the exact versions you need, and get quota in writing before you design around it.
5. Compute cost per useful token at expected utilization, not list price. Include egress and idle reserve.
6. Pick per workload, not per company. A single organization can sensibly run all three patterns at once.

The decision flips at predictable points. Hosted endpoints lose ground as volume rises and as you need control the provider will not give. In-region GPUs lose ground when your team cannot staff the on-call burden or your traffic is too small to keep hardware busy. On-prem loses ground when demand is volatile or you cannot absorb the lead time, and wins when no other option satisfies the rule.

So what’s the key message then?

For the engineering lead: build the region into your evaluation harness. Benchmark time to first token and tail latency from each user geography against each candidate placement, not just quality on a fixed prompt set. A model that scores two points higher but sits 100 ms further away is not an upgrade for a voice or chat product.

For the business owner: ask for three numbers per region before signing off on a model change: cost per thousand requests at realistic utilization, p95 time to first token from where your customers actually are, and the residency basis for each data flow. If a vendor or team cannot produce those, the comparison you were shown is incomplete, however good the model looks.

Frontier hosted models, open-weights on rented GPUs and on-prem clusters are all legitimate answers here. They answer different questions: access to capability, control over the stack, and control over the perimeter. Placement is the part that tells you which question you are actually asking.