Indic Language Tokenization: Why Hindi and Tamil Prompts Cost More and How to Measure Quality and Cost Per Language

In my work with teams shipping assistants for Indian users, the budget conversation usually goes the same way. Someone prices the product using English test prompts, the numbers look fine, and the launch goes out in Hindi, Tamil and Bengali as well. A month later the invoice and the latency graphs tell a different story, and nobody can say why, because the same feature, the same model and the same code are behind both.

The reason is mundane. Language models do not read characters or words. They read tokens, and the tokenizer that turns text into tokens was fitted to some corpus, usually one dominated by English and a handful of European languages. Text in a script the tokenizer saw rarely gets chopped into smaller and smaller pieces, sometimes down to individual bytes. More pieces means more billable units, more sequential decoding steps and less room left in the context window.

I am not going to give you a single multiplier, because there isn’t one. It depends on the tokenizer, the script, the domain of the text and even whether you write in formal or colloquial register. What I can give you is why the gap exists, what the research says about it, and a measurement method you can run this week on your own traffic.

Why a Hindi sentence costs more than its English twin

A tokenizer is a vocabulary of fragments, built by a procedure such as byte-pair encoding that repeatedly merges the most frequent adjacent pieces in the training text. Frequent strings become single tokens. Rare strings stay in pieces. If English made up most of the corpus, then “the”, “ing” and “tion” each earn a slot in the vocabulary, while a Devanagari conjunct or a Tamil syllable-plus-vowel-sign combination may not.

Indic scripts add a second wrinkle. They are abugidas: a consonant carries an inherent vowel, and vowel signs, conjunct-forming marks and the virama attach to it. In Unicode a single visible syllable can be several code points. In UTF-8 each of those code points takes three bytes. When a tokenizer has no merge for a rarely seen syllable, it falls back toward code points or raw bytes, and one syllable on the screen becomes several tokens in the bill.

Two research papers are worth reading if you want the evidence rather than my summary. Petrov and colleagues, in “Language Model Tokenizers Introduce Unfairness Between Languages” (arXiv 2305.15425), measured how the number of tokens for equivalent text varies widely across languages and tokenizers, and argued that this turns into unequal cost, latency and usable context. Ahia and colleagues, in “Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models” (arXiv 2305.13707), looked at the same problem from the pricing side and found that speakers of some languages pay more for the same service. Both predate the current generation of models, and several vendors have since widened their vocabularies, so treat their numbers as history and their method as the useful part.

That last point matters. Newer tokenizers with larger, more multilingual vocabularies have narrowed the gap for some languages and some models. Models trained with Indian languages as a priority often ship tokenizers designed for them. Others have not changed much. This is exactly why I will not hand you a ratio: any figure I printed would be stale or wrong for the model you actually call.

Three costs, not one

When people say “Hindi costs more”, they usually mean the invoice. There are three separate effects, and they hit different owners in the company.

Money. Hosted APIs bill per input and output token. If the same instruction takes more tokens in Tamil, the price per request rises roughly in proportion, for both the prompt and the response. The system prompt, the retrieved documents and the conversation history all inflate together, which is why the effect is bigger in retrieval-heavy products than in short chat.

Time. Decoding is sequential. A model emits one token at a time, so an answer that needs more tokens takes longer to finish streaming, even if time to first token is unchanged. A user reading a Telugu answer that arrives noticeably slower than the English version experiences a worse product, and that is a quality issue as much as a cost one.

Room. A context window is counted in tokens. If your retrieval pipeline stuffs eight passages into the prompt and each passage is longer in tokens in Hindi, then you either truncate sooner or fit fewer passages. The model sees less evidence per query in one language than in another, and answer quality can drop for reasons that have nothing to do with the model’s reasoning ability.

For self-hosted deployments the same three effects show up differently. You do not get an invoice per token, but you pay for GPU seconds, and more tokens per answer means fewer concurrent users per GPU and a longer KV cache per session. The economics I walked through in the piece on inference cost architecture apply here with one more variable: the token count is a function of language, so your capacity plan needs a language mix, not a single average.

The executive version

If you run the business, the point is simple. Your per-user cost, response time and effective product quality are not uniform across your user base. Without measurement, you will underprice the Indic-language tier, overspend on the model for those users, or ship a slower and less accurate experience to the customers you most want to win. Any of those is a decision you can make deliberately, but only if the numbers are in front of you.

The reverse mistake is also real. Do not assume the answer is to switch every Indic request to a specialised model. A model trained for Indian languages may handle tokenization and local idiom better, and it may also have a smaller capability ceiling on tool use, long-context reasoning or code. A large hosted model may tokenize Hindi inefficiently and still produce the best answer. An open-weights model you host may give you control over the tokenizer and the cost curve, at the price of running the evaluation and serving yourself. Which one wins depends on your languages, your task and your team. That is the whole argument for measuring.

A measurement method you can run this week

The method has four steps. None needs special infrastructure.

Step 1: Build a parallel sample from real traffic

Pull a few hundred representative prompts per language you serve, with consent and privacy handling that fits your policy. Include the awkward cases: code-mixed Hinglish, romanised Hindi typed in Latin script, mixed numerals, product names in English inside a Tamil sentence. Real users do all of these, and tokenizers treat each differently.

For a clean comparison of token efficiency, also keep a small parallel set: the same content professionally translated into each language. Parallel text lets you compute a ratio against English without the confound of different content.

Step 2: Count tokens with each model’s own tokenizer

This step is cheap and surprisingly revealing. Use the tokenizer that ships with each candidate model, not a stand-in. For open-weights models this is a local library call. For hosted models, use the vendor’s published tokenizer or token-counting endpoint, and cross-check against the usage field in real API responses, since that is what you are billed on.

from collections import defaultdict
import statistics

def tokens_per_char(texts, count_tokens):
    # count_tokens: a function str -> int for ONE model's tokenizer
    ratios = [count_tokens(t) / max(len(t), 1) for t in texts]
    return statistics.median(ratios)

def vs_english(parallel, count_tokens):
    # parallel: {"en": [...], "hi": [...], "ta": [...]} same content, same order
    base = sum(count_tokens(t) for t in parallel["en"])
    return {lang: sum(count_tokens(t) for t in texts) / base
            for lang, texts in parallel.items()}

Pin whichever tokenizer library and model revision you test with and record them next to the result. A tokenizer ratio is only meaningful for a named version, and vendors do change them between model releases.

Report the ratio against English from the parallel set, and report the median tokens per request from your real sample. They answer different questions: the first isolates the tokenizer, the second reflects your actual prompts, including system prompt overhead that may be in English anyway.

Step 3: Score quality in the language, not in translation

This is where teams cut corners. Translating the Hindi output back into English and grading it there hides the failures you care about: unnatural phrasing, wrong honorific level, mangled script, a drift into English mid-answer. Grade in the language, using a mix of methods.

• A small human-labelled set by native speakers for the tasks that matter, scored on correctness, fluency and register.
• Task-specific automatic checks where possible: exact-match for extraction, schema validity for structured output, retrieval hit rate for RAG.
• Public benchmarks as a sanity check only. Suites such as IndicXTREME and IndicGenBench exist for Indic languages, and they are useful for ruling out a model that is plainly weak, but they will not tell you how a model handles your support tickets.
• If you use a model as a judge, validate it against your human labels per language first. A judge that is reliable in English may be noisy in Odia.

Step 4: Combine into cost and latency per successful task

Now put the numbers together. For each model and language, compute:

• Median input and output tokens per task.
• Price per task from the actual rate card or your GPU cost per second.
• Time to complete the answer, not only time to first token.
• Success rate from step 3.
• Cost per successful task: price per task divided by success rate.

That last figure is the one to compare. A cheaper model that fails a quarter more often in Tamil may cost more per resolved ticket once you count retries and human escalations. A pricier model with a more efficient tokenizer and higher quality may cost less. Neither outcome is visible if you look at the rate card alone.

A worked example, in shape

Imagine a support assistant that answers in English, Hindi and Tamil, with three candidates: a large hosted model, an open-weights model you serve yourself, and a model trained with Indian languages in mind. I am deliberately not putting numbers in the table, because they would be invented.

What usually comes out of a table like this is not a single champion. You see a model that is excellent in Hindi and merely adequate in Tamil. You see one with a wasteful tokenizer but the best answers, which pays off for high-value queries and not for FAQ lookups. You see that your system prompt, written in English, is a fixed cost that dominates short Hindi questions. Those are routing decisions, and the routing logic is the practical output: send each language and task type to the model whose cost per successful task is best, and keep the evaluation running so the routing can change when a vendor ships a new tokenizer.

Practical levers once you have numbers

You will rarely fix this by changing the tokenizer, but there are several things you control.

Trim the fixed overhead. System prompts, tool schemas and few-shot examples are paid on every request. Keep them in whichever language is cheapest for the model while still producing correct output in the target language, and test that choice instead of assuming it. Some models follow English instructions and answer well in Hindi; others do better with instructions in the user’s language.

Budget context by tokens, not by passages. If your retriever packs a fixed number of chunks, make the chunk budget a token budget per language so Tamil users do not silently get less evidence.

Cache what repeats. Prompt caching, where your provider or serving stack supports it, reduces the cost of long shared prefixes regardless of language, which softens the penalty on retrieval-heavy flows.

Set output limits per language. A max-tokens limit that is generous in English can truncate a Malayalam answer mid-sentence. Set limits from measured lengths.

Consider the serving stack. If you self-host, check whether your inference engine’s tokenizer and chat template match the model’s reference implementation. Mismatches quietly degrade multilingual output, and they are easy to miss because the English tests still pass.

Re-measure on every model change. Tokenizers change between versions. Make the four-step run a job that executes whenever you swap or upgrade a model.

Where the evidence is thin

Be honest about what this approach cannot tell you. Token counts are exact; quality scores are estimates built on small samples and imperfect raters. Rater agreement among native speakers varies by task, and dialect matters: Hindi in Delhi, Lucknow and Mumbai differs in vocabulary and register. A few hundred examples will separate a clearly poor model from a clearly good one, but they will not rank two close candidates with confidence. Where the call is close, run a larger sample or a live A/B test with real users and watch resolution rate and escalation rate.

Also, the research I cited measured earlier tokenizers. The direction of the finding, that tokenization efficiency is uneven across scripts and affects cost, has held up as a method of analysis. The size of the gap for any current model is something you have to check yourself.

What to do on Monday

Pick your two highest-traffic non-English languages. Take a hundred real prompts in each and a parallel English set. Count tokens with each candidate model’s tokenizer, then put the ratios next to your current spend by language. Even before any quality work, that one table will tell you whether language is a rounding error or a line item. If it is a line item, build the full evaluation and let cost per successful task, per language, drive the model choice. The right answer might be one model everywhere, or three models behind a router. Either is defensible once it comes from your own data.