1 November 2026 • 6 min read
LLM APIs bill by the token, not by the word. A token is a piece of text that the model's tokenizer cuts out of your input. If a language is cut into more pieces for the same sentence, the same message costs more, uses up more of the context window and takes longer to process. For a product serving Hindi and Tamil users, that is a cost line worth knowing before you set a per-user budget. This page checks the claim against published measurements, shows what they say, and tells you what is out of date.
Yes, with a large difference between the two languages and between tokenizers. The cleanest published numbers come from a model card by Sarvam AI, an Indian model maker, so they are a vendor's own measurement and should be read that way. The card reports "fertility", which is the average number of tokens per word, for four tokenizers.
| Language | GPT-4o | Gemma-2 | Llama-3.1 | Sarvam-2B |
|---|---|---|---|---|
| English | 1.23 | 1.23 | 1.24 | 1.43 |
| Hindi | 1.65 | 1.96 | 2.67 | 1.4 |
| Tamil | 3.17 | 4.19 | 12.39 | 2.17 |
Tokens per word, from the Sarvam-2B model card (August 2024). Lower is cheaper.
Divide each Indian-language figure by the English figure for the same tokenizer and you get the extra cost of the same number of words. These ratios are our arithmetic:
So Hindi is a modest premium on GPT-4o's tokenizer, and Tamil is a large one. Sarvam's own model flips the picture: its Hindi figure of 1.4 is lower than its English figure of 1.43, because the tokenizer was built for Indian languages. In an earlier post on its first model, Sarvam said existing multilingual models often need 4 to 8 tokens per word for Indic scripts, against 1.4 for English. That was Sarvam's description of the models it compared against. Its own table puts Hindi on the other tokenizers between 1.65 and 2.67, below that range, though Tamil on Llama-3.1 is above it.
A tokenizer learns its vocabulary from a training text. Common character sequences in that text become single tokens; rare ones are split into smaller pieces. If the text is mostly English, English words get whole-word tokens, while Tamil words are broken into many small fragments.
A NeurIPS 2023 paper by Petrov and co-authors measured this across languages. It found that the same text translated into different languages can differ in tokenized length by up to 15 times in some cases, and that the disparity persists even for tokenizers built for multilingual use. The authors link it to the corpus: tokenizers are heavily influenced by the biases of their source text. A second 2023 paper, by Ahia and co-authors, analysed OpenAI's API on 22 languages and concluded that speakers of many languages are overcharged while getting poorer results.
Both papers used the tokenizers of their day, which are older than the models you will call in 2026, so we do not quote their language-by-language figures.
Three effects, all from the same cause.
The size of each effect depends on your model. Work it out from your own traffic, not from a table in a blog post.
The Sarvam card dates from August 2024 and covers four tokenizers that were current then. Vendors change tokenizers between model generations. Anthropic's token-counting guide says Claude 4.7 and later models use a newer tokenizer, and that the same input text produces about 30 percent more tokens than on earlier models. That statement is about all text, not Indian languages in particular, but it shows that counts move between generations, and that you should recount against the exact model you plan to call. We found no vendor-published Hindi or Tamil figure for the 2026 models, so the premium for those models is unmeasured here.
For a cross-vendor view of prices, see the LLM API comparison for India.
On GPT-4o's tokenizer, per Sarvam's 2024 table, 1.65 tokens per Hindi word against 1.23 for English, about 1.34 times. Other tokenizers in the same table range up to 2.67 for Hindi.
On every general-purpose tokenizer in that table, yes: Tamil is 3.17 tokens per word on GPT-4o and up to 12.39 on Llama-3.1, against 1.65 and 2.67 for Hindi.
We could not confirm it. The published Indian-language figures date from 2024 and earlier. Count tokens on the model you plan to use.
No. Vendors price per token. The per-word figure is a way to compare languages; your invoice follows the token count.
Nine stories, every morning at 09:15 IST.
Get the brief free →