What Is an AI Token? A Developer's Guide

Every API call to an LLM is measured in tokens. Not words, not characters, not sentences. Tokens. They determine what you pay, how much context fits in a single request, and where your application hits its limits. Understanding what is an AI token is foundational to controlling costs and writing efficient prompts when building on any major LLM provider.

This guide covers how tokenization works under the hood, how to count tokens programmatically, what tokens mean for your API bill, and how semantic caching can cut your token consumption.

From Text to Tokens: How Tokenization Works

Large language models do not process raw text. According to OpenAI's tokenizer documentation, models process text using tokens, which are common sequences of characters found in a set of text. The models learn to understand the statistical relationships between these tokens and excel at producing the next token in a sequence.

A practical rule of thumb from OpenAI's tokenizer page: one token generally corresponds to about 4 characters of common English text. This translates to roughly three-quarters of a word, so 100 tokens equals approximately 75 words.

Where Token Boundaries Fall

Tokenization is not intuitive. Common words often map to a single token, but less frequent words get split into multiple tokens. Here is how it plays out in practice:

• "hello" is typically one token
• "indistinguishable" becomes multiple tokens because the tokenizer splits less common words into subword pieces
• Whitespace and punctuation consume tokens too
• Code tends to tokenize less efficiently than prose because variable names, brackets, and syntax characters each take individual tokens
• Non-English text generally uses more tokens per word than English because tokenizer vocabularies are trained predominantly on English corpora

OpenAI's tokenizer page shows different encodings for different model generations: GPT-5.x and O1/O3 models use one encoding, while GPT-4 and GPT-3.5 (legacy) and GPT-3 (legacy) each use their own. You can experiment with these differences on OpenAI's interactive tokenizer to see exactly how your text splits.

Counting Tokens Before You Send a Request

Knowing your token count before making an API call lets you estimate costs, stay within context window limits, and make routing decisions.

Using tiktoken for Local Token Counting

OpenAI's tokenizer page recommends their tiktoken package for Python as a programmatic interface for tokenizing text. It runs locally, so there is no network latency and no API cost:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
tokens = enc.encode("The quick brown fox jumps over the lazy dog.")
print(len(tokens))  # number of tokens

# Get the encoding for a specific model:
enc = tiktoken.encoding_for_model("gpt-4o")

For JavaScript, OpenAI notes that the community-supported @dbdq/tiktoken package works with most GPT models.

Provider-Side Token Counting

Beyond local counting with tiktoken, major LLM providers offer server-side token counting endpoints. These are useful when you need an exact count that matches what the provider will bill, especially for requests that include images, tool definitions, or system prompts where local estimation gets tricky.

The pattern is similar across providers: send the same structured input you would send to the generation endpoint, and get back a token count without triggering actual inference.

How Tokens Drive API Costs

Every major LLM provider prices API access per million tokens, with separate rates for input (what you send) and output (what the model generates). Output tokens are always more expensive because they require the model to run inference step by step, generating one token at a time.

This pricing structure has three practical implications:

Token Budgeting in Practice

Here is a simple framework for estimating monthly token costs for any LLM feature:

1. Measure average tokens per request. Use tiktoken to count a representative sample of your prompts and measure output lengths from test runs.
2. Multiply by daily volume. A customer support bot handling 10,000 queries per day with 500 input tokens and 300 output tokens per query consumes 5 million input tokens and 3 million output tokens daily.
3. Apply your model's rate. Check your provider's pricing page and multiply. The difference between model tiers can turn a $500/month feature into a $3,000/month one on the same volume.
4. Factor in caching discounts. All major providers offer discounted rates for cached inputs (tokens the model has already processed in a recent request), typically at a steep discount to standard rates.

The math is straightforward. The hard part is step 1: most teams underestimate how many tokens their system prompts, tool definitions, and retrieved context actually consume.

How Semantic Caching Reduces Token Consumption

The most direct way to cut token costs is to avoid sending duplicate requests. Traditional caching (hash the prompt, store the response) only catches exact matches. Semantic caching goes further.

Consider two prompts: "What is the capital of France?" and "what's the capital of france." These are semantically identical, but a hash-based cache treats them as completely different queries because the strings differ. A semantic cache converts prompts into vector embeddings and compares their meaning, returning the stored response when similarity exceeds a threshold.

When a cached response serves the answer, you skip both input and output token charges entirely. This is different from provider-level prompt caching, which discounts repeated prefixes but still charges for the API call. Semantic caching intercepts the request before it reaches the provider at all.

You can test how this works with different prompts in the shim playground.

Choosing the Right Token Strategy

Token management is not a single decision. It is a stack of choices that compound:

1. Pick the right model tier. Match model capability to task complexity. Use frontier models for tasks that require deep reasoning. Route simpler tasks to smaller models.
2. Count before you send. Use tiktoken or your provider's count endpoint to validate prompt size before making API calls. This prevents context window overflows and surprise bills.
3. Minimize input tokens. Trim system prompts, compress retrieved context, and strip unnecessary formatting. Every token in your prompt is a token you pay for.
4. Cache at every layer. Use provider-level prompt caching for repeated prefixes and semantic caching for repeated queries. The two operate at different levels and stack.
5. Monitor token usage per endpoint. Aggregate token counts by feature, customer, or use case. Without per-endpoint visibility, you cannot identify which part of your application is driving costs.

Tokens are the unit of account for everything in the LLM economy. Knowing how they work, how to count them, and how to spend fewer of them is the difference between an AI feature that scales and one that gets pulled because the bill came in too high.

const openai = new OpenAI({
  baseURL: "https://api.getshim.tech/v1",
  apiKey: process.env.SHIM_API_KEY
})