A token is the basic unit a language model uses to process text — it can be a whole word, part of a longer word, or a punctuation mark — and it underlies how AI models measure cost and context limits.

Key Points

  • Text is split into tokens before being processed by a language model, through a process called tokenization.
  • In English, a token represents on average roughly 0.75 words; in languages with more complex morphology, the ratio can vary.
  • Usage cost for many AI models is calculated based on the number of tokens processed, both input and output.
  • A model context window is measured in tokens, not in words or characters.

Practical Example

A simple sentence may split into 8-10 tokens, while a long technical word or a rare term can split into more tokens than you might expect at first glance.

Why It Matters

Understanding tokenization helps estimate the cost of using AI APIs and plan content so it fits within context window limits.

FAQ

Is a token equal to a character?

No, a token is usually longer than a character and can represent a whole word or part of a word.

Why does token count matter for cost?

Most AI APIs charge based on the number of tokens sent and generated, not the number of words.

Related Terms

Sources

← All terms in GEO & AEO Full glossary