snori
snori
Log in
DE EN

The AI glossary AI basics

Token

A token is the smallest unit of text a language model processes — often a word fragment, a short whole word, or a punctuation mark. A German sentence typically contains more tokens than words, since compound or rare words get split into several tokens. Tokens are also the billing unit for many AI providers.

Source: OpenAI — Tokenizer

← Back to the glossary