00 · IN THREE MINUTES
The answer in three steps
- 1A token can be a whole word, part of a word, punctuation or even a byte sequence.
- 2The converts text to integer IDs; the model converts those IDs to vectors.
- 3Token counts affect context length, cost and performance, but they are not a direct measure of meaning.
01 · TEXT FIRST BECOMES PIECES
Text first becomes pieces
A tokenizer uses a fixed vocabulary of recurring character or byte sequences. Common strings may fit in one token while unfamiliar names, spelling variants or some scripts are divided into several pieces.
02 · PIECES BECOME NUMBERS
Pieces become numbers
Each vocabulary entry has an integer ID. The model looks up a learned vector for that ID and combines it with position and context; the number itself carries no natural meaning outside that tokenizer.
Text first becomes pieces
A tokenizer uses a fixed vocabulary of recurring character or byte sequences. Common strings may fit in one token while unfamiliar names, spelling variants or some scripts are divided into several pieces.
text → piecesPieces become numbers
Each vocabulary entry has an integer ID. The model looks up a learned vector for that ID and combines it with position and context; the number itself carries no natural meaning outside that tokenizer.
pieces → IDsBoundaries are engineered
Subword methods reduce the need for a separate entry for every possible word. The trade-off is uneven segmentation: the same idea can consume different numbers of tokens across languages, spellings and formatting.
IDs → vectorsThe model predicts another token
After processing the context, the network produces a probability distribution over its vocabulary. Sampling or selecting from that distribution yields the next token, which is appended and processed again.
probability → token03 · BOUNDARIES ARE ENGINEERED
Boundaries are engineered
Subword methods reduce the need for a separate entry for every possible word. The trade-off is uneven segmentation: the same idea can consume different numbers of tokens across languages, spellings and formatting.
04 · THE MODEL PREDICTS ANOTHER TOKEN
The model predicts another token
After processing the context, the network produces a probability distribution over its vocabulary. Sampling or selecting from that distribution yields the next token, which is appended and processed again.
05 · TOKENS ARE NOT THOUGHTS
Tokens are not thoughts
A token is an interface between text and computation. It helps explain context windows and billing, but one token is not one concept, and a longer token sequence is not automatically a deeper argument.
06 · SOURCES AND EVIDENCE
Sources and evidence
Claims are linked to foundational papers, standards or the primary study behind the update.
- 01Neural Machine Translation of Rare Words with Subword UnitsPRIMARY STUDY ↗
Supports a defined mechanism, measurement or evidence boundary in this article.
- 02SentencePiece: A simple and language independent subword tokenizerPRIMARY STUDY ↗
Supports a defined mechanism, measurement or evidence boundary in this article.