I. What is a Token?

In the field of large language models (LLMs), a token is the smallest unit for text processing—much like the basic brick used to build a grand structure. Think of language as a complex skyscraper: tokens are the individual, unique bricks that make up this building. They come in various forms:

  • Complete words: In language systems like English, common words are often treated as single tokens. For example, words such as “apple” and “book” stand alone, each carrying a clear and distinct meaning. In linguistic expression, they act like sturdy small bricks, holding basic semantic information.
  • Word fragments: For more complex words, a splitting strategy is used. Take “hesitate” as an example—under specific processing methods, it may be split into “hesit” and “ate”. This splitting is not random; its purpose is to help the model better learn the structural rules of words and the semantic relationships within them. For instance, common affixes like “un-” and “-tion” become easier to understand through splitting. This lets the model grasp how these affixes influence a word’s overall meaning—similar to figuring out how bricks of different shapes fit together in construction.
  • Punctuation marks: Punctuation is indispensable in linguistic expression. It acts like connecting parts in a building, giving text rhythm and logic. In LLMs, each punctuation mark (e.g., “.”, “,”, “!”, “?”) counts as a separate token. Take the sentence “I love reading books.” as an example: the period “.” is an independent token. It helps the model recognize the end of a sentence and reflects the logical pause of a complete statement.
  • Spaces: In some LLM setups, spaces are also categorized as tokens. Spaces themselves don’t have any actual meaning. But they play a key role in text structure. Their job is to separate different words and phrases. This is like gaps in a building that distinguish linguistic units. For example, take the sentence “I like apples”. Spaces here clearly separate “I”, “like”, and “apples”—the core elements. This makes it easier for the model to process the text later.

Computers cannot directly understand human natural language; their “thinking” relies on numerical operations. Therefore, LLMs need an effective way to convert human language into a format computers can process—and tokenization is the key step to make this happen.

When a text is input into an LLM, the model does not process the entire text directly. First, it performs tokenization, splitting the text into individual tokens. For example, if the input text is “Artificial intelligence drives technological development”, the model will split it into tokens like “Artificial”, “intelligence”, “drives”, “technological”, and “development”.

These tokens are then converted into numerical IDs. For instance, “Artificial” might be assigned ID 1001, “intelligence” ID 1002, and so on. These numerical IDs become the actual data the model operates on—similar to bricks sorted by specific numbers in a construction worker’s hands. Finally, the model feeds these numerical IDs into a neural network for in-depth computation and processing. This allows the model to understand the text and complete subsequent generation tasks.

II. The Important Role of Tokens in LLMs

(I) Core Role as Input Units

When a user inputs text into an LLM, the model’s first step is to convert this text into tokens. Take the input sentence “What will the weather be like tomorrow, and is it suitable for going out?” as an example. The model may split it into tokens such as “What”, “will”, “the”, “weather”, “be”, “like”, “tomorrow”, “,”, “and”, “is”, “it”, “suitable”, “for”, “going”, “out”, “?”.

Next, the model converts these tokens into vectors. A vector is a mathematical representation that assigns each token a unique position and set of features in a high-dimensional space. This enables the model to perform complex calculations on these vectors via a neural network and output corresponding results.

In an intelligent Q&A scenario, for example, the model generates answers about the weather and outdoor suitability by analyzing these token vectors. It can be said that tokens, as input units, form the first “gateway” for LLMs to understand user input. Their accurate splitting and conversion lay the foundation for subsequent complex computations and intelligent responses.

(II) Significant Impact on Computational Costs

There is a direct, close relationship between an LLM’s required computation and the number of tokens in the text. Generally, the more tokens a text has, the longer the model takes to process it and the more computing power it consumes.

For example: The simple greeting “Hello” contains only 1 token, so the model spends relatively little time and power processing it. In contrast, a more complex word like “Unbelievable” may split into 3 tokens under specific rules, requiring more computational resources.

Consider a longer English text: “Today’s weather is exceptionally sunny, making it perfect for going out for a walk and enjoying the beautiful outdoor time”. After tokenization, it will produce many tokens. Compared to short texts, processing such long, complex texts significantly increases the model’s computational load.

This is like building a small house versus a large palace: the number of building materials (tokens) differs, leading to huge differences in construction time and labor costs (computational costs). In practical use—such as when using ChatGPT—users may notice token limits for each conversation. The reason is that processing large numbers of tokens consumes massive computing resources; setting token limits is a necessary measure to ensure stable system operation and efficient service.

(III) Profound Influence on Generation Quality

When an LLM does text generation tasks (e.g., writing articles or stories), it uses a strategy of predicting the next token one by one. For example, if the model gets the input “Artificial intelligence is transfor”, its task is to predict the most likely next token. It makes this prediction based on existing tokens and the linguistic knowledge and patterns it has learned. In the end, it generates complete, logical text like “Artificial intelligence is transforming the world”.

During this prediction process, the model does not deterministically choose one token. Instead, it calculates multiple possible tokens and their respective probabilities. Continuing the example above, the model might predict “ming” with an 80% probability, another context-specific “ming” with 10%, and yet another with 5% (note: adjusted for clarity).

Typically, the model selects the token with the highest probability to continue generating text. However, in scenarios requiring diverse outputs, it may also consider tokens with lower probabilities to make the generated text richer and more flexible.

From this process, it is clear that tokens during LLM text generation are like choosing each piece of a puzzle. Each token prediction directly affects the quality, coherence, and logic of the final text—making tokens one of the core factors determining generation quality.

III. Practical Examples of Tokenization

(I) Characteristics and Methods of English Tokenization

English words have rich morphological variations, so subword splitting is often used in tokenization. Take “running” as an example: it may be split into “run” and “ning”. Here, “run” is the core part of the word, retaining its basic meaning, while “ning” (as a suffix) changes the word’s tense or part of speech.

Through this splitting, the model can better learn the derivative relationships between words and how meanings evolve. Another example is the complex word “unbelievable”, which may split into “un”, “belie”, and “able”. “Un-” is a common negative prefix, and “-able” is a suffix meaning “capable of being…”. This splitting helps the model understand how these affixes influence the word’s overall meaning.

This allows the model to infer the meaning of other words containing these subwords, improving its grasp of semantics. Subword splitting also effectively reduces the number of tokens and boosts the model’s learning efficiency.

For instance, without subword splitting, every different form of a word would need to be learned as an independent token—leading to an extremely large vocabulary. With subword splitting, however, the model can understand and process countless word forms by learning a limited set of subwords and their combinations. This is like building diverse structures with a limited number of building blocks.

(II) Special Tokens and Their Unique Uses

In LLMs, special tokens are introduced to handle specific tasks. They act like specialized components in a building, playing key roles when the model performs particular tasks.

  • [CLS] (Classification Token): Mainly used in classification tasks such as sentiment analysis. When the model needs to determine if a text expresses positive, negative, or neutral sentiment, it adds the special [CLS] token at the start of the text. By learning and analyzing the relationships between each token in the text and [CLS], the model finally performs sentiment classification based on the output vector corresponding to [CLS].

For example, when analyzing the sentiment of the sentence “This movie has a wonderful plot and excellent acting; I really enjoyed it”, the model focuses on the connections between [CLS] and positive sentiment-related tokens (e.g., “wonderful”, “excellent”, “enjoyed”). This lets it determine that the text expresses positive sentiment.

  • [SEP] (Separator Token): Plays an important role in question-answering tasks, where it separates different sentences. For example, in the question-answer pair “Question: What will the weather be like tomorrow? Answer: Tomorrow will be sunny”, the model may add the [SEP] token between the question and the answer.

This clearly distinguishes between different text segments, helping the model better understand the correspondence between the question and the answer—thus processing the question-answering task more accurately.

  • [PAD] (Padding Token): Its role is to align text lengths. When processing a batch of text data, texts of varying lengths would waste computational resources and increase processing difficulty if input directly into the model. This is where the [PAD] token helps.

For example, let’s take two sentences into consideration. One sentence is “I enjoy reading”, and the other is longer. The longer one is “I love sitting by the window on a sunny afternoon, quietly reading an interesting book”. We want the model to process these two sentences in a uniform way. So, we add [PAD] tokens to the end of the shorter sentence. This addition helps make the lengths of the two sentences consistent.

Assuming a unified length of 20 tokens, “I enjoy reading” might be padded to “I enjoy reading [PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD][PAD]”. This allows the model to perform efficient parallel computing on this batch of uniformly sized texts, improving processing efficiency.

IV. The In-Depth Impact of Tokens on LLM Logical Processing

(I) The Encoding Process of Input Tokens

When a text is input into an LLM, it is first split into individual tokens (the tokenization process mentioned earlier). Immediately after, these tokens are encoded into vectors. There are various encoding methods, such as the commonly used One-Hot Encoding and Word Embedding.

Take Word2Vec (a type of Word Embedding) as an example: it maps each token to a low-dimensional vector space. In this space, tokens with similar meanings are positioned closer together. For instance, the vectors for “car” and “automobile” will be relatively close, while the vector distance between “car” and “apple” will be much larger.

Through this encoding, text information is converted into a numerical format the model can understand and process. This is similar to translating the various symbols on a construction blueprint into specific material specifications and location details that construction workers can recognize and act on. This lays the foundation for the model to perform complex computations and learning in the neural network.

(II) The Model’s Mechanism for Learning Token Relationships

LLMs typically use a Self-Attention mechanism to learn connections between different tokens. This mechanism is like a special “perspective” the model has: when processing each token, it can focus on how closely the current token is related to other tokens in the text.

For example, take the sentence “Xiao Ming flew a kite in the park; the kite flew very high”. When the model processes the token “kite”, the Self-Attention mechanism starts working. It helps the model capture relationships between “kite” and other tokens. These tokens include “Xiao Ming”, “park”, and “flew” from the first part. The first part here is “Xiao Ming flew a kite in the park”. Besides that, the mechanism also captures other relationships. These are between “kite” and tokens like “flew” and “very high”. These two tokens come from the second part: “the kite flew very high”.

The model calculates attention weights between different tokens to determine each token’s importance in the current context. This helps it better understand the sentence’s overall meaning. This mechanism lets the model overcome the limitations of traditional sequence models (e.g., Recurrent Neural Networks) in handling long-distance dependencies. This helps the model grasp logical connections between text parts more accurately. It’s similar to how components in a building are linked. These links rely on precise structural design. Together, the components form a stable and meaningful whole.

(III) Token-Based Text Generation Process

For generation tasks (e.g., writing articles or stories), LLMs gradually predict the next token and expand the text incrementally. Starting from the input text fragment, the model calculates the most likely next token. It does this based on its understanding of token relationships (mentioned earlier) and the linguistic patterns and knowledge it acquired during training.

For example, if the model receives the input “On a beautiful morning”, it will predict possible next tokens like “sunlight”, “birds”, or “breeze”. It uses its existing linguistic knowledge and understanding of this context to make these predictions.

The model then adds the predicted token to the existing text sequence and predicts the next token again based on the updated sequence. This cycle repeats, gradually generating a complete text.

In this process, tokens are like “inspiration fragments” in the creative process. By continuously selecting appropriate tokens and combining them, the model builds coherent, logical, and meaningful text. This is similar to an artist gradually combining various elements into a complete work of art according to their vision.