How Large Language Models Work

Models cannot read text directly. First, a tokenizerTokenizerThe component that splits text into tokens (subword units) that the model processes. Common types: BPE, SentencePiece. splits text into subword fragments called tokensTokenA basic unit of text (such as a word or subword fragment) processed by a language model.. Common prefixes or suffixes (like ##ing or BPE space markers Ġ) are isolated to allow the model to recognize grammatical roots [Radford et al. 2018].

Each tokenTokenA basic unit of text (such as a word or subword fragment) processed by a language model. is mapped to a unique integerIntegerA whole number representing a token's index position inside a vocabulary table. ID. Then, a lookup table translates these IDs into embeddingsEmbeddingA high-dimensional vector representing the semantic meaning of a token, grouping similar words close together in vector space.: dense vectorsVectorA list of numerical values representing a coordinate in high-dimensional space, used to encode semantic meanings.(often 4096+ dimensions) that position words in a semantic space. In this space, words with similar meanings (like "king" and "queen") are grouped close together.

The core engine of the TransformerTransformerNeural network architecture using self-attention to process entire sequences in parallel. Replaced RNNs and enabled massive scaling. is Self-AttentionSelf-AttentionAn attention mechanism relating different positions of a single sequence to compute a representation of the sequence. [Vaswani et al. 2017]. In a sentence, words depend on contextContextThe text or prompt preceding a generation that the model uses to understand what it is currently processing.. For example, in "The llmLLMLarge Language Model — a deep learning model trained on massive text corpora to understand and generate human language. read the promptPromptThe input text or instruction provided to a language model to guide its output. because "it" was relevant", what does "it" refer to?

Self-Attention calculates a set of queryQueryIn self-attention, the vector representing the current token seeking context from other parts of the sequence., keyKeyIn self-attention, the vector representing what information a token contains, matched against queries to compute attention weights., and value vectorsVectorA list of numerical values representing a coordinate in high-dimensional space, used to encode semantic meanings.. The query for "it" is compared with keysKeyIn self-attention, the vector representing what information a token contains, matched against queries to compute attention weights. for all other words. The model computes weights representing how much "attentionAttentionA mechanism in Transformers that determines how much focus to place on other words in a sequence when processing the current word." to pay to each word.

TransformersTransformerNeural network architecture using self-attention to process entire sequences in parallel. Replaced RNNs and enabled massive scaling. run multiple Attention HeadsHeadsIndependent attention mechanisms running in parallel, allowing the model to capture multiple relationships at once. in parallel. This allows the model to capture different relations simultaneously (e.g. tracking pronouns in one head, verbs in another).

At inferenceInferenceUsing a trained model to generate predictions or outputs (as opposed to training it)., an LLMLLMLarge Language Model — a deep learning model trained on massive text corpora to understand and generate human language. outputs logits (raw scores) for every tokenTokenA basic unit of text (such as a word or subword fragment) processed by a language model. in its vocabulary. These are turned into probabilities using SoftmaxSoftmaxA mathematical function that converts a vector of raw scores (logits) into a probability distribution summing to 1.. We control the randomness and diversity of outputs using two parametersParameterA variable or weight inside a neural network that is adjusted during training to store knowledge.:

• 🌡️ TemperatureTemperatureA hyperparameter controlling the randomness of outputs; lower values make outputs more deterministic, higher values make them more creative.:Controls distribution scale. Low temperature (< 0.5) compresses values, sharpening the peaks so the model repeatedly picks the absolute highest score (factual/deterministicDeterministicAn execution mode or output that is completely predictable, always yielding the identical response for a given input.). High temperature (> 1.0) flattens the curves, increasing diversity (creative/unpredictable).
• 🎯 Top-P (Nucleus Sampling): TruncatesTruncateTo cut off or discard the tail end of a text sequence or a probability distribution. the probability distribution to include only a subset whose cumulative sum is under P. TokensTokenA basic unit of text (such as a word or subword fragment) processed by a language model. in the tail whose sum exceedsP are completely discarded.