How Language Models Generate Text: Temperature and Top-k in Python

The concept in plain words

A language model outputs a score (logit) for every possible next token. Sampling settings decide how those scores become a choice; temperature controls randomness and top-k keeps only the best candidates.

See how a language model picks the next token: layer normalization, temperature and top-k sampling in plain Python.

Key ideas

A temperature below 1 makes the output more predictable; above 1 makes it more creative.
Top-k sampling removes unlikely tokens so the text stays on topic.
Text is generated one token at a time, feeding each new token back in.
Layer normalization keeps values in a stable range as they pass through many layers.

When should you use it?

Understand what happens inside neural networks and AI libraries.
Learn the idea before you implement it in the matching challenges.
Prepare for machine learning courses, interviews and projects.

See it in action

Press the dice button to sample the next word. Lower the temperature or top-k to make the choice more predictable.

Example code

Run code →

main.py

import math
import random

# Layer normalization rescales a vector to mean 0 and variance 1.
def layer_norm(x, eps=1e-5):
    mean = sum(x) / len(x)
    var = sum((v - mean) ** 2 for v in x) / len(x)
    return [(v - mean) / math.sqrt(var + eps) for v in x]
print("layer norm:", [round(v, 3) for v in layer_norm([2.0, 4.0, 6.0, 8.0])])

# Next-token scores from a model; temperature and top-k shape the choice.
words = ["cat", "dog", "car", "sky"]
logits = [2.0, 1.5, 0.3, -1.0]

def probabilities(temperature=1.0, top_k=None):
    scores = [l / temperature for l in logits]
    if top_k:  # keep only the k best scores
        cutoff = sorted(scores, reverse=True)[top_k - 1]
        scores = [s if s >= cutoff else -1e9 for s in scores]
    exps = [math.exp(s - max(scores)) for s in scores]
    return [e / sum(exps) for e in exps]

for t, k in [(1.0, None), (0.5, None), (1.0, 2)]:
    print(f"temperature={t} top_k={k}:", [round(p, 2) for p in probabilities(t, k)])

random.seed(7)
print("sampled:", random.choices(words, weights=probabilities(0.8, 3), k=5))

Expected output

layer norm: [-1.342, -0.447, 0.447, 1.342]
temperature=1.0 top_k=None: [0.54, 0.33, 0.1, 0.03]
temperature=0.5 top_k=None: [0.71, 0.26, 0.02, 0.0]
temperature=1.0 top_k=2: [0.62, 0.38, 0.0, 0.0]
sampled: ['cat', 'cat', 'dog', 'cat', 'cat']

How it works

layer_norm() rescales a vector to mean 0 and variance 1, as Transformers do inside every block. probabilities() divides the logits by the temperature and applies softmax, so a low temperature makes the top word more likely. Top-k sets every score outside the best k to almost −∞ before the softmax.

Change the values and run the program in the CodeUtility online Python compiler without installing Python locally.

Practice exercises

Modify the runnable example with the exercises below to build understanding beyond copying the result.

Change the inputs or weights and predict the output before running.
Print the intermediate values to follow each step of the calculation.
Solve the linked challenges below to implement the idea yourself.

Run in Python IDE →

Practice with a challenge

Use what this lesson covers to solve a real problem. Each challenge runs in your browser with instant tests and an AI review.

All challenges