Step 01 / 02

How we choose the next token

Module map

Once you have a probability distribution, you still have to choose. Different strategies change the personality of the output.

  • Greedy decoding — always pick the current highest-probability token. Simple, often repetitive.
  • Temperature — reshapes the distribution. Lower = sharper / more concentrated. Higher = flatter / more room for lower-probability options.
  • Top-k — only consider the k highest-scoring candidates.
  • Top-p (nucleus) — consider the smallest set of candidates whose probabilities add up to p.
Do not teach yourself “high temperature = creative.” More precisely: higher temperature increases sampling randomness. Whether that helps quality depends on the task.