greedy decoding

Greedy decoding is how an AI model writes a sentence by always choosing, at every step, the one word with the highest probability and attaching it to what came before. Think of a traveler who, at every fork in the road, picks only the path that looks most promising right now and never looks back. Another path might lead to a better outcome later, but greedy decoding doesn't weigh that possibility—it only takes the best option directly in front of it.

The advantage of this method is that the result is always the same. Feed in the same question any number of times, and since it always picks the number-one probability word, you get exactly the same answer every time. That's why it's often used as a baseline to check whether two models truly produce the same answer, or whether a new technique only sped things up without changing the content of the answer.

On the other hand, when a chatbot in an actual service answers with different phrasing each time, that's not greedy decoding—it's because a different method is being used that mixes in randomness based on probability. Greedy decoding matters in situations where reproducibility and verification are needed, rather than diversity.

The article explains that "with greedy decoding at temperature 0, the sentences produced are literally identical to running the original model alone." To verify a claim that attaching a draft model nearly tripled the speed, you need evidence that the content of the answers didn't change—and greedy decoding became that benchmark. A common misconception is that greedy decoding always produces the best answer, but in fact it only picks what's best at each individual step, so it can miss a better answer when the sentence is considered as a whole.

If your chatbot lets you lower the temperature setting to 0, try feeding in the same prompt two or three times. If the setting is close to greedy decoding, you can check whether the answer comes out identical down to the last word.