Definition
Token Probability measures how likely each word or word fragment in a piece of text is, given what came before it. AI-generated text tends to have high average token probability because language models are trained to select the most statistically likely next word at each step.
Token probability is one of the core signals underlying AI text detection. Every word choice in a piece of writing can be assigned a probability: given the words that came before, how likely was this particular word to appear next? In human writing, this probability varies widely – writers make unexpected, creative, or personal choices that a language model would not have predicted. In AI-generated text, token probabilities tend to be consistently high because the model was trained to favor the most likely word at each step.
The concept is closely related to perplexity – a measure of how surprised a language model is by a piece of text. Low perplexity corresponds to consistently high token probabilities. When detectors describe text as “highly predictable,” token probability is the underlying signal being measured.
How It Works
To calculate token probability, a reference language model processes a piece of text and assigns a probability score to each token based on the preceding context. These scores are aggregated – often as a geometric mean – to produce an overall measure of the text’s predictability. High average token probability suggests the text follows patterns the model expected. Low average probability suggests the text made choices the model did not anticipate.
AI detection systems use token probability as one of several signals. A document where nearly every word choice falls within the top predictions of a language model is a strong candidate for AI-generated text, regardless of content or topic.
Why It Matters for AI Detection
Token probability is one of the most direct and reliable signals available to AI detection systems. It is not based on style or topic – it is a mathematical property of the text itself. This makes it robust to many common evasion attempts: adding synonyms, changing sentence order, or inserting filler phrases does not substantially change the underlying token probability profile if the core word choices remain model-typical.
For educators, understanding token probability helps explain why AI detection is not fooled by superficial edits. It also explains why some human writing – particularly formulaic or highly structured text – can score high on AI detection metrics even when genuinely human-authored.
FAQs
Can students lower their token probability score by using unusual words?
To some extent – using rare vocabulary can lower average token probability. But doing so systematically typically produces text that reads as unnatural or forced, which may itself raise flags. Detectors also use multiple signals alongside token probability, so changing vocabulary alone does not reliably evade detection.
Is token probability the same as perplexity?
They are closely related but not identical. Perplexity is calculated from token probabilities and measures how unpredictable the text is to a reference model. High token probability corresponds to low perplexity. Perplexity is the more commonly cited metric in detection contexts, but it is derived directly from token probability calculations.