View source for Perplexity

{{Short description|Concept in information theory}}
{{other uses|Perplexity (disambiguation)}}

{{More citations needed|date=July 2022}}
{{Wiktionary|perplexity}}

In [[information theory]], '''perplexity''' is a measure of uncertainty in the value of a sample from a discrete probability distribution. The larger the perplexity, the less likely it is that an observer can guess the value which will be drawn from the distribution. Perplexity was originally introduced in 1977 in the context of [[speech recognition]] by [[Frederick Jelinek]], [[Robert Mercer | Robert Leroy Mercer]], Lalit R. Bahl, and [[James K. Baker]].<ref>{{Cite journal |last1=Jelinek |first1=F. |last2=Mercer |first2=R. L. |last3=Bahl |first3=L. R. |last4=Baker |first4=J. K. |date=1977 |title=Perplexity—a measure of the difficulty of speech recognition tasks |journal=The Journal of the Acoustical Society of America |volume=62 |issue=S1 |pages=S63 |doi=10.1121/1.2016299 |issn=0001-4966|doi-access=free |bibcode=1977ASAJ...62Q..63J }}</ref>

== Perplexity of a probability distribution ==

The perplexity ''PP'' of a discrete [[probability distribution]] ''p'' is a concept widely used in information theory, [[machine learning]], and statistical modeling. It is defined as

:<math>\mathit{PP}(p) := 2^{H(p)}=2^{-\sum_x p(x)\log_2 p(x)}=\prod_x p(x)^{-p(x)}</math>

where ''H''(''p'') is the [[entropy (information theory)|entropy]] (in [[Bit|bits]]) of the distribution, and ''x'' ranges over the [[Event (probability theory)|events]]. The base of the logarithm need not be 2: The perplexity is independent of the base, provided that the entropy and the exponentiation use the same base. In some contexts, this measure is also referred to as the ''[[Diversity_index|(order-1 true) diversity]]''.

Perplexity of a [[random variable]] ''X'' may be defined as the perplexity of the distribution over its possible values ''x''. It can be thought of as a measure of uncertainty or "surprise" related to the outcomes.

For a probability distribution ''p'' where exactly ''k'' outcomes each have a probability of ''1/k'' and all other outcomes have a probability of zero, the perplexity of this distribution is simply ''k''. This is because the distribution models a fair ''k''-sided [[Dice|die]], with each of the ''k'' outcomes being equally likely. In this context, the perplexity ''k'' indicates that there is as much uncertainty as there would be when rolling a fair ''k''-sided die. Even if a random variable has more than ''k'' possible outcomes, the perplexity will still be ''k'' if the distribution is uniform over ''k'' outcomes and zero for the rest. Thus, a random variable with a perplexity of ''k'' can be described as being "''k''-ways perplexed," meaning it has the same level of uncertainty as a fair ''k-''sided die.

Perplexity is sometimes used as a measure of the difficulty of a prediction problem. It is, however, generally not a straight forward representation of the relevant probability. For example, if you have two choices, one with probability 0.9, your chances of a correct guess using the optimal strategy are 90 percent. Yet, the perplexity is 2<sup>−0.9 log<sub>2</sub> 0.9 - 0.1 log<sub>2</sub> 0.1</sup>= 1.38. The inverse of the perplexity, 1/1.38 = 0.72, does not correspond to the 0.9 probability.

The perplexity is the exponentiation of the entropy, a more straightforward quantity. Entropy measures the expected or "average" number of bits required to encode the outcome of the random variable using an optimal [[variable-length code]]. It can also be regarded as the expected information gain from learning the outcome of the random variable, providing insight into the uncertainty and complexity of the underlying probability distribution.

== Perplexity of a probability model ==

A model of an unknown probability distribution ''p'', may be proposed based on a training sample that was drawn from ''p''.  Given a proposed probability model ''q'', one may evaluate ''q'' by asking how well it predicts a separate test sample ''x''<sub>1</sub>, ''x''<sub>2</sub>, ..., ''x<sub>N</sub>'' also drawn from ''p''.  The perplexity of the model ''q'' is defined as

:<math>b^{- \frac{1}{N} \sum_{i=1}^N \log_b q(x_i)} = \left(\prod_i q(x_i) \right)^{-1/N}</math>

where <math>b</math> is customarily 2. Better models ''q'' of the unknown distribution ''p'' will tend to assign higher probabilities ''q''(''x<sub>i</sub>'') to the test events. Thus, they have lower perplexity because they are less surprised by the test sample. This is equivalent to saying that better models have higher [[Likelihood function|likelihoods]] for the test data, which leads to a lower perplexity value.

The exponent above may be regarded as the average number of bits needed to represent a test event ''x<sub>i</sub>'' if one uses an optimal code based on ''q''.  Low-perplexity models do a better job of [[Data compression|compressing]] the test sample, requiring few bits per test element on average because ''q''(''x<sub>i</sub>'') tends to be high.

The exponent <math>- \tfrac{1}{N} \sum_{i=1}^N \log_b q(x_i)</math> may also be interpreted as a [[cross-entropy]]:

:<math>H(\tilde{p},q) = -\sum_x \tilde{p}(x) \log_b q(x)</math>

where <math>\tilde{p}</math> denotes the [[Empirical distribution function|empirical distribution]] of the test sample (i.e., <math>\tilde{p}(x) = n/N</math> if ''x'' appeared ''n'' times in the test sample of size ''N'').

By the definition of [[KL divergence]], it is also equal to <math display="block">H(\tilde p) + D_{KL}(\tilde p \| q)</math>which is <math>\geq H(\tilde p)</math>. Consequently, the perplexity is minimized when <math>q = \tilde p</math>.

== Perplexity per token ==
In [[natural language processing]] (NLP), a '''[[Corpus linguistics|corpus]]''' is a structured collection of [[Text corpus|texts]] or documents, and a '''[[language model]]''' is a probability distribution over entire texts or documents. Consequently, in NLP, the more commonly used measure is '''perplexity per token''' (word or, more frequently, sub-word), defined as:<math display="block">\left(\prod_{i=1}^n q(s_i) \right)^{-1/N}</math> where <math>s_1, ..., s_n</math> are the <math>n</math> documents in the corpus and <math>N</math> is the number of ''tokens'' in the corpus. This normalizes the perplexity by the length of the text, allowing for more meaningful comparisons between different texts or models rather than documents.

Suppose the average text ''x<sub>i</sub>'' in the corpus has a probability of <math>2^{-190}</math> according to the language model. This would give a model perplexity of 2<sup>190</sup> per sentence. However, in NLP, it is more common to normalize by the length of a text. Thus, if the test sample has a length of 1,000 tokens, and could be coded using 7.95 bits per token, one could report a model perplexity of 2<sup>7.95</sup> = 247 ''per token.'' In other words, the model is as confused on test data as if it had to choose uniformly and independently among 247 possibilities for each token.

There are two standard evaluation metrics for language models: perplexity or word error rate(WER). The simpler of these measures, WER, is simply the percentage of erroneously recognized words E (deletions, insertions, substitutions) to total number of words N, in a speech recognition task i.e.<math display="block">WER = 
\left ( \frac{E}{\N} \right ) *100%</math>The second metric, perplexity (per token), is an information theoretic measure that evaluates the similarity of proposed model ''m'' to the original distribution ''p''. It can be computed as a inverse of (geometric) average probability of test set ''T''

<math>PPL(D) = \sqrt[N]{\frac{1}{m(T)}} = 2^{-\frac{1}{N} \log_2\big(m(T)\big)}</math>

where ''N'' is the number of tokens in test set ''T''. This equation can be seen as the exponentiated cross entropy, where cross entropy H (''p'';''m'') is approximated as

<math>H(p;m) = -\frac{1}{N} \log_2\big(m(T)\big)</math>

=== Recent advances in language modeling ===
Since 2007, significant advancements in language modeling have emerged, particularly with the advent of [[deep learning]] techniques. Perplexity per token, a measure that quantifies the predictive power of a language model, has remained central to evaluating models such as the dominant [[transformer (machine learning model)|transformer]] models like Google's [[BERT (language model)|BERT]], OpenAI's [[GPT-4]] and other [[Large language model|large language models]] (LLMs).

This measure was employed to compare different models on the same dataset and guide the optimization of [[Hyperparameter (machine learning)|hyperparameters]], although it has been found sensitive to factors such as linguistic features and sentence length.<ref>{{Cite conference |last1=Miaschi |first1=Alessio |last2=Brunato |first2=Dominique |last3=Dell'Orletta |first3=Felice |last4=Venturi |first4=Giulia |date=2021 |title=What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity |book-title=Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures |url=https://aclanthology.org/2021.deelio-1.5 |doi=10.18653/v1/2021.deelio-1.5 |pages=40–47 |doi-access=free |access-date=2023-08-24 |archive-date=2023-10-24 |archive-url=https://web.archive.org/web/20231024045622/https://aclanthology.org/2021.deelio-1.5/ |url-status=live }}</ref> 

Despite its pivotal role in language model development, perplexity has shown limitations, particularly as an inadequate predictor of [[speech recognition]] performance, [[overfitting]] and [[Generalization error|generalization]],<ref>{{Cite journal |last1=Klakow |first1=Dietrich |last2=Peters |first2=Jochen |date=2002 |title=Testing the correlation of word error rate and perplexity |journal=Speech Communication |volume=38 |issue=1–2 |pages=19–28 |issn=0167-6393 |url=https://doi.org/10.1016/S0167-6393(01)00041-3 |doi=10.1016/S0167-6393(01)00041-3}}</ref><ref>{{Cite journal |last1=Chen |first1=Stanley F |last2=Beeferman |first2=Douglas |last3=Rosenfeld |first3=Roni |date=2018 |title=Evaluation Metrics For Language Models |journal=Carnegie Mellon University |doi=10.1184/R1/6605324.v1 |url=https://doi.org/10.1184/R1/6605324.v1}}</ref> raising questions about the benefits of blindly optimizing perplexity alone. 
=== Brown Corpus ===
The lowest perplexity that had been published on the [[Brown Corpus]] (1 million words of American English of varying topics and genres) as of 1992 is indeed about 247 per word/token, corresponding to a cross-entropy of log<sub>2</sub>247 = 7.95 bits per word or 1.75 bits per letter<ref>{{cite journal |last=Brown |first=Peter F. |date=March 1992 |title=An Estimate of an Upper Bound for the Entropy of English |journal=Computational Linguistics |volume=18 |issue=1 |url=https://www.cs.cmu.edu/~roni/11761/PreviousYearsHandouts/gauntlet.pdf |access-date=2007-02-07 |display-authors=etal |archive-date=2021-09-17 |archive-url=https://web.archive.org/web/20210917151307/https://www.cs.cmu.edu/~roni/11761/PreviousYearsHandouts/gauntlet.pdf |url-status=live }}</ref> using a [[n-gram|trigram]] model. While this figure represented the state of the art (SOTA) at the time, advancements in techniques such as deep learning have led to significant improvements in perplexity on other benchmarks, such as the One Billion Word Benchmark.<ref>Jozefowicz, Rafal, et al. "Exploring the limits of language modeling." arXiv preprint arXiv:1602.02410 (2016). [https://arxiv.org/abs/1602.02410] {{Webarchive|url=https://web.archive.org/web/20210504232535/https://arxiv.org/abs/1602.02410|date=2021-05-04}}</ref>

In the context of the [[Brown Corpus]], simply guessing that the next word is "the" will achieve an accuracy of 7 percent, contrasting with the 1/247 = 0.4 percent that might be expected from a naive use of perplexity. This difference underscores the importance of the [[statistical model]] used and the nuanced nature of perplexity as a measure of predictiveness.<ref>Wilcox, Ethan Gotlieb, et al. "On the predictive power of neural language models for human real-time comprehension behavior." arXiv preprint arXiv:2006.01912 (2020). [https://arxiv.org/abs/2006.01912] {{Webarchive|url=https://web.archive.org/web/20230825104354/https://arxiv.org/abs/2006.01912|date=2023-08-25}}</ref> The guess is based on unigram statistics, not on the trigram statistics that yielded the perplexity of 247, and utilizing trigram statistics would further refine the prediction.

==See also==
*[[Cross-entropy]]
*[[Statistical model validation]]

==References==
<references />

{{Machine learning evaluation metrics}}

[[Category:Entropy and information]]
[[Category:Language modeling]]