View source for XLNet

{{Multiple issues|
{{Sources exist|date=September 2024}}
{{original research|date=September 2024}}{{refimprove|date=September 2024}}
}}
{{Short description|A large language model developed by Google AI}}
{{Infobox software
| name = XLNet
| logo = 
| screenshot = 
| screenshot size = 
| caption = 
| author = [[Google AI]]
| developer = 
| released = {{Start date and age|19 June 2019}}
| latest release version = 
| latest release date = 
| repo = https://github.com/zihangdai/xlnet/
| programming language = 
| operating system = 
| replaces = 
| replaced_by = 
| genre = {{ indented plainlist |
*[[Large language model]]
*[[Transformer (deep learning architecture)]]
}}
| license = [[Apache-2.0]]
| website =
}}

The '''XLNet''' was an autoregressive [[Transformer (deep learning architecture)|Transformer]] designed as an improvement over [[BERT (language model)|BERT]], with 340M parameters and trained on 33 billion words. It was released on 19 June, 2019, under the [[Apache License|Apache 2.0]] license.<ref name="xlnet">{{cite web|work=GitHub|title=xlnet|url=https://github.com/zihangdai/xlnet/|access-date=2 January 2024}}</ref> It achieved state-of-the-art results on a variety of natural language processing tasks, including language modeling, question answering, and natural language inference.

== Architecture ==
The main idea of XLNet is to model language autoregressively like the [[Generative pre-trained transformer|GPT models]], but allow for ''all possible [[Permutation|permutations]]'' of a sentence.<ref name=":0">{{Cite web |title=Pretrained models — transformers 2.0.0 documentation |url=https://huggingface.co/transformers/v2.0.0/pretrained_models.html |access-date=2024-08-05 |website=huggingface.co}}</ref> Concretely, consider the following sentence:<blockquote>My dog is cute.</blockquote>In standard autoregressive language modeling, the model would be tasked with predicting the probability of each word, conditioned on the previous words as its context: 

We factorize the joint probability of a sequence of words  <math>x_1, \ldots, x_T</math> using the chain rule:<math display="block">
\Pr(x_1, \ldots, x_T) = \Pr(x_1) \Pr(x_2 | x_1) \Pr(x_3 | x_1, x_2) \ldots \Pr(x_T | x_1, \ldots, x_{T-1}). 
</math>

For example, the sentence "My dog is cute" is factorized as:

<math display=block>
\Pr(\text{My}, \text{dog}, \text{is}, \text{cute}) = \Pr(\text{My}) \Pr(\text{dog} | \text{My}) \Pr(\text{is} | \text{My}, \text{dog}) \Pr(\text{cute} | \text{My}, \text{dog}, \text{is}).
</math>

Schematically, we can write it as

<math display="block">
\texttt{<MASK> } \texttt{<MASK> } \texttt{<MASK> } \texttt{<MASK> } \to \text{My } \texttt{<MASK> } \texttt{<MASK> } \texttt{<MASK> } \to \text{My dog }\texttt{<MASK> } \texttt{<MASK> } \to \text{My dog is }\texttt{<MASK> } \to \text{My dog is cute}. 
</math>

However, for XLNet, the model is required to predict the words in a randomly generated order. Suppose we have sampled a randomly generated order 3241, then schematically, the model is required to perform the following prediction task:

<math display="block">\texttt{<MASK> } \texttt{<MASK> } \texttt{<MASK> } \texttt{<MASK> } 
\to \texttt{<MASK> } \texttt{<MASK> } \text{is }\texttt{<MASK> } 
\to \texttt{<MASK> } \text{dog is }\texttt{<MASK> } 
\to \texttt{<MASK> }\text{dog is cute} 
\to \text{My dog is cute}</math>

By considering all permutations, XLNet is able to capture longer-range dependencies and better model the bidirectional context of words.

=== Two-Stream Self-Attention ===

To implement permutation language modeling, XLNet uses a two-stream self-attention mechanism. The two streams are:

* '''Content stream:''' This stream encodes the content of each word, as in standard causally masked self-attention.
* '''Query stream:''' This stream encodes the content of each word in the context of what has gone before. In more detail, it is a '''masked cross-attention mechanism''', where the queries are from the query stream, and the key-value pairs are from the content stream. 
The content stream uses the causal mask<math display="block">M_{\text{causal}} = \begin{bmatrix}
0 & -\infty & -\infty & \dots  & -\infty \\
0 & 0 & -\infty & \dots  & -\infty \\
0 & 0 & 0 & \dots  & -\infty \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
0 & 0 & 0 & \dots  & 0
\end{bmatrix}
</math>permuted by a random [[permutation matrix]] to <math>P M_{\text{causal}} P^{-1}
</math>. 

The query stream uses the cross-attention mask <math>P (M_{\text{causal}} - \infty I) P^{-1}
</math>, where the diagonal is subtracted away specifically to avoid the model "cheating" by looking at the content stream for what the current masked token is.

Like the causal masking for GPT models, this two-stream masked architecture allows the model to train on all tokens in one forward pass.

== Training ==
Two models were released:<ref name="xlnet" /><ref name=":0" />

* XLNet-Large, cased: 110M parameters, 24-layer, 1024-hidden, 16-heads
* XLNet-Base, cased: 340M parameters, 12-layer, 768-hidden, 12-heads.

It was trained on a dataset that amounted to 32.89 billion tokens after tokenization with SentencePiece. The dataset was composed of [[BookCorpus|BooksCorpus]], and English Wikipedia, Giga5, ClueWeb 2012-B, and [[Common Crawl]].

It was trained on 512 TPU v3 chips, for 5.5 days. At the end of training, it still under-fitted the data, meaning it could have achieved lower loss with more training. It took 0.5 million steps with an [[Stochastic gradient descent|Adam optimizer]], linear learning rate decay, and a batch size of 8192.<ref name="LX3rI">{{cite arXiv |eprint=1906.08237 |class=cs.CL |first1=Zhilin |last1=Yang |first2=Zihang |last2=Dai |title=XLNet: Generalized Autoregressive Pretraining for Language Understanding |date=2 January 2020 |last3=Yang |first3=Yiming |last4=Carbonell |first4=Jaime |last5=Salakhutdinov |first5=Ruslan |last6=Le |first6=Quoc V.}}</ref>

== See also ==

* [[BERT (language model)]]
* [[Transformer (machine learning model)]]
* [[Generative pre-trained transformer]]

== Reference ==
<references />

{{Google AI}}
{{Google LLC}}
{{Natural Language Processing}}
{{Differentiable computing}}

[[Category:Google software]]
[[Category:Large language models]]
[[Category:Software using the Apache license]]