View source for Attention (machine learning)

{{Short description|Machine learning technique}}
{{Machine learning}}
[[File:Attention_mechanism_overview.svg|thumb|Attention mechanism, overview]]
'''Attention''' is a [[machine learning]] method that determines the relative importance of each component in a sequence relative to the other components in that sequence. In [[natural language processing]], importance is represented by [https://pmc.ncbi.nlm.nih.gov/articles/PMC9007265/ "soft"] weights assigned to each word in a sentence. More generally, attention encodes vectors called [[lexical token|token]] [[Word embedding|embeddings]] across a fixed-width [[context window|sequence]] that can range from tens to millions of tokens in size.

Unlike "hard" weights, which are computed during the backwards training pass, "soft" weights exist only in the forward pass and therefore change with every step of the input. Earlier designs implemented the attention mechanism in a serial [[recurrent neural network]] (RNN) language translation system, but a more recent design, namely the [[Transformer (machine learning model)|transformer]], removed the slower sequential RNN and relied more heavily on the faster parallel attention scheme.

Inspired by ideas about [[attention|attention in humans]], the attention mechanism was developed to address the weaknesses of leveraging information from the [[hidden layer]]s of recurrent neural networks. Recurrent neural networks favor more recent information contained in words at the end of a sentence, while information earlier in the sentence tends to be [[Vanishing gradient problem|attenuated]]. Attention allows a token equal access to any part of a sentence directly, rather than only through the previous state.

==History==
{{See also|Timeline of machine learning}}
Academic reviews of the history of the attention mechanism are provided in Niu et al.<ref name=":0">{{Cite journal |last1=Niu |first1=Zhaoyang |last2=Zhong |first2=Guoqiang |last3=Yu |first3=Hui |date=2021-09-10 |title=A review on the attention mechanism of deep learning |url=https://www.sciencedirect.com/science/article/pii/S092523122100477X |journal=Neurocomputing |volume=452 |pages=48–62 |doi=10.1016/j.neucom.2021.03.091 |issn=0925-2312}}</ref> and Soydaner.<ref name=":1">{{Cite journal |last=Soydaner |first=Derya |date=August 2022 |title=Attention mechanism in neural networks: where it comes and where it goes |url=https://link.springer.com/10.1007/s00521-022-07366-3 |journal=Neural Computing and Applications |language=en |volume=34 |issue=16 |pages=13371–13385 |doi=10.1007/s00521-022-07366-3 |issn=0941-0643|arxiv=2204.13154 }}</ref>

=== Predecessors ===
[[Attentional control|Selective attention]] in humans had been well studied in neuroscience and cognitive psychology.<ref>{{Cite book |last1=Kramer |first1=Arthur F. |url=http://www.oxfordscholarship.com/view/10.1093/acprof:oso/9780195305722.001.0001/acprof-9780195305722 |title=Attention: From Theory to Practice |last2=Wiegmann |first2=Douglas A. |last3=Kirlik |first3=Alex |date=2006-12-28 |publisher=Oxford University Press |isbn=978-0-19-530572-2 |chapter=1 Attention: From History to Application |doi=10.1093/acprof:oso/9780195305722.003.0001}}</ref> In 1953, [[Colin Cherry]] studied selective attention in the context of audition, known as the [[cocktail party effect]].<ref name="Cherry 1953">{{cite journal |vauthors=Cherry EC |year=1953 |title=Some Experiments on the Recognition of Speech, with One and with Two Ears |url=http://www.ee.columbia.edu/~dpwe/papers/Cherry53-cpe.pdf |journal=The Journal of the Acoustical Society of America |volume=25 |issue=5 |pages=975–79 |bibcode=1953ASAJ...25..975C |doi=10.1121/1.1907229 |issn=0001-4966 |hdl-access=free |hdl=11858/00-001M-0000-002A-F750-3}}</ref>

In 1958, [[Donald Broadbent]] proposed the [[Broadbent's filter model of attention|filter model of attention]].<ref name="Broadbent">{{cite book |last=Broadbent |first=D |author-link=Donald Broadbent |title=Perception and Communication |publisher=Pergamon Press |year=1958 |location=London}}</ref> Selective attention of vision was studied in the 1960s by [[George Sperling]]'s [[Iconic memory#Sperling's partial report procedure|partial report paradigm]]. It was also noticed that [[Saccade|saccade control]] is modulated by cognitive processes, insofar as the eye moves preferentially towards areas of high [[Salience (neuroscience)|salience]]. As the fovea of the eye is small, the eye cannot sharply resolve the entire visual field at once. The use of saccade control allows the eye to quickly scan important features of a scene.<ref>{{Cite journal |last1=Kowler |first1=Eileen |last2=Anderson |first2=Eric |last3=Dosher |first3=Barbara |last4=Blaser |first4=Erik |date=1995-07-01 |title=The role of attention in the programming of saccades |url=https://dx.doi.org/10.1016/0042-6989%2894%2900279-U |journal=Vision Research |volume=35 |issue=13 |pages=1897–1916 |doi=10.1016/0042-6989(94)00279-U |pmid=7660596 |issn=0042-6989}}</ref>

These research developments inspired algorithms such as the [[Neocognitron]] and its variants.<ref>{{Cite journal |last=Fukushima |first=Kunihiko |date=1987-12-01 |title=Neural network model for selective attention in visual pattern recognition and associative recall |url=https://opg.optica.org/abstract.cfm?URI=ao-26-23-4985 |journal=Applied Optics |language=en |volume=26 |issue=23 |pages=4985–4992 |doi=10.1364/AO.26.004985 |pmid=20523477 |bibcode=1987ApOpt..26.4985F |issn=0003-6935}}</ref><ref>{{cite arXiv |last1=Ba |first1=Jimmy |title=Multiple Object Recognition with Visual Attention |date=2015-04-23 |eprint=1412.7755 |last2=Mnih |first2=Volodymyr |last3=Kavukcuoglu |first3=Koray|class=cs.LG }}</ref> Meanwhile, developments in neural networks had inspired circuit models of biological visual attention.<ref>{{cite book |last1=Koch |first1=Christof |chapter=Shifts in Selective Visual Attention: Towards the Underlying Neural Circuitry |date=1987 |title=Matters of Intelligence: Conceptual Structures in Cognitive Neuroscience |pages=115–141 |editor-last=Vaina |editor-first=Lucia M. |chapter-url=https://doi.org/10.1007/978-94-009-3833-5_5 |access-date=2024-08-06 |place=Dordrecht |publisher=Springer Netherlands |language=en |doi=10.1007/978-94-009-3833-5_5 |isbn=978-94-009-3833-5 |last2=Ullman |first2=Shimon}}</ref><ref name=":1" /> One well-cited network from 1998, for example, was inspired by the [[Visual cortex|low-level primate visual system]]. It produced [[saliency map]]s of images using handcrafted (not learned) features, which were then used to guide a second neural network in processing patches of the image in order of reducing saliency.<ref name="li">{{Cite journal |last1=Itti |first1=L. |last2=Koch |first2=C. |last3=Niebur |first3=E. |date=November 1998 |title=A model of saliency-based visual attention for rapid scene analysis |url=https://ieeexplore.ieee.org/document/730558 |journal=IEEE Transactions on Pattern Analysis and Machine Intelligence |volume=20 |issue=11 |pages=1254–1259 |doi=10.1109/34.730558}}</ref>

A key aspect of attention mechanism can be written (schematically) as: <math display="block">\sum_i \langle(\text{query})_i, (\text{key})_i\rangle (\text{value})_i</math>where the angled brackets denote dot product. This shows that it involves a multiplicative operation. Multiplicative operations within artificial neural networks had been studied under the names of [[Group method of data handling|Group Method of Data Handling]] (1965)<ref name="ivak1965">{{cite book |last=Ivakhnenko |first=A. G. |author-link=Alexey Grigorevich Ivakhnenko |url={{google books |plainurl=y |id=FhwVNQAACAAJ}} |title=Cybernetic Predicting Devices |publisher=CCM Information Corporation |year=1973}}</ref><ref name="ivak1967">{{cite book |last1=Ivakhnenko |first1=A. G. |author-link=Alexey Grigorevich Ivakhnenko |url={{google books |plainurl=y |id=rGFgAAAAMAAJ}} |title=Cybernetics and forecasting techniques |last2=Grigorʹevich Lapa |first2=Valentin |publisher=American Elsevier Pub. Co. |year=1967}}</ref> (where Kolmogorov-Gabor polynomials implement multiplicative units or "gates"<ref name="DLhistory">{{cite arXiv |eprint=2212.11279 |class=cs.NE |first=Jürgen |last=Schmidhuber |author-link=Jürgen Schmidhuber |title=Annotated History of Modern AI and Deep Learning |date=2022}}</ref>), ''[[higher-order neural network]]s'',<ref>{{Cite journal |last1=Giles |first1=C. Lee |last2=Maxwell |first2=Tom |date=1987-12-01 |title=Learning, invariance, and generalization in high-order neural networks |url=https://opg.optica.org/abstract.cfm?URI=ao-26-23-4972 |journal=Applied Optics |language=en |volume=26 |issue=23 |pages=4972–4978 |doi=10.1364/AO.26.004972 |pmid=20523475 |issn=0003-6935}}</ref> ''multiplication units'',<ref>{{Cite journal |last1=Feldman |first1=J. A. |last2=Ballard |first2=D. H. |date=1982-07-01 |title=Connectionist models and their properties |url=https://www.sciencedirect.com/science/article/pii/S0364021382800013 |journal=Cognitive Science |volume=6 |issue=3 |pages=205–254 |doi=10.1016/S0364-0213(82)80001-3 |issn=0364-0213}}</ref> ''sigma-pi units'',<ref name="PDP" /> ''fast weight controllers'',<ref name="transform1992" /> and ''hyper-networks''.<ref>{{cite arXiv |last1=Ha |first1=David |title=HyperNetworks |date=2016-12-01 |eprint=1609.09106 |last2=Dai |first2=Andrew |last3=Le |first3=Quoc V.|class=cs.LG }}</ref>

In ''fast weight controller'' ([[Jürgen Schmidhuber|Schmidhuber]], 1992), one of its two networks has  "fast weights" or "dynamic links" (1981).<ref name="malsburg1981">Christoph von der Malsburg: The correlation theory of brain function. Internal Report 81-2, MPI Biophysical Chemistry, 1981. http://cogprints.org/1380/1/vdM_correlation.pdf See Reprint in Models of Neural Networks II, chapter 2, pages 95-119. Springer, Berlin, 1994.</ref><ref name="feldman1982">Jerome A. Feldman, "Dynamic connections in neural networks," Biological Cybernetics, vol. 46, no. 1, pp. 27-39, Dec. 1982.</ref><ref>{{Cite journal |last1=Hinton |first1=Geoffrey E. |last2=Plaut |first2=David C. |date=1987 |title=Using Fast Weights to Deblur Old Memories |url=https://escholarship.org/uc/item/0570j1dp |journal=Proceedings of the Annual Meeting of the Cognitive Science Society |language=en |volume=9}}</ref> A slow neural network learns by gradient descent to generate keys and values for computing the weight changes of the fast neural network which computes answers to queries.<ref name="transform1992"/> This was later shown to be equivalent to the unnormalized linear Transformer.<ref name="schlag2021" /> A follow-up paper developed a similar system with active weight changing.<ref name="attention1993">{{cite conference |last1=Schmidhuber |first1=Jürgen |author-link1=Jürgen Schmidhuber |date=1993 |title=Reducing the ratio between learning complexity and number of time-varying variables in fully recurrent nets |publisher=Springer |pages=460–463 |book-title=ICANN 1993}}</ref>

=== Recurrent attention ===
During the deep learning era, attention mechanism was developed to solve similar problems in encoding-decoding.<ref name=":0" />

In machine translation, the [[seq2seq]] model, as it was proposed in 2014,<ref name="sequence">{{cite arXiv |eprint=1409.3215 |class=cs.CL |first1=Ilya |last1=Sutskever |first2=Oriol |last2=Vinyals |title=Sequence to sequence learning with neural networks |last3=Le |first3=Quoc Viet |year=2014}}</ref> would encode an input text into a fixed-length vector, which would then be decoded into an output text. If the input text is long, the fixed-length vector would be unable to carry enough information for accurate decoding. An attention mechanism was proposed to solve this problem.

An image captioning model was proposed in 2015, citing inspiration from the seq2seq model.<ref>{{Cite web |last1=Vinyals |first1=Oriol |last2=Toshev |first2=Alexander |last3=Bengio |first3=Samy |last4=Erhan |first4=Dumitru |date=2015 |title=Show and Tell: A Neural Image Caption Generator |url=https://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Vinyals_Show_and_Tell_2015_CVPR_paper.html |pages=3156–3164}}</ref> that would encode an input image into a fixed-length vector. Xu et al (2015),<ref>{{Cite journal |last1=Xu |first1=Kelvin |last2=Ba |first2=Jimmy |last3=Kiros |first3=Ryan |last4=Cho |first4=Kyunghyun |last5=Courville |first5=Aaron |last6=Salakhudinov |first6=Ruslan |last7=Zemel |first7=Rich |last8=Bengio |first8=Yoshua |date=2015-06-01 |title=Show, Attend and Tell: Neural Image Caption Generation with Visual Attention |url=https://proceedings.mlr.press/v37/xuc15.html |journal=Proceedings of the 32nd International Conference on Machine Learning |language=en |publisher=PMLR |pages=2048–2057}}</ref> citing Bahdanau et al (2014),<ref name=":2">{{cite arXiv |last1=Bahdanau |first1=Dzmitry |title=Neural Machine Translation by Jointly Learning to Align and Translate  |date=19 May 2016 |eprint=1409.0473 |last2=Cho |first2=Kyunghyun |last3=Bengio |first3=Yoshua|class=cs.CL }} (orig-date 1 Sep 2014)</ref> applied the attention mechanism as used in the seq2seq model to image captioning.

=== Transformer ===
{{See also|Transformer (deep learning architecture)#History}}
One problem with seq2seq models was their use of recurrent neural networks, which are not parallelizable as both the encoder and the decoder must process the sequence token-by-token. ''Decomposable attention<ref name="parikh2" />'' attempted to solve this problem by processing the input sequence in parallel, before computing a "soft alignment matrix" (''alignment'' is the terminology used by Bahdanau et al<ref name=":2" />) in order to allow for parallel processing.

The idea of using the attention mechanism for self-attention, instead of in an encoder-decoder (cross-attention), was also proposed during this period, such as in [[differentiable neural computer]]s<ref name="Graves2016" /> and [[neural Turing machine]]s.<ref>{{cite arXiv |last1=Graves |first1=Alex |title=Neural Turing Machines |date=2014-12-10 |eprint=1410.5401 |last2=Wayne |first2=Greg |last3=Danihelka |first3=Ivo|class=cs.NE }}</ref> It was termed ''intra-attention''<ref name="parikh">{{cite arXiv |last1=Cheng |first1=Jianpeng |title=Long Short-Term Memory-Networks for Machine Reading |date=2016-09-20 |eprint=1601.06733 |last2=Dong |first2=Li |last3=Lapata |first3=Mirella|class=cs.CL }}</ref> where an LSTM is augmented with a memory network as it encodes an input sequence.

These strands of development were brought together in 2017 with the [[Transformer architecture]], published in the ''[[Attention Is All You Need]]'' paper.

== Overview ==
The attention network was designed to identify high correlations patterns amongst words in a given sentence, assuming that it has learned word correlation patterns from the training data.  This correlation is captured as neuronal weights learned during training with [[backpropagation]].{{Image frame|content=[[File:attention-qkv.png|700px]]|width=700|align=center|caption=The diagram shows the Attention forward pass calculating correlations of the word "that" with other words in "See that girl run."  Given the right weights from training, the network should be able to identify "girl" as a highly correlated word.  Some things to note:
* This example focuses on the attention of a single word "that".  In practice, the attention of each word is calculated in parallel to speed up calculations.  Simply changing the lowercase "x" vector to the uppercase "X" matrix will yield the formula for this.
* Softmax scaling {{math|qW<sub>k</sub><sup>T</sup>}} / {{radic|100}} prevents a high variance in {{math|qW<sub>k</sub><sup>T</sup>}} that would allow a single word to excessively dominate the softmax resulting in attention to only one word, as a discrete hard max would do.
* <u>Notation</u>: the commonly written row-wise {{math|softmax}} formula above assumes that vectors are rows, which runs contrary to the standard math notation of column vectors. More correctly, we should take the transpose of the context vector and use the column-wise {{math|softmax}}, resulting in the more correct form
<math display="block">\begin{align} (XW_v)^T * {[ (W_k X^T) * { (\underline{x}W_q)^T } ]_{sm} } \end{align}</math>.}}

This attention scheme has been compared to the Query-Key analogy of relational databases.  That comparison suggests an '''asymmetric''' role for the Query and Key vectors, where '''one''' item of interest (the Query vector "that") is matched against '''all''' possible items (the Key vectors of each word in the sentence).   However, both Self and Cross Attentions' parallel calculations matches all tokens of the K matrix with all tokens of the Q matrix; therefore the roles of these vectors are '''symmetric'''.  Possibly because the simplistic database analogy is flawed, much effort has gone into understand Attention further by studying their roles in focused settings, such as in-context learning,<ref name="zhang2024" /> masked language tasks,<ref name="rende2023" /> stripped down transformers,<ref name="bhe2023" /> bigram statistics,<ref name="timnguyen2024" /> N-gram statistics,<ref name="tcircuits" /> pairwise convolutions,<ref name="algosimple" /> and arithmetic factoring.<ref name="charton2023" />

== Machine translation ==
[[File:Self-attention_in_CNN,_RNN,_and_self-attention.svg|thumb|Comparison of the data flow in CNN, RNN, and self-attention]]
The seq2seq method developed in the early 2010s uses two neural networks:  an encoder network converts an input sentence into numerical vectors, and a decoder network converts those vectors to sentences in the target language.  The Attention mechanism was grafted onto this structure in 2014 and shown below.  Later it was refined into the Transformer design (2017).
[[File:Attention-animated.webm|thumb|A step-by-step sequence of a language translation]]
{{plain image with caption|File:attention-1-sn.png|Encoder-decoder with attention.<ref name="bdritz2017" /> Numerical subscripts (100, 300, 500, 9k, 10k) indicate vector sizes while lettered subscripts i and i − 1 indicate time steps.  Pinkish regions in H matrix and w vector are zero values.  See Legend for details.|align=none|width=500px}}
{| class="wikitable mw-collapsible mw-collapsed"
|+Legend
!Label
!Description
|-
|100
|Max. sentence length
|-
|300
|[[Word embedding|Embedding]] size (word dimension)
|-
|500
|Length of hidden vector
|-
|9k, 10k
|Dictionary size of input & output languages respectively.
|-
|<u>x</u>, <u>Y</u>
|9k and 10k [[1-hot]] dictionary vectors.  <u>x</u> → x implemented as a [[lookup table]] rather than vector multiplication.  <u>Y</u> is the 1-hot maximizer of the linear Decoder layer D; that is, it takes the argmax of D's linear layer output.
|-
|x
|300-long word embedding vector.  The vectors are usually pre-calculated from other projects such as [[GloVe]] or [[Word2Vec]].
|-
|h
|500-long encoder hidden vector.  At each point in time, this vector summarizes all the preceding words before it.  The final h can be viewed as a "sentence" vector, or a [[thought vector]] as Hinton calls it.
|-
|s
|500-long decoder hidden state vector.
|-
|E
|500 neuron [[recurrent neural network]] encoder.  500 outputs.  Input count is 800–300 from source embedding + 500 from recurrent connections.  The encoder feeds directly into the decoder only to initialize it, but not thereafter; hence, that direct connection is shown very faintly.
|-
|D
|2-layer decoder.  The recurrent layer has 500 neurons and the fully-connected linear layer has 10k neurons (the size of the target vocabulary).<ref name="pytorch_s2s" />   The linear layer alone has 5 million (500 × 10k) weights – ~10 times more weights than the recurrent layer.
|-
|score
|100-long alignment score
|-
|w
|100-long vector attention weight.  These are "soft" weights which changes during the forward pass, in contrast to "hard" neuronal weights that change during the learning phase.
|-
|A
|Attention module – this can be a dot product of recurrent states, or the query-key-value fully-connected layers.  The output is a 100-long vector w.
|-
|H
|500×100. 100 hidden vectors h concatenated into a matrix
|-
|c
|500-long context vector = H * w.  c is a linear combination of h vectors weighted by w.
|}

=== Interpreting Attention weights ===
In translating between languages, alignment is the process of matching words from the source sentence to words of the translated sentence. In the ''I love you'' example above, the second word ''love'' is aligned with the third word ''aime''. Stacking soft row vectors together for ''je'', ''t''', and ''aime'' yields an [[Statistical machine translation#Word alignment|alignment matrix]]:
{| class="wikitable"
!
!I
!love
!you
|-
!je
| style="background-color:black; color:white;" |0.94
|0.02
|0.04
|-
!t'
|0.11
|0.01
| style="background-color:grey ; color:white;" |0.88
|-
!aime
|0.03
| style="background-color:black; color:white;" |0.95
|0.02
|}
Sometimes, alignment can be multiple-to-multiple. For example, the English phrase ''look it up'' corresponds to ''cherchez-le''. Thus, "soft" attention weights work better than "hard" attention weights (setting one attention weight to 1, and the others to 0), as we would like the model to make a context vector consisting of a weighted sum of the hidden vectors, rather than "the best one", as there may not be a best hidden vector.

This view of the attention weights addresses some of the neural network [[Explainable artificial intelligence|explainability]] problem. Networks that perform verbatim translation without regard to word order would show the highest scores along the (dominant) diagonal of the matrix. The off-diagonal dominance shows that the attention mechanism is more nuanced. On the first pass through the decoder, 94% of the attention weight is on the first English word ''I'', so the network offers the word ''je''. On the second pass of the decoder, 88% of the attention weight is on the third English word ''you'', so it offers ''t'''. On the last pass, 95% of the attention weight is on the second English word ''love'', so it offers ''aime''.

== seq2seq ==
=== Problem statement ===
[[File:Seq2seq_with_RNN_and_attention_mechanism.gif|thumb|Animation of seq2seq with RNN and attention mechanism]]

Consider the seq2seq language English-to-French translation task. To be concrete, let us consider the translation of "the zone of international control <end>", which should translate to "la zone de contrôle international <end>". Here, we use the special <end> token as a [[control character]] to delimit the end of input for both the encoder and the decoder.

An input sequence of text <math>x_0, x_1, \dots</math> is processed by a neural network (which can be an LSTM, a Transformer encoder, or some other network) into a sequence of real-valued vectors <math>h_0, h_1, \dots</math>, where <math>h</math> stands for "hidden vector".

After the encoder has finished processing, the decoder starts operating over the hidden vectors, to produce an output sequence <math>y_0, y_1, \dots</math>, autoregressively. That is, it always takes as input both the hidden vectors produced by the encoder, and what the decoder itself has produced before, to produce the next output word:

# (<math>h_0, h_1, \dots</math>, "<start>") → "la"
# (<math>h_0, h_1, \dots</math>, "<start> la") → "la zone"
# (<math>h_0, h_1, \dots</math>, "<start> la zone") → "la zone de"
# ...
# (<math>h_0, h_1, \dots</math>, "<start> la zone de contrôle international") → "la zone de contrôle international <end>"

Here, we use the special <start> token as a [[control character]] to delimit the start of input for the decoder. The decoding terminates as soon as "<end>" appears in the decoder output.

=== Attention weights ===
[[File:Attention_mechanism_output.svg|thumb|Attention mechanism with attention weights, overview.]]As hand-crafting weights defeats the purpose of machine learning, the model must compute the attention weights on its own. Taking analogy from the language of [[Database|database queries]], we make the model construct a triple of vectors: key, query, and value. The rough idea is that we have a "database" in the form of a list of key-value pairs. The decoder sends in a '''query''', and obtains a reply in the form of a weighted sum of the '''values''', where the weight is proportional to how closely the query resembles each '''key'''.

The decoder first processes the "<start>" input partially, to obtain an intermediate vector <math>h^d_0</math>, the 0th hidden vector of decoder. Then, the intermediate vector is transformed by a linear map <math>W^Q</math> into a '''query''' vector <math>q_0 = h_0^d W^Q </math>. Meanwhile, the hidden vectors outputted by the encoder are transformed by another linear map <math>W^K</math> into '''key''' vectors <math>k_0 = h_0 W^K, k_1 = h_1 W^K,  \dots </math>. The linear maps are useful for providing the model with enough freedom to find the best way to represent the data.

Now, the query and keys are compared by taking dot products: <math>q_0 k_0^T, q_0 k_1^T, \dots</math>. Ideally, the model should have learned to compute the keys and values, such that <math>q_0 k_0^T</math> is large, <math>q_0 k_1^T </math> is small, and the rest are very small. This can be interpreted as saying that the attention weight should be mostly applied to the 0th hidden vector of the encoder, a little to the 1st, and essentially none to the rest.

In order to make a properly weighted sum, we need to transform this list of dot products into a probability distribution over <math>0, 1, \dots </math>. This can be accomplished by the [[softmax function]], thus giving us the attention weights:<math display="block">(w_{00}, w_{01}, \dots) = \mathrm{softmax}(q_0 k_0^T, q_0 k_1^T, \dots) </math>This is then used to compute the '''context vector''':<math display="block">c_0 = w_{00} v_0 + w_{01} v_1 + \cdots </math>

where <math>v_0 =  h_0 W^V, v_1 = h_1 W^V , \dots </math> are the '''value''' vectors, linearly transformed by another matrix to provide the model with freedom to find the best way to represent values. Without the matrices <math>W^Q, W^K, W^V </math>, the model would be forced to use the same hidden vector for both key and value, which might not be appropriate, as these two tasks are not the same.[[File:Encoder_cross-attention.png|thumb|Computing the attention weights by dot-product. This is the "decoder cross-attention".]]This is the dot-attention mechanism. The particular version described in this section is "decoder cross-attention", as the output context vector is used by the decoder, and the input keys and values come from the encoder, but the query comes from the decoder, thus "cross-attention".

More succinctly, we can write it as<math display="block">c_0 = \mathrm{Attention}(h_0^d W^Q, HW^K, H W^V) = \mathrm{softmax}((h_0^d W^Q) \; (H W^K)^T) (H W^V) </math>where the matrix <math>H </math> is the matrix whose rows are <math>h_0, h_1, \dots </math>. Note that the querying vector, <math>h_0^d</math>, is not necessarily the same as the key-value vector <math>h_0</math>. In fact, it is theoretically possible for query, key, and value vectors to all be different, though that is rarely done in practice.

== Variants ==
Many variants of attention implement soft weights, such as

* fast weight programmers, or fast weight controllers (1992).<ref name="transform1992" /> A "slow" [[neural network]] outputs the "fast" weights of another neural network through [[outer product]]s. The slow network learns by gradient descent. It was later renamed as "linearized self-attention".<ref name="schlag2021" />
* Bahdanau-style attention,<ref name="bahdanau" /> also referred to as ''additive attention'',
* Luong-style attention,<ref name="xy-dot" /> which is known as ''multiplicative attention'',
* highly parallelizable ''self-attention'' introduced in 2016 as ''decomposable attention''<ref name="parikh" /> and successfully used in [[Transformer (machine learning model)|transformers]] a year later,
* ''positional attention'' and ''factorized positional attention''.<ref name="luo" />

For [[convolutional neural network]]s, attention mechanisms can be distinguished by the dimension on which they operate, namely: spatial attention,<ref name="xzhu1" /> channel attention,<ref name="jhu1" /> or combinations.<ref name="psanghyun1" /><ref name="mgeorgescu" />

Much effort has gone into understand Attention further by studying their roles in focused settings, such as in-context learning,<ref name="zhang2024" /> masked language tasks,<ref name="rende2023" /> stripped down transformers,<ref name="bhe2023" /> bigram statistics,<ref name="timnguyen2024" /> N-gram statistics,<ref name="tcircuits" /> pairwise convolutions,<ref name="algosimple" /> and arithmetic factoring.<ref name="charton2023" /><!--
There are many variants of attention: dot product, query-key-value,<ref name="allyouneed" /> hard, soft, self, cross, Luong,<ref name="xy-dot" /> and Bahdanau<ref name=Bahdanau /> to name a few.
-->

These variants recombine the encoder-side inputs to redistribute those effects to each target output. Often, a correlation-style matrix of dot products provides the re-weighting coefficients.  In the figures below, W is the matrix of context attention weights, similar to the formula in Core Calculations section above.

{| class="wikitable"
|-
! 1. encoder-decoder dot product
! 2. encoder-decoder QKV
! 3. encoder-only dot product
! 4. encoder-only QKV
! 5. Pytorch tutorial
|-
| [[File:attn-xy-dot.png |thumb|Both encoder & decoder are needed to calculate attention.<ref name="xy-dot" />]]
| [[File:attn-xy-qkv.png |thumb|Both encoder & decoder are needed to calculate attention.<ref name="xy-qkv" />]]
| [[File:attn-xx-dot.png |thumb|Decoder is ''not'' used to calculate attention. With only 1 input into corr, W is an auto-correlation of dot products. w<sub>'''ij'''</sub> = x<sub>'''i'''</sub> x<sub>'''j'''</sub>.<ref name="xx-dot" />]]
| [[File:attn-xx-qkv.png |thumb|Decoder is ''not'' used to calculate attention.<ref name="xx-qkv" />]]
| [[File:attn-pytorch-tutorial.png |thumb|A fully-connected layer is used to calculate attention instead of dot product correlation.<ref name="pytorch-tutorial" />]]
|}

{| class="wikitable mw-collapsible mw-collapsed"
|+ Legend
|-
! Label !! Description
|- 
| Variables X, H, S, T || Upper case variables represent the entire sentence, and not just the current word. For example, H is a matrix of the encoder hidden state—one word per column.
|-
| S, T || S, decoder hidden state; T, target word embedding. In the [[Pytorch]] Tutorial variant training phase, T alternates between 2 sources depending on the level of [[teacher forcing]] used. T could be the embedding of the network's output word; i.e. embedding(argmax(FC output)). Alternatively with teacher forcing, T could be the embedding of the known correct word which can occur with a constant forcing probability, say 1/2.
|-
| X, H || H, encoder hidden state; X, input word embeddings.
|-
| W || Attention coefficients
|-
| Qw, Kw, Vw, FC || Weight matrices for query, key, value respectively. FC is a fully-connected weight matrix.
|-
| ⊕, ⊗ || ⊕, vector concatenation; ⊗, matrix multiplication.
|-
| corr || Column-wise softmax(matrix of all combinations of dot products). The dot products are '''x<sub>i</sub> * x<sub>j</sub>''' in variant #3, '''h<sub>i</sub>* s'''<sub>j</sub> in variant 1, and column&nbsp;<sub>'''i'''</sub> ( Kw * H ) * column&nbsp;<sub>'''j'''</sub> ( Qw * S ) in variant 2, and column&nbsp;<sub>'''i'''</sub> ( Kw * X ) * column&nbsp;<sub>'''j'''</sub> ( Qw * X ) in variant 4. Variant 5 uses a fully-connected layer to determine the coefficients. If the variant is QKV, then the dot products are normalized by the {{math|{{radic|d}}}} where {{math|d}} is the height of the QKV matrices.
|}

=== Self-attention ===
[[File:Encoder self-attention, block diagram.png|thumb|Encoder self-attention, block diagram]]

[[File:Encoder self-attention, detailed diagram.png|thumb|Encoder self-attention, detailed diagram]]Self-attention is essentially the same as cross-attention, except that query, key, and value vectors all come from the same model. Both encoder and decoder can use self-attention, but with subtle differences.

For encoder self-attention, we can start with a simple encoder without self-attention, such as an "embedding layer", which simply converts each input word into a vector by a fixed [[lookup table]]. This gives a sequence of hidden vectors <math>h_0, h_1, \dots </math>. These can then be applied to a dot-product attention mechanism, to obtain<math display="block">\begin{aligned}
h_0' &= \mathrm{Attention}(h_0 W^Q, HW^K, H W^V) \\ 
h_1' &= \mathrm{Attention}(h_1 W^Q, HW^K, H W^V) \\
&\cdots
\end{aligned}  </math>or more succinctly, <math>H' = \mathrm{Attention}(H W^Q, HW^K, H W^V)   </math>. This can be applied repeatedly, to obtain a multilayered encoder. This is the "encoder self-attention", sometimes called the "all-to-all attention", as the vector at every position can attend to every other.

=== Masking ===
[[File:Decoder self-attention with causal masking, detailed diagram.png|thumb|Decoder self-attention with causal masking, detailed diagram]]For decoder self-attention, all-to-all attention is inappropriate, because during the autoregressive decoding process, the decoder cannot attend to future outputs that has yet to be decoded. This can be solved by forcing the attention weights <math>w_{ij} = 0 </math> for all <math>i < j </math>, called "causal masking". This attention mechanism is the "causally masked self-attention".

== Optimizations ==

=== Flash attention ===
The size of the attention matrix is proportional to the square of the number of input tokens. Therefore, when the input is long, calculating the attention matrix requires a lot of [[GPU]] memory. Flash attention is an implementation that reduces the memory needs and increases efficiency without sacrificing accuracy. It achieves this by partitioning the attention computation into smaller blocks that fit into the GPU's faster on-chip memory, reducing the need to store large intermediate matrices and thus lowering memory usage while increasing computational efficiency.<ref>{{Cite web |last=Mittal |first=Aayush |date=2024-07-17 |title=Flash Attention: Revolutionizing Transformer Efficiency |url=https://www.unite.ai/flash-attention-revolutionizing-transformer-efficiency/ |access-date=2024-11-16 |website=Unite.AI |language=en-US}}</ref>

== Mathematical representation ==

=== Standard Scaled Dot-Product Attention ===
For matrices: <math>\mathbf{Q}\in\mathbb{R^{m\times d_k}}, \mathbf{K}\in\mathbb{R^{n\times d_k}}</math> and <math>\mathbf{V}\in\mathbb{R^{n\times d_v}}</math>, the scaled dot-product, or '''QKV attention''' is defined as:
<math display="block">
    \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}\right)\mathbf{V}\in\mathbb{R}^{m\times d_v}
</math>
where <math>{}^T</math> denotes [[matrix transpose|transpose]] and the [[softmax function]] is applied independently to every row of its argument. The matrix <math>\mathbf{Q}</math> contains <math>m</math> queries, while matrices <math>\mathbf{K}, \mathbf{V}</math> jointly contain an ''unordered'' set of <math>n</math> key-value pairs. Value vectors in matrix <math>\mathbf{V}</math> are weighted using the weights resulting from the softmax operation, so that the rows of the <math>m</math>-by-<math>d_v</math> output matrix are confined to the [[convex hull]] of the points in <math>\mathbb{R}^{d_v}</math> given by the rows of <math>\mathbf{V}</math>.

To understand the '''permutation invariance''' and '''permutation equivariance''' properties of QKV attention,<ref name="SetTransformer">{{cite arXiv |last1=Lee |first1=Juho |last2=Lee |first2=Yoonho |last3=Kim |first3=Jungtaek |last4=Kosiorek |first4=Adam R |last5=Choi |first5=Seungjin |last6=Teh |first6=Yee Whye |title=Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks |date=2018 |class=cs.LG |eprint=1810.00825 }}</ref> let <math>\mathbf{A}\in\mathbb{R}^{m\times m}</math> and <math>\mathbf{B}\in\mathbb{R}^{n\times n}</math> be [[permutation matrix|permutation matrices]]; and <math>\mathbf{D}\in\mathbb{R}^{m\times n}</math> an arbitrary matrix. The softmax function is '''permutation equivariant''' in the sense that:
:<math>
\text{softmax}(\mathbf{A}\mathbf{D}\mathbf{B}) = \mathbf{A}\,\text{softmax}(\mathbf{D})\mathbf{B}
</math>
By noting that the transpose of a permutation matrix is also its inverse, it follows that:
:<math>
\text{Attention}(\mathbf{A}\mathbf{Q}, \mathbf{B}\mathbf{K}, \mathbf{B}\mathbf{V}) = \mathbf{A}\,\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V})
</math>
which shows that QKV attention is [[equivariance|equivariant]] with respect to re-ordering the queries (rows of <math>\mathbf{Q}</math>); and [[Invariant (mathematics)|invariant]] to re-ordering of the key-value pairs in <math>\mathbf{K},\mathbf{V}</math>. These properties are inherited when applying linear transforms to the inputs and outputs of QKV attention blocks. For example, a simple '''self-attention''' function defined as:
:<math>
\mathbf{X}\mapsto\text{Attention}(\mathbf{X}\mathbf{T}_q, \mathbf{X}\mathbf{T}_k, \mathbf{X}\mathbf{T}_v)
</math>  
is permutation equivariant with respect to re-ordering the rows of the input matrix <math>X</math> in a non-trivial way, because every row of the output is a function of all the rows of the input. Similar properties hold for ''multi-head attention'', which is defined below.

=== Masked Attention ===
When QKV attention is used as a building block for an autoregressive decoder, and when at training time all input and output matrices have <math>n</math> rows, a '''masked attention''' variant is used:
<math display="block">
    \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}+\mathbf{M}\right)\mathbf{V}
</math>
where the mask, <math>\mathbf{M}\in\mathbb{R}^{n\times n}</math> is a [[triangular matrix|strictly upper triangular matrix]], with zeros on and below the diagonal and <math>-\infty</math> in every element above the diagonal. The softmax output, also in <math>\mathbb{R}^{n\times n}</math> is then ''lower triangular'', with zeros in all elements above the diagonal. The masking ensures that for all <math>1\le i<j\le n</math>, row <math>i</math> of the attention output is independent of row <math>j</math> of any of the three input matrices. The permutation invariance and equivariance properties of standard QKV attention do not hold for the masked variant.

=== Multi-Head Attention ===

[[File:Encoder cross-attention, multiheaded version.png|thumb|Decoder multiheaded cross-attention]]
Multi-head attention
<math display="block">
    \text{MultiHead}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{Concat}(\text{head}_1, ..., \text{head}_h)\mathbf{W}^O
</math>
where each head is computed with QKV attention as:
<math display="block">
    \text{head}_i = \text{Attention}(\mathbf{Q}\mathbf{W}_i^Q, \mathbf{K}\mathbf{W}_i^K, \mathbf{V}\mathbf{W}_i^V)
</math>
and <math>\mathbf{W}_i^Q, \mathbf{W}_i^K, \mathbf{W}_i^V</math>, and <math>\mathbf{W}^O</math> are parameter matrices.

The permutation properties of (standard, unmasked) QKV attention apply here also. For permutation matrices, <math>\mathbf{A}, \mathbf{B}</math>:
:<math>
\text{MultiHead}(\mathbf{A}\mathbf{Q}, \mathbf{B}\mathbf{K}, \mathbf{B}\mathbf{V}) = \mathbf{A}\,\text{MultiHead}(\mathbf{Q}, \mathbf{K}, \mathbf{V})
</math>
from which we also see that '''multi-head self-attention''':
:<math>
\mathbf{X}\mapsto\text{MultiHead}(\mathbf{X}\mathbf{T}_q, \mathbf{X}\mathbf{T}_k, \mathbf{X}\mathbf{T}_v)
</math>  
is equivariant with respect to re-ordering of the rows of input matrix <math>X</math>.

=== Bahdanau (Additive) Attention ===
<math display="block">
    \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}(\tanh(\mathbf{W}_Q\mathbf{Q} + \mathbf{W}_K\mathbf{K})\mathbf{V})
</math>
where <math>\mathbf{W}_Q</math> and <math>\mathbf{W}_K</math> are learnable weight matrices.<ref name=bahdanau />

=== Luong Attention (General) ===
<math display="block">
    \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}(\mathbf{Q}\mathbf{W}\mathbf{K}^T)\mathbf{V}
</math>
where <math>\mathbf{W}</math> is a learnable weight matrix.<ref name="xy-dot" />

== See also ==
* [[Recurrent neural network]]
* [[seq2seq]]
* [[Transformer (deep learning architecture)]]
* [[Attention]]
* [[Dynamic neural network]]

== References ==
{{Reflist|refs=
<ref name="pytorch_s2s">{{ cite web | url = https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html | title = Pytorch.org seq2seq tutorial | access-date=December 2, 2021 }}</ref>

<ref name=PDP>{{Cite book| publisher = MIT Press | isbn = 978-0-262-68053-0 |editor1-last=Rumelhart | editor1-first = David E. |editor2-last=Hinton |editor2-first=G. E. | editor3=PDP Research Group  |last1=Rumelhart | first1 = David E. |last2=Hinton |first2=G. E. | last3 = Mcclelland| first3 = James L. |chapter=A General Framework for Parallel Distributed Processing | title = Parallel Distributed Processing, Volume 1: Explorations in the Microstructure of Cognition: Foundations | location = Cambridge, Massachusetts | date = 1987-07-29|chapter-url = https://stanford.edu/~jlmcc/papers/PDP/Chapter2.pdf| language = en}}</ref>

<ref name="transform1992">{{Cite journal |last1=Schmidhuber|first1=Jürgen|author-link1=Jürgen Schmidhuber|date=1992|title=Learning to control fast-weight memories: an alternative to recurrent nets.|journal=Neural Computation|volume=4|issue=1 |pages=131–139|doi=10.1162/neco.1992.4.1.131 |s2cid=16683347 }}</ref>

<ref name="schlag2021">
{{Cite conference | last1=Schlag|first1=Imanol| last2=Irie|first2=Kazuki| last3=Schmidhuber|first3=Jürgen|author-link1=Juergen Schmidhuber|title=  Linear Transformers Are Secretly Fast Weight Programmers|publisher=Springer|date=2021 |pages=9355–9366 |book-title= ICML 2021}}</ref>

<ref name="timnguyen2024">
{{ cite arXiv |eprint=2407.12034 |last1=Nguyen |first1=Timothy |title=Understanding Transformers via N-gram Statistics |date=2024 |class=cs.CL }}</ref>

<ref name=Graves2016>{{Cite journal | date=2016-10-12|title=Hybrid computing using a neural network with dynamic external memory|journal=Nature|language=en|volume=538|issue=7626|doi=10.1038/nature20101|issn=1476-4687|pages=471–476|pmid=27732574|bibcode=2016Natur.538..471G|s2cid=205251479|url=https://ora.ox.ac.uk/objects/uuid:dd8473bd-2d70-424d-881b-86d9c9c66b51|last1=Graves|first1=Alex|last2=Wayne|first2=Greg|last3=Reynolds|first3=Malcolm|last4=Harley|first4=Tim|last5=Danihelka|first5=Ivo|last6=Grabska-Barwińska|first6=Agnieszka|last7=Colmenarejo|first7=Sergio Gómez|last8=Grefenstette|first8=Edward|last9=Ramalho|first9=Tiago|last10=Agapiou|first10=John|last11=Badia|first11=Adrià Puigdomènech|last12=Hermann|first12=Karl Moritz|last13=Zwols|first13=Yori|last14=Ostrovski|first14=Georg|last15=Cain|first15=Adam|last16=King|first16=Helen|last17=Summerfield|first17=Christopher|last18=Blunsom|first18=Phil|last19=Kavukcuoglu|first19=Koray|last20=Hassabis|first20=Demis}}</ref>

<!-- <ref name=allyouneed>{{cite journal |last1=Vaswani |first1=Ashish |author1-link= Ashish Vaswani |last2=Shazeer |first2=Noam |last3=Parmar |first3=Niki |last4=Uszkoreit |first4=Jakob |last5=Jones |first5=Llion |last6=Gomez |first6=Aidan N  |author6-link= Aidan Gomez |last7=Kaiser |first7=Łukasz |last8=Polosukhin |first8=Illia |title=Attention is All you Need |journal=Advances in Neural Information Processing Systems |date=2017 |volume=30 |url=https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf |publisher=Curran Associates, Inc.}}</ref> -->

<ref name=bdritz2017>
{{cite arXiv|last1=Britz|first1=Denny|last2=Goldie|first2=Anna|last3=Luong|first3=Minh-Thanh|last4=Le|first4=Quoc|date=2017-03-21|title=Massive Exploration of Neural Machine Translation Architectures|class=cs.CV|eprint=1703.03906}}</ref>

<ref name=xy-dot>
{{ cite arXiv|last1=Luong|first1=Minh-Thang| date=2015-09-20 |title=Effective Approaches to Attention-Based Neural Machine Translation |class=cs.CL| eprint=1508.04025v5}}</ref>

<ref name=xy-qkv>
{{cite AV media |people = Neil Rhodes |date=2021 |title= CS 152 NN—27: Attention: Keys, Queries, & Values | url=https://www.youtube.com/watch?v=rA28vBqN4RM  | access-date=2021-12-22 | time=06:30 }}</ref>

<ref name=xx-dot>
{{cite AV media |people = Alfredo Canziani & Yann Lecun |date=2021 |title= NYU Deep Learning course, Spring 2020 | url=https://www.youtube.com/watch?v=f01J0Dri-6k | access-date=2021-12-22 | time=05:30 }}</ref>

<ref name=xx-qkv>
{{cite AV media |people = Alfredo Canziani & Yann Lecun |date=2021 |title= NYU Deep Learning course, Spring 2020 | url=https://www.youtube.com/watch?v=f01J0Dri-6k | access-date=2021-12-22 | time=20:15 }}</ref>

<ref name=pytorch-tutorial>
{{ cite web|url=https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html | title = NLP From Scratch: Translation With a Sequence To Sequence Network and Attention | last=Robertson|first=Sean|  website=pytorch.org|access-date=2021-12-22 }}</ref>

<ref name=bahdanau>
{{cite arXiv |last1=Bahdanau |first1=Dzmitry |last2=Cho |first2=Kyunghyun |last3=Bengio |first3=Yoshua |title=Neural Machine Translation by Jointly Learning to Align and Translate |date=2014 |class=cs.CL |eprint=1409.0473}}</ref>

<ref name=psanghyun1>
{{Cite arXiv |last1=Woo |first1=Sanghyun |last2=Park |first2=Jongchan |last3=Lee |first3=Joon-Young |last4=Kweon |first4=In So |date=2018-07-18 |title=CBAM: Convolutional Block Attention Module |class=cs.CV |eprint=1807.06521 }}</ref>

<ref name=mgeorgescu>
{{Cite arXiv|last1=Georgescu |first1=Mariana-Iuliana |last2=Ionescu |first2=Radu Tudor |last3=Miron |first3=Andreea-Iuliana |last4=Savencu |first4=Olivian |last5=Ristea |first5=Nicolae-Catalin |last6=Verga |first6=Nicolae |last7=Khan |first7=Fahad Shahbaz |date=2022-10-12 |title=Multimodal Multi-Head Convolutional Attention with Various Kernel Sizes for Medical Image Super-Resolution |class=eess.IV |eprint=2204.04218 }}</ref>

<ref name=jhu1>
{{Cite book |last1=Hu |first1=Jie |last2=Shen |first2=Li |last3=Sun |first3=Gang |title=2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition |chapter=Squeeze-and-Excitation Networks |year=2018 |chapter-url=https://ieeexplore.ieee.org/document/8578843 |pages=7132–7141 |doi=10.1109/CVPR.2018.00745|arxiv=1709.01507 |isbn=978-1-5386-6420-9 |s2cid=206597034 }}</ref>

<ref name=xzhu1>
{{Cite book |last1=Zhu |first1=Xizhou |last2=Cheng |first2=Dazhi |last3=Zhang |first3=Zheng |last4=Lin |first4=Stephen |last5=Dai |first5=Jifeng |title=2019 IEEE/CVF International Conference on Computer Vision (ICCV) |chapter=An Empirical Study of Spatial Attention Mechanisms in Deep Networks |year=2019 |chapter-url=https://ieeexplore.ieee.org/document/9009578 |pages=6687–6696 |doi=10.1109/ICCV.2019.00679|arxiv=1904.05873 |isbn=978-1-7281-4803-8 |s2cid=118673006 }}</ref>

<ref name="parikh2">{{Cite book |last1=Parikh |first1=Ankur |last2=Täckström |first2=Oscar |last3=Das |first3=Dipanjan |last4=Uszkoreit |first4=Jakob |chapter=A Decomposable Attention Model for Natural Language Inference |date=2016 |pages=2249–2255 |title=Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing |chapter-url=http://dx.doi.org/10.18653/v1/d16-1244 |location=Stroudsburg, PA, USA |publisher=Association for Computational Linguistics |doi=10.18653/v1/d16-1244|arxiv=1606.01933 }}</ref>

<ref name=luo>
{{Cite web|url=https://www.catalyzex.com/paper/learning-positional-attention-for-sequential|title=Learning Positional Attention for Sequential Recommendation
|website=catalyzex.com}}</ref>

<ref name=zhang2024>{{cite journal |last1=Zhang |first1=Ruiqi |title= Trained Transformers Learn Linear Models In-Context
 |journal=Journal of Machine Learning Research 1-55 |date=2024 |volume=25 |arxiv=2306.09927 |url=https://jmlr.org/papers/volume25/23-1042/23-1042.pdf}}</ref>

<ref name="rende2023">{{cite journal|arxiv=2304.07235|last1=Rende |first1=Riccardo |title=Mapping of attention mechanisms to a generalized Potts model |journal=Physical Review Research |year=2024 |volume=6 |issue=2 |page=023057 |doi=10.1103/PhysRevResearch.6.023057 |bibcode=2024PhRvR...6b3057R }}</ref>

<ref name="bhe2023">{{cite arXiv|eprint=2311.01906|last1=He |first1=Bobby |title=Simplifying Transformers Blocks |year=2023 |class=cs.LG }}</ref>

<ref name=tcircuits>{{ cite web|url=https://transformer-circuits.pub | title = Transformer Circuits |website=transformer-circuits.pub }}</ref>

<ref name="charton2023">{{cite arXiv|eprint=2308.15594|last1=Charton |first1=François |title=Learning the Greatest Common Divisor: Explaining Transformer Predictions |year=2023 |class=cs.LG }}</ref>

<ref name=algosimple>{{cite AV media |date=2023 |title=Transformer Neural Network Derived From Scratch | url=https://www.youtube.com/watch?v=kWLed8o5M2Y&t=330s | access-date=2024-04-07 | time=05:30 }}</ref>

}}

== External links ==

* {{cite journal |last1=Olah |first1=Chris |last2=Carter |first2=Shan |date=September 8, 2016 |title=Attention and Augmented Recurrent Neural Networks |url=https://distill.pub/2016/augmented-rnns/ |journal=Distill |publisher=Distill Working Group |volume=1 |issue=9 |doi=10.23915/distill.00001}}
* [[Dan Jurafsky]] and James H. Martin (2022) [https://web.stanford.edu/~jurafsky/slp3/ ''Speech and Language Processing'' (3rd ed. draft, January 2022)], ch. 10.4 Attention and ch. 9.7 Self-Attention Networks: Transformers
* [[Alex Graves (computer scientist)|Alex Graves]] (4 May 2020), [https://www.youtube.com/watch?v=AIiwuClvH6k&vl=en-GB Attention and Memory in Deep Learning] (video lecture), [[DeepMind]] / [[University College London|UCL]], via YouTube

{{Artificial intelligence navbox}}

[[Category:Machine learning]]