View source for GloVe

{{Short description|Algorithm for obtaining vector representations of words}}
{{distinguish|Glove (disambiguation){{!}}Glove}}
'''GloVe''', coined from Global Vectors, is a model for distributed word representation. The model is an [[unsupervised learning]] algorithm for obtaining [[Word embedding|vector representation]]s for words. This is achieved by mapping words into a meaningful space where the distance between words is related to semantic similarity.<ref name=":1">{{Cite journal |last=Pennington |first=Jeffrey |last2=Socher |first2=Richard |last3=Manning |first3=Christopher |date=October 2014 |editor-last=Moschitti |editor-first=Alessandro |editor2-last=Pang |editor2-first=Bo |editor3-last=Daelemans |editor3-first=Walter |title=GloVe: Global Vectors for Word Representation |url=https://aclanthology.org/D14-1162 |journal=Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) |location=Doha, Qatar |publisher=Association for Computational Linguistics |pages=1532–1543 |doi=10.3115/v1/D14-1162}}</ref> Training is performed on aggregated global word-word [[co-occurrence]] [[statistics]] from a corpus, and the resulting representations showcase interesting linear substructures of the [[word vector space]]. As log-bilinear regression model for unsupervised learning of word representations, it combines the features of two model families, namely the global matrix factorization and local context window methods.

It is developed as an [[Open-source software|open-source]] project at [[Stanford University|Stanford]]<ref>[https://www.aclweb.org/anthology/D14-1162 ''GloVe: Global Vectors for Word Representation'' (pdf)] {{Webarchive|url=https://web.archive.org/web/20200903022100/https://www.aclweb.org/anthology/D14-1162/|date=2020-09-03}} "We use our insights to construct a new model for word representation which we call GloVe, for Global Vectors, because the global corpus statistics are captured directly by the model."</ref> and was launched in 2014. It was designed as a competitor to [[word2vec]], and the original paper noted multiple improvements of GloVe over word2vec. {{As of|2022}}, both approaches are outdated, and [[Transformer (deep learning architecture)|Transformer]]-based models, such as [[BERT (language model)|BERT]], which add multiple neural-network attention layers on top of a word embedding model similar to Word2vec, have come to be regarded as the state of the art in NLP.<ref>{{Cite journal |last1=Von der Mosel |first1=Julian |last2=Trautsch |first2=Alexander |last3=Herbold |first3=Steffen |date=2022 |title=On the validity of pre-trained transformers for natural language processing in the software engineering domain |url=https://ieeexplore.ieee.org/document/9785808 |journal=IEEE Transactions on Software Engineering |volume=49 |issue=4 |pages=1487–1507 |arxiv=2109.04738 |doi=10.1109/TSE.2022.3178469 |issn=1939-3520 |s2cid=237485425}}</ref>

== Definition ==
<blockquote>You shall know a word by the company it keeps (Firth, J. R. 1957:11)<ref>{{Cite book |last=Firth |first=J. R. |url=https://cs.brown.edu/courses/csci2952d/readings/lecture1-firth.pdf |title=Studies in Linguistic Analysis |publisher=Wiley-Blackwell |year=1957}}</ref></blockquote>The idea of GloVe is to construct, for each word <math>i</math>, two vectors <math>w_i, \tilde w_i</math>, such that the relative positions of the vectors capture part of the statistical regularities of the word <math>i</math>. The statistical regularity is defined as the co-occurrence probabilities. Words that resemble each other in meaning should also resemble each other in co-occurrence probabilities.

=== Word counting ===
Let the '''vocabulary''' be <math>V</math>, the set of all possible words (aka "tokens"). Punctuation is either ignored, or treated as vocabulary, and similarly for capitalization and other typographical details.<ref name=":1" />

If two words occur close to each other, then we say that they occur in the '''context''' of each other. For example, if the context length is 3, then we say that in the following sentence<blockquote>GloVe<sub>1</sub>, coined<sub>2</sub> from<sub>3</sub> Global<sub>4</sub> Vectors<sub>5</sub>, is<sub>6</sub> a<sub>7</sub> model<sub>8</sub> for<sub>9</sub> distributed<sub>10</sub> word<sub>11</sub> representation<sub>12</sub></blockquote>the word "model<sub>8</sub>" is in the context of "word<sub>11</sub>" but not the context of "representation<sub>12</sub>".

A word is not in the context of itself, so "model<sub>8</sub>" is not in the context of the word "model<sub>8</sub>", although, if a word appears again in the same context, then it does count.

Let <math>X_{ij}</math> be the number of times that the word <math>j</math> appears in the context of the word <math>i</math> over the entire corpus. For example, if the corpus is just "I don't think that that is a problem." we have <math>X_{\text{that}, \text{that}} = 2</math> since the first "that" appears in the second one's context, and vice versa.

Let <math>X_i = \sum_{j \in V} X_{ij}</math> be the number of words in the context of all instances of word <math>i</math>. By counting, we have<math display="block">X_i = 2 \times(\text{context size}) \times \# (\text{occurrences of word }i)</math>(except for words occurring right at the start and end of the corpus)

=== Probabilistic modelling ===
Let <math display="block">P_{ik} := P(k | i) := \frac{X_{ik}}{X_{i}}</math>be the '''co-occurrence probability'''. That is, if one samples a random occurrence of the word <math>i</math> in the entire document, and a random word within its context, that word is <math>k</math> with probability <math>P_{ik}</math>. Note that <math>P_{ik} \neq P_{ki}</math> in general. For example, in a typical modern English corpus, <math>P_{\text{ado}, \text{much}}</math> is close to one, but <math>P_{\text{much}, \text{ado}}</math> is close to zero. This is because the word "ado" is almost only used in the context of the archaic phrase "[[Much Ado About Nothing|much ado about]]", but the word "much" occurs in all kinds of contexts.

For example, in a 6 billion token corpus, we have

{| class="wikitable"
|+ Table 1 of <ref name=":1" />
! Probability and Ratio 
! <math>k=\text{ solid }</math>
! <math>k=\text{ gas }</math>
! <math>k=\text{ water }</math>
! <math>k=\text{ fashion }</math>
|-
| <math>P(k \mid \text { ice })</math> 
| <math>1.9 \times 10^{-4}</math>
| <math>6.6 \times 10^{-5}</math>
| <math>3.0 \times 10^{-3}</math>
| <math>1.7 \times 10^{-5}</math>
|-
| <math>P(k \mid \text { steam })</math>
| <math>2.2 \times 10^{-5}</math>
| <math>7.8 \times 10^{-4}</math>
| <math>2.2 \times 10^{-3}</math>
| <math>1.8 \times 10^{-5}</math>
|-
| <math>P(k \mid \text { ice }) / P(k \mid \text { steam })</math>
| <math>8.9</math>
| <math>8.5 \times 10^{-2}</math>
| <math>1.36</math>
| <math>0.96</math>
|}
Inspecting the table, we see that the words "ice" and "steam" are indistinguishable along the "water" (often co-occurring with both) and "fashion" (rarely co-occurring with either), but distinguishable along the "solid" (co-occurring more with ice) and "gas" (co-occurring more with "steam").

The idea is to learn two vectors <math>w_i, \tilde w_i</math> for each word <math>i</math>, such that we have a [[multinomial logistic regression]]:<math display="block">w_i^T \tilde w_j + b_i + \tilde b_j \approx \ln P_{ij}</math>and the terms <math>b_i, \tilde b_j</math> are unimportant parameters.

This means that if the words <math>i, j</math> have similar co-occurrence probabilities <math>(P_{ik})_{k \in V} \approx (P_{jk})_{k \in V} </math>, then their vectors should also be similar: <math>w_i \approx w_j</math>.

=== Logistic regression ===
Naively, logistic regression can be run by minimizing the squared loss:<math display="block">L = \sum_{i, j \in V} (w_i^T \tilde w_j + b_i + \tilde b_j - \ln P_{ij})^2</math>However, this would be noisy for rare co-occurrences. To fix the issue, the squared loss is weighted so that the loss is slowly ramped-up as the absolute number of co-occurrences <math>X_{ij}</math> increases:<math display="block">L = \sum_{i, j \in V} f(X_{ij}) (w_i^T \tilde w_j + b_i + \tilde b_j - \ln P_{ij})^2
 </math>where<math display="block">f(x)=\left\{\begin{array}{cc}
\left(x / x_{\max }\right)^\alpha & \text { if } x<x_{\max } \\
1 & \text { otherwise }
\end{array}\right.</math>and <math>x_{\max }, \alpha</math> are [[Hyperparameter (machine learning)|hyperparameters]]. In the original paper, the authors found that <math>x_{\max } = 100, \alpha = 3/4</math> seem to work well in practice.

=== Use ===
Once a model is trained, we have 4 trained parameters for each word: <math>w_i, \tilde w_i, b_i, \tilde b_i</math>. The parameters <math> b_i, \tilde b_i</math> are irrelevant, and only <math>w_i, \tilde w_i</math> are relevant.

The authors recommended using <math>w_i + \tilde w_i</math> as the final representation vector for word <math>i</math>, because empirically it worked better than <math>w_i</math> or <math>\tilde w_i</math> alone.

== Applications ==
GloVe can be used to find relations between words like synonyms, company-product relations, zip codes and cities, etc. However, the unsupervised learning algorithm is not effective in identifying homographs, i.e., words with the same spelling and different meanings. This is as the unsupervised learning algorithm calculates a single set of vectors for words with the same morphological structure.<ref>{{cite journal |last=Wenig |first= Phillip |date= 2019 |title=Creation of Sentence Embeddings Based on Topical Word Representations: An approach towards universal language understanding|journal=Towards Data Science }}</ref> The algorithm is also used by the [[SpaCy]] library to build semantic word embedding features, while computing the top list words that match with distance measures such as [[cosine similarity]] and [[Euclidean distance]] approach.<ref>{{Cite book|title=Advances in Computing and Data Sciences: Second International Conference, ICACDS 2018, Dehradun, India, April 20-21, 2018, Revised Selected Papers|last=Singh|first=Mayank|last2=Gupta|first2=P. K.|last3=Tyagi|first3=Vipin|last4=Flusser|first4=Jan|last5=Ören|first5=Tuncer I.|publisher=Springer|year=2018|isbn=9789811318122|location=Singapore|pages=171}}</ref> GloVe was also used as the word representation framework for the online and offline systems designed to detect psychological distress in patient interviews.<ref name=":0">{{Cite book |last=Abad |first=Alberto |title=Advances in Speech and Language Technologies for Iberian Languages: Third International Conference, IberSPEECH 2016, Lisbon, Portugal, November 23-25, 2016, Proceedings |last2=Ortega |first2=Alfonso |last3=Teixeira |first3=António |last4=Mateo |first4=Carmen |last5=Hinarejos |first5=Carlos |last6=Perdigão |first6=Fernando |last7=Batista |first7=Fernando |last8=Mamede |first8=Nuno |publisher=Springer |year=2016 |isbn=9783319491691 |location=Cham |pages=165 }}</ref>

== See also ==
* [[ELMo]]
* [[BERT (language model)|BERT]]
* [[Word2vec]]
* [[fastText]]
* [[Natural language processing]]

==References==
{{reflist}}

==External links==
* [http://nlp.stanford.edu/projects/glove/ GloVe] {{Webarchive|url=https://web.archive.org/web/20161219132150/http://nlp.stanford.edu/projects/glove/ |date=2016-12-19 }}
*[https://deeplearning4j.org/api/latest/org/deeplearning4j/models/glove/Glove.html Deeplearning4j GloVe] {{Webarchive|url=https://web.archive.org/web/20190202154323/https://deeplearning4j.org/api/latest/org/deeplearning4j/models/glove/Glove.html |date=2019-02-02 }}

{{Natural Language Processing}}
{{Artificial intelligence navbox}}

[[Category:Computational linguistics]]
[[Category:Natural language processing toolkits]]