View source for Dilution (neural networks)

{{Disputed|date=April 2020}}
[[File:Dropout_mechanism.png|thumb|On the left is a fully connected neural network with two hidden layers. On the right is the same network after applying dropout.]]
'''Dilution''' and '''dropout''' (also called '''DropConnect'''<ref>{{Cite journal|last1=Wan|first1=Li|last2=Zeiler|first2=Matthew|last3=Zhang|first3=Sixin|last4=Le Cun|first4=Yann|last5=Fergus|first5=Rob|date=2013|title=Regularization of Neural Networks using DropConnect|url=https://proceedings.mlr.press/v28/wan13.html|journal=Proceedings of the 30th International Conference on Machine Learning, PMLR|volume=28|issue=3|pages=1058–1066|via=PMLR}}</ref>) are [[Regularization (mathematics)|regularization]] techniques for reducing [[overfitting]] in [[artificial neural network]]s by preventing complex co-adaptations on [[Training, validation, and test sets|training data]]. They are an efficient way of performing model averaging with neural networks.<ref name="MyUser_Arxiv.org_July_26_2015c">{{cite arXiv |eprint=1207.0580|title= Improving neural networks by preventing co-adaptation of feature detectors |last1= Hinton |first1=Geoffrey E. |last2=Srivastava |first2=Nitish |last3=Krizhevsky |first3=Alex |last4=Sutskever |first4=Ilya |last5= Salakhutdinov |first5=Ruslan R. |class=cs.NE |year=2012 }}</ref> ''Dilution'' refers to thinning weights,<ref name=":1" /> while ''dropout'' refers to randomly "dropping out", or omitting, units (both hidden and visible) during the training process of a neural network.<ref name="MyUser_Jmlr.org_July_26_2015c">{{cite web |url=http://jmlr.org/papers/v15/srivastava14a.html |title=Dropout: A Simple Way to Prevent Neural Networks from Overfitting |newspaper=Jmlr.org |date=  |author= |accessdate= July 26, 2015}}</ref><ref>{{cite arXiv|last1=Warde-Farley|first1=David|last2=Goodfellow|first2=Ian J.|last3=Courville|first3=Aaron|last4=Bengio|first4=Yoshua|date=2013-12-20|title=An empirical analysis of dropout in piecewise linear networks|eprint=1312.6197|class=stat.ML}}</ref><ref name="MyUser_Arxiv.org_July_26_2015c" /> Both trigger the same type of regularization.

== Types and uses ==
Dilution is usually split in ''weak dilution'' and ''strong dilution''. Weak dilution describes the process in which the finite fraction of removed connections is small, and strong dilution refers to when this fraction is large. There is no clear distinction on where the limit between strong and weak dilution is, and often the distinction is dependent on the precedent of a specific use-case and has implications for how to solve for exact solutions.

Sometimes dilution is used for adding damping noise to the inputs. In that case, weak dilution refers to adding a small amount of damping noise, while strong dilution refers to adding a greater amount of damping noise. Both can be rewritten as variants of weight dilution.

These techniques are also sometimes referred to as random pruning of weights, but this is usually a non-recurring one-way operation. The network is pruned, and then kept if it is an improvement over the previous model. Dilution and dropout both refer to an iterative process. The pruning of weights typically does not imply that the network continues learning, while in dilution/dropout, the network continues to learn after the technique is applied.

== Generalized linear network ==

Output from a layer of linear nodes, in an artificial neural net can be described as

{{NumBlk|:|<math>y_i = \sum_j w_{ij} x_j</math>|{{EquationRef|1}}}}

:* <math>y_i</math> – output from node <math>i</math>
:* <math>w_{ij}</math> – real weight before dilution, also called the Hebb connection strength
:* <math>x_j</math> – input from node <math>j</math>

This can be written in vector notation as

{{NumBlk|:|<math>\mathbf{y} = \mathbf{W} \mathbf{x}</math>|{{EquationRef|2}}}}

:* <math>\mathbf{y}</math> – output vector
:* <math>\mathbf{W}</math> – weight matrix
:* <math>\mathbf{x}</math> – input vector

Equations (1) and (2) are used in the subsequent sections.

== Weak dilution ==
During weak dilution, the finite fraction of removed connections (the weights) is small, giving rise to a tiny uncertainty. This edge-case can be solved exactly with [[mean field theory]]. In weak dilution the impact on the weights can be described as

{{NumBlk|:|<math>\hat{w_{ij}} = 
\begin{cases} 
w_{ij},  & \mbox{with } P(c) \\
0, & \mbox{otherwise}
\end{cases}</math>|{{EquationRef|3}}}}

:*<math>\hat{w_{ij}}</math> – diluted weight
:* <math>w_{ij}</math> – real weight before dilution
:* <math>P(c)</math> – the probability of <math>c</math>, the probability of keeping a weight

The interpretation of probability <math>P(c)</math> can also be changed from keeping a weight into pruning a weight.

In vector notation this can be written as

{{NumBlk|:|<math>\hat{\mathbf{W}} = \operatorname{g} \left ( \mathbf{W}, c \right )</math>|{{EquationRef|4}}}}

where the function <math>\operatorname{g} ( \cdot )</math> imposes the previous dilution.

In weak dilution only a small and fixed fraction of the weights are diluted. When the number of terms in the sum goes to infinite (the weights for each node) it is still infinite (the fraction is fixed), thus [[mean field theory]] can be applied. In the notation from Hertz et al.<ref name=":1" /> this would be written as

{{NumBlk|:|<math>\left \langle h_i \right \rangle = c \sum_j w_{ij} \left \langle S_j \right \rangle</math>|{{EquationRef|5}}}}

:* <math>\left \langle h_i \right \rangle</math> the mean field temperature
:* <math>c</math> – a scaling factor for the temperature from the probability of keeping the weight
:* <math>w_{ij}</math> – real weight before dilution, also called the Hebb connection strength
:* <math>\left \langle S_j \right \rangle</math> – the mean stable equilibrium states

There are some assumptions for this to hold, which are not listed here.<ref>{{Citation|last=Sompolinsky|first=H.|title=The theory of neural networks: The Hebb rule and beyond|date=1987|work=Heidelberg Colloquium on Glassy Dynamics|series=Lecture Notes in Physics|volume=275|pages=485–527|place=Berlin, Heidelberg|publisher=Springer Berlin Heidelberg|doi=10.1007/bfb0057531|bibcode=1987LNP...275..485S|isbn=978-3-540-17777-7}}</ref><ref>{{Cite journal|last1=Canning|first1=A|last2=Gardner|first2=E|date=1988-08-07|title=Partially connected models of neural networks|journal=Journal of Physics A: Mathematical and General|volume=21|issue=15|pages=3275–3284|doi=10.1088/0305-4470/21/15/016|bibcode=1988JPhA...21.3275C|issn=0305-4470}}</ref>

== Strong dilution ==
When the dilution is strong, the finite fraction of removed connections (the weights) is large, giving rise to a huge uncertainty.

== Dropout ==

Dropout is a special case of the previous weight equation ({{EquationNote|3}}), where the aforementioned equation is adjusted to remove a whole row in the vector matrix, and not only random weights

{{NumBlk|:|<math>\hat{\mathbf{w}_{j}} = 
\begin{cases} 
\mathbf{w}_{j},  & \mbox{with } P(c) \\
\mathbf{0}, & \mbox{otherwise}
\end{cases}</math>|{{EquationRef|6}}}}

:* <math>P(c)</math> – the probability <math>c</math> to keep a row in the weight matrix
:* <math>\mathbf{w}_{j}</math> – real row in the weight matrix before dropout
:* <math>\hat{\mathbf{w}_{j}}</math> – diluted row in the weight matrix

Because dropout removes a whole row from the vector matrix, the previous (unlisted) assumptions for weak dilution and the use of mean field theory are not applicable.

The process by which the node is driven to zero, whether by setting the weights to zero, by “removing the node”, or by some other means, does not impact the end result and does not create a new and unique case. If the neural net is processed by a high-performance digital array-multiplicator, then it is likely more effective to drive the value to zero late in the process graph. If the net is processed by a constrained processor, perhaps even an analog neuromorphic processor, then it is likely a more power-efficient solution is to drive the value to zero early in the process graph.

== Google's patent ==
Although there have been examples of randomly removing connections between [[Artificial neuron|neurons]] in a neural network to improve models,<ref name=":1">{{Cite book|last1=Hertz|first1=John|title=Introduction to the Theory of Neural Computation|last2=Krogh|first2=Anders|last3=Palmer|first3=Richard|publisher=Addison-Wesley Pub. Co.|year=1991|isbn=0-201-51560-1|location=Redwood City, California|pages=45–46}}</ref> this technique was first introduced with the name ''dropout'' by [[Geoffrey Hinton]], et al. in 2012.<ref name="MyUser_Arxiv.org_July_26_2015c" /> [[Google]] currently holds the patent for the dropout technique.<ref name="pat">{{cite patent|country=US|number=9406017B2|title=System and method for addressing overfitting in a neural network|status=|pubdate=2016-08-02|gdate=2016-08-02|invent1=Hinton|inventor1-first=Geoffrey E.|url=https://patents.google.com/patent/US9406017B2/en}}</ref><ref group="note">The patent is most likely not valid due to previous art. “Dropout” has been described as “dilution” in previous publications. It is described by Hertz, Krogh, and Palmer in ''Introduction to the Theory of Neural Computation'' (1991) {{ISBN|0-201-51560-1}}, pp. 45, ''Weak Dilution''. The text references Sompolinsky ''The Theory of Neural Networks: The Hebb Rules and Beyond'' in ''Heidelberg Colloquium on Glossy Dynamics'' (1987) and Canning and Gardner ''Partially Connected Models of Neural Networks'' in ''Journal of Physics'' (1988). It goes on to describe strong dilution. This predates Hinton's paper.</ref>

==See also==
* [[AlexNet]]
* {{slink|Convolutional neural network|Dropout}}

==Notes==
<references group="note"/>

==References==
{{Reflist}}

[[Category:Deep learning]]