Smooth maximum

In mathematics, a smooth maximum of an indexed family x₁, ..., x_n of numbers is a smooth approximation to the maximum function $\max (x_1,\dots ,x_n),$ meaning a parametric family of functions $m_{\alpha }(x_1,\dots ,x_n)$ such that for every Template:Mvar, the function Template:Tmath is smooth, and the family converges to the maximum function Template:Tmath as Template:Tmath. The concept of smooth minimum is similarly defined. In many cases, a single family approximates both: maximum as the parameter goes to positive infinity, minimum as the parameter goes to negative infinity; in symbols, Template:Tmath as Template:Tmath and Template:Tmath as Template:Tmath. The term can also be used loosely for a specific smooth function that behaves similarly to a maximum, without necessarily being part of a parametrized family.

Examples

Boltzmann operator

Smoothmax of (−x, x) versus x for various parameter values. Very smooth for $\alpha$=0.5, and more sharp for $\alpha$=8.

For large positive values of the parameter $\alpha >0$, the following formulation is a smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

${𝒮}_{\alpha }(x_1,\dots ,x_n)=\frac{\sum _{i=1}^n{x}_i{e}^{\alpha x_i}}{\sum _{i=1}^n{e}^{\alpha x_i}}$

${𝒮}_{\alpha }$ has the following properties:

1. ${𝒮}_{\alpha }\to \max$ as $\alpha \to \mathrm{\infty }$
2. ${𝒮}_0$ is the arithmetic mean of its inputs
3. ${𝒮}_{\alpha }\to \min$ as $\alpha \to -\mathrm{\infty }$

The gradient of ${𝒮}_{\alpha }$ is closely related to softmax and is given by

${\mathrm{\nabla }}_{x_i}{𝒮}_{\alpha }(x_1,\dots ,x_n)=\frac{e^{\alpha x_i}}{\sum _{j=1}^n{e}^{\alpha x_j}}[1+\alpha (x_i-{𝒮}_{\alpha }(x_1,\dots ,x_n))].$

This makes the softmax function useful for optimization techniques that use gradient descent.

This operator is sometimes called the Boltzmann operator, after the Boltzmann distribution.

LogSumExp

Template:Main Another smooth maximum is LogSumExp:

${\mathrm{L}\mathrm{S}\mathrm{E}}_{\alpha }(x_1,\dots ,x_n)=\frac{1}{\alpha }\log \sum _{i=1}^n\exp \alpha x_i$

This can also be normalized if the $x_i$ are all non-negative, yielding a function with domain $[0,\mathrm{\infty }{)}^n$ and range $[0,\mathrm{\infty })$:

$g(x_1,\dots ,x_n)=\log (\sum _{i=1}^n\exp x_i-(n-1))$

The $(n-1)$ term corrects for the fact that $\exp (0)=1$ by canceling out all but one zero exponential, and $\log 1=0$ if all $x_i$ are zero.

Mellowmax

The mellowmax operator is defined as follows:

${\mathrm{m}\mathrm{m}}_{\alpha }(x)=\frac{1}{\alpha }\log \frac{1}{n}\sum _{i=1}^n\exp \alpha x_i$

It is a non-expansive operator. As $\alpha \to \mathrm{\infty }$, it acts like a maximum. As $\alpha \to 0$, it acts like an arithmetic mean. As $\alpha \to -\mathrm{\infty }$, it acts like a minimum. This operator can be viewed as a particular instantiation of the quasi-arithmetic mean. It can also be derived from information theoretical principles as a way of regularizing policies with a cost function defined by KL divergence. The operator has previously been utilized in other areas, such as power engineering.

p-Norm

Template:Main Another smooth maximum is the p-norm:

$\Vert (x_1,\dots ,x_n){\Vert }_p={(\sum _{i=1}^n|x_i{|}^p)}^{\frac{1}{p}}$

which converges to $\Vert (x_1,\dots ,x_n){\Vert }_{\mathrm{\infty }}=\underset{1\le i\le n}{\max }|x_i|$ as $p\to \mathrm{\infty }$.

An advantage of the p-norm is that it is a norm. As such it is scale invariant (homogeneous): $\Vert (\lambda x_1,\dots ,\lambda x_n){\Vert }_p=|\lambda |\cdot \Vert (x_1,\dots ,x_n){\Vert }_p$, and it satisfies the triangle inequality.

Smooth maximum unit

The following binary operator is called the Smooth Maximum Unit (SMU):

$\begin{array}{cc}\underset{\epsilon }{\max }(a,b)& =\frac{a+b+|a-b{|}_{\epsilon }}{2}\\ & =\frac{a+b+\sqrt{(a-b{)}^2+\epsilon }}{2}\end{array}$

where $\epsilon \ge 0$ is a parameter. As $\epsilon \to 0$, $|\cdot {|}_{\epsilon }\to |\cdot |$ and thus $\underset{\epsilon }{\max }\to \max$.

See also

• LogSumExp
• Softmax function
• Generalized mean

References

https://www.johndcook.com/soft_maximum.pdf

M. Lange, D. Zühlke, O. Holz, and T. Villmann, "Applications of lp-norms and their smooth approximations for gradient based learning vector quantization," in Proc. ESANN, Apr. 2014, pp. 271-276. (https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2014-153.pdf)