View source for Rate–distortion theory

{{Short description|Branch of information theory which provides the theoretical foundations for lossy data compression}}
{{Information theory}}

{{More citations needed|date=March 2012}}

'''Rate–distortion theory''' is a major branch of [[information theory]] which provides the theoretical foundations for [[lossy data compression]]; it addresses the problem of determining the minimal number of bits per symbol, as measured by the rate ''R'', that should be communicated over a channel, so that the source (input signal) can be approximately reconstructed at the receiver (output signal) without exceeding an expected distortion ''D''.

== Introduction ==
[[File:Rate distortion theory problem setup.svg|thumb|512px|right|Rate distortion encoder and decoder. An encoder <math>f_n</math> encodes a sequence <math>X^n</math>. The encoded sequence <math>Y^n</math> is then fed to a decoder <math>g_n</math> which outputs a sequence <math>\hat{X}^n</math>. We try to minimize the distortion between the original sequence <math>X^n</math> and the reconstructed sequence <math>\hat{X}^n</math>.]]

Rate–distortion theory gives an analytical expression for how much compression can be achieved using lossy compression methods. Many of the existing audio, speech, image, and video compression techniques have transforms, quantization, and bit-rate allocation procedures that capitalize on the general shape of rate–distortion functions.

Rate–distortion theory was created by [[Claude Shannon]] in his foundational work on information theory.

In rate–distortion theory, the ''rate'' is usually understood as the number of [[bit]]s per data sample to be stored or transmitted. The notion of ''distortion'' is a subject of on-going discussion.<ref>{{cite conference |last=Blau |first=Y. |last2=Michaeli |first2=T. |url=http://proceedings.mlr.press/v97/blau19a/blau19a.pdf |title=Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff |book-title=Proceedings of the International Conference on Machine Learning |date=2019 |publisher=PMLR |pages=675–685 |arxiv=1901.07821}}</ref> In the most simple case (which is actually used in most cases), the distortion is defined as the expected value of the square of the difference between input and output signal (i.e., the [[mean squared error]]). However, since we know that most [[lossy compression]] techniques operate on data that will be perceived by human consumers (listening to [[music]], watching pictures and video) the distortion measure should preferably be modeled on human [[perception]] and perhaps [[aesthetics]]:  much like the use of [[probability]] in [[lossless compression]], distortion measures can ultimately be identified with [[loss function]]s as used in Bayesian [[estimation theory|estimation]] and [[decision theory]]. In audio compression, perceptual models (and therefore perceptual distortion measures) are relatively well developed and routinely used in compression techniques such as [[MP3]] or [[Vorbis]], but are often not easy to include in rate–distortion theory. In image and video compression, the human perception models are less well developed and inclusion is mostly limited to the [[JPEG]] and [[MPEG]] weighting ([[quantization (signal processing)|quantization]], [[Normalization (image processing)|normalization]]) matrix.

== Distortion functions ==
Distortion functions measure the cost of representing a symbol <math>x</math> by an approximated symbol <math>\hat{x}</math>. Typical distortion functions are the Hamming distortion and the Squared-error distortion.

=== Hamming distortion ===
:<math> d(x,\hat{x}) = \begin{cases} 
  0 & \text{if } x = \hat{x} \\
  1 & \text{if } x \neq \hat{x}
                      \end{cases}
</math>

=== Squared-error distortion ===
:<math> d(x,\hat{x})=\left( x-\hat{x}\right)^2 </math>

== Rate–distortion functions ==
The functions that relate the rate and distortion are found as the solution of the following minimization problem:

:<math>\inf_{Q_{Y\mid X}(y\mid x)} I_Q(Y;X) \text{ subject to } D_Q \le D^*.</math>

Here <math>Q_{Y\mid X}(y\mid x)</math>, sometimes called a test channel, is the [[conditional probability|conditional]] [[probability density function]] (PDF) of the communication channel output (compressed signal) <math>Y</math> for a given input (original signal) <math>X</math>, and <math>I_Q(Y;X)</math> is the '''[[mutual information]]''' between <math>Y</math> and <math>X</math> defined as

:<math>I(Y;X) = H(Y) - H(Y\mid X) \, </math>

where <math>H(Y)</math> and <math>H(Y\mid X)</math> are the entropy of the output signal ''Y'' and the [[conditional entropy]] of the output signal given the input signal, respectively:

:<math> H(Y) = - \int_{-\infty}^\infty P_Y (y) \log_{2} (P_Y (y))\,dy </math>

:<math> H(Y\mid X) = 
       - \int_{-\infty}^\infty \int_{-\infty}^\infty Q_{Y\mid X}(y\mid x) P_X (x) \log_2 (Q_{Y\mid X} (y\mid x))\, dx\, dy. </math>

The problem can also be formulated as a distortion–rate function, where we find the [[Infimum and supremum|infimum]] over achievable distortions for given rate constraint. The relevant expression is:

:<math>\inf_{Q_{Y\mid X}(y\mid x)} E[D_Q[X,Y]] \text{ subject to } I_Q(Y;X)\leq R. </math>

The two formulations lead to functions which are inverses of each other.

The mutual information can be understood as a measure for 'prior' uncertainty the receiver has about the sender's signal (''H''(''Y'')), diminished by the uncertainty that is left after receiving information about the sender's signal (<math>H(Y\mid X)</math>). Of course the decrease in uncertainty is due to the communicated amount of information, which is <math>I \left(Y;X \right)</math>.

As an example, in case there is ''no'' communication at all, then <math>H(Y\mid X) = H (Y)</math> and <math>I(Y;X) = 0</math>. Alternatively, if the communication channel is perfect and the received signal <math>Y</math> is identical to the signal <math>X</math> at the sender, then <math>H(Y\mid X) = 0</math> and <math>I(Y;X) = H(X) = H(Y)</math>.

In the definition of the rate–distortion function, <math>D_Q</math> and <math>D^{*}</math> are the distortion between <math>X</math> and <math>Y</math> for a given <math>Q_{Y\mid X}(y\mid x)</math> and the prescribed maximum distortion, respectively. When we use the [[mean squared error]] as distortion measure, we have (for [[amplitude]]-[[continuous signal]]s):

:<math>D_Q = \int_{-\infty}^\infty \int_{-\infty}^\infty
          P_{X,Y}(x,y) (x-y)^2\, dx\, dy = \int_{-\infty}^\infty \int_{-\infty}^\infty
          Q_{Y\mid X}(y\mid x)P_{X}(x) (x-y)^2\, dx\, dy. </math>

As the above equations show, calculating a rate–distortion function requires the stochastic description of the input <math>X</math> in terms of the PDF <math>P_X (x)</math>, and then aims at finding the conditional PDF <math>Q_{Y\mid X}(y\mid x)</math> that minimize rate for a given distortion <math>D^{*}</math>. These definitions can be formulated measure-theoretically to account for discrete and mixed random variables as well.

An [[Analytical expression|analytical]] solution to this [[Optimization problem|minimization problem]] is often difficult to obtain except in some instances for which we next offer two of the best known examples. The rate–distortion function of any source is known to obey several fundamental properties, the most important ones being that it is a [[Continuous function|continuous]], [[monotonically decreasing]] [[Convex function|convex]] (U) [[function (mathematics)|function]] and thus the shape for the function in the examples is typical (even measured rate–distortion functions in real life tend to have very similar forms).

Although analytical solutions to this problem are scarce, there are upper and lower bounds to these functions including the famous [[Shannon lower bound]] (SLB), which in the case of squared error and memoryless sources, states that for arbitrary sources with finite differential entropy,

:<math> R(D) \ge h(X) - h(D) \, </math>

where ''h''(''D'') is the differential entropy of a Gaussian random variable with variance D. This lower bound is extensible to sources with memory and other distortion measures. One important feature of the SLB is that it is asymptotically tight in the low distortion regime for a wide class of sources and in some occasions, it actually coincides with the rate–distortion function. Shannon Lower Bounds can generally be found if the distortion between any two numbers can be expressed as a function of the difference between the value of these two numbers.

The [[Blahut–Arimoto algorithm]], co-invented by [[Richard Blahut]], is an elegant iterative technique for numerically obtaining rate–distortion functions of arbitrary finite input/output alphabet sources and much work has been done to extend it to more general problem instances.

The computation of the rate-distortion function requires knowledge of the underlying distribution, which is often unavailable in contemporary applications in data-science and machine learning. However, this challenge can be addressed using deep learning-based estimators of the rate-distortion function.<ref>{{Cite journal |last=Tsur |first=Dor |last2=Huleihel |first2=Bashar |last3=Permuter |first3=Haim H. |date=2024 |title=On Rate Distortion via Constrained Optimization of Estimated Mutual Information |url=https://ieeexplore.ieee.org/document/10681556/ |journal=IEEE Access |volume=12 |pages=137970–137987 |doi=10.1109/ACCESS.2024.3462853 |issn=2169-3536|doi-access=free }}</ref> These estimators are typically referred to as 'neural estimators', involving the optimization of a parametrized variational form of the rate distortion objective. 

When working with stationary sources with memory, it is necessary to modify the definition of the rate distortion function and it must be understood in the sense of a limit taken over sequences of increasing lengths.
:<math> 
R(D) = \lim_{n \rightarrow \infty} R_n(D)
</math>
where
:<math>
R_n(D) = \frac{1}{n} \inf_{Q_{Y^n\mid X^n} \in \mathcal{Q}} I(Y^n, X^n)
</math>
and
:<math>
\mathcal{Q} = \{ Q_{Y^n\mid X^n}(Y^n\mid X^n,X_0): E[d(X^n,Y^n)] \leq D \}
</math>
where superscripts denote a complete sequence up to that time and the subscript 0 indicates initial state.

===Memoryless (independent) Gaussian source with squared-error distortion===
If we assume that <math>X</math> is a [[normal distribution|Gaussian]] random variable with [[variance]] <math>\sigma^2</math>, and if we assume that successive samples of the signal <math>X</math> are [[stochastically independent]] (or equivalently, the source is ''[[memorylessness|memoryless]]'', or the signal is ''uncorrelated''), we find the following [[analytical expression]] for the rate–distortion function:

:<math> R(D) = \begin{cases}
  \frac{1}{2}\log_2(\sigma_x^2/D ), & \text{if } 0 \le D \le \sigma_x^2 \\
              0, & \text{if } D > \sigma_x^2. 
                      \end{cases}
</math>&nbsp;&nbsp;&nbsp;<ref>{{harvnb|Cover|Thomas|2012|p=310}}</ref>

The following figure shows what this function looks like:

[[File:Rate distortion function.png|400px]]

Rate–distortion theory tell us that 'no compression system exists that performs outside the gray area'. The closer a practical compression system is to the red (lower) bound, the better it performs. As a general rule, this bound can only be attained by increasing the coding block length parameter. Nevertheless, even at unit blocklengths one can often find good (scalar) [[Quantization (signal processing)|quantizers]] that operate at distances from the rate–distortion function that are practically relevant.<ref>{{cite book| first = Thomas M. |last=Cover |first2=Joy A. |last2=Thomas |chapter=10. Rate Distortion Theory | title = Elements of Information Theory | publisher = Wiley |orig-year=2006 |year=2012 |chapter-url={{GBurl|VWq5GG6ycxMC|p=301}} |isbn=978-1-118-58577-1 |edition=2nd}}</ref>

This rate–distortion function holds only for Gaussian memoryless sources. It is known that the Gaussian source is the most "difficult" source to encode: for a given mean square error, it requires the greatest number of bits. The performance of a practical compression system working on&mdash;say&mdash;images, may well be below the <math>R \left(D \right)</math> lower bound shown.

===Memoryless (independent) Bernoulli source with Hamming distortion===
The rate-distortion function of a [[Bernoulli random variable]] with Hamming distortion is given by:
:<math> R(D) = \left\{ \begin{matrix} 
  H_b(p)-H_b(D), & 0 \le D \le \min{(p,1-p)} \\
  0,         & D > \min{(p,1-p)}
                      \end{matrix} \right.
</math>
where <math>H_b</math> denotes the [[binary entropy function]].

Plot of the rate-distortion function for <math>p=0.5</math>:

[[File:Rate distortion function Bernoulli.png|400px]]

== Connecting rate-distortion theory to channel capacity  ==
Suppose we want to transmit information about a source to the user with a distortion not exceeding ''D''. Rate–distortion theory tells us that at least <math>R(D)</math> bits/symbol of information from the source must reach the user. We also know from Shannon's channel coding theorem that if the source entropy is ''H'' bits/symbol, and the [[channel capacity]] is ''C'' (where <math>C < H</math>), then <math>H-C</math> bits/symbol will be lost when transmitting this information over the given channel. For the user to have any hope of reconstructing with a maximum distortion ''D'', we must impose the requirement that the information lost in transmission does not exceed the maximum tolerable loss of <math>H-R(D)</math> bits/symbol. This means that the channel capacity must be at least as large as <math>R(D)</math>.<ref name="BergerRateDistortion">{{cite book |title=Rate Distortion Theory: A Mathematical Basis for Data Compression |publisher=Prentice Hall |first=Toby |last=Berger |year=1971 |url=https://archive.org/details/ratedistortionth0000berg/ |url-access=registration |lccn=75-148254 |isbn=978-0-13-753103-5 |oclc=156968}}</ref>

== See also ==
* {{annotated link|Blahut–Arimoto algorithm}}
* {{annotated link|Data compression}}
* {{annotated link|Decorrelation}}
* {{annotated link|Rate–distortion optimization}}
* {{annotated link|Sphere packing}}
* {{annotated link|White noise}}

== References ==
{{Reflist}}

==External links==
*{{cite web |first=Sarah |last=Marzen |first2=Simon |last2=DeDeo |title=PyRated: a python package for rate distortion theory |url=https://sites.santafe.edu/~simon/styled-13/ |quote=PyRated is a very simple Python package to do the most basic calculation in rate-distortion theory: the determination of the “codebook” and the transmission rate ''R'', given a utility function (distortion matrix) and a Lagrange multiplier ''beta''.}} 
*[https://web.archive.org/web/20011006065854/http://www-ict.its.tudelft.nl/vcdemo VcDemo Image and Video Compression Learning Tool]

{{Compression Methods}}
{{Authority control}}

{{DEFAULTSORT:Rate-distortion theory}}
[[Category:Data compression]]
[[Category:Information theory]]