Tensor reshaping

Template:More citations needed In multilinear algebra, a reshaping of tensors is any bijection between the set of indices of an order-$M$ tensor and the set of indices of an order-$L$ tensor, where $L<M$. The use of indices presupposes tensors in coordinate representation with respect to a basis. The coordinate representation of a tensor can be regarded as a multi-dimensional array, and a bijection from one set of indices to another therefore amounts to a rearrangement of the array elements into an array of a different shape. Such a rearrangement constitutes a particular kind of linear map between the vector space of order-$M$ tensors and the vector space of order-$L$ tensors.

Definition

Given a positive integer $M$, the notation $[M]$ refers to the set $\{1,\dots ,M\}$ of the first Template:Mvar positive integers.

For each integer $m$ where $1\le m\le M$ for a positive integer $M$, let $V_m$ denote an $I_m$-dimensional vector space over a field $F$. Then there are vector space isomorphisms (linear maps)

$$\begin{array}{cc}{V}_1\otimes \cdots \otimes V_M& \simeq F^{I_1}\otimes \cdots \otimes F^{I_M}\\ & \simeq F^{I_{{\pi }_1}}\otimes \cdots \otimes F^{I_{{\pi }_M}}\\ & \simeq F^{I_{{\pi }_1}{I}_{{\pi }_2}}\otimes F^{I_{{\pi }_3}}\otimes \cdots \otimes F^{I_{{\pi }_M}}\\ & \simeq F^{I_{{\pi }_1}{I}_{{\pi }_3}}\otimes F^{I_{{\pi }_2}}\otimes F^{I_{{\pi }_4}}\otimes \cdots \otimes F^{I_{{\pi }_M}}\\ & ⋮\\ & \simeq F^{I_1{I}_2\dots I_M},\end{array}$$

where $\pi \in {𝔖}_M$ is any permutation and ${𝔖}_M$ is the symmetric group on $M$ elements. Via these (and other) vector space isomorphisms, a tensor can be interpreted in several ways as an order-$L$ tensor where $L\le M$.

Coordinate representation

The first vector space isomorphism on the list above, $V_1\otimes \cdots \otimes V_M\simeq F^{I_1}\otimes \cdots \otimes F^{I_M}$, gives the coordinate representation of an abstract tensor. Assume that each of the $M$ vector spaces $V_m$ has a basis $\{v_1^m,v_2^m,\dots ,v_{I_m}^m\}$. The expression of a tensor with respect to this basis has the form $$𝒜=\sum _{i_1=1}^{I_1}\dots \sum _{i_M=1}^{I_M}{a}_{i_1,i_2,\dots ,i_M}{v}_{i_1}^1\otimes v_{i_2}^2\otimes \cdots \otimes v_{i_M}^M,$$ where the coefficients $a_{i_1,i_2,\dots ,i_M}$ are elements of $F$. The coordinate representation of $𝒜$ is $$\sum _{i_1=1}^{I_1}\dots \sum _{i_M=1}^{I_M}{a}_{i_1,i_2,\dots ,i_M}{𝐞}_{i_1}^1\otimes {𝐞}_{i_2}^2\otimes \cdots \otimes {𝐞}_{i_M}^M,$$where ${𝐞}_i^m$ is the $i^{\text{th}}$ standard basis vector of $F^{I_m}$. This can be regarded as a M-way array whose elements are the coefficients $a_{i_1,i_2,\dots ,i_M}$.

General flattenings

For any permutation $\pi \in {𝔖}_M$ there is a canonical isomorphism between the two tensor products of vector spaces $V_1\otimes V_2\otimes \cdots \otimes V_M$ and $V_{\pi (1)}\otimes V_{\pi (2)}\otimes \cdots \otimes V_{\pi (M)}$. Parentheses are usually omitted from such products due to the natural isomorphism between $V_i\otimes (V_j\otimes V_k)$ and $(V_i\otimes V_j)\otimes V_k$, but may, of course, be reintroduced to emphasize a particular grouping of factors. In the grouping, $$(V_{\pi (1)}\otimes \cdots \otimes V_{\pi (r_1)})\otimes (V_{\pi (r_1+1)}\otimes \cdots \otimes V_{\pi (r_2)})\otimes \cdots \otimes (V_{\pi (r_{L-1}+1)}\otimes \cdots \otimes V_{\pi (r_L)}),$$ there are $L$ groups with $r_l-r_{l-1}$ factors in the $l^{\text{th}}$ group (where $r_0=0$ and $r_L=M$).

Letting $S_l=(\pi (r_{l-1}+1),\pi (r_{l-1}+2),\dots ,\pi (r_l))$ for each $l$ satisfying $1\le l\le L$, an $(S_1,S_2,\dots ,S_L)$-flattening of a tensor $𝒜$, denoted ${𝒜}_{(S_1,S_2,\dots ,S_L)}$, is obtained by applying the two processes above within each of the $L$ groups of factors. That is, the coordinate representation of the $l^{\text{th}}$ group of factors is obtained using the isomorphism $(V_{\pi (r_{l-1}+1)}\otimes V_{\pi (r_{l-1}+2)}\otimes \cdots \otimes V_{\pi (r_l)})\simeq (F^{I_{\pi (r_{l-1}+1)}}\otimes F^{I_{\pi (r_{l-1}+2)}}\otimes \cdots \otimes F^{I_{\pi (r_l)}})$, which requires specifying bases for all of the vector spaces $V_k$. The result is then vectorized using a bijection ${\mu }_l:[I_{\pi (r_{l-1}+1)}]\times [I_{\pi (r_{l-1}+2)}]\times \cdots \times [I_{\pi (r_l)}]\to [I_{S_l}]$ to obtain an element of $F^{I_{S_l}}$, where $I_{S_l}:={\prod }_{i=r_{l-1}+1}^{r_l}{I}_{\pi (i)}$, the product of the dimensions of the vector spaces in the $l^{\text{th}}$ group of factors. The result of applying these isomorphisms within each group of factors is an element of $F^{I_{S_1}}\otimes \cdots \otimes F^{I_{S_L}}$, which is a tensor of order $L$.

Vectorization

By means of a bijective map $\mu :[I_1]\times \cdots \times [I_M]\to [I_1\cdots I_M]$, a vector space isomorphism between $F^{I_1}\otimes \cdots \otimes F^{I_M}$ and $F^{I_1\cdots I_M}$ is constructed via the mapping ${𝐞}_{i_1}^1\otimes \cdots {𝐞}_{i_m}^m\otimes \cdots \otimes {𝐞}_{i_M}^M\mapsto {𝐞}_{\mu (i_1,i_2,\dots ,i_M)},$ where for every natural number $i$ such that $1\le i\le I_1\cdots I_M$, the vector ${𝐞}_i$ denotes the ith standard basis vector of $F^{i_1\cdots i_M}$. In such a reshaping, the tensor is simply interpreted as a vector in $F^{I_1\cdots I_M}$. This is known as vectorization, and is analogous to vectorization of matrices. A standard choice of bijection $\mu$ is such that

$$\mathrm{vec}(𝒜)={\left[\begin{array}{ccccccc}{a}_{1,1,\dots ,1}& a_{2,1,\dots ,1}& \cdots & a_{n_1,1,\dots ,1}& a_{1,2,1,\dots ,1}& \cdots & a_{I_1,I_2,\dots ,I_M}\end{array}\right]}^T,$$

which is consistent with the way in which the colon operator in Matlab and GNU Octave reshapes a higher-order tensor into a vector. In general, the vectorization of $𝒜$ is the vector $[a_{{\mu }^{-1}(i)}{]}_{i=1}^{I_1\cdots I_M}$.

The vectorization of $𝒜$ denoted with $vec(𝒜)$ or ${𝒜}_{[:]}$ is an $[S_1,S_2]$-reshaping where $S_1=(1,2,\dots ,M)$ and $S_2=\mathrm{\varnothing }$.

Mode-m Flattening / Mode-m Matrixization

Let $𝒜\in F^{I_1}\otimes F^{I_2}\otimes \cdots \otimes F^{I_M}$ be the coordinate representation of an abstract tensor with respect to a basis. Mode-m matrixizing (a.k.a. flattening) of $𝒜$ is an $[S_1,S_2]$-reshaping in which $S_1=(m)$ and $S_2=(1,2,\dots ,m-1,m+1,\dots ,M)$. Usually, a standard matrixizing is denoted by

$${𝐀}_{[m]}={𝒜}_{[S_1,S_2]}$$

This reshaping is sometimes called matrixizing, matricizing, flattening or unfolding in the literature. A standard choice for the bijections ${\mu }_1,{\mu }_2$ is the one that is consistent with the reshape function in Matlab and GNU Octave, namely

$${𝐀}_{[m]}:=\left[\begin{array}{cccc}{a}_{1,1,\dots ,1,1,1,\dots ,1}& a_{2,1,\dots ,1,1,1,\dots ,1}& \cdots & a_{I_1,I_2,\dots ,I_{m-1},1,I_{m+1},\dots ,I_M}\\ a_{1,1,\dots ,1,2,1,\dots ,1}& a_{2,1,\dots ,1,2,1,\dots ,1}& \cdots & a_{I_1,I_2,\dots ,I_{m-1},2,I_{m+1},\dots ,I_M}\\ ⋮& ⋮& & ⋮\\ a_{1,1,\dots ,1,I_m,1,\dots ,1}& a_{2,1,\dots ,1,I_m,1,\dots ,1}& \cdots & a_{I_1,I_2,\dots ,I_{m-1},I_m,I_{m+1},\dots ,I_M}\end{array}\right]$$

Definition Mode-m Matrixizing: $$[{𝐀}_{[m]}{]}_{jk}=a_{i_1\dots i_m\dots i_M},\text{ where }j=i_m\text{ and }k=1+\sum _{\genfrac{}{}{0ex}{}{n=0}{n\ne m}}^M(i_n-1)\prod _{\genfrac{}{}{0ex}{}{l=0}{l\ne m}}^{n-1}{I}_l.$$ The mode-m matrixizing of a tensor $𝒜\in F^{I_1\times ...I_M},$ is defined as the matrix ${𝐀}_{[m]}\in F^{I_m\times (I_1\dots I_{m-1}{I}_{m+1}\dots I_M)}$. As the parenthetical ordering indicates, the mode-m column vectors are arranged by sweeping all the other mode indices through their ranges, with smaller mode indexes varying more rapidly than larger ones; thus