3D Morphable Model

Template:Short descriptionIn computer vision and computer graphics, the 3D Morphable Model (3DMM) is a generative technique that uses methods of statistical shape analysis to model 3D objects. The model follows an analysis-by-synthesis approach over a dataset of 3D example shapes of a single class of objects (e.g., face, hand). The main prerequisite is that all the 3D shapes are in a dense point-to-point correspondence, namely each point has the same semantical meaning over all the shapes. In this way, we can extract meaningful statistics from the dataset and use it to represent new plausible shapes of the object's class. Given a 2D image, we can represent its 3D shape via a fitting process or generate novel shapes by directly sampling from the statistical shape distribution of that class.

The question that initiated the research on 3DMMs was to understand how a visual system could handle the vast variety of images produced by a single class of objects and how these can be represented. The primary assumption in developing 3DMMs was that prior knowledge about object classes was crucial in vision. 3D Face Morphable Models are the most popular 3DMMs since they were the first to be developed in the field of facial recognition. It has also been applied to the whole human body, the hand, the ear, cars, and animals.

3D Face Morphable Model

In computer vision and computer graphics, the 3D Face Morphable Model (3DFMM) is a generative technique for modeling textured 3D faces. The generation of new faces is based on a pre-existing database of example faces acquired through a 3D scanning procedure. All these faces are in dense point-to-point correspondence, which enables the generation of a new realistic face (morph) by combining the acquired faces. A new 3D face can be inferred from one or multiple existing images of a face or by arbitrarily combining the example faces. 3DFMM provides a way to represent face shape and texture disentangled from external factors, such as camera parameters and illumination.

The 3D Morphable Model (3DMM) is a general framework that has been applied to various objects other than faces, e.g., the whole human body, specific body parts, and animals. 3DMMs were first developed to solve vision tasks by representing objects in terms of the prior knowledge that can be gathered from that object class. The prior knowledge is statistically extracted from a database of 3D examples and used as a basis to represent or generate new plausible objects of that class. Its effectiveness lies in the ability to efficiently encode this prior information, enabling the solution of otherwise ill-posed problems (such as single-view 3D object reconstruction).

Historically, face models have been the first example of morphable models, and the field of 3DFMM remains a very active field of research as today. In fact, 3DFMM has been successfully employed in face recognition, entertainment industry (gaming and extended reality, virtual try on, face replacement, face reenactment), digital forensics, and medical applications.

Modeling

In general, 3D faces can be modeled by three variational components extracted from the face dataset:

• shape model - model of the distribution of geometrical shape across different subjects
• expression model - model of the distribution of geometrical shape across different facial expressions
• appearance model - model of the distribution of surface textures (color and illumination)

Shape modeling

Visualization of the mean shape (center) and the first three principal components at +2 and -2 standard deviations of a 3DFMM

The 3DFMM uses statistical analysis to define a statistical shape space, a vectorial space equipped with a probability distribution, or prior. To extract the prior from the example dataset, all the 3D faces must be in a dense point-to-point correspondence. This means that each point has the same semantical meaning on each face (e.g., nose tip, edge of the eye). In this way, by fixing a point, we can, for example, derive the probability distribution of the texture's red channel values over all the faces. A face shape $S$ of $n$ vertices is defined as the vector containing the 3D coordinates of the $n$ vertices in a specified order, that is $S\in {ℝ}^{3n}$. A shape space is regarded as a $d$-dimensional space that generates plausible 3D faces by performing a lower-dimensional ($d\ll n$) parametrization of the database. Thus, a shape $S$ can be represented through a generator function $𝐜:{ℝ}^d\to {ℝ}^{3n}$ by the parameters $𝐰\in {ℝ}^d$, $𝐜(𝐰)=S\in {ℝ}^{3n}$. The most common statistical technique used in 3DFMM to generate the shape space is Principal Component Analysis (PCA), that generates a basis that maximizes the variance of the data. Performing PCA, the generator function is linear and defined as $$𝐜(𝐰)=\overline{𝐜}+𝐄𝐰$$where $\overline{𝐜}$ is the mean over the training data and $𝐄\in {ℝ}^{3n\times d}$ is the matrix that contains the $d$ most dominant eigenvectors.

Using a unique generator function for the whole face leads to the imperfect representation of finer details. A solution is to use local models of the face by segmenting important parts such as the eyes, mouth, and nose.

Expression modeling

The modeling of the expression is performed by explicitly subdividing the representation of the identity from the facial expression. Depending on how identity and expression are combined, these methods can be classified as additive, multiplicative, and nonlinear.

The additive model is defined as a linear model and the expression is an additive offset with respect to the identity $$𝐜({𝐰}^s,{𝐰}^w)=\overline{𝐜}+{𝐄}^s{𝐰}^s+{𝐄}^e{𝐰}^e$$where ${𝐄}^s$,${𝐄}^e$ and ${𝐰}^s$,${𝐰}^e$ are the matrices basis and the coefficients vectors of the shape and expression space, respectively. With this model, given the 3D shape of a subject in a neutral expression ${𝐜}_{ne}$ and in a particular expression ${𝐜}^{exp}$, we can transfer the expression to a different subject by adding the offset ${\mathrm{\Delta }}_{𝐜}={𝐜}^{exp}-{𝐜}^{ne}$. Two PCAs can be performed to learn two different spaces for shape and expression.

In a multiplicative model, shape and expression can be combined in different ways. For example, by exploiting $d_e$ operators ${𝐓}_j:{ℝ}^{3n}\to {ℝ}^{3n}$ that transform a neutral expression into a target blendshape we can write$$𝐜({𝐰}^s,{𝐰}^e)=\sum _{j=1}^{d_e}{w}_j^e{𝐓}_j(𝐜({𝐰}^s)+{\mathit{\delta }}^s)+{\mathit{\delta }}_j^e$$where ${\mathit{\delta }}^s$ and ${\mathit{\delta }}_j^s$ are vectors to correct to the target expression.

The nonlinear model uses nonlinear transformations to represent an expression.

Appearance modeling

The color information id often associated to each vertex of a 3D shape. This one-to-one correspondence allows us to represent appearance analogously to the linear shape model $$𝐝({𝐰}^t)=\overline{𝐝}+{𝐄}^t{𝐰}^t$$where ${𝐰}^t$ is the coefficients vector defined over the basis matrix ${𝐄}^t$. PCA can be again be used to learn the appearance space.

History

Facial recognition can be considered the field that originated the concepts that later on converged into the formalization of the morphable models. The eigenface approach used in face recognition represented faces in a vector space and used principal component analysis to identify the main modes of variation. However, this method had limitations: it was constrained to fixed poses and illumination and lacked an effective representation of shape differences. As a result, changes in the eigenvectors did not accurately represent shifts in facial structures but caused structures to fade in and out. To address these limitations, researchers added an eigendecomposition of 2D shape variations between faces. The original eigenface approach aligned images based on a single point, while new methods established correspondences on many points. Landmark-based face warping was introduced by Craw and Cameron (1991), and the first statistical shape model, Active Shape Model, was proposed by Cootes et al. (1995). This model used shape alone, but Active Appearance Model by Cootes et al. (1998) combined shape and appearance. Since these 2D methods were effective only for fixed poses and illumination, they were extended by Vetter and Poggio (1997) to handle more diverse settings. Even though separating shape and texture was effective for face representation, handling pose and illumination variations required many separate models. On the other hand, advances in 3D computer graphics showed that simulating pose and illumination variations was straightforward. The combination of graphics methods with face modeling led to the first formulation of 3DMMs by Blanz and Vetter (1999). The analysis-by-synthesis approach enabled the mapping of the 3D and 2D domains and a new representation of 3D shape and appearance. Their work is the first to introduce a statistical model for faces that enabled 3D reconstruction from 2D images and a parametric face space for controlled manipulation.

In the original definition of Blanz and Vetter, the shape of a face is represented as the vector $S=(X_1,Y_1,Z_1,...,X_n,Y_n,Z_n{)}^T\in {ℝ}^{3n}$ that contains the 3D coordinates of the $n$ vertices. Similarly, the texture is represented as a vector $T=(R_1,G_1,B_1,...,R_n,G_n,B_n{)}^T\in {ℝ}^{3n}$ that contains the three RGB color channels associated with each corresponding vertex. Due to the full correspondence between exemplar 3D faces, new shapes ${𝐒}_{models}$ and textures ${𝐓}_{models}$ can be defined as a linear combination of the $m$ example faces:$${𝐒}_{model}=\sum _{i=1}^m{a}_i{𝐒}_i{𝐓}_{model}=\sum _{i=1}^m{b}_i{𝐓}_i\text{with}\sum _{i=1}^m{a}_i=\sum _{i=1}^m{b}_i=1$$Thus, a new face shape and texture is parametrized by the shape $𝐚=(a_1,a_2,...,a_m{)}^T$ and texture coefficients $𝐛=(b_1,b_2,...,b_m{)}^T$. To extract the statistics from the dataset, they performed PCA to generate the shape space of dimension to $d$ and used a linear model for shape and appearance modeling. In this case, a new model can be generated in the orthogonal basis using the shape and the texture eigenvector $s_i$ and $t_i$, respectively:

${𝐒}_{model}=\overline{𝐒}+\sum _{i=1}^m{a}_i{𝐬}_i{𝐓}_{model}=\overline{𝐓}+\sum _{i=1}^m{b}_i{𝐭}_i$

where $\overline{𝐒}$ and $\overline{𝐓}$ are the mean shape and texture of the dataset.

Publicly available databases

In the following table, we list the publicly available databases of human faces that can be used for the 3DFMM.

Publicly available databases of human faces

	Year	Geometry	Appearance	Size	Download	Institution
Basel Face Model 2009	2009	shape	per-vertex	100 individuals in neutral expression	Link	University of Basel
FaceWarehouse	2014	shape, expression	-	150 individuals in 20 different expressions	Link	Zhejiang University
Large Scale Facial Model (LSFM)	2016	shape	-	9,663 individuals	Link	Imperial College London
Surrey Face Model	2016	shape, expression (multi-resolution)	per-vertex	169 individuals	Link	University of Surrey
Basel Face Model 2017	2017	shape, expression	per-vertex	200 individuals and 160 expression scans	Link	University of Basel
Liverpool-York Head Model (LYHM)	2017	shape (full head - no hair, no eyes)	per-vertex	1,212 individuals	Link	University of York, Alder Hey Hospital
Faces Learned with an Articulated Model and Expressions (FLAME)	2017	shape (full head - no hair), expression, head pose	texture	3,800 individuals for shape, 8,000 for head pose, 21,000 for expression	Link	University of Southern California, Max Planck Institute for Intelligent Systems
Convolutional Mesh Autoencoder (CoMA)	2018	shape (full head - no hair), expression	-	2 individuals in 12 extreme expressions	Link	Max Planck Institute for Intelligent Systems
Morphable Face Albedo Model	2020	-	per-vertex diffuse and specular albedo	73 individuals	Link	University of York
FaceVerse	2022	shape	texture	128 individuals in 21 different expressions	Link	Tsinghua University

See also

• Statistical shape analysis

External links

• Template:Cite web