Own faces

Eigenfaces (also Engl. Eigenfaces called) are the result of a method for the automated face recognition . As a method of machine vision , self -faces enable the recognition of a previously trained set of human identities in real time - however, they require a very high degree of homogeneity in terms of lighting conditions, size of the images, rotation and scaling of the face. Due to this high sensitivity for variation between the training data set and the face actually to be classified, self-faces are used less and less for face recognition in practice. The procedure, developed in 1991, is considered the first fully automated face recognition technology and provided the basis for a multitude of further developments, industrial applications and inspirations for alternative approaches. The method is based on principal component analysis and, in contrast to more modern procedures using convolutional neural networks , falls within the unsupervised learning category .

History of the process

Basic development

A core problem of machine vision is that an image with the dimensions can be viewed as a vector in a high-dimensional space with dimensions and this immense number of degrees of freedom (there are already gray-scale 8-bit images of the size of pixels), purely statistically speaking, becomes a very difficult object detection Problem. The principal component analysis (PCA) is a smallest possible vectorial a method subspace to find that represents the greatest amount of variance in the data, and eliminates redundant correlations. Looking for a more compact representation of images of faces, Sirovich and Kirby first used principal component analysis in 1987 to use the most important principal components as basis vectors to generate a smaller-dimensional representation whose linear combination could effectively reconstruct the original images. The directions of the greatest variance are extracted using PCA by calculating the eigenvectors and eigenvalues ​​of the covariance matrix from the probability distribution over the high-dimensional space of possible facial images (to put it simply, the most characteristic features of the respective faces). Four years later, Turk and Pentland had further developed the process so that faces could be classified and categorized in near real-time, which was a breakthrough in face recognition technology. In general, the calculation of the eigenvectors of the correlation matrix is ​​very time-consuming, since the size of this square matrix depends on the image quality. Crucial for the real-time calculation was that Turk and Pentland found a way to calculate the eigenvectors of this high-dimensional correlation matrix by calculating the eigenvectors of a related matrix, the size of which only scales with the amount of images in the data set (details in the section Description of the procedure ). ${\displaystyle M\times N}$ ${\displaystyle MN}$ ${\displaystyle 10^{32}}$ ${\displaystyle 100\times 100}$

Further developments

The original method has a number of obvious weaknesses ( details below ) that have been sought to compensate for through improvements. The following are mentioned as examples:

• Pentland et al. presented in 1994 with View-Based Eigenfaces an approach that creates separate eigenspaces for different facial expressions of faces so that face recognition is not restricted to neutral expressions.
• In the same work, a modular Eigenfeature Framework was described, which calculates different eigenspaces for certain components of the face and uses the resulting own noses , own eyes and other features to refine a “rough” representation of the own face .
• To classification of faces to optimize (rather than maximum compact representations to produce) may Fisher (Engl. Faces fisherfaces ) rather than be used by own faces - based on linear discriminant analysis .
• Local Binary Patterns Histograms (LBPH). Eigenfaces and fisherfaces are heavily dependent on the lighting conditions. This disadvantage is overcome with the help of LBPH face recognition.

software

In the free software - library for image processing OpenCV own Faces, Faces and Fisher LBPH are implemented.

Description of the procedure

Eigen faces are generated using principal component analysis (PCA).

Instructions for practical implementation

Suppose you have a set of images of faces so that each image has the dimensions (for example , so each image is made up of pixels). For the method to work, the images must have been taken under similar lighting conditions and the same components (eyes, nose, mouth, etc.) must be in the same positions. Then the procedure is as follows: ${\displaystyle K=16}$ ${\displaystyle M\times N}$ ${\displaystyle M=N=300}$ ${\displaystyle 90.000}$

1. The individual images are transformed from a matrix to a vector with 90,000 dimensions, so that . ${\displaystyle I_{1},...,I_{K}}$ ${\displaystyle I_{i}\in \mathbb {R} ^{MN}\forall i\leq K}$

2. An average face is calculated from the entire training data set: ${\displaystyle {\bar {I}}}$

${\displaystyle {\bar {I}}={\frac {1}{K}}\sum _{i=1}^{K}I_{i}}$.

3. This average face is subtracted from each image from the training data set:

${\displaystyle \Phi _{i}=I_{i}-{\bar {I}}}$.

4. The difference faces are now concatenated to form a matrix . ${\displaystyle \Phi _{i}}$ ${\displaystyle A=[\Phi _{1}\Phi _{2}\cdots \Phi _{K}]\in \mathbb {R} ^{MN\times K}}$

5. The covariance matrix is calculated: ${\displaystyle C}$

${\displaystyle C=AA^{T}}$

Now saves the variances of the individual pixels over the entire data set on the main diagonal and the covariance of pixel with pixel over the entire data set off the main diagonal at Position . ${\displaystyle C\in \mathbb {R} ^{MN\times MN}}$ ${\displaystyle i,j}$ ${\displaystyle i}$ ${\displaystyle j}$

6. The next step is to calculate the eigenvectors and eigenvalues ​​of the covariance matrix . The eigenvectors of the covariance matrix have the same dimensionality as the original images. Turk and Pentland called them self-faces because of their face-like appearance . ${\displaystyle C}$ ${\displaystyle C}$

7. The eigenvectors are sorted in descending order according to the absolute size of the eigenvalues, so that the first eigenvectors cover the most variance. The set of eigenfaces taken into account is now made quite arbitrarily on the basis of a threshold value for the cumulative explained variance. ${\displaystyle k<K}$

8. The own faces can now be used to compactly represent known or new faces and to reconstruct them later or to determine the identity of a person.

Speed ​​problem

The calculation of the eigenvectors from is usually not easily possible for two reasons. First: Since , even with images of the size of pixels, a covariance matrix of the size is generated, the eigenvectors of which cannot be calculated in a satisfactory time. Second, the general rule is that . Because in the special case of the calculation of eigenfaces the rule is that the training set contains fewer images than each image pixel, has no full rank and accordingly some eigenvalues ​​are equal to zero. As a rule there are exactly actual eigenvectors. Instead of calculating these with extreme computational effort , Turk and Pentland used a trick that enables real-time localization and recognition of faces: The eigenvectors of a new matrix are calculated , which is defined as follows: ${\displaystyle C}$ ${\displaystyle C\in \mathbb {R} ^{MN\times MN}}$ ${\displaystyle 300\times 300}$ ${\displaystyle 90.000}$ ${\displaystyle \mathrm {rang} (C)=\mathrm {rang} (AA^{T})=\mathrm {rang} (A)\leq \min\{MN,K\}}$ ${\displaystyle K<MN}$ ${\displaystyle C}$ ${\displaystyle K-1}$ ${\displaystyle C}$ ${\displaystyle K-1}$ ${\displaystyle L}$

${\displaystyle L:=A^{T}A\in \mathbb {R} ^{K\times K}}$

The eigenvectors of can be calculated in a fraction of the time because it has a much smaller dimension than . At a training set of images the size is only the size , rather than the original size of . The calculation of the eigenvectors of on shifting applies because of the following: ${\displaystyle L}$ ${\displaystyle L}$ ${\displaystyle C}$ ${\displaystyle K=16}$ ${\displaystyle 300\times 300}$ ${\displaystyle L}$ ${\displaystyle 16\times 16}$ ${\displaystyle 90.000\times 90.000}$ ${\displaystyle C}$ ${\displaystyle L}$

Let be the eigenvalue decomposition of given by ${\displaystyle C}$

${\displaystyle Cv_{i}=AA^{T}v_{i}=\lambda _{i}v_{i}}$

searched. Because the matrix is ​​too large, we consider instead the eigenvalue decomposition of : ${\displaystyle AA^{T}\in \mathbb {R} ^{MN\times MN}}$ ${\displaystyle L}$

${\displaystyle Lu_{i}=A^{T}Au_{i}=\lambda _{i}u_{i}}$

If the equation is multiplied by on the left, the result is ${\displaystyle A}$

${\displaystyle AA^{T}Au_{i}=\lambda _{i}Au_{i}}$

Now , the eigenvalue decomposition of unambiguously results in the eigenvalue decomposition of . The vectors thus obtained are the eigenvectors of , whereby we are only interested in those with the highest eigenvalues . The 's must be orthonormal , i.e. H. they still need to be normalized. ${\displaystyle v_{i}:=Au_{i}}$ ${\displaystyle L}$ ${\displaystyle C}$ ${\displaystyle v_{i}}$ ${\displaystyle C}$ ${\displaystyle K'}$ ${\displaystyle v}$ ${\displaystyle u}$

Interpretation of the own faces

Eigenface examples generated from a large database

The eigenfaces are the resulting eigenvectors of the correlation matrix . Eigen faces are generic faces in the sense that actual faces can be reconstructed by linear combinations of their own faces. Given a sufficiently large data set, each human face could therefore be represented by a combination of these standardized faces. Based on the right hand shown eigenfaces that various eigenfaces various properties (becomes clear features ) of the training images encode. While the upper faces reflect parts of the hairstyle, the face at the bottom left represents the illumination of the background and the face at the bottom right the direction of the incidence of light. ${\displaystyle v_{i}}$ ${\displaystyle C}$

In order to visualize the Eigen faces, they must first be converted from vector to matrix notation. It is customary to order the eigenfaces according to the absolute amount of the associated eigenvalues , so that the first eigenfaces represent the positions of greatest variance and the relevance of the following eigenfaces gradually decreases. From a mathematical point of view, PCA rotates the coordinate system in such a way that the axes of the new coordinate system (the eigen faces) lie in the direction of the greatest variance of the manifold given by the training data . ${\displaystyle \lambda _{i}}$

Compression and reconstruction of faces

Eigen faces make it possible to save images in a very compact form by projecting each training image into the eigen space spanned by the Eigen faces. This can be achieved by simply multiplying the image vector by the eigenface matrix and results in a vector of length instead of the original matrix of size . Since, as a rule, only the most important faces are projected, the actual representation of an image is even more compact. The actual training images can now be reconstructed through a linear combination of the most important individual faces. The reconstruction error decreases the more own faces are used for the reconstruction. For example, the following could apply to the first training image: (i.e. 15% of the first self-face plus 42% of the second and so on). ${\displaystyle K}$ ${\displaystyle M\times N}$ ${\displaystyle k<K}$ ${\displaystyle k<K}$ ${\displaystyle \Phi _{i}}$ ${\displaystyle \Phi _{i}=0.15v_{1}+0.42v_{2}+...+0.2v_{k}}$

The graphic opposite (will follow soon) illustrates the influence of a growing number of related self-faces for the reconstruction. An attempt was made to reconstruct the same image in each column. In the first line, only the first Eigenface was used, in the second line the first 5 Eigenface, in the third line the first 10 Eigenface and in the last line all Eigenfaces were used. Although the faces are easier to see below, it is noticeable even in this example that the reconstruction error quickly reaches a saturation point.

Own faces for identity recognition

Eigen faces can be used in a very elegant way to recognize pre-trained identities. This is done in that a recording of a known person, unknown to the algorithm, is projected into the eigenspace generated by the eigenface . The resulting vector can now be compared with all images from the training database by means of classic, vectorial distance measurement (e.g. Euclidean norm ). The identity of the unknown recording can then be determined using K nearest neighbors or similar classification methods.

Generalization of self-faces to self-images

The concept of self-faces can easily be generalized to all other types of images ( Eigenimages ). The term self- faces has only established itself due to the fact that the first application investigated using self-images was face recognition. For example, the self-images can be extracted from a set of images that show the same object from different angles, distances and lighting conditions in order to generate an object representation that is as transformation-invariant as possible, which can then be used for top-down object recognition algorithms such as template matching or wireframe models. Another example would be a sequence of images that shows how an observed, static scene moves back and forth on the retina due to microsaccades . Here, too, the self-images show the particularly invariant components of the scene. Further applications of self-images are in lip reading , speaker authentication , imaging processes or handwriting recognition .

Summary

Per

• The training process is fully automatic, unsupervised and easy to implement.
• Once the training process is complete, face recognition can be done in real time.
• Own faces can deal with large databases of training images (scalability).
• Eigenfaces are optimally (in terms of variance) compressed representations of high-dimensional images of faces.

Cons

• High sensitivity to variation in lighting conditions, rotation and scaling of the face, facial expression, obscuration and size of the image. Accordingly, the conditions under which the images are taken must be controlled very precisely.
• In practical applications, the first self-faces usually represent variation in the lighting conditions and not variation in the faces themselves. In order to improve the precision of the algorithm, the first 3 self-faces are often ignored.

Individual evidence

1. ^ Shang Zeng and Matthew Turk: Eigenfaces. Scholarpedia article, 2008, accessed January 1, 2018 .  
2. ↑ Lawrence Sirovich, Michael Kirby: Low-dimensional procedure for the characterization of human faces. (PDF) Journal of the Optical Society of America, 1987, pp. 509-524 , accessed January 1, 2018 .  
3. ↑ Matthew Turk, Alex Petland: Self Faces for Recognition. (PDF) Journal of Cognitive Neuroscience, 1991, pp. 71–86 , accessed on January 1, 2018 (English).  
4. ^ A. Pentland et al .: View-based and modular eigenspaces for face recognition. Proceedings of CVPR, 1994, accessed January 1, 2018 .  
5. ↑ PB Belhumeur et al .: Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection. (PDF) IEEE Transactions on Pattern Analysis and Machine Intelligence, July 1997, accessed January 1, 2018 .  
6. ↑ Ramiz Raja: Face Detection using OpenCV and Python. Super Data Science, August 3, 2017, accessed January 24, 2019 . 
7. ↑ OpenCV documentation: Face Recognition with OpenCV. Retrieved January 24, 2019 . 
8. ^ Born, J. et al .: Hebbian Learning of Hand-Centered Represenations in a Hierarchical Neural Network Model of the Primate Visual System. PLOS ONE, May 2017, accessed January 1, 2018 .