Download Facial Expression Recognition-Computer Sciences Applications-Project Report and more Study Guides, Projects, Research Applications of Computer Sciences in PDF only on Docsity!
iv
- Figure 1: Abstract Level Representation of the System ....................... List of Figures
- Figure 2: Simple Facial Expression Framework ...................................
- Figure 3: Principle Component Analysis Basic Steps ...........................
- Figure 4: Training using LVQ Network - Leave One Subject Out .....
- Figure 5: Result Improvement using Principle Components 30-
- and 500 epochs ........................................................................................
- Figure 6: Result Improvement using Principle Components 30-
- and 150 epochs ........................................................................................
- Figure 7: Result Improvement using Principle Components 20-
- and 500 epochs ........................................................................................
- Figure 8: 2-D Gabor Filter.....................................................................
- Figure 9: Framework for facial Expression Recognition ...................
- Figure 10: Results Using Gabor filtering .............................................
- Figure 11: Improved Results Using Gabor Filtering ..........................
v
Abstract
Facial Expression system is a system that is used to recognize a set of expressions either from a file or from a camera attached to the system. This software can be integrated into any other system that requires an expression recognition module. Expression recognition from the static images is a challenging problem as compared to images form a live camera. In static images we have less information as compared to the live stream. The Expression recognition system described here has both the functionality of input image either from a file or from a camera. Different techniques are applied on input image for expression recognition.
1
Introduction:
Purpose
The name of the product to be developed is Facial Expression Recognition System. And the objective of the software to be developed is to make computer so much intelligent that it can recognize different facial expressions of different persons. Facial expressions are an important means of our nonverbal communications. It has been a significant research area for behavioral scientists. The main purpose of the Automatic Facial Expression Recognition System (AFERS) is to recognize the facial expressions from the images. The system takes images as an input and apply feature extraction techniques to extract a set of features of the image and then based on the knowledge it classify the images as to which expression they belong.
Figure 1: Abstract Level Representation of the System
The scientific study of emotion in facial expressions started with the work of Charles Darwin published in 1872. Suwa first attempted to automatically analyze facial expressions by tracking the motion of 20 identified spots in 1978. A Facial Expression Recognition System is a computer system that when given an input image calculates the feature changes and uses this knowledge to output the recognized facial expression. The recent advances in motion analysis, face detection, tracking and recognition enables us to build a facial expression recognition system.
2
Overview
This document describes the work carried out during 7 th^ semester. Section 2 describes the work done before 7th^ semester mid term. Section 3 describes the work done after 7 th^ semester mid term.
4
Expression Recognition Techniques:
There is a long history of interest in the recognition of emotion from facial expressions, influenced by Darwin’s pioneering work and extensive studies on face perception. Face perception is very important component of human cognition as faces are rich in information about individual identity, but also about mood and mental state. Indeed, a continuous effort has been put towards constructing automatic systems, which recognize successfully human emotions from static images or/and sequences of images. People in computer vision and pattern recognition have been working on automatic recognition from characteristics in human faces for the last twenty years [^2 ]. A number of techniques have been developed by researchers. I have worked on two techniques for recognition of facial expressions from the static images.
Facial Expression Recognition Using
Principle Component Analysis:
Principal Components Analysis (PCA) is a technique for simplifying a dataset, by reducing multidimensional datasets to lower dimensions for analysis [^3 ]. PCA involves a mathematical procedure that transforms a number of (possibly) correlated variables into a (smaller) number of uncorrelated variables called principal components. The first principal component accounts for as much of the variability in the data as possible, and each succeeding component accounts for as much of the remaining variability as possible [^4 ]. Main objectives of PCA are as follows
To discover or to reduce the dimensionality of the data set To identify new meaningful underlying variables
Background Statistics & Algebra:
Idea behind the statistics is that you have a big set of data and you have to analyze the sets and finding relationships between the individuals of dataset. Following are some of the formulas which are helpful in understanding PCA [ 5 ].
5
Mean:
Simple mean of the dataset is calculated as follows
n
X
n
i i
X
^1
The symbol X is called the mean of the set X.
Standard Deviation:
Standard deviation is known as “The Average distance between mean of dataset and a point”. Mathematically known as
( 1 )
1
_^2
n
S
n
i Xi^ X
Where s is called the Standard deviation and X is called the mean of set X.
Variance:
Variance is the square of standard deviation, mathematically written as
( 1 )
1
_^2 2
n
S
n
i Xi^ X
Variance and Standard deviation are the measure of spread of data.
Covariance:
We always measure this between two dimensions. If we calculate covariance between one dimension and itself then we get the variance. Formula for covariance is
cov ,^1
n
X X Y Y
X Y
n
i
i i
7
value by its mean value. The purpose of this is to produce data whose mean value is zero.
Step 3:
Then we calculate the covariance matrix defined as follows. For three x, y, z, the covariance matrix will be
zx z y z z
yx yy yz
xx xy xz C cov , cov , cov ,
cov , cov , cov ,
cov , cov , cov ,
This values at diagonal are the variances along x, y and z.
Step 4:
As the covariance matrix is square, we can calculate the eigenvector and Eigen values of the matrix. We generate eigenvectors whose length is 1.
Step 5:
We sort the vector according to their value and Eigenvector with the highest Eigen value is called the Principle Component of the data. We choose a subset of these vectors for further processing.
Results Using PCA & LVQ:
In my project I have used Principle component analysis (PCA) for feature extraction and then Learning Vector Quantization for feature classification. I am using JAFFE database for expression recognition. There are 213 images of size 256x256 each. First of all I resize the image in order to minimize the background noise.
8
256x256 140x
Leave One Subject Out:
In our database we have total 10 individuals. We shall create two datasets i.e. one for testing and one for training. By leaving one subject out we have _ images in training dataset and _ images in testing dataset. We apply PCA on both datasets and then make an LVQ network. Then train the network using out training dataset. Training parameters for neural network are as follows,
Table 1: Training Using LVQ Network
And the training results are as follows,
Training Using LVQ Network No. Of Training Samples 191 No. Of Testing Samples 22 Sample Size 140x
Expression Labels
Happy Fear Sad Surprise Disgust Anger Natural No. of Epochs 150
Training Results No. of Misclassifications 36/ Recognition Rate 80.15% Error Rate 18.85%
Table 2: Training Results using first 50 Principle Components
10
Then we subtract the mean from the testing data, and calculate the outputs from the network, the result for the new subject are
Confusion matrix of the testing result is shown below Anger Disgust Fear Happy Natural Sad Surprise Anger 2/3 = 66.66%
0% 0% 0% 1/3 = 33.33%
0% 0%
Disgust 0% 2/3 = 66.66%
0% 0% 0% 1/3 = 33.33%
0%
Fear 0% 0% 2/4 = 50%
0% 1 / 4 = 25%
0% 1 / 4 = 25% Happy 0% 0% 0% 100% 0% 0% 0% Natural 0% 0% 0% 0% 2/3 = 66.66%
0% 1/3 = 33.33% Sad 0% 0% 0% 0% 2/3 = 66.66%
1/3 = 33.33%
0%
Surprise 0% 0% 0% 0% 0% 0% 100% Table 5: Confusion Matrix for the Testing Results
Results Improvement:
PCA is also used for face recognition applications. The Principle components that have higher Eigen values may or may not correspond to varying expression. With this concept in mind we train network using different range of principle components. In the method mentioned we have used first 50 Principle components. But in order to improve the result we have tried different ranges of principle components. The
Testing Results No. of Misclassifications 7/ Recognition Rate 68.18% Error Rate 31.82%
Table 4: Testing using an unknown Subject
11
training results on whole database using different ranges of principle components are shown below,
Principle Components Range 30-60:
Figure 5: Result Improvement using Principle Components 30-60 and 500 epochs
Figure 6: Result Improvement using Principle Components 30-60 and 150 epochs
Results using 500 Epochs and 150 Epochs are same in this case but the results are improved by changing the Principle components range from 1-50 to 30-60.
Table 6: Recognition Using 30- PC's and 500 Epochs No. of Misclassifications
14/
Recognition Rate 93.43% Error Rate 6.57%
No. of Misclassifications
14/
Recognition Rate 93.43% Error Rate 6.57%
Table 7: Recognition Using 30- PC's and 150 Epochs
13
Facial Expression Recognition Using 2-D
Gabor Wavelets:
A Gabor filter is a linear filter whose impulse response is defined by a harmonic function multiplied by a Gaussian function [^7 ]. Facial expression images are coded using a multi-orientation, multi-resolution set of Gabor filters which are topographically ordered and aligned approximately with the face. The similarity space derived from this representation is compared with one derived from semantic ratings of the images by human observers. The results show that it is possible to construct a facial expression classifier with Gabor coding of the facial images as the input stage. The Gabor representation shows a significant degree of psychological plausibility, a design feature which may be important for human-computer interfaces [^8 ].
An example of 2-D Gabor filter is shown below
Figure 8: 2-D Gabor Filter
Mathematical Background:
The formula for complex Gabor function in space domain is as follows [ 9 ]
g x , y s x , ywr x , y
Where s x , y is a complex sinusoidal also known as carrier and wr x , y is a 2-D
Gaussian-shaped function, known as envelop. The complex sinusoidal is further defined as
s x , y exp j 2 0 x 0 y P
14
Where 0 , 0 and P define the spatial frequency and phase of sinusoidal
respectively. And the Gaussian envelop is defined as follows 2 0
(^22) 0 , .exp^2 wr x y K a x x r b y y r
Where x 0 (^) , y 0 are peek of the function, a and b are scaling parameters of Gaussian
and the (^) r stands for a rotation operation such that
x x 0 r x x 0 cos y y 0 sin
y y 0 r x x 0 sin y y 0 cos
Basic Framework:
Basic Framework for facial expression recognition using Gabor wavelets [ 10 ], as defined by me is, shown below,
Figure 9: Framework for facial Expression Recognition
This is basically a five step process. First of all we perform Gabor filtering on the input image and then we ignore the imaginary part of the result. And just take the real part of the result. Then we resize the filtered image in order to redue the background noise. Then we generate a vector from the 2-D matrix of the image and reduce its dimensionality using Principle Component Analysis. And then we use a classifier in order to classify the results into one of the seven basic expressions.
16
Figure 11: Improved Results Using Gabor Filtering
17
References:
[1] Gargesha and P. Kuchi, “ Facial Expression Recognition Using Artificial Neural Networks ”, EEE 551- Artificial Neural Computation Systems, (Spring 2002) [2] B. Fasel and Juergen Luettin, “ Automatic Facial Expression Analysis: A Survey” , Journal of the Pattern Recognition Society, accepted 15 February 2002. [3] “ Principal component analysis ”, URL : http://en.wikipedia.org/wiki/Karhunen-Lo%C3%A8ve_transform [4] Djmw, “ Principal component analysis ”, (March 23, 1999) URL : http://www.fon.hum.uva.nl/praat/manual/Principal_component_analysis.html [5] Lindsay I Smith, “ A Tutorial on Principle Component Analysis ” , (February 2002), URL : http://csnet.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf [6] Abdi, H. " Eigen-decomposition: eigenvalues and eigenvecteurs. In N.J. Salkind (Ed.): Encyclopedia of Measurement and Statistics. Thousand Oaks (CA): Sage. ", URL : http://www.utdallas.edu/~herve/Abdi-EVD2007-pretty.pdf, (2007) [7] “ Gabor Filter ”, URL : http://en.wikipedia.org/wiki/Gabor_filter [8] Michael Lyons and Shigeru Akamatsu, “ Coding Facial Expressions with Gabor Wavelets ”, Proceedings, Third IEEE International Conference on Automatic Face and Gesture Recognition, (April 14-16 1998), Nara Japan, IEEE Computer Society, pp. 200-205. [9] Javier, R. Mavellan, “ Tutorial on Gabor Filters ” URL: http://mplab.ucsd.edu/tutorials/pdfs/gabor.pdf [10] Michael J. Lyons, Julien Budynek, Andre Plante , Shigeru Akamatsu “ Classifying Facial Attributes using a 2-D GaborWavelet Representation and Discriminant Analysis ”, Proceedings of the 4th International Conference on Automatic Face and Gesture Recognition ,(28-30 March, 2000), Grenoble France, IEEE Computer Society, pp. 202-207.
