Download Exploratory Factor Analysis - Review Sheet | STAT 530 and more Study notes Statistics in PDF only on Docsity!
Exploratory Factor Analysis
Brian Habing - University of South Carolina â Last Updated: October 4, 2005
FA is not worth the time necessary to understand it and carry it out. -Hills, 1977
Factor analysis should not be used in most practical situations. -Chatfield and Collins, 1980, pg. 89.
At the present time, factor analysis still maintains the flavor of an art, and no single strategy should yet be "chiseled into stone". -Johnson and Wichern, 2002, pg. 517.
38 - The number of articles ISI Web of Science shows with the topic âFactor Analysisâ for the week ending October 1, 2005.
1213 - The number of FA articles in 2005 through October 1st^.
References:
Chatfield, C. & Collins, A. J. (1980). Introduction to Multivariate Analysis. London: Chapman and Hall.
Hair, J.F. Jr. , Anderson, R.E., Tatham, R.L., & Black, W.C. (1998). Multivariate Data Analysis, (5th Edition). Upper Saddle River, NJ: Prentice Hall.
Hatcher, L. (1994). A Step-by-Step Approach to Using the SAS System for Factor Analysis and Structural Equation Modeling. Cary, NC: SAS Institute Inc.
Hills, M. (1977). Book Review, Applied Statistics, 26 , 339-340.
Johnson, D.E. (1998). Applied Multivariate Methods for Data Analysis. Pacific Grove, CA: Brooks/Cole Publishing.
Johnson, R.A. & Wichern, D.W. (2002). Applied Multivariate Statistical Analyses (5th^ Edition). Upper Saddle River, NJ: Prentice Hall.
Mardia, K.V., Kent, J.T., & Bibby, J.M. (1979). Multivariate Analysis. New York: Academic Press.
Sharma, S. (1996). Applied Multivariate Techniques. United States: John Wiley & Sons.
Stevens, J. (2002). Applied Multivariate Statistics for the Social Sciences (4 th^ Edition). Mahwah, NJ: Lawrence Erlbaum Associates.
Thompson, B. (2000). Q-Technique Factor Analysis: One Variation on the Two-Mode Factor Analysis of Variables. In Grimm, L.G. & Yarnold, P., Reading and Understanding More Multivariate Statistics. Washington, D.C.: American Psychological Association.
Goal: To explain the variance in the observed variables in terms of underlying latent factors.
The Model: The following assumes that the p observed variables (the Xi ) that have been measured for each of the n subjects have been standardized.
q q qk k q
k k
k k
X F F u
X F F u
X F F u
λ λ
λ λ
λ λ
L
M
L
L
1 1
2 21 1 2 2
1 11 1 1 1
The Fj are the k common factors, the ui are the q unique (or specific) errors, and the λ ij are the q x k factor
loadings. The Fj have mean zero and standard deviation one, and are generally assumed to be independent. (We will assume this orthogonality below, but it is not true for oblique rotations.) The ui are also independent and the Fj and ui are mutually independent of each other.
In matrix form this can be written as:
X (^) q Ă 1 = Î q Ă k F k Ă 1 +u q Ă 1
which is equivalent to
ÎŁ=ÎÎT^ +cov(u)
where ÎŁ q x q is the correlation matrix of X (^) q x 1. Since the errors are assumed to be independent, cov(u) should be a q x q diagonal matrix. This implies that:
Var ( ) Var( ) 1
2 i
k
j
X (^) i = (^) â ij + u
The sum of Xi 's squared factor loadings is called its communality (the variance it has in common with the other variables through the common factors). The i th^ error variance is called the uniqueness or specificity of Xi (the variance that is specific to variable i ).
Path Diagram: Factor Analytic models are often represented by path diagrams. Each latent variable is represented by a circle, and each manifest variable is represented by a square. An arrow indicates causality⊠which can get to be a pretty complicated subject in some models!
The diagram at the left is for a 2-factor model of 3 variables ( k =2, q =3), where the first variable
loads only on factor 1 ( λ 12 =0), the second variable
loads on both factors, and the third variable loads
only on factor 2 ( λ 31 =0).
How many factors? There are several rules for determining how many factors are appropriate for your data.
Mardia, Kent, and Bibby (1979, pg. 258) point out that there is a limit to how many factors you can have and actually end up with a model that is simpler than what your raw data. The quantity s is the difference between the number of unique values in your data's q x q correlation matrix and the number of parameters in the k factor model:
s = ( q â k ) â ( q + k ) 2
It only makes sense to perform a factor analysis if s >0, and some programs will not let you estimate the factor analytic model if it is not true. Even though you could always exactly fit a 5 factor model to 5 variables using principal components and no specificity, s >0 only for one or two factors. The minimum number of variables required for different numbers of factors are:
Factors 2 3 4 5 6
Variables Required 5 7 8 9 11
In general this is not something we worry about too much since we usually want to have a much smaller number of factors than variables. Also, recall that we want several variables loading on each factor before we can actually trust that factor to be meaningful anyway.
Kaiser's Criterion / Eigen Value > 1 â Take as many factors as there are eigenvalues > 1 for the correlation matrix. Hair, et.al. (1998, pg. 103) reports that this rule is good if there are 20 to 50 variables, but it tends to take too few if there are <20 variables, and to many if there are >50. Stevens (2002, pg. 389) reports that it tends to take too many if there are > 40 variables and their communalities are around 0.4. It tends to be accurate with 10-30 variables and their communalities are around 0.7.
Scree Plot â Take the number of factors corresponding to the last eigenvalue before they start to level off. Hair, et.al. (1998, pg. 104) reports that it tends to keep one or more factors more than Kaiser's criterion. Stevens (2002, pg. 390) reports that both Kaiser and Scree are accurate if n >250 and communalities â„ 0.6.
Fixed % of Variance Explained â Keep as many factors as are required to explain 60%, 70%, 80-85%, or 95%. There is no general consensus and one should check what is common in your field. It seems reasonable that any decent model should have at least 50% of the variance in the variables explained by the common factors.
A priori â If you have a hypothesis about the number of factors that should underlie the data, then that is probably a good (at least minimum) number to use.
In practice there is no single best rule to use and a combination of them is often used, so if you have no a priori hypothesis check all three and use the closest thing to a majority decision.
There are also a variety of other methods out there that are very popular with some authors: Minimum average partial correlation, parallel analysis, and modified parallel analysis are three of them. Three that require multivariate normality are: likelihood ratio test, AIC (Akaike's information criterion), and SBC (Schwartzâs Bayesian criterion).
Methods for Fitting the Factor Analysis Model:
Principal Components Factor Analysis â Just take the first m loadings from the principal components solution and multiply by the square root of the corresponding eigen value. This is not really appropriate since it attempts to explain all of the variance in the variables and not just the common variance. It therefore will often have highly correlated errors. However, Hair, et.al. (1998, pg. 103) reports that it often gives similar results to other methods if there are â„30 variables or if most variables have communalities > 0.6.
Principal Factor Factor Analysis â (a.k.a. Principal Axis Factoring and sometimes even Principal Components Factoring!) Come up with initial estimates of the communality for each variable and replace the diagonals in the correlation matrix with those. Then do principal components and take the first m loadings. Because you have taken out the specificity the error matrix should be much closer to a diagonal matrix. There are various initial estimates used for the initial communalities: the absolute value of the maximum correlation of that variable with any of the others, the squared multiple correlation coefficient for predicting that variable from the others in multiple regression, and the corresponding diagonal element from the inverse of the correlation matrix. There seems to be no agreement on which is best⊠but the first is a slight bit easier to program.
Iterated Principal Factor Factor Analysis - This begins the same way as the principal factor method, but you use the fitted model to get better estimates of the communalities, and then you keep repeating the process. This method will often fail to fit because of the Heywood case (estimated communalities > 1, negative error variances) though.
Maximum Likelihood Factor Analysis â Requires the assumption of multivariate normality and is difficult to program. It does allow for various tests of hypotheses however.
Other Methods â Alpha Factoring, Image Factoring, Harris Factoring, Rao's Canonical Factoring, Unweighted Least Squares
I would recommend using the iterated principal factor solution if it does not have impossible communalities or error variances. If the iterated method fails then I would use the non-iterated principal factor method if the error covariances do not seem unreasonably large. If that too fails you either need more factors, or the factor analytic model is possibly inappropriate for your data set.
Rotation: It is often considered best to use an orthogonal rotation because then all of the mathematical representations we've used so far will work. Two of the most popular are:
Varimax â Maximizes the sum of the variances of the squared factor loadings within the columns. This tends to force each variable to load highly on as few factors as possible. Ideally it will cause each variable to load on only one factor and thus point out good summed scores that could be made. If there is an overall underlying general factor that is working on all of the variables this rotation will not find it. (e.g. The math test has geometry, trigonometry, and algebra⊠but does it also have an overall general math ability too?)
Quartimax â Whereas varimax focuses on the columns, quartimax focuses on the rows. Sharma (1996, pg.
- reports that: it tends to have all of the variables load highly on one factor, and then each variable will tend to load highly on one other factor and near zero on the rest; unlike varimax this method will tend to keep the general factor. It sometimes doesnât seem to differ much from the initial solution.
There are also oblique rotations that don't keep the independence between the factors. Popular ones include Promax, Oblimin, and Orthoblique. Interpret these with caution!
