Download Factor Analysis: Extracting Loadings & Rotating Factors from Correlation Matrix and more Study notes Mathematical Statistics in PDF only on Docsity!
1
Extraction of Initial Factors
Principal Components Extraction (PC)
The matrix of factor loadings based on factor m is
Λ (^) m = Ω (^) m Γ 1 2 m
where
Ω ω ω ω Γ
m m m m
1 2 1 2
diag , , ,
K
K
1 6 2 γ^ γ^ γ 7
The communality of variable i is given by
hi j ij j
m
=
∑γ^ ω^
2 1
Analyzing a Correlation Matrix
γ 1 ≥ γ 2 ≥ K ≥ γ m are the eigenvalues and ω (^) i are the corresponding eigenvectors of R , where R is the correlation matrix.
Analyzing a Covariance Matrix
γ 1 ≥ γ 2 ≥ K ≥ γ m are the eigenvalues and ω (^) i are the corresponding eigenvectors of Σ , where Σ = ( σ (^) ij ) n × n is the covariance matrix.
The rescaled loadings matrix is Λ (^) mR = [ diag Σ ]− 1/2^ Λ m. The rescaled communality of variable i is h (^) iR = σ (^) ii −^1 h^ i.
Principal Axis Factoring
Analyzing a Correlation Matrix
An iterative solution for communalities and factor loadings is sought. At iteration i , the communalities from the preceding iteration are placed on the diagonal of R , and the resulting R is denoted by R i. The eigenanalysis is performed on R i and the new communality of variable j is estimated by
h (^) j i k i jk i j
m 0 5 = 0 5 0 5 =
∑γ^ ω^
2 1
The factor loadings are obtained by
Λ (^) m i 0 5 = Ω (^) m i 0 5 Γ 1 2 m i 0 5
Iterations continue until the maximum number (default 25) is reached or until the maximum change in the communality estimates is less than the convergence criterion (default 0.001).
Analyzing a Covariance Matrix
This analysis is the same as analyzing a correlation matrix, except Σ is used instead of the correlation matrix R. Convergence is dependent on the maximum change of rescaled communality estimates. At iteration i , the rescaled loadings matrix is Λ^ m i R ( ) =^ [ diag^ Σ^ ]−1/2^ Λ m i ( ). The rescaled communality of variable i is h (^) j i R ( ) = σ (^) ii −^1 h^ j i ( ).
Maximum Likelihood (ML)
The maximum likelihood solutions of Λ and ψ 2 are obtained by minimizing
F = tr^ ′ + p !
" $#
− − ΛΛ ψ 2 ΛΛ ψ
1 2 1 4 9 R^ log 4 9 R
with respect to Λ and ψ , where p is the number of variables, Λ is the factor loading matrix, and ψ 2 is the diagonal matrix of unique variances.
f
f x
f x x
f x
k k k m
p
i
k ik k m
p
i j
ij i
ik jk k^ n k n
in jn ij n
m
k m
p
1 6^ ψ (^4 )
4 9
= − + +^ − − +
− = +
− = +
= + =
∑
∑
∑ ∑
log γ γ
γ
∂ ∂ δ^
∂ ω^ ω
γ γ γ γ ω^ ω^ δ
1 1
1 2 1
2
1 1
ω
where
δ (^) ij
i j = (^) i j
% & '
if if
The approximate second-order derivatives
∂ ∂ ∂
ω ω
2
1
2 f x (^) i xj (^) k m ik^ jk
p ≅
= +
∑
are used in the initial step and when the matrix of the exact second-order derivatives is not positive definite or when all elements of the vector d are greater than 0.1. If ∂ 2 f ∂ x (^) i^2^ < 0 05. (Heywood variables), the diagonal element is replaced by 1 and the rest of the elements of that column and row are set to 0. If the
value of f ( ψ ) is not decreased by step d , the step is halved and halved again until
the value of f ( ψ ) decreases or 25 halvings fail to produce a decrease. (In this case,
the computations are terminated.) Stepping continues until the largest absolute value of the elements of d is less than the criterion value (default 0.001) or until the maximum number of iterations (default 25) is reached. Using the converged value of ψ (denoted by ψ$ ), the eigenanalysis is performed on the matrix ψ$^ R −^1 ψ$. The factor loadings are computed as
Λ^ $^ m = ψΩ$ (^) m 4 Γ m −^1^ − I m 9 1 2
where
Γ Ω ω ω ω
m m m m
diag γ 1 γ 2 γ 1 2
K
K
1 6 1 6
Unweighted and Generalized Least Squares (ULS, GLS)
The same basic algorithm is used in ULS and GLS methods as in maximum likelihood, except that
f
k k m
p
k k m
1 6ψ = (^) p 1 − 6
%
&
K KK
'
K KK
= +
= +
∑
∑
γ
γ
2
(^1 )
1
for ULS
for GLS
for the ULS method, the eigenanalysis is performed on the matrix R − ψ 2 , where γ 1 ≥ γ 2 ≥ K ≥ γ p are the eigenvalues. In terms of the derivatives, for ULS
γ ω
ω ω γ^ γ γ γ
ω ω δ γ^ ω
f x
x
f x x
x x x
i
i k ik k m
p
i j
i j ik jk k m
p k n k n
ik jk n
m ij i k^ ik k m
p
!
"
$
= +
= + = = +
∑
∑ ∑ ∑
2 1
2
1 1
2 2 1
and
∂ ∂ ∂ ω^ ω
2
1
2 f 4 x (^) i x (^) j x xi^ j^ k m ik^ jk
p
= +
∑
Alpha (Harman, 1976)
Iteration for Communalities
At each iteration i :
• The eigenvalues 4 r 0 5 i 9 and eigenvectors 0 Ω ( i ) 5 of
H 1 2 0 i − 1 5 0 R − I H 5 1 2 0 i − 15 + I are computed.
- The new communalities are
h (^) k i j i kj i h j
m
0 5 γ^ 0 5 ω^ 0 5 k i 0 5
2 1
1
The initial values of the communalities, H 0 , are
h
r h io r
ii (^) io
j ij
&K
'K
1 1 R 10 − 8 and all 0 1 max otherwise
where r ii^ is the i th diagonal entry of R −^1. If R ≥ 10 − 8 and all r ii^ are equal to one, the procedure is terminated. If for some i , max j ij
r > 1 , the procedure is terminated.
- Iteration stops if any of the following are true:
max k k i^ k i
k i
h h
i
h k
0 5 0 5
0 5
− 1
EPS
MAX
for any
Final Communalities and Factor Pattern Matrix
The communalities are the values when iteration stops, unless the last termination criterion is true, in which case the procedure terminates. The factor pattern matrix is
F m (^) = H 0 5^ 1 2 f^ Ω m f (^) 0 5 Γ1 2 m f 0 5
where f is the final iteration.
Image (Kaiser, 1963)
Factor Analysis of a Correlation Matrix
- Eigenvalues and eigenvectors of S −^1 RS^ −^1 are found.
S
R
2 11 1
diag th diagonal element of
r r r i
nn ii
4 ,^ K, 9
Factor Rotations
Orthogonal Rotations (Harman, 1976)
Rotations are done cyclically on pairs of factors until the maximum number of iterations is reached or the convergence criterion is met. The algorithm is the same for all orthogonal rotations, differing only in computations of the tangent values of the rotation angles.
- The factor pattern matrix is normalized by the square root of communalities:
Λ ∗ m^ = H −1 2Λ m
where
Λ (^) m = 2 λ 1 , K,λ m 7 is the factor pattern matrix
H = diag 1 h 1 (^) , K, hn 6 is the diagonal matrix of communalities
- The transformation matrix T is initialized to I m
- At each iteration i (1) The convergence criterion is
SV (^) i n (^) kj i n k
n kj i k
n
j
m 0 5 =^ 0 5 − 0 5
∗
∗ = =
∑ ∑ λ ∑λ
4 1
2 1
2 2 1
where the initial value of Λ^ ∗ m^ 0 5 1 is the original factor pattern matrix. For subsequent iterations, the initial value is the final value of Λ∗ m i^0 − 15 when all factor pairs have been rotated. (2) For all pairs of factors 3 λ (^) j ,λ k 8 where k > j , the following are computed:
(a) Angle of rotation
P = 1 4 tan −^11 X Y 6
where
X
D AB n D mAB n D
C A B n C m A B n C
u f f v^ f^ f^ p n
A u (^) B v
C u v (^) D u v
p i pj i pk i p i^ pj i^ pk i
p i p
n p i p
n
p i p i p
n p i p i p
n
% &K 'K
− − − −
%
&
KK
'
K K
= − =^ =
∗ ∗ ∗^ ∗
= (^) =
= (^) =
∑ (^) ∑
∑ (^) ∑
2 2 2 2
2 2
(^1 )
2 2 (^1 )
Varimax Equimax Quartimax
Y =
Varimax Equimax Quartimax
4 9 4 9
0 5 0 5 0 5 0 5^ 0 5^ 0 5
0 5 (^) 0 5
0 5 0 5 0 5 0 5
, K,
If sin 0 5 P ≤ 10 −^15 , no rotation is done on the pair of factors.
(b) New rotated factors
~ (^) , ~ (^) , cos sin sin cos
λ (^) j i λ (^) k i λ (^) j i λ k i P^ P (^4) 0 5 0 5 9 4 (^) 0 5 0 5 (^9) P P
0 5 0 5 0 5 0 5
= ∗^ ∗ −
where λ ∗ j i 0 5^ are the last values for factor j calculated in this iteration.
(c) Accrued rotation transformation matrix
~ , ~ (^) , cos^ sin sin cos
t t t t
P P
(^3) j k 8 3 (^) j k (^8) P P
0 5 0 5 0 5 0 5
where t (^) j and t (^) k are the last calculated values of the j th and k th columns of T.
Oblique Rotations
The direct oblimin method (Jennrich and Sampson, 1966) is used for oblique
rotation. The user can choose the parameter δ. The default value is δ = 0.
(a) The factor pattern matrix is normalized by the square root of the communalities
Ω (^) m^ *^ = H −1 2Λ m
where
h (^) j jk k
m
=
∑λ
2 1
If no Kaiser is specified, this normalization is not done. (b) Initializations The factor correlation matrix C is initialized to I m. The following are also computed:
s (^) h k n
u i^ m
v
x v n u
D u
G x
H s n D
FO H G
k (^) k
i ji j
n
i ji j
n
i i i
i i
m
i i
m
i k
n
= %& '
∗ = ∗ = = = = ∑
∑
∑
∑
∑
1 4 1 2 1 1 2 1
2
if Kaiser if no Kaiser ,^ ,
K
λ K
λ
δ
δ
1 6
1 6
(c) At each iteration, all possible factor pairs are rotated. For a pair of factors λ ∗ p
and λ (^) q ∗^0 p ≠ q 5 , the following are computed:
- A root, a , of the equation
b^3 + P b ′^2 + Q b ′ + R = 0 is computed, as well as:
A c a a t A t a t
= + (^) pq +
=
1
1 2
2 1
- The rotated pair of factors is
~ λ (^) , λ~ (^) λ ,λ p q p q
∗^ ∗^ ∗ ∗ t^ a
=^
(^4 9 01 )
These replace the previous factor values.
- New values are computed for
~
~ ~^ ~
,
u A u
x A x
v
u
x v n u
S S
D D u u
G G x x
p p
p p
q iq i
n
q iq i
n
q q q
k pq k kp kq
pq p q
pq p q
∗
∗
∗ ∗
∑
∑
2 4 1 2 1 2
2 2
λ
λ
δ
λ λ
1 6
All values designated as V ~ replaces V and are used in subsequent calculations.
- The new factor correlations with factor p are
c t c t c i p
c c
c
ip ip iq
pi ip
pp
1 −^1
0 5
- After all factor pairs have been rotated, iteration is terminated if
MAX iterations have been done or F (^1) 0 5 i − F (^1 0) i − 15 < 0 FO (^) 50 EPS 5
where
F H G
H s n D
F FO
i
k k
n
2 1
2
0
0 5
0 5
1 6
=
∑
~ (^) ~ (^) δ ~
Otherwise, the factor pairs are rotated again.
- The final rotated factor pattern matrix is
λ~^ λ~ m =^^ H 1 2 ∗ m
where ~ λ (^) m is the value in the final iteration.
and the diagonal elements do not equal 1, we must modify the rotated factor to f (^) pro max = C fpro max_ temp
where C = { diag (( Q Q ′ )− 1 )}−1/
The rotated factor pattern is Λ (^) pro max = Λvar max i QC −^1
The correlation matrix of the factors is R (^) ff = C Q Q ( ′ )− 1 C ′
The factor structure matrix is Λ (^) S = Λ pro max R ff
Factor Score Coefficients (Harman, 1976)
Regression
1
W
R S
% &
K
'
K
− − −
m m m m m m
1 1
PC without rotation PC with rotation otherwise
1 6
where S S
m m m
factor structure matrix Λ for orthogonal rotations
For PC without rotation if any γ (^) i ≤ 10 −^8 , factor score coefficients are not computed. For PC with rotation, if the determinant of Λ ′ m Λ m is less than 10 −^8 , the coefficients are not computed. Otherwise, if R is singular, factor score coefficients are not computed.
1 This algorithm applies to SPSS 7.0 and later releases.
Bartlett
W = J −^1 Λ′ U −^2
where J U U R R
Λ −^2 Λ
Anderson Rubin
W = ′ U −^ RU −^ ′ U
− (^) − Λ 2 2 Λ Λ
1 2 (^2) 4 9
where the symmetric square root of the parenthetical term is taken.
Optional Statistics (Dziubin and Shirkey, 1974)
- The anti-image covariance matrix A = (^) 3 8 aij is given by
a r ij r r
ij = ii jj
- The chi-square value for Bartlett’s test of sphericity is
χ 2 1 2 5 6
= − W − − p^ + log R
with p p 0 − 15 2 degrees of freedom.
- The Kaiser-Mayer-Olkin measure of sample adequacy is
KMO
r
r a
KMO
r
r a
j
ij i j ij i j
ij i j
ij i j ij i j
ij i j
≠
≠
∗ ≠
≠
≠
∗ ≠
∑
∑ ∑
∑∑
∑ ∑ ∑∑
2
2 2
2
2 2
where aij^ ∗^ is the anti-image correlation coefficient.
