Download Discovering Statistics Using SPSS: Analyzing Repeated-Measures Designs and more Study notes Statistics in PDF only on Docsity!
Chapter 14: Repeated-measures designs
Smart Alex’s Solutions
Task 1
It is common that lecturers obtain reputations for being ‘hard’ or ‘light’ markers (or to use the students’ terminology, ‘evil manifestations from Beelzebub’s bowels’ and ‘nice people’) but there is often little to substantiate these reputations. A group of students investigated the consistency of marking by submitting the same essays to four different lecturers. The mark given by each lecturer was recorded for each of the eight essays. The independent variable was the lecturer who marked the report and the dependent variable was the percentage mark given. The data are in the file TutorMarks.sav. Conduct a one-‐way ANOVA on these data by hand Essay Tutor 1 (Prof Field) Tutor 2 (Prof Smith) Tutor 3 (Prof Scrote) Tutor 4 (Prof Death) Mean S^2 1 62 58 63 64 61.75 6. 2 63 60 68 65 64.00 11. 3 65 61 72 65 65.75 20. 4 68 64 58 61 62.75 18. 5 69 65 54 59 61.75 43. 6 71 67 65 50 63.25 84. 7 78 66 67 50 65.25 132. 8 75 73 75 45 67.00 216. Mean 68.875 64.25 65.25 57. There were eight essays, each marked by four different lecturers. Their marks are shown in the table above. In addition, the mean mark given by each lecturer is shown in the table, and also the mean mark that each essay received and the variance of marks for a particular essay. Now,
the total variance within essays will in part be caused by the fact that different lecturers are harder or softer markers (the manipulation), and in part by the fact that the essays themselves will differ in quality (individual differences).
The total sum of squares (SST)
Remember from one-‐way independent ANOVA that SST is calculated using the following equation: SS! = 𝑠!"#$%^!^ 𝑁 − 1 Well, in repeated-‐measures designs the total sum of squares is calculated in exactly the same way. The grand variance in the equation is simply the variance of all scores when we ignore the group to which they belong. So if we treated the data as one big group it would look as follows: 62 58 63 64 63 60 68 65 65 61 72 65 68 64 58 61 69 65 54 59 71 67 65 50 78 66 67 50 75 73 75 45 Grand Mean = 63. The variance of these scores is 55.028 (try this on your calculator). We used 32 scores to generate this value, and so N is 32. As such the equation becomes: SS! = 𝑠!"#$%^!^ 𝑁 − 1 = 55. 028 32 − 1 = 1705. 868
The n s simply represent the number of scores on which the variances are based (i.e. the number of experimental conditions, or in this case the number of lecturers). All of the variances we need are in the table, so we can calculate SSW as: SS! = 𝑠!""#$%^!^ 𝑛! − 1 + 𝑠!""#$%^!^ 𝑛! − 1 + 𝑠!""#$%^!^ 𝑛! − 1 + ⋯ + 𝑠!""#$^!^! 𝑛! − 1 = 6. 92 4 − 1 + 11. 33 4 − 1 + 20. 92 4 − 1 + 18. 25 4 − 1
- 58 4 − 1 + 84. 25 4 − 1 + 132. 92 4 − 1 + 216 4 − 1 = 20. 76 + 34 + 62. 75 + 54. 75 + 130. 75 + 252. 75 + 398. 75 + 648 = 1602. The degrees of freedom for each person are n – 1 (i.e. the number of conditions minus 1). To get the total degrees of freedom we add the df for all participants. So, with eight participants (essays) and four conditions (i.e. n = 4) we get 8 × 3 = 24 degrees of freedom.
The model sum of squares (SSM)
So far, we know that the total amount of variation within the data is 1705.868 units. We also know that 1602.5 of those units are explained by the variance created by individuals’ (essays’) performances under different conditions. Now some of this variation is the result of our experimental manipulation, and some of this variation is simply random fluctuation. The next step is to work out how much variance is explained by our manipulation and how much is not. In independent ANOVA, we worked out how much variation could be explained by our experiment (the model SS) by looking at the means for each group and comparing these to the overall mean. So, we measured the variance resulting from the differences between group means and the overall mean. We do exactly the same thing with a repeated-‐measures design. First we calculate the mean for each level of the independent variable (in this case the mean mark given by each lecturer) and compare these values to the overall mean of all marks. So, we calculate this SS in the same way as for independent ANOVA:
- Calculate the difference between the mean of each group and the grand mean.
- Square each of these differences.
- Multiply each result by the number of subjects within that group ( ni ).
- Add the values for each group together: SS! = 𝑛! 𝑥! − 𝑥!"#$% ! Using the means from the essay data, we can calculate SSM as follows:
SSM = 8(68.875 – 63.9375)^2 + 8(64.25 – 63.9375)^2 + 8(65.25 – 63.9375)^2
+ 8(57.375 – 63.9375)^2
= 8(4.9375)^2 + 8(0.3125)^2 + 8(1.3125)^2 + 8(–6.5625)^2
For SSM, the degrees of freedom ( df M) are again one less than the number of things used to calculate the sum of squares. For the model sums of squares we calculated the sum of squared errors between the four means and the grand mean. Hence, we used four things to calculate these sums of squares. So, the degrees of freedom will be 3. So, as with independent ANOVA, the model degrees of freedom is always the number of groups ( k ) minus 1: 𝑑𝑓! = 𝑘 − 1 = 3
The residual sum of squares (SSR)
We now know that there are 1706 units of variation to be explained in our data, and that the variation across our conditions accounts for 1602 units. Of these 1602 units, our experimental manipulation can explain 554 units. The final sum of squares is the residual sum of squares (SSR), which tells us how much of the variation cannot be explained by the model. This value is the amount of variation caused by extraneous factors outside of experimental control (such as natural variation in the quality of the essays). Knowing SSW and SSM already, the simplest way to calculate SSR is to subtract SSM from SSW: SS! = SS! − SS! = 1602. 5 − 554. 125 = 1048. 375 The degrees of freedom are calculated in a similar way: 𝑑𝑓! = 𝑑𝑓! − 𝑑𝑓! = 24 − 3 = 21
The mean squares
SSM tells us how much variation the model (e.g., the experimental manipulation) explains and SSR tells us how much variation is due to extraneous factors. However, because both of these
Doing the analysis
To conduct an ANOVA using a repeated-‐measures design, activate the define factors dialog box by selecting. In the define factors dialog box ( Error! Reference source not found. ) you are asked to supply a name for the within-‐subject (repeated-‐measures) variable. In this case the repeated-‐measures variable was the tutor marking the essay, so replace the word factor1 with the word TUTOR. Next, you have to tell SPSS how many levels there were (i.e., how many experimental conditions there were). In this case, there were four tutors, so enter the number 4 into the box labelled Number of Levels. Click on to add this variable to the list of repeated-‐measures variables. This variable will now appear in the white box at the bottom of the dialog box as TUTOR(4). The finished dialog box is shown in the left-‐hand side of Figure. Next, click on to go to the main dialog box. The main dialog box (see the right-‐hand side of Figure ) has a space labelled Within-‐Subjects Variables that contains a list of four question marks followed by a number. These question marks are for the variables representing the four levels of the independent variable. The variables corresponding to these levels should be selected and placed in the appropriate space. We have only four variables in the data editor, so it is possible to select all four variables at once (by clicking on the variable at the top, pressing the Shift key and then clicking on the last variable that you want to select). The selected variables can then be dragged to the box labelled Within-‐Subjects Variables (or click on ). When all four variables have been transferred, you can select various options for the analysis. There are several options that can be accessed with the buttons at the side of the main dialog box. These options are similar to the ones we have already encountered.
Figure 1 If you click on in the main dialog box you can access the contrasts dialog box ( Error! Reference source not found. ). There is no particularly good contrast for the data we have (the simple contrast is not very useful because we have no control category) so let’s use the repeated contrast, which will compare each tutor’s marks against the previous tutor. When you have selected this contrast, click on to return to the main dialog box. Figure 2
(that’s because I’m so nice, you see … or it could be because I’m stupid and so have low academic standards?). Professor Death, on the other hand, gave very low grades. These mean values are useful for interpreting any effects that may emerge from the main analysis. Output 1 Output 2 contains information about Mauchly’s test. This test should be non-‐significant if we are to assume that the condition of sphericity has been met. The output shows Mauchly’s test for the tutor data, and the important column is the one containing the significance value. The significance value (.043) is less than the critical value of .05, so we accept that the variances of the differences between levels are significantly different. In other words, the assumption of sphericity has been violated. Knowing that we have violated this assumption, a pertinent question is: how should we proceed? Output 2 SPSS produces three corrections based upon the estimates of sphericity advocated by Greenhouse and Geisser (1959) and Huynh and Feldt (1976). Both of these estimates give rise to a correction factor that is applied to the degrees of freedom used to assess the observed F -‐ ratio_._ The Greenhouse–Geisser correction varies between 1/( k − 1 ) (where k is the number of repeated-‐measures conditions) and 1. The closer that is to 1.00, the more homogeneous the variances of differences, and hence the closer the data are to being spherical. In a situation in which there are four conditions (as with our data) the lower limit of will be 1/(4−1), or 0.33 (known as the lower-‐bound estimate of sphericity). The calculated value of in the output is .558. This is closer to the lower limit of 0.33 than it is to the upper limit of 1, and it Within-Subjects Factors Measure: MEASURE_ TUTOR TUTOR TUTOR TUTOR TUTOR 1 2 3 4 Dependent Variable Mauchly's Test of Sphericitya Measure: MEASURE_ .131 11.628 5 .043 .558 .712. Within Subjects Effect TUTOR Mauchly's W Approx. Chi-Square df Sig. Greenhouse-Geisser Huynh-Feldt Lower-bound Epsilonb Tests the null hypothesis that the error covariance matrix of the orthonormalized transformed dependent variables is proportional to an identity matrix. a. Design: Intercept Within Subjects Design: TUTOR May be used to adjust the degrees of freedom for the averaged tests of significance. Corrected tests are displayed in the layers (by default) of the Tests of Within Subjects Effects table. b. εˆ εˆ εˆ
therefore represents a substantial deviation from sphericity. We will see how these values are used in the next section.
The main ANOVA
Output 3 shows the results of the ANOVA for the within-‐subjects variable. This table can be read much the same as for one-‐way between-‐group ANOVA. There is a sum of squares for the repeated-‐measures effect of tutor , which tells us how much of the total variability is explained by the experimental effect. Note the value is 554.125, which is the model sum of squares (SSM) that we calculated in Task 1. There is also an error term, which is the amount of unexplained variation across the conditions of the repeated-‐measures variable. This is the residual sum of squares (SSR) that was calculated earlier, and note the value is 1048.375 (which is the same value as calculated). As I explained earlier, these sums of squares are converted into mean squares by dividing by the degrees of freedom. As we saw before, the df for the effect of tutor are simply k − 1, where k is the number of levels of the independent variable. The error df are ( n − 1)( k − 1), where n is the number of participants (or in this case, the number of essays) and k is as before. The F -‐ratio is obtained by dividing the mean squares for the experimental effect (184.708) by the error mean squares (49.923). As with between-‐group ANOVA, this test statistic represents the ratio of systematic variance to unsystematic variance. The value of F (184.71/49.92 = 3.70) is then compared against a critical value for 3 and 21 degrees of freedom. SPSS displays the exact significance level for the F -‐ratio. The significance of F is .028, which is significant because it is less than the criterion value of .05. We can, therefore, conclude that there was a significant difference between the marks awarded by the four lecturers. However, this main test does not tell us which lecturers differed from each other in their marking. Output 3 Tests of Within-Subjects Effects Measure: MEASURE_ 554.125 3 184.708 3.700. 554.125 1.673 331.245 3.700. 554.125 2.137 259.329 3.700. 554.125 1.000 554.125 3.700. 1048.375 21 49. 1048.375 11.710 89. 1048.375 14.957 70. 1048.375 7.000 149. Sphericity Assumed Greenhouse-Geisser Huynh-Feldt Lower-bound Sphericity Assumed Greenhouse-Geisser Huynh-Feldt Lower-bound Source TUTOR Error(TUTOR) Type III Sum of Squares df Mean Square F Sig. a. Computed using alpha =.
Therefore, we should probably go with the Greenhouse–Geisser correction and conclude that the F -‐ratio is non-‐significant. These data illustrate how important it is to use a valid critical value of F : it can mean the difference between a statistically significant result and a non-‐significant result. More important, it can mean the difference between making a Type I error and not. Had we not used the corrections for sphericity we would have concluded erroneously that the markers gave significantly different marks. However, I should quantify this statement by saying that this example also highlights how arbitrary it is that we use a .05 level of significance. These two corrections produce significance values only marginally less than or more than .05, and yet they lead to completely opposite conclusions! So, we might be well advised to look at an effect size to see whether the effect is substantive regardless of its significance. We also saw earlier that a final option, when you have data that violate sphericity, is to use multivariate test statistics (MANOVA) because they do not make this assumption (see O’Brien & Kaiser, 1985). The repeated-‐measures procedure in SPSS automatically produces multivariate test statistics. Output 4 shows the multivariate test statistics for this example. The column displaying the significance values clearly shows that the multivariate tests are non-‐ significant (because p is .063, which is greater than the criterion value of .05). Bearing in mind the loss of power in these tests, this result supports the decision to accept the null hypothesis and conclude that there are no significant differences between the marks given by different lecturers. The interpretation of these results should stop now because the main effect is non-‐ significant. However, we will look at the output for contrasts to illustrate how these tests are displayed in the SPSS Viewer. Output 4 The transformation matrix requested in the options is shown in Output 5, and we have to draw on our knowledge of contrast coding to interpret this table. The first thing to remember is that a code of 0 means that the group is not included in a contrast. Therefore, contrast 1 (labelled Level 1 vs. Level 2 in the table) ignores Prof Scrote and Prof Death. The next thing to remember is that groups with a negative weight are compared to groups with a positive Multivariate Testsa .741 4.760c^ 3.000 5.000. .259 4.760c^ 3.000 5.000. 2.856 4.760c^ 3.000 5.000. 2.856 4.760c^ 3.000 5.000. Pillai's Trace Wilks' Lambda Hotelling's Trace Roy's Largest Root Effect TUTOR Value F Hypothesis df Error df Sig. Design: Intercept Within Subjects Design: TUTOR a. b. Computed using alpha =. c. Exact statistic
weight. In this case this means that the first contrast compares Prof Field against Prof Smith. Using the same logic, contrast 2 (labelled Level 2 vs. Level 3 ) ignores Prof Field and Prof Death and compares Prof Smith and Prof Scrote. Finally, contrast 3 ( Level 3 vs. Level 4 ) compares Prof Death with Prof Scrote. This pattern of contrasts is consistent with what we expect to get from a repeated contrast (i.e. all groups except the first are compared to the preceding category). The transformation matrix, which appears at the bottom of the output, is used primarily to confirm what each contrast represents. Output 5 Above the transformation matrix, we should find a summary table of the contrasts (Output 6). Each contrast is listed in turn, and as with between-‐group contrasts, an F -‐test is performed that compares the two chunks of variation. So, looking at the significance values from the table, we could say that Prof Field marked significantly more highly than Prof Smith ( Level 1 vs. Level 2 ), but that Prof Smith’s marks were roughly equal to Prof Scrote’s ( Level 2 vs. Level 3 ) and Prof Scrote’s marks were roughly equal to Prof Death’s ( Level 3 vs. Level 4 ). However, the significant contrast should be ignored because of the non-‐significant main effect (remember that the data did not obey sphericity). The important point to note is that the sphericity in our data has led to some important issues being raised about correction factors, and about applying discretion to your data (it’s comforting to know that the computer does not have all of the answers, but it’s slightly alarming to realize that this means we have to actually know some of the answers ourselves). In this example we would have to conclude that no significant differences existed between the marks given by different lecturers. However, the ambiguity of our data might make us consider running a similar study with a greater number of essays being marked.
(indicating a low level of variability in our data). However, there is a very low correlation between the marks given by Prof Death and myself (indicating a high level of variability between our marks). It is this large variability between Prof Death and myself that has produced the non-‐significant result despite the average marks being very different (this observation is also evident from the standard errors). Task 3 Calculate the effect sizes for the analysis in Task 1. In repeated-‐measures ANOVA, the equation for ω^2 is: 𝜔!^ =
MSM − MSR
MSR +
MSB − MSR
𝑘 +^
𝑛𝑘 MSM^ −^ MSR
SPSS doesn’t give us SSW in the output, but we know that this is made up of SSM and SSR, which we are given. By substituting these terms, and rearranging the equation, we get: SST = SSB + SSM + SSR SSB = SS! − SSM − SSR The next problem is that SPSS, which is clearly trying to hinder us at every step, doesn’t give us SST and, I’m afraid (unless I’ve missed something in the output), you’re just going to have to calculate it by hand. From the values we calculated earlier, you should get: SSB = 1705. 868 − 554. 125 − 1048. 375 = 103. 37 The next step is to convert this to a mean squares by dividing by the degrees of freedom, which in this case are the number of people in the experiment minus 1 ( n – 1): MSB=
SSB
dfB
SSB
Having done all this and probably died of boredom in the process we must now resurrect ourselves with renewed vigour for the effect size equation, which becomes:
𝜔!^ =
8 × 4
4 +^
8 × 4 184.^71 −^49.^92
So, we get ω^2 = .24. If you calculate it the same way as for the independent ANOVA you should get a slightly bigger answer (.25 in fact). I’ve mentioned at various other points that it’s actually more useful to have effect size measures for focused comparisons anyway (rather than the main ANOVA), and so a slightly easier approach to calculating effect sizes is to calculate them for the contrasts we did. For these we can use the equation that we’ve seen before to convert the F -‐values (because they all have 1 degree of freedom for the model) to r : 𝑟 = 𝐹 1 , dfR 𝐹 1 , dfR + dfR For the three comparisons we did, we would get: 𝑟Field vs. Smith =
𝑟Smith vs. Scrote =
𝑟Scrote vs. Death =
Therefore, the differences between Profs Field and Smith and between Scrote and Death were both large effects, but the differences between Profs. Smith and Scrote were small.
Reporting one-way repeated-measures ANOVA
We could report the main finding as follows:
- The results show that the mark of an essay was not significantly affected by the lecturer who marked it, F (1.67, 11.71) = 3.70, p = .063. If you choose to report the sphericity test as well, you should report the chi-‐square approximation, its degrees of freedom and the significance value. It’s also nice to report the
The error bar chart of the roving eye data in Figure 4 shows the mean number of women who were eyed up after different doses of alcohol. It’s clear from this chart that the mean number of women is pretty similar between 1 and 2 pints, and for 3 and 4 pints, but there is a jump after 2 pints. Output 8 Output 8 shows the initial diagnostic statistics. First, we are told the variables that represent each level of the independent variable. This box is useful to check that the variables were entered in the correct order. The next table provides basic descriptive statistics for the four levels of the independent variable. This table confirms what we saw in Figure 4. Output 9 Output 9 contains Mauchly’s test, and we hope to find that it’s non-‐significant if we are to assume that the condition of sphericity has been met. However, the significance value (.022) is less than the critical value of .05, so we accept that the assumption of sphericity has been violated. Within-Subjects Factors Measure: MEASURE_ PINT PINT PINT PINT ALCOHOL 1 2 3 4 Dependent Variable Descriptive Statistics 11.7500 4.31491 20 11.7000 4.65776 20 15.2000 5.80018 20 14.9500 4.67327 20 1 Pint 2 Pints 3 Pints 4 Pints Mean Std. Deviation N Mauchly's Test of Sphericityb Measure: MEASURE_ .477 13.122 5 .022 .745 .849. Within Subjects Effect ALCOHOL Mauchly's W Approx. Chi-Square df Sig. Greenhouse- Geisser Huynh-Feldt Lower-bound Epsilona Tests the null hypothesis that the error covariance matrix of the orthonormalized transformed dependent variables is proportional to an identity matrix. May be used to adjust the degrees of freedom for the averaged tests of significance. Corrected tests are displayed in the Tests of Within-Subjects Effects table. a. Design: Intercept Within Subjects Design: ALCOHOL b.
Output 1 Output 10 shows the main result of the ANOVA. The significance of F is .005, which is significant because it is less than the criterion value of .05. We can, therefore, conclude that alcohol had a significant effect on the average number of women that were eyed up. However, this main test does not tell us which quantities of alcohol made a difference to the number of women eyed up. This result is all very nice, but as of yet we haven’t done anything about our violation of the sphericity assumption. This table contains several additional rows giving the corrected values of F for the three different types of adjustment (Greenhouse–Geisser, Huynh–Feldt and lower-‐ bound). First we decide which correction to apply, and to do this we need to look at the estimates of sphericity: if the Greenhouse–Geisser and Huynh–Feldt estimates are less than .75 we should use Greenhouse–Geisser, and if they are above .75 we use Huynh–Feldt. We discovered in the book that based on these criteria we should use Huynh–Feldt here. Using this corrected value we still find a significant result because the observed p (.008) is still less than the criterion of .05. Output 2 Tests of Within-Subjects Effects Measure: MEASURE_ 225.100 3 75.033 4.729. 225.100 2.235 100.706 4.729. 225.100 2.547 88.370 4.729. 225.100 1.000 225.100 4.729. 904.400 57 15. 904.400 42.469 21. 904.400 48.398 18. 904.400 19.000 47. Sphericity Assumed Greenhouse-Geisser Huynh-Feldt Lower-bound Sphericity Assumed Greenhouse-Geisser Huynh-Feldt Lower-bound Source ALCOHOL Error(ALCOHOL) Type III Sum of Squares df Mean Square F Sig. Pairwise Comparisons Measure: MEASURE_ 5.000E-02 .742 1.000 -2.133 2. -3.450 1.391 .136 -7.544. -3.200 1.454 .242 -7.480 1. -5.000E-02 .742 1.000 -2.233 2. -3.500* 1.139 .038 -6.853 -. -3.250 1.420 .202 -7.429. 3.450 1.391 .136 -.644 7. 3.500* 1.139 .038 .147 6. .250 1.269 1.000 -3.485 3. 3.200 1.454 .242 -1.080 7. 3.250 1.420 .202 -.929 7. -.250 1.269 1.000 -3.985 3. (J) ALCOHOL 2 3 4 1 3 4 1 2 4 1 2 3 (I) ALCOHOL 1 2 3 4 Mean Difference (I-J) Std. Error Sig.a^ Lower Bound Upper Bound 95% Confidence Interval for Differencea Based on estimated marginal means *. The mean difference is significant at the .05 level. a. Adjustment for multiple comparisons: Bonferroni.
