East Carolina University
Department of Psychology
Multiple Comparisons Among Proportions
My friend, Laura Anderlozzi, has conducted a 2 x 6 test and obtained significant results. The editor of the journal to which she submitted the manuscript asked her to make pairwise comparisons. What is she to do now (message Professor Karl on Facebook and ask for help). There are several options.
What Comparisons are to Be Made?
There are six columns, A, B, C, D, E, and F, and two rows, 1 and 2. Suppose we wanted to consider only columns A and B. We compute for column A the proportion of scores in row 1 (call this p1), and for column B we compute the proportion of scores in row 1 (call this p2). We want to test the null hypothesis that p1 = p2. The usual 2 x 2 Pearson Chi-Square does exactly this (as do several other procedures). The 2 x 6 Pearson Chi-square tests the null that p1 = p2 = p3 = p4 = p5 = p6. After you reject that six-proportion null, you still do not know which proportions differ significantly from which others.
Run Multiple 2 x 2 Pearson Chi-Square Analyses
Here you would have to run 15 contingency table analyses, rows x (AB, AC, AD, AE, AF, BC, BD, BE, BF, CD, CE, CF, DE, DF, EF), each comparing the proportion in one column with that in another column. Aside from this being rather tedious, one may be concerned about familywise error rate when conducting so many comparisons. You could use the Bonferroni or the Sidak inequality to cap familywise error rate.
Find an Online Calculator for Comparing Two Proportions
There are lots of such calculators out there. Make each of the (in this case 15) comparisons. Possibly less tedious than the multiple 2 x 2 Chi-Square analyses, but the familywise error concern remains.
Use My SPSS Macro for Comparing Two Proportions
This should be less tedious than the methods above. You enter one line of summary data (proportions and sample sizes) for each comparison and then run the syntax file. The output is confidence intervals. If an interval does not contain zero, the comparison is significant.
Google "Marascuilo Procedure"
You will get lots of hits. Doing this by hand is tedious. I was unable to find an online calculator to do it.
Use the Multiple Comparison Test for Proportions in a 2xc Crosstabulation in SAS Macro
If you are comfortable with SAS, this seems like a good solution. The approach taken is very similar to that which Tukey developed for making pairwise comparisons among means (HSD). When interpreting the output, remember that the differences between group proportions (Diff) is after the arcsin transformation that is applied prior to analysis. These will not be the same as the differences between the untransformed group proportions. Also note that the output may indicate that a difference between proportions is significant even when their difference is less than that of a difference already declared not significant. For example, for ordered means A, B, C, D, E, F, the output may indicate that the difference between B and D is significant even though the difference between A and D is not. In such cases, I declare the difference between B and D not to be significant. APA-style reference: Elliott, A. C., & Reisch, J. S. (2006, March). Implementing a multiple comparison test for proportions in a 2xc crosstabulation in SASŪ. Paper presented at the SAS Users Group International Conference, San Francisco, CA. Retrieved from http://www2.sas.com/proceedings/sugi31/204-31.pdf
Small Expected Frequencies
If you have a column with small total counts, comparisons involving that column will lack power. My usual advice for dealing with this is to delete any such column from the analysis or combine it with another column (if that makes sense).
Back to the Stat Help Page
Contact Information for the Webmaster,
Dr. Karl L. Wuensch
This page most recently revised on the 4th of March, 2015.