PSYC 6430 Readings Associated with Chapter 8 (Power) in Howell

   You should be prepared to answer, on the final exam in PSYC 6430, questions relating to the readings listed below.  Below the listings I have a several comments which you should review.  If you have any questions about these readings, please ask me, electronically or in class.

• State statistical hypotheses that address the research question.
• Select an appropriate test statistic.
• Specify the alpha and the beta with which you are comfortable.
• Specify the minimum nontrivial effect size.
• Determine the number of research units you need to be able to detect an effect of that size with the alpha and beta specified earlier.
• Collect the data, screen the data for problems, transform if necessary, and compute the test statistic.
• Obtain the exact p value.  If p ≤ the criterion of statistical significance, reject the null hypothesis.
• Compute a strength of effect estimate.
• Write up the results.

    Cohen's (1992) power primer provides for common statistical tests the required number of research units necessary to have 80% power for small, medium, or large effects.  Essentially the same information can be found in my document Estimating the Sample Size Necessary to Have Enough Power.

    Tachibana (1980) pointed out that most research in Psychology has so little power that when attempting to replicate an effect previously reported as statistically significant the expected result is a Type II error.  Psychologists are know to write all sorts of drivel trying to explain the failure to replicate -- maybe the effect is different for the population from which my research participants were drawn, maybe current events have modified the effect, maybe the effect interacts with the phase of the moon, and so on.  Apparently the gatekeepers of the published literature in psychology did not want to hear this -- Perceptual and Motor Skills is not exactly a top-notch journal.

    Nelson, Rosenthal, and Rosnow (1986) replicated and extended the earlier research of Rosenthal (1963).  Those earlier results showed that psychologists are more impressed with significant results from research with large sample sizes than with significant results from research with small sample sizes, even when the obtained p is held constant.  In other words, psychologists really do not understand hypothesis testing very well.  Consider this example:  Researcher A reports a significant effect (p = .047) based on data from 15 subjects.  Research B reports a significant effect (p = .047) based on data from 1,500 subjects.  Which result is more impressive?  Most psychologists are more impressed with B, but they should be more impressed with A.  Since A obtained a significant result despite having lower power (due to the small sample size), A has probably found a large effect.  B had so much power that e would be expected to find significant results even if the effect was so small as to be trivial -- in fact, with 1,500 subjects and a p of .047, the effect size would be so small as to be trivial.  The research reported by A should be conducted again, on a larger sample size, to be better able to estimate the size of the effect.  With a small sample size, a confidence interval for the effect will be rather wide.

    Consider also this next example:  Researcher C reports a nonsignificant effect (p = .47) based on data from 15 subjects.  Researcher D reports a nonsignificant effect (p = .47) based on data from 1,500 subjects.  Which result is more impressive?  Now the large sample research is more impressive.  D has so much power that e would be almost certain to obtain significant results if the effect were nontrivial in magnitude, so e's nonsignificant results allow em to assert that the null hypothesis is true or very close to true.  D's confidence interval will be narrow and will include zero.  C has so little power that the most likely reason for nonsignificance is a Type II error.  C's confidence interval will be wide, ranging from a large effect in one direction to a large effect in the opposite direction.

    Note that Nelson et al. suggest that researchers always report an exact p and an estimate of effect size.  Also note that the Rosenthal (1963) article appeared in a journal that gets little respect (the Journal of Psychology).  At the time he was a junior faculty member at the University of North Dakota, and the guardians of the publication gate did not want to hear what he had to say.  He is now at Harvard, and publishes in the most prestigious of journals, such as the American Psychologist.

   Wuensch (1987) attempted to publish a short manuscript that would explain, in language simple enough that the typical psychologist could understand it, that the frequency of Type I errors in the published literature is not equal to the criterion of statistical significance (5%).  American Psychologists did not want to hear that.  The American Psychologist would rather publish a short report that served to perpetuate that delusion.

    Rosnow and Rosenthal (1989) remind us that p is a continuous variable, and that it makes little sense to discriminate between research for which p = .049 and research for which p = .051.  They also point out that the famous physicist Enrico Fermi thought that p = .10 was a wise operational definition of a "miracle."  I guess psychologists employ a higher standard than do physicists.  Rosenthal and Rosnow also recommend that researchers always report effect size estimates, for both effects which are statistically significant and effects that are not statistically significant.

    Frank Schmidt (1996), in his presidential address (the Division of Evaluation, Measurement, and Statistics, American Psychological Association) argued that NHST (null hypothesis statistical testing) has impeded scientific progress in psychology and should be abandoned.  Among the points made by Schmidt are the following:

• The unconditional probability of a Type I error is likely zero for almost all hypotheses tested.  We should be much more concerned with the probability of a Type II error, which is typically high.
•  Given the low power typical of psychological research, studies which yield significant results are most often those where random error led to a sample in which the apparent size of the effect is greater than what it really is in the population.  This leads to overestimation of effect sizes for those studies.  At the same time, the majority of those who attempt to replicate such research will get better estimates of the effect size but nonsigificant results, which may be misinterpreted as meaning there is no effect.  These researchers may then go on a wild goose chase trying to figure out what moderator variables made their results nonsignificant when the earlier research was significant.
• Increased use of procedures which are designed to control familywise error rates have reduced power from an average of about 46% to a mere 37% over a 22 year period.
• We would be better off reporting point estimates and confidence intervals, not p values.
• It is not reasonable to expect individual studies to have sample sizes large enough to have high power.  The goal of individual studies should be to produce the data which will subsequently be included in meta-analyses.
• Psychologists hold a number of false believes about the value of hypothesis testing, and this (along with their belief that their manuscripts will not be accepted for publication unless they include p values) accounts for the continued use of NHST despite the large number of valid arguments against it.
• False Belief 1:  A low p tells you that if someone tries to replicate the study, then they too will probably obtain significant results.  As pointed out earlier by Tachibana, this is just not so.
• False Belief 2:  A low p tells you that your result is important and your effect size large.
• False Belief 3:  If p > .05, then there is no effect or the effect is so small as to be trivial.

    Nickerson (2000) also lists false beliefs about hypothesis testing:

• False Belief 1:  p is the probability that the null hypothesis is true and 1-p the probability that it is not true.  As you know, p is a conditional probability, the probability of obtaining results as more discrepant with the null hypothesis than are those obtained, were the null hypothesis true.
• False Belief 2:  Rejecting a null hypothesis establishes the truth of the theory which predicted the null to be false.  In fact, there are usually other alternative explanations for the falsity of the null hypothesis.
• False Belief 3:  A low p tells you that if someone tries to replicate the study, then they too will probably obtain significant results.  As pointed out earlier by Tachibana, this is just not so.
• False Belief 4:  A low p tells you that your effect is large.
• False Belief 5:  A low p tells you that your effect is important.
• False Belief 6:  Alpha is the absolute (unconditional) probability that one has made a Type I error when rejecting that null hypothesis.
• False Belief 7:  The .05 criterion of statistical significance means that there is a 5% chance that you will make a Type I error when you test the null hypothesis.
• False Belief 8:   The .05 criterion of statistical significance means that  5% of published significant results are, in fact, Type I errors.  Recall Wuensch's (1987) rejected manuscript on this topic.
• False Belief 9:  If you have not rejected the null hypothesis, β is the probability that the alternative hypothesis is true.  There are also other false beliefs about β which parallel those about α.
• False Belief 10:  If p > .05, then the null hypothesis is true.
• False Belief 11:  Research which produces results which are not "statistically significant" are of little importance.

    Nickerson also made the following important points:

• The unconditional probability of a Type I error is the product of p and the probability that the null hypothesis is true.  Since null hypotheses are rarely ever true, the unconditional probability of a Type I error is very close to zero.
• The availability of easy to use statistical software has resulted in statistical analyses being done by persons who have almost no understanding of the analyses.
• If your analysis has sufficient power, you can assert that your null hypothesis is close to true.
• Point null hypotheses are rarely if ever true.  Range null hypotheses (such as "the difference between these two population means is less than 1/10 of a standard deviation, so small as to be trivial.")
• No matter how small the effect, you can find it to "statistically significant" if your sample size is sufficiently large.
• One should not uncritically adopt the .05 criterion of statistical significance, but rather should chose a criterion after weighing the relative seriousness of Type I versus Type II errors.  See Wuensch's (1994) article on this topic.
• Even if you take into account the "file drawer problem," the probability that a published significant result represents a Type I error is considerably less than .05.
• We need to worry a lot more about Type II errors and less about Type I errors, and accordingly should consider using a criterion of statistical significance that is greater than the common .05.
• NHST is more appropriate for testing non-nil null hypothesis than for testing nil null hypotheses.  For example, if your theory predicts that the in a certain circumstance the value of some parameter should be 17.3, then use that prediction as a null hypothesis.  Of course, psychological theories are rarely able to generate such quantitative predictions.
• Do not test point null hypotheses, instead test range null hypotheses.
• Report exact p values, not just that p < .05 or < .01 or > .05.  If p is very small, it is OK just to say p < .0001 rather than something like p = .00000082.
• Report confidence intervals and estimates of effect size, whether or not the effect is "statistically significant."
• If you want to interpret a nonsignificant result as being due to the truth or near truth of the null hypothesis, you better be able to demonstrate that you had high power for detecting even small effects.
• If you have to base a practical decision on the results of your research, use a "preponderance of evidence" criterion rather than a tradition decision rule based on p.
• The word "significant" in statistics should not imply "important."  Some have suggested "reliable" as an alternative, but that implies that a low p means a replication attempt is highly likely also to produce a low p, but that is simply not true.

     After you read the reports from the APA Task Force on Statistical Inference, read my summary of the main points made in those reports.

    And now for a little chuckle, thanks to the late J. Cohen:

    From Cohen, J. (1994).  The earth is round (p < .05).  American Psychologist, 49: 997-1003.
 
     "And we, as teachers, consultants, authors, and otherwise perpetrators of quantitative methods, are responsible for the ritualization of null hypothesis significance testing (NHST; I resisted the temptation to call it
Statistical Hypothesis Inference Testing)......
 

Back to the Readings for Students in Graduate Statistics Page

Visit Karl's Index Page

Contact Information for the Webmaster,
Dr. Karl L. Wuensch
 

This page most recently revised on 13. November 2004.