Resampling.txt I have placed on my SAS programs page a program that will do approximate randomization tests. These are similar to "bootstrap" or "resampling" statistics - they have absolutely no assumptions. Conducting an exact randomization test involves obtaining all of the possible permutations of the obtained data. This can use up enormous amounts of computer time. An approximate randomization test saves time by randomly drawing a sample from the population of all possible permutations and estimating p, the exact significance level, from that sample. For example, suppose we wish to test the null hypothesis that two populations have identical means. From our sample data we first compute Mo, the difference between the two sample means. If the null hypothesis is correct, then rearranging the scores between the two groups should have little effect on the difference between the two group's means. If the null hypothesis is true, then Mo should not be very extreme (should not be in the tails of) in the distribution of M, the distribution of the differences between means for all possible permutations of the observed (sample) data. The exact significance level, p, is the proportion of all of the values in M that are equal to or more extreme than is Mo. The program ART-INDT SAS on my 192 D-disk does an approximate randomization test to test the null hypothesis that two independent samples come from populations with identical means. The program came from the article "SAS procedures for approximate randomization tests," Behavior Research Methods, Instruments, & Computers, 1993, 25: 406-409, which was authored by Ru San Chen and William P. Dunlap. The program selects a sample of 1,000 of the possible permutations (you could modify it to obtain more) and then estimates p from that pseudo sampling distribution. Keep in mind that the p here is an estimate of the exact significance level. Although we can just use it in the usual way (if p is less than or equal to some criterion, we reject the null hypothesis), some suggest an additional step. Given that we have an estimated p, we can test the null hypothesis that the actual, exact p (referred to as phi) is less than or equal to some given value (such as .05 or .01). If the resulting probability value is large (> .95), then we can with confidence reject the null hypothesis. The data in the File ART-INDT DATA are those from Table 1 of Chen & Dunlap. Below is the output from four runnings of the program. Notice that one does not obtain the same results each time -- one would not expect to, since one is estimating p rather than obtaining an exact value of p. I think there is a clear danger here -- the user running the program many times and reporting the output from only one unusual run. -------------------------------------------------------------------------------- Approximate Randomization Test, Equality of Means, 2 Independent Samples 1 15:01 Friday, April 21, 1995 Original Statistic = 0.583 (Diff between the 2 sample means) Estimated Significance Level p (two-tailed) = 0.035 (Reject null) Prob(phi <= 0.05) = 0.9860258169 (Reject null at .05) Prob(phi <= 0.01) = 1.223979E-10 (but not at .01) -------------------------------------------------------------------------------- Approximate Randomization Test, Equality of Means, 2 Independent Samples 1 15:04 Friday, April 21, 1995 Original Statistic = 0.583 Estimated Significance Level p (two-tailed) = 0.046 (Reject null) Prob(phi <= 0.05) = 0.6909935817 (don't reject null) Prob(phi <= 0.01) = 1.158826E-17 " -------------------------------------------------------------------------------- Approximate Randomization Test, Equality of Means, 2 Independent Samples 1 15:07 Friday, April 21, 1995 Original Statistic = 0.583 Estimated Significance Level p (two-tailed) = 0.035 (Reject null) Prob(phi <= 0.05) = 0.9860258169 (Reject) Prob(phi <= 0.01) = 1.223979E-10 (Retain) -------------------------------------------------------------------------------- Approximate Randomization Test, Equality of Means, 2 Independent Samples 1 15:14 Friday, April 21, 1995 Original Statistic = 0.583 Estimated Significance Level p (two-tailed) = 0.042 (Reject null) Prob(phi <= 0.05) = 0.8644091221 (Retain) Prob(phi <= 0.01) = 5.830131E-15 (Retain) ------------------------------------------------------------------------------- Approximate Randomization Test, Equality of Means, 2 Independent Samples 1 15:22 Friday, April 21, 1995 Original Statistic = 0.583 Estimated Significance Level p (two-tailed) = 0.026 (Reject null) Prob(phi <= 0.05) = 0.9998981019 (Reject) Prob(phi <= 0.01) = 5.6706913E-6 (Retain) ------------------------------------------------------------------------------ Here are some more traditional stats on the original data using Minitab: MTB > read 'art-indt' data c1 c2 MTB > name c1 'x' c2 'y' MTB > dotp c2; SUBC> by c1. x . 1 . :.: : . +---------+---------+---------+---------+---------+-------y x 2 :. : . . .. . +---------+---------+---------+---------+---------+-------y 0.00 0.70 1.40 2.10 2.80 3.50 MTB > twot c2 c1 TWOSAMPLE T FOR y x N MEAN STDEV SE MEAN 1 10 0.412 0.206 0.065 2 10 0.99 1.02 0.32 95 PCT CI FOR MU 1 - MU 2: (-1.324, 0.16) TTEST MU 1 = MU 2 (VS NE): T= -1.78 P=0.11 DF= 9 MTB > twot c2 c1; SUBC> pooled. TWOSAMPLE T FOR y x N MEAN STDEV SE MEAN 1 10 0.412 0.206 0.065 2 10 0.99 1.02 0.32 95 PCT CI FOR MU 1 - MU 2: (-1.271, 0.11) TTEST MU 1 = MU 2 (VS NE): T= -1.78 P=0.092 DF= 18 POOLED STDEV = 0.732 MTB > unstack c2 into c3 c4; SUBC> subscripts c1. MTB > mann c3 c4 Mann-Whitney Confidence Interval and Test C3 N = 10 Median = 0.440 C4 N = 10 Median = 0.715 Point estimate for ETA1-ETA2 is -0.350 95.5 pct c.i. for ETA1-ETA2 is (-0.720,0.010) W = 80.0 Test of ETA1 = ETA2 vs. ETA1 n.e. ETA2 is significant at 0.0640 The test is significant at 0.0636 (adjusted for ties) Cannot reject at alpha = 0.05