What kind of significance tests do we use for doing inference on the mean of an obviously non-normal population? If the sample is large, we can still use the t-test since the sampling distribution of the sample mean is close to normal and the t-procedure is robust. If the sample size is small and the underlying distribution is clearly not normal (e.g. is extremely skewed), what significance test do we use? Let’s take the example of a matched pairs data problem. The matched pairs t-test is to test the hypothesis that there is “no difference” between two continuous random variables and that are paired. If the underlying distributions are normal or if the sample size is large, the matched pairs t-test are an excellent test. However, absent normality or large samples, the sign test is an alternative to the matched pairs t-test. In this post, we discuss how the sign test works and present some examples. Examples 1 and 2 are shown in this post. Examples 3, 4 and 5 are shown in the next post
The sign test, more examples.
The sign test and the confidence intervals for percentiles (discussed in the previous post Confidence intervals for percentiles) are examples of distribution-free inference procedures. They are called distribution-free because no assumptions are made about the underlying distributions of the data measurements. For more information about distribution-free inferences, see [Hollander & Wolfe].
We discuss two types of problems for which the sign test is applicable – one-sample location problems and matched pairs data problems. In the one-sample problems, the sign test is to test whether the location (median) of the data has shifted. In the matched pairs problems, the sign test is to test whether the location (median) of one variable has shifted in relation to the matched variable. Thus, the test hypotheses must be restated in terms of the median if the sign test is to be used as an alternative to the t-test. With the sign test, the question is “has the median changed?” whereas the question is “has the mean changed?” for the t-test.
The sign test is one of the simplest distribution-free procedures. It is an excellent choice for a significance test when the sample size is small and the data are highly skewed or have outliers. In such cases, the sign test is preferred over the t-test. However, the sign test is generally less powerful than the t-test. For the matched pairs problems, the sign test only looks at the signs of the differences of the data pairs. The magnitude of the differences is not taken into account. Because the sign test does not use all the available information contained in the data, it is less powerful than the t-test when the population is close to normal.
How the sign test works
Suppose that is a pair of continuous random variables. Suppose that a random sample of paired data is obtained. We omit the observations with . Let be the number of pairs for which . For each of these pairs, we make a note of the sign of the difference ( if and if ). Let be the number of signs out of these pairs. The sign test gets its name from the fact that the statistic is the test statistic of the sign test. Thus we are only considering the signs of the differences in the paired data and not the magnitude of the differences. The sign test is also called the binomial test since the statistic has a binomial distribution.
Let . Note that this is the probability that a data pair has a sign. If , then any random pair has an equal chance of being a or a sign. The null hypothesis is the hypothesis of “no difference”. Under this hypothesis, there is no difference between the two measurements and . The sign test is test the null hypothesis against any one of the following alternative hypotheses:
The statistic can be considered a series of independent trials, each of which has probability of success . Thus . When is true, . Thus the binomial distribution is used for calculating significance. The left-tailed P-value is of the form and the right-tailed P-value is . Then the two-tailed P-value is twice the one-sided P-value.
The sign test can also be viewed as testing the hypothesis that the median of the differences is zero. Let be the median of the differences . The null hypothesis is equivalent to the hypothesis . For the alternative hypotheses, we have the following equivalences:
A running club conducts a 6-week training program in preparing 20 middle aged amateur runners for a 5K running race. The following matrix shows the running times (in minutes) before and after the training program. Note that five kilometers = 3.1 miles.
The difference is taken to be pre-training time minus post-training time. Use the sign test to test whether the training program improves run time.
For a given runner, let be a random pre-training running time and be a random post-training running time. The hypotheses to be tested are:
Under the null hypothesis , there is no difference between the pre-training run time and post-training run time. The difference is equally likely to be a plus sign or a minus sign. Let be the number of runners in the sample for which . Then . The observed value of the statistic is . Since this is a right-tailed test, the following is the P-value:
Because of the small P-value, the result of 15 out of 20 runners having improved run time cannot be due to random chance alone. So we reject and we have good reason to believe that the training program reduces run time.
A car owner is curoius about the effect of oil changes on gas mileage. For each of 17 oil changes, he recorded data for miles per gallon (MPG) prior to the oil change and after the oil change. The following matrix shows the data:
Regular oil changes are obviously crucial to maintaining the overall health of the car. It seems to make sense that oil changes would improve gas mileage. Is there evidence that this is the case? Do the analysis using the sign test.
In this example we set the hypotheses in terms of the median. For a given oil change, let be the post oil change MPG and be the pre oil change MPG. Consider the differences . Let be the median of the differences . We test the null hypothesis that there is no change in MPG before and after oil change against the alternative hypothesis that the median of the post oil change MPG has shifted to the right in relation to the pre oil change MPG. We have the following hypotheses:
Let be the number of oil changes with positive differences in MPG (post minus pre). Then . The observed value of the statistic is . Since this is a right-tailed test, the following is the P-value:
At the significance level of , we reject the null hypothesis. However, we would like to add a caveat. The value of this example is that it is an excellent demonstration of the sign test. The 17 oil changes are not controlled. For example, the data are just records of mileage and gas usage for 17 oil changes (both pre and post). No effort was made to make sure that the driving conditions are similar for the pre oil change MPG and post oil change MPG (freeway vs. local streets, weather conditions, etc). With more care in producing the data, we can conceivably derive a more definite answer.
Myles Hollander and Douglas A. Wolfe, Non-parametric Statistical Methods, Second Edition, Wiley (1999)