Chapter 10: The t Test for Two Independent Samples
so far, we’ve only discussed single-sample inferential techniques, but IRL, most studies require comparison across 2+ sets of data. when the scores are numerical values, the research question involves mean differences between data sets. the designs of such studies [during which 2 sets of data are obtained] fall into 2 categories:
the two sets of data could come from entirely separate groups of participants. eg group 1 being men and group 2 being women, or group 1 being students given laptops and group 2 being students not given laptops. this is known as an independent-measures research design or a between-subjects design. subject of this chapter.
the 2 sets of data could come from the same group of participants ut at different points in time/before and after an intervention/etc. eg, group before therapy vs same group after 6 weeks of therapy. this is called a repeated-measures research design or a within-subjects design. subject of next chapter.
independent-measures research design/between-subjects design: a research design that uses a separate, independent group of participants for each treatment condition (or for each population); comparison occurs between two groups of individuals.
since we’re working with 2 samples, we use subscripts to denote which data goes with which sample (M1=mean for sample 1; n2=number of participants in sample 2; etc)
“the goal of an independent-measures research study is to evaluate the mean difference between two populations (or between two treatment conditions).” thus:
we state our null hypothesis as H0: μ1-μ2 = 0. note that the null hypothesis is equivalent to μ1 = μ2, but we write it the way we initially did because it produces a specific numerical value used in the calculation of the t statistic.
our alternative hypothesis is H1: μ1-μ2 ≠ 0. alternatively, we can state that μ1 ≠ μ2
the independent-measures hypothesis test uses another version of the t statistic. it has the same general structure as the one from the previous chapter, but to distinguish the 2, we call last chapter’s the single-sample t statistic and this one the independent-measures t statistic. it includes data from 2 separate samples and hypotheses about 2 different populations, but it has commonalities with last chapter’s t stat:
the basic structure of the t stat is the same for both independent-measures and single-sample hypothesis tests. in both cases, .
the independent-measures t is basically a two-sample t that doubles all the elements of the single-sample t formulas.
recalling from last chapter that the single-sample hypothesis test’s t was calculated with , we just slightly modify this to give us the independent-measures t formula, . we obtain M1 and M2 from the sample data, and we get μ1-μ2 from the null hypothesis (0). this means we can further simplify t as being . in this case, the t stat is once again a simple ratio comparing actual mean difference with the difference that would be expected by chance.
while sM measured the standard error between sample mean and population mean, we’re now working with sM1-M2, which is the standard error between a sample mean difference and a population mean difference.
note: this follows the general notation guidelines, so don’t let it trip you up! the standard error takes the form sstatistic; when comparing means, we use sM, and now that we’re using mean difference, we’re using sM1-M2. that’s all!
the estimated standard error of M1-M2 that appears in the earlier denominator can be interpreted in 2 ways:
(1) we define it as a measure of the standard/average distance between a sample stat (M1-M2) and the corresponding population parameter (μ1-μ2).
(2) we consider it as measuring how far, on average, the sample mean difference is from 0 (since the null hypothesis states that, in a perfect world, it should be 0).
we can also interpret sM1-M2 in two ways:
(1) it measures the standard difference between (M1-M2) and (μ1-μ2).
(2) when the null hypothesis is true, it measures the standard, or average size of (M1-M2). that is, it measures how much difference is reasonable to expect between 2 sample means.
we develop the formula for sM1-M2 by considering 3 factors:
each of the 2 sample means represents its own population mean, but in each case, there is some error. in other words, M1 approximates μ1 with some error, and M2 approximates μ2 with some error. this means there are 2 sources of error!!
the amount of error associated with each sample mean is measured by the estimated standard error of M. using the equation from last chapter to determine standard error, we know that:
for M1,
for M2,
for the independent-measures t statistic, we want to know the total amount of error involved in using two sample means to approximate two population means. to do this, we find the error from each sample separately and then add them together. the resulting overall equation for standard error is: .
note: we add the error because we have to consider not only each population’s range, but also the range between each population. eg, if population I has a range from 50 to 70 and population II has a range from 20 to 30, the smallest range between them is 20 (a score of 30 from population II and a score of 50 from population I), and the biggest is 50 (20 and 70). this means that the range of possible disparities is 30 points. we could also have found this by considering that the range for population I (20) plus the range for population 2 (10) adds to equal 30. thus the variability for the difference in scores is found by adding together the variability for each of the two populations.
note: that above equation only works for samples wherein n1=n2!!! when the samples are of different sizes, the formula is biased since it treats both sample variances equally despite them not being equally good!!
one way to correct this is with pooled variance (sp2), where we combine the 2 sample variances into a single value by averaging the 2 variances using a procedure allowing the bigger sample to carry more weight in the final value’s determination. recalling that sample variance was computed as SS/df for a single sample, we can determine that for the independent-measures t stat, we consider both samples’ calculations and combine them into the formula .
we can also use a slightly modified version of this formula, which is especially useful if sample data are summarized as means and variances (s2): .
using all this ^^^, we can find a formula that uses pooled variance instead of individual sample variances: .
conceptually, this standard error measures how accurately the difference between 2 sample means represents the difference between the 2 population means.
also note that the pooled variance from the 2 samples is used to compute the standard error for the 2 samples.
we can still use our table of t stats vs df and alpha levels—just make sure to use (df1+df2) rather than just one or the other!
as in chapter 9, they recommend splitting the calculations into 3 steps: find pooled variance, use that to compute standard error, and finally, compute t stat.
as usual, we can also do a one-tailed test with this procedure! in that case, your hypotheses become H0: μ1 ≤ μ2 and H1: μ1 > μ2 (if expecting increase). notice that we aren’t using 0 here like we recommended above. despite that, we still use 0 as our μ1-μ2 because our null hypothesis is still expecting no change!!
remember that we’re doing M1-M2, so an increase in mean will lead to a negative t stat and a decrease in mean will lead to a positive t stat!
we’ll write our results as, eg, “Reported scores were significantly higher for students in the dimly lit room, t(14) = -2.67, p < .01, one-tailed.”
3 assumptions must be satisfied before using independent-measures t formula for hypothesis testing:
(1) the observations w/in each sample must be independent.
(2) the two populations from which the samples are selected must be normal. (when there is cause to suspect that the populations are far from normal, you should compensate with quite large samples.)
(3) the two populations from which the samples are selected must have equal variances.
this is referred to as homogeneity of variance and states that the 2 populations being compared have the same variance. this is similar to the concept from ch8 that stated that for the z-score hypothesis test, the treatment effect was assumed to add a constant value to each individual score, keeping population standard deviation the same even after treatment. this is essentially the same assumption but in terms of variance.
“Recall that the pooled variance in the t-statistic formula is obtained by averaging together the two sample variances. It makes sense to average these two values only if they both are estimating the same population variance—that is, if the homogeneity of variance assumption is satisfied. If the two sample variances are estimating different population variances, then the average is meaningless. (Note: If two people are asked to estimate the same thing—for example, your weight—it is reasonable to average the two estimates. However, it is not meaningful to average estimates of two different things. If one person estimates your weight and another estimates the number of beans in a pound of whole-bean coffee, it is meaningless to average the two numbers.)”
homogeneity of variance is most important when sample sizes are highly disparate. since one of our numbers from the equation comes from our null hypothesis (0), we already have a questionable number, but if we have a questionable number for our M1-M2, too, our data is incredibly flawed. you have no way to determine which of those 2 numbers caused the bad data, so you can neither accept nor reject the null hypothesis.
to determine whether homogeneity of variance assumption is satisfied, we can either (1) just look at the samples’ variances and see whether they’re close to each other [as they should be] or not, or (2) if one sample variance is more than 3 or 4 times larger than the other, we can use Hartley’s F-max test (which has the added bonus of being able to check homogeneity of variance for more than 2 independent samples, which we’ll use later in ch 12). for this computation (which assumes that sample variance provides unbiased estimate of population variance)…
(1) compute the sample variance, , for each of the separate samples.
(2) select the largest and the smallest of these sample variances and compute . a relatively large F-max indicates a large difference between the sample variances. in this case, the data suggest that the population variances are different and that the homogeneity assumption has been violated. on the other hand, a small value of F-max (near 1.00) indicates that the sample variances are similar and that the homogeneity assumption is reasonable.
(3) the F-max value computed for the sample data is compared to the critical value found in Table B.3 (Appendix B). if the sample value is larger than the table value, you conclude that the variances are different and that the homogeneity assumption is not valid.
to locate the critical value in the table, you need to know
k = number of separate samples (for the independent-measures t test, k = 2)
df = n-1 for each sample variance. the Hartley test assumes that all samples are the same size. (eg: for two samples, both of which are n = 10, we look up df = 9.
the alpha level. the table provides critical values for ɑ=.05 and ɑ=.01. generally a test for homogeneity would use the larger alpha level.
if your F-max < critical value, your assumption is good.
note that the goal with the F-max test, unlike our other hypothesis tests, is to fail to reject H0 (which states that the 2 population variances are the same) because that means the difference between sample variances is insignificant and thus our data works! we want our null hypothesis (that there is no real difference between our sample variances) to be true!
if F-max rejects the null hypothesis or you have some other reason to suspect the homogeneity of variance assumption isn’t justified, do not compute an independent-measures t statistic using pooled variance! we have an alternate procedure that doesn’t pool the 2 samples and doesn’t require the homogeneity assumption:
(1) the standard error is computed using the 2 separate sample variances as in our earlier equations.
(2) the value of degrees of freedom for the t stat is adjusted using the equation , where and .
decimal values for df should be rounded down to the next lower integer.
this lowers the value of df, pushing the boundaries for the critical region farther out and thus making the test more demanding and therefore correcting for the same bias problem that the pooled variance attempts to avoid.
also note that many computer programmes, including SPSS, calculate 2 values of the independent-measures t statistic: one with pooled variance (with equal variances assumed) and one with this adjustment (with equal variances not assumed). it’s your job to decide which is more appropriate for the given situation.
like last chapter, since sample size influences how likely you are ot reject your null hypothesis, we need to accompany our data with a measure of the absolute magnitude of treatment effect independent of the sample size. we used Cohen’s d last chapter and the equation . for the independent-measures research study, we use M1-M2 as our best estimate of the mean difference between the 2 populations, and the pooled standard deviation (square root of pooled variance) to estimate population standard deviation; thus we use the formula .
we generally report Cohen’s d as a positive value [even when it’s negative based on how you input your Ms).
we also use the same criteria for Cohen’s d here as we did before (0.2, 0.5, and 0.8 being small, medium, and large effects, respectively)
independent-measures t hypothesis also lets us compute r2, the percentage of variance accounted for. like last chapter, r2 measures exactly how much of the variability can be explained. our formula is the same as before: . we also use the same outlines from last chapter (0.01, 0.09, and 0.25 being small, medium, and large effects, respectively).
can also compute confidence interval! with a single-sample t, we used a single sample mean to estimate a population mean; here, we use a sample mean difference to estimate a population mean difference. we use the equation . we know M1-M2 from the data, as well as s(M1-M2). though we don’t know t, we can use the degrees of freedom for the t stat and the t distribution table to estimate the t value. thus we can then solve for μ1-μ2. this gives us the range of possible μ scores between which we can say with x% confidence (depending on the confidence interval we choose) the μ will fall. note that M1-M2 should be located exactly at the center of this interval.
influenced by other factors! eg, larger confidence interval = larger range, and larger sample size means narrower interval.
since the confidence interval is related to sample size, it’s not a pure measure of effect size like Cohen’s d or r2.
we can also use the confidence interval ^ to help determine if we need to reject H0! if H0 is outside of the range we create for our confidence interval, we can safely reject it!
we report the results of an independent measures t test in the following manner: “The students who were tested in a dimly lit room reported higher performance scores (M=12, SD=2.93) than the students who were tested in the well-lit room (M=8, SD=3.07). The mean difference was significant, t(14)=2.67, p<.05, d=1.33.”
also, as in the last chapter, if an exact probability is available from a computer analysis, you should report it.
finally, if a confidence interval is reported to describe effect size, it appears immediately after the results from the hypothesis test, eg: “The difference was significant, t(14)=2.67, p=.018, 95% CI [0.782, 7.218].”
recall from last chapter that certain variables affect t tests:
higher variability = higher magnitude of estimated standard error in denominator of t stat = smaller value for t stat = less likely to find significant result
smaller sample size = higher standard error = smaller t value = decreased likelihood of rejecting H0
also recall that variance affects Cohen’s d and r2 (larger variance = smaller measures of effect size), but sample size doesn’t affect Cohen’s d and only has a very minor affect on r2.