INFERENTIAL STATS

Statistical Significance

The quality of a result from data being unlikely, but not impossible, to be due to chance variation, or sampling error. It is used to determine whether or not the null hypothesis should be retained or rejected.

Null Hypothesis (H₀)

A type of hypothesis that states that the treatment has no effect and that there is no change, difference, or relationship. The independent variable has no effect on the dependent variable. It indicates that the results are not statistically significant. If a obtained value does not fall beyond the critical region boundaries, there is not enough evidence to reject the null hypothesis, so it is retained. It is assumed to be true but still unproven.

Alternative Hypothesis (H₁)

A type of hypothesis that states that there is a change, a difference, or a relationship for the general population. It indicates that the results are statistically significant. The null hypothesis is rejected and the alternative hypothesis is supported if, according to the chosen significance criterion, it is very unlikely that the difference between groups is due to chance.

Alpha Level (α)

A significance criterion and the common convention of determining what is too unlikely to be considered a chance difference between means by setting critical region boundaries. Statistically significant differences are usually signified if p < 0.05, with an alpha level of α = 0.05.

If M exceeds the critical region boundaries, what are the 2 possibilities?

Sampling error or significant differences of the sample from the population, which can be due to experimental manipulation or other factors. Deciding between these possibilities requires determining the probability of getting a sample mean that is different from the population merely by chance or other meaningful factors.

P (Probability)-Value

TA statistical measure of the probability of rejecting the null hypothesis if it is true. The p-value that is greater than the standard significance level (p > 0.05) indicates strong evidence for the null hypothesis. However, a p-value that is less than the standard significance level (p < 0.05) is statistically significant.

Power

The statistical definition for the probability of rejecting a null hypothesis if it is false.

Type I (Alpha) Error

Rejecting a null hypothesis that is actually true. The researcher concludes that there is an effect, when in reality there is no effect. The alpha level can more or less be equivalent to the probability that a test will lead to a type I error, as it determines the probability of obtaining sample data in the critical region, even though the null hypothesis is true.

What are the determining factors of power?

The significance criterion in place, as increases in the alpha level, and subsequently the critical region, are correlated with increases in power; sample size, as increases in N result in a decrease in the sample error of the mean, and in turn, an increase in power; effect size, which can be maximized to increase differences between conditions and remove extraneous variability.

Effect Size

A measure indicating the strength of the relationship between the independent variable and the dependent variable. It measures the absolute magnitude of a treatment effect, independent of sample size. Different measures are used to measure effect size, such as Cohen’s d and r².

Cohen’s d

The most commonly used measure of effect size.

d = (μ_treatment (M) - μ_{no treatment} (μ)) / (σ or s)

(for Cohen’s d, a measure of 0.2 and smaller indicates a “small” effect, a measure of around 0.5 indicates a “medium” effect, and a measure of 0.8 and higher indicates a “large” effect.)

Type II Error

Failing to reject a null hypothesis that is actually false. The researcher fails to detect an effect. The probability that a test will lead to a type II error is not easily identified.

Sampling Error

The discrepancy that exists between the results gathered from a sample and the results that could be gathered if the entire population was tested (eg. margin of error in polls, etc.).

Correlation

The degree to which two variables are associated with each other. Correlations between two variables in a sample can be due to chance factors/sampling error or statistically significant relationships reflected between them. Relationships between numerical scores are usually measured and described using correlation, while relationships between non-numerical categories are usually measured and described using summary tables.

Pearson’s Product Moment Correlation (Pearson’s r)

The most common measure of the correlation of two variables for interval and ratio data. Conveys the strength and the direction of the relationship between two variables. It is mainly used as a sample statistic. It will always range from -1.0 to 1.0. If there is no relationship, the coefficient will be close to 0. It only measures linear relationships.

r = Σ((zx)(zy)) / N

(N stands for the number of pairs; df = N - 2)

Restriction of Range

Discerning the correlation of a sample within a certain and very specific range of factors instead of a more wider one. It can result in the lowering of Pearson’s r, which underestimates the magnitude of the relationship between two variables and does not allow for the “true” correlation to emerge.

Regression Analysis

Using the known variable X, the predictor variable, to predict a score (Y’) on the unknown variable Y, the criterion, based on the correlation between them. It involves making a regression line, an equation of a line predicting scores of Y. The accuracy of the prediction depends entirely on the correlation of the two variables. The closer Pearson’s r is to 1 or -1, the greater the level of prediction. Having no relationship (r = 0) present means that x cannot predict y.

Regression Line

An equation of a line predicting scores (Y’) of the criterion, Y, during regression analysis. The standard error of the estimate (S_y’) can measure how much the actual scores of Y would deviate from what the equation predicts (Y’). The best-fitting line is one in which the total amount of error is the smallest.

Y’ = bX + a

(b stands for the slope of the regression line; a stands for the y-intercept of a regression line)

Slope of a Regression Line (b)

b = r(S_y / S_x)

Y-Intercept of a Regression Line (a)

a = M_y - (b(M_x))

(b stands for the slope of the regression line)

Standard Error of the Estimate (S_y’)

The average amount actual Y scores would deviate from what a regression equation would have predicted (Y’) in a regression analysis. This error in analysis is also called the residual.

S_y’ = S_y √(1 - r²)

r²

The square of Pearson’s r. Signifies the amount of covariance, or the amount of shared variance, between two variables.

Simple Probability

Expressed as a number between 0 and 1, typically through a proportion. The probability of an event, usually expressed through a letter, is written as P((event)). Probability is, in essence, the same as relative frequency.

P((event)) = (# of outcomes including event A) / (total # of outcomes).

Distribution of Sample Means

A theoretical probability distribution that reflects the probability of obtaining different sample means from a population, given a particular sample size (N), a known population mean (μ) and a known population standard deviation (σ). The mean of a distribution of sample means is the grand expected mean, and the standard deviation is the sample error of the mean (σ_M).

Grand (Expected) Mean

The mean derived from a distribution of sample means. It will be equal to the population mean (μ).

Central Limit Theorem

States that as long as N is about 30 or larger, sample means drawn from a population will be normally distributed, even if the parent population is not normally distributed.

Standard Error of the Mean (σ_M)

The standard deviation derived from a distribution of sample means, measuring how far sample means deviate from the grand mean. It will be much smaller than the standard deviations of each sample, since sample means tend to be much more consistent than raw scores in a data set. Multiple samples drawn from the same population will have similar means, which means that the standard error of the mean would be very small, especially when samples are large.

σ_{M =} σ / √(N)

One Sample Z-Test

The method of determining the statistical significance of a sample mean. 

Z_obt = (M - μ) / σ_M

(σ_M stands for the standard error of the mean)

One-Sample T-Test

The method of determining the statistical significance of a sample mean, but the population standard deviation is not known.

t_obt = (M - μ) / s_M

(s_M stands for the estimated standard error of the mean, df = n - 1)

Estimated Standard Error of the Mean (s_M)

The estimate of how far a sample mean would deviate from the grand mean.

s_{M =} s / √(N)

Two-Sample T-Test

The method of determining the statistical significance of the responses of 2 groups, but the population standard deviation and the population mean is not known. There are 2 types that encompass the different ways to manipulate an independent variable: a between-subjects (independent-samples) design, where the two samples consist of entirely different people tested under separate conditions, and a within-subjects (paired-samples) design, where one sample is tested twice under different conditions.

Independent-Samples (Between-Subjects) Test

A two-sample t-test where different groups of people are exposed to a different condition. Two means are obtained from different samples.

t_obt = ((M₁ - M₂)-(μ₁ - μ₂)) / s_M1-M2

((μ₁ - μ₂) is almost always set to zero; s_M1-M2 stands for the standard error of the difference, df = (n₁ -1) + (n₂ - 1))

Estimated Standard Error of the Difference (s_M1-M2)

The average difference between the means of two samples drawn from the same population. Used to calculate independent-samples t-tests.

s_{M1-M2 =} √((s²_p / n₁) + (s²_p / n₂))

(s²_p stands for the pooled variance)

Pooled Variance

The average of two or more group variances used to calculate independent-samples t-tests.

s_p² = (SS₁ + SS₂) / (df₁ + df₂)

Paired/Related-Samples (Within-Subjects/Repeated-Measures) T-Test

A two-sample t-test where each subject in one group is tested twice under different conditions. Two means are obtained from the same sample.

t_obt = (M_D - μ_D) / s_MD

(μ_D is almost always set to zero; s_MD stands for the standard error of the difference, df = n - 1)

Estimated Standard Error of the Difference (s_MD)

The average difference between the means of two samples drawn from the same population. Used to calculate paired-samples t-tests.

s_MD = √(s² / n)

s² = SS / df

SS = Σ(D - M_D)²

M_D = ΣD / n

(s² stands for the variance of the difference between pairs of scores; SS stands for the squared sum of difference scores; M_D stands for mean difference)

Confidence Intervals

Intervals that measure the estimated range of the values where the population parameter being estimated is likely to lie within. They capture where the true value probably is, based on sample data, with a specified, and usually high (95-99%) level of confidence. They are used to test significance in paired-samples t-tests.

CI or diff = M_D ± (t_crit)(s_MD)

What is one determining factor for confidence intervals?

Sample size, as larger samples yield narrower confidence intervals and more certainty, and variability, as more variability leads to wider intervals and less certainty.

Why is a paired-samples t-test the most powerful two-sample t-test?

There is more of a probability of rejecting the null hypothesis if the independent variable really has an effect. This is because there is less noise, or extraneous variability, in the data due to subjects being tested twice.

What is one disadvantage of a paired-samples t-test?

Carry-over effects, such as drug effects, are possible in this test.

Homogeneity of Variance

An assumption that must be implemented for any independent-samples t-test, stating that all comparison groups must have the same variance. It is primarily tested through Hartley’s F-max test.

Hartley’s F-Max Test

A test used to determine the homogeneity of variance. Large F-max values indicate a large and significant difference between sample variances, while small F-max values (around 1) indicate a small difference. If sample information suggests a violation of the homogeneity of variance, the degrees of freedom must be adjusted.

F-max = s² (largest) / s² (smallest)

Analysis of Variance (ANOVA)

The method of assessing the difference between the responses of 3 or more groups by comparing the variances across them. There are many types that vary depending on the number of independent variables involved.

One-Way ANOVA

An analysis of variance used when there is a clear independent and dependent variable, along with 3 or more levels. It is tested through the F-test.

F-Test

A test used in the one-way ANOVA that compares the variances of 3 or more sample means. If all means are statistically similar, the null hypothesis is retained.

F_obt = MS_between / MS_within

MS = SS / df

(df_between = k - 1; df_within = N - k; df_total = k - 1; k stands for the levels of the independent variable; MS stands for the mean of squared deviations/variance; N stands for the sample size summed across all groups)

Sum of Squared Deviations

Values used in the F-test of the one-way ANOVA.

SS_total = Σx²_total - ((Σx_total)² / N)

SS_between = Σ((sum of scores for each group)² / (n for group)) - ((Σx_total)² / N)

SS_within = SS_total - SS_between

(N stands for the sample size summed across all groups)

Post-Hoc Test

A test only performed when the null hypothesis is rejected in an analysis of variance. It signifies specifically which groups are different from the other groups. There are many post-hoc tests, with more than 8.

Tukey’s Honestly Significant Difference (HSD)

A post-hoc test that determines which value of the dependent variable the difference between group means must exceed in order to be a significant difference. To use this test, the sample sizes (n) of all the groups must be equal.

HSD = q_k (√(MS_within / n))

(q_k stands for the studentized range statistic)

η²

A post-hoc test that determines the proportion of variance accounted for by the independent variable. It is analogous to r² by correlation.

η² = SS_between / SS_total

Parametric Statistics

Statistics that rely on underlying population parameters, such as μ and σ. They assume normal distribution according to the central limit theorem, similar variance among groups, and interval/ratio scales.

Nonparametric Statistics

Statistics that do not rely on underlying population parameters. They are used for nominal/ordinal scales, abnormally distributed data (skewed data, etc.), and data with heterogeneous variance. There are many different types of nonparametric statistics, and many parametric tests have a nonparametric alternative, such as the Mann-Whitney U-test instead of the t-test.

χ²-Test

A non-parametric statistical test. It is used when the dependent variable is nominal. It tests the frequency of falling into different nominal categories and whether the observed frequencies differ significantly from what would be expected by chance or from a theorized pattern. Because there are no post-hoc tests, avoid contingency tables with greater than one degree of freedom.

χ² = Σ((f_o - f_e)² /  f_e)

f_e = ((column total)(row total)) / (grand total)

(df = k - 1)