Advanced Statistics in Psychology - Exam 2

Data Screening

Used to check for errors, missing data, outliers (known as “data cleaning)

Model Diagnostics

Checking the assumptions of the model and ensuring that there are no issues with it

In what order should you complete data screening and diagnostics?

1. Accuracy/errors

2. Missing data

3. Outliers

   1. The first three steps need to be done first as it affects the assumptions surrounding the data

4. Assumptions

What does accuracy look like when screening data?

Looks for typos in data

• Can use descriptive statistics to check for numbers that shouldn’t be in the dataset

  • Min, max, mean, SD, and valid/missing values

• This can be changed or deleted

  • Can also be checked using frequency tables

What should you do with data that has labels?

Highlight and delete incorrect values

• This is only done if you are nto sure and can’t change the data

Accuracy

Checking data for typos; use descriptive statistics and frequency tables to help with this

Missing Data

This can be due to forgetting to enter data, typos, or if someone forgot to answer a question

This can limit statistical power depending on how much is missing.

Use the filter function on JASP to see where data is missing

What are the three types of missing data?

MCAR/MAR and MNAR

MCAR/MAR

Data that is missing randomly; this is more ideal to have

MNAR

Data that isn’t missing randomly; this is less ideal and can be due to things such as fatigue or question sensitivity

How much missing data can you have?

It depends:

• < 5% of data is okay

• 5-10% is when this would be considered borderline

• >20% is when this data needs to be fixed

What can you do when there’s missing data?

• Split groups and test characteristics of people with missing data vs. those that don’t

• Filter out people with missing data and run analyses

• Delete the variable or person

• Exclude people in a pairwise or listwise manner

• Fill in data with “estimation” a.k.a the mean

Pairwise exclusion

Only excludes people when they have missing values for a specific analysis

Listwise exclusion

Excludes them for all analyses; this can reduce the statistical power

Outlier

A case with an extreme value on one variable or multiple variables; this can influence parameter estimates or errors

What are the the two types of outliers?

Univariate and multivariate

Univariate Outlier

For basic statistics, can be screened by looking at Z-score cutoffs (+/- 3 Z-scores)

Multivariate Outlier

Outliers for advanced or multivariable statistics; to spot outliers in this context, you must run a regression and a Mahalanobis test (use Chi-square cutoff score to help eliminate outliers)

• Can also use a Leverage test to see if there are ones that are statistically significant

Assumptions

Conditions that the GLM believes (assumes) are true about the data (or more correct - the residuals/error)

What are the five major assumptions?

1. Additivity and linearity (data assumptions)

2. Normality

3. Homogeneity of Variance/Homoscedasticity

4. Independence of errors

5. Spherical errors

Additivity

When variables are too correlated to be used together and there may be too much overlap

How to check for additivity?

Run a bivariate correlation and develop a heat map

• Look at the scores

  • For multicollinearity: r > 0.90

  • For singularity: r > 0.95

  • 
    

Linearity

The relationship between the predictor and outcome variable is straight; can look at a scatterplot or linear model to determine whether this assumptions has been met

How to check for linearity?

Generate a residual vs. predicted plot from the linear regression

• Tip: If you have a categorical predictor with two groups, you will ALWAYS meet linearity

Normality

Assumes that the model residuals (directly) are normally distributed

How to check for normality?

Create a standardized residuals histogram; ideally this should look like a normal bell curve with minimal skew on both ends

Homogeneity of Variance

When the variance of an outcome variable is stable/similar at all levels of the predictor variable

What are violations of homogeneity called?

Heteroscedasticity

How to check for homogeneity of variance?

Using a standardized residuals vs. predicted plot and looking for a random cloud/box shape

Independence of errors

When one person’s score shouldn’t influence another persons’s score; this is more of a design issue

What is a consequence of independence of errors if this is violated?

Autocorrelation/Autoregression

Autocorrelation/Autoregression

Responses at one point in time will likely correlate with responses at another time

(Example: Family/twin studies)

When should t-tests be used?

When there is one predictor variable that is categorical (that has two levels) and an outcome variable that is continuous; essentially it is used to compare means/groups

Independent t-test

Between subjects/independent designs; this compares two means based on independent data

Dependent/paired samples t-test

Repeated measures/within subjects/dependent designs; this compares two means based on RELATED data

Matched samples

When data is collected from the same people at different time points; typically associated with a dependent t-test

Why would we use a t-test?

To see an observed difference between sample means

To see if there was a difference between the sample means that we would expect to obtain (eventually) if there was NO effect (i.e., if the null hypothesis was true)

How to tell if there was a significant difference between groups visually?

If there is no overlap between the confidence intervals and bell curves

Stats wise: When there is clear difference in the means and SD

This can be checked using descriptive or raincloud plots

What role does the standard error play in a t-test?

If the observed difference between sample means becomes larger, the more confidence there is that there is a genuine difference between samples - signal to noise ratio

Why would we use t-scores instead of z-scores to compare groups?

Standard error can increase since group comparisons are typically done with smaller samples, which may make this less accurate

T-distributions were created to account for this uncertainty in SE estimates from sample data

What assumptions mainly concern independent t-tests?

Normality (check Q-Q plots) and Homogeneity of Variance (can plot the raw data and look for obvious differences between two groups)

What is the difference between a one-way hypothesis and a two-way hypothesis?

One-way: Directional and has more statistical power

Two-way: Nondirectional and leverages both hypotheses; though it has low power it won’t necessarily impact the interpretation of the results since it would be based on “random chance”

Effect size

Helps quantifies the magnitude of the effect produced by statistical significance

What are the three types of effect sizes used in t-tests?

Cohen’s d, Hedge’s g, and Glass’s d (delta)

Cohen’s d

Best for qual variance between groups when samples are larger than 30

Hedge’s g

Best for equal variance between groups when samples are smaller than 20

Glass’s d (delta)

Best for unequal variance between groups and/or when you have no obvious control group

What is the interpretation for all effect sizes?

The value is X number of SDs that the two means are apart from each other

What are the cutoffs according to Cohen’s d?

• Small - 0.2

• Medium - 0.5

• Large - 0.8

How to write an interpretation of an independent t-test?

Make sure to include the sample mean (M) and standard deviation (SD) for both groups that were compared; indicate whether there was a significant difference using values that were obtained from either the student or Welch’s test

These values include mean difference, 95% CI [lower, upper] of mean difference, t(df) = t-statistic, p = value, Cohen’s d (d = x [lower, upper] 95% CI for Cohen’s d)

Dependent t-test

Compares scores from same participants at different times

• Accounts for individual variation, which means that there’s less noise and increased power to detect true differences

• Can use smaller sample size as everyone is there own group

How to visually demonstrate group differences for a dependent t-test?

Using descriptive plots, raincloud plots, and bar plots

What tests should you do if assumptions are violated in an independent t-test?

Mann-Whitney Test (used for non-normal data) and Welch’s t-test (for heteroscedasticity/unequal variances b/w groups)

What tests should you do if assumptions are violated in a dependent t-test?

Wilcoxon Signed-Rank Test

What are the limitations of using t-tests?

Can really only be used to compare 2 means at a time (with either two levels of measurement or measured 2x) and can only be used with one predictor/IV

This is when we can use an ANOVA!

When to use an ANOVA?

When there’s a categorical predictor with more than 2 levels of measurement, when there’s more than one predictor variable, and when there’s one outcome variable that’s continuous

Why can’t we use t-tests to measure multiple variables/ levels of measurement?

Because the more t-tests that are conducted, the greater likelihood that a Type 1 error rate increases; this is known as a family-wise error rate and can increase exponentially to 100%