Research Methods and Statistics

theory

organized set of beliefs about a phenomenon

hypothesis

predictions about associations between/among variables that often derive from a larger theoretical framework

independent variable

The experimental factor that is manipulated; the variable whose effect is being studied.

dependent variable

The outcome factor; the variable that may change in response to manipulations of the independent variable.

efficacy trials

the impact of an intervention under optimal conditions

effectiveness trials

if the intervention works under real-world conditions

intent-to-treat analysis

Analysis of all subjects in a study regardless of whether they completed or dropped out of the study.

quasi-experimental studies

experiments that lack random assignment of participants to conditions (i.e., self-select into treatment conditions; participants can choose whether they want medication or not) and when groups are differentially exposed to treatment conditions); less control

correlational studies

non-experimental method; a type of research that is mainly statistical in nature; determines the relationship between two variables

case-control design

compare a group of participants who possess a certain characteristic (e.g., diagnosis of attention deficit hyperactivity disorder [ADHD]) with a group of participants who do not possess the characteristic.

cohort design

a longitudinal study in which data are collected at two or more points in time from individuals in a cohort

single-case experiment

assess the causal influence of an intervention on an outcome (i.e., assessment before, during, and after intervention)

ABAB design

single-case design that alternates the baseline A phase (intervention absent) with an intervention B phase (intervention present).

multiple baseline design

A type of single-subject design in which a treatment is instituted at successive points in time for two or more persons, settings, or behaviors.

level shifts

comparison of the last data point in an immediately prior phase to the first data point in an immediately subsequent phase

slope

rate of behavior change

case study

a descriptive technique in which one individual or group is studied in depth in the hope of revealing universal principles

internal validity

the extent to which the association between x and y is causal in nature

Nonspuriousness

when the hypothesized cause is responsible for the effect (e.g., psychotherapy is responsible for decreased depression symptoms as opposed to the natural progression of symptoms which may remit or decrease in severity over time independent of any treatment).

External validity

extent to which the causal association can be generalized to or across variations in study instances (e.g., individuals, treatments, outcomes, settings, and times).

statistical conclusion validity

examines whether there is a statistical association between x and y, and the magnitude of this association

construct validity

extent to which a test measures the hypothetical trait (construct) it is intended to measure (i.e., conducting a factor analysis)

maturation

when naturally occurring changes are mistaken for an intervention effect (e.g., when symptoms remit because of the passage of time rather than the effects of an intervention).

history

occurs when some event (or constellation of events) takes place during the study and impacts the results in a manner mistaken for an intervention effect

statistical regression (regression to the mean)

when extreme scores tend to revert to the mean on a subsequent evaluation

attrition

when the pattern of participant drop-out impacts the way one might interpret the results as an intervention effect

testing

when exposing individuals to the pretest changes them in ways that might be mistaken for an intervention effect (i.e., a pretest that asks detailed questions about the personal and social consequences of one's alcohol use might inspire individuals to lower their consumption)

instrumentation

when the measurement tool changes during the course of the study and impacts the results in a manner mistaken for an intervention effect

selection

when systematic differences among intervention groups can be mistaken for an intervention effect (i.e., groups differ at the beginning of the study because of the way subjects were assigned to groups and is a potential threat whenever subjects are not randomly assigned to groups

classical test theory

the variance of an observed measure comprises true score variance and random error variance

true score variance

reflection of the construct of interest

random error variance

reflects all other factors that vary randomly over testing occasions to impact an individual's score (e.g., test fatigue)

internal consistency

measure reliability on a single testing occasion by examining the degree of inter-item correlation in multiple-item tests (i.e., Cronbach's alpha)

Kuder Richardson formula

Used to calculate interitem consistency when items are dichotomous (yes/no, true/false)

split-half reliability

A measure of reliability in which a test is split into two parts and an individual's scores on both halves are compared.

test-retest reliability

measures the reliability of a measure over time in repeated administrations

alternate-forms reliability

a procedure for testing the reliability of responses to survey questions in which subjects' answers are compared after the subjects have been asked slightly different versions of the questions or when randomly selected halves of the sample have been administered slightly different versions of the questions (i.e., between the Beck Depression Inventory and the Patient Health Questionnaire)

interrater reliability

extent to which coding is consistent or stable among raters

face validity

extent to which items appear to measure the construct of interest

content validity

examines whether test items adequately represent the content domains for the relevant construct

structural validity

is the extent to which the structure of the measure is consistent with the theorized factor structure of the construct

criterion validity

an empirical form of measurement validity that establishes the extent to which a measure is correlated with a behavior or concrete outcome that it should be related to

concurrent validity

the degree to which the measures gathered from one tool agree with the measures gathered from other assessment techniques

convergent validity

scores on the measure are related to other measures of the same construct

discriminant validity

scores on the measure are not related to other measures that are theoretically different (i.e., a measure of psychological well-being should not correlate highly with a measure of impostor syndrome)

generalizability theory

based on the idea that a person's test scores vary from testing to testing because of variables in the testing situation

item response theory

theory and practice of analyzing each individual item for its ability to discriminate (i.e., can distinguish those with low vs. high intelligence)

exploratory factor analysis

Researchers do not propose a formal hypothesis about the factors that underlie a set of test scores, but instead use the procedure broadly to help identify underlying components

confirmatory factor analysis

A procedure in which researchers, using factor analysis, consider the theory associated with a test and propose a set of underlying factors that they expect the test to contain; they then conduct a factor analysis to see whether the factors they proposed do indeed exist.

nominal measurement

a measure for which different scores represent different, but not ordered, categories

ordinal measurement

a measurement in which attributes are ranked by assigning numbers in ascending or descending order

interval measurement

a measure for which a one-unit difference in scores is the same throughout the range of the measure (no true zero point)

ratio measurement

A measurement level with equal distances between scores and a true meaningful zero point (e.g., weight).

descriptive statistics

numerical data used to measure and describe characteristics of groups (i.e., measures of central tendency and measures of variation)

central tendency

used to identify the center of a distribution of scores (i.e., mean, median, mode)

variability

describe the scatter or dispersion of scores in a distribution

variance

standard deviation squared

standard deviation

a measures of dispersion (variability) of scores around the mean of the distributions; captures the average distance of scores from the mean (square root of sum of x-mean squared divided by n -1); square root of variance

z-score

a type of standard score that tells us how many standard deviation units a given score is above or below the mean for that group (i.e., normal distribution)

z-score conversion

square root of x-mean over standard deviation)

Positive skewed distribution

Most of the scores are bunched towards the left indicating low scores; mean is greater than the median which is then greater than the mode

negative skewed distribution

Most of the scores are bunched towards the right; mode is greater than the median which is greater than the mean

Leptokurtic distribution

Distribution curve is very tall, thin and peaked.

platykurtic distribution

Flatter and more spread out than a normal curve.

(Memory: 'Plat' sounds like 'flat')

kurtosis

measures the relative peakedness of the distribution

parametric statistics

statistics that make more distributional assumptions (e.g., that the distribution is normal), assume data are measured on an interval or ratio scale, are conducted on actual data (as opposed to on ranks derived from data), and allow researchers to test more specific hypotheses about the populations from which observations are drawn

nonparametric statistics

typically used when assumptions of parametric approaches are (seriously) violated and/or when the data cannot be used to compute necessary quantities for parametric statistics

inferential statistics

use sample data to test hypotheses about presumed population parameters

probability theory

used to quantify the likelihood of the data (e.g., the sample mean), given the presumed population parameter, (i.e., the population mean)

Null hypothesis significance testing (NHST)

the researcher specifies two mutually exclusive hypotheses (null and alternative) regarding the population parameter of interest. On the other hand, the alternative hypothesis (or the researcher hypothesis) subsumes all other possible outcomes

Statistically significant results

Results with a p value of less than .05 and rejecting the null hypothesis

Not statistically significant results

failing to reject the null hypothesis

type 1 error (alpha)

rejecting the null hypothesis when it is true (false positive); set prior to data collection

Type 2 error (beta)

accepting the null hypothesis when it is false

statistical power

Probability of correctly rejecting a false null hypothesis (1-Beta); increases when having a large sample, maximizing the effects of the IV, increasing the size of alpha, and reducing error

Cohen's d

reflects the standardized mean difference between two groups (e.g., treatment and control).

Cohen's d small effect size

0.2

Cohen's d moderate effect size

0.5

Cohen's d large effect size

0

Confidence Interval (CI)

indicates a range in which the population mean is believed to be found

Clinical significance

whether or not the difference was meaningful for those affected

number needed to treat (NNT)

quantifies the number of additional patients in the treatment and control conditions needed to generate one additional positive outcome in the treatment group relative to the control group

one sample z-test

used to compare a sample mean to a population mean when the population standard deviation is known

one sample z-test computation

sample mean (M = 110) minus the population mean (μ = 100). divided by standard deviation of the sample

statistical significance

data that occurs relatively infrequently under the null hypothesis

one sample t-test

Used to determine if a single sample mean is different from a hypothesized population mean if the standard deviation is unknown

independent samples t-test

used to test mean differences between two populations when the population distributions are normal (i.e., one might want to test whether older adults and younger adults differ significantly in their average level of need for emotional support)

paired samples t-test

examines mean differences across the observation pairs

ANOVA

used to compare multiple sample means

ANOVA assumptions

an interval- or ratio-level dependent variable, the independence assumption, the normality assumption, and homogeneity of variance

ANOVA main effects

examine the unique effect of an independent variable on an outcome

ANOVA interaction

examine whether the effects of one predictor on an outcome vary significantly as a function of another predictor

omnibus statistical test

statistical test that examines whether there are any significant differences between multiple groups overall

post hoc tests

additional significance tests conducted to determine which means are significantly different for a main effect

Bonferroni correction

A multiple comparison procedure in which the familywise error rate is divided by the number of comparisons

one-way between subjects ANOVA

can be used to contrast two or more treatment groups on a dependent variable

one-way within subjects ANOVA

This can be used to examine a single cohort's symptom levels or two or more assessments (e.g., pre-test, posttest, and follow-up measures for individuals exposed to a single intervention.)

Sphericity

assumes that the variances of the differences of the various factor levels are equal

Greenhouse-Geisser correction

increases the P value when sphericity is not met