Research Methods Quiz 2

Reliability

The extent that measured values can be obtained consistently in the face of repetition

Systematic Error

A predictable error that is the result of a consistent flaw in the experimental design, such as measurement tool calibration. It is unidirectional and a threat to validity but not to reliability.

Random Error

An unpredictable error that can be bidirectional and is a matter of chance, such as electrical noise or temperature drift. Several repetitions can reduce the impact of these errors.

Internal Consistency

Type of reliability that reflects the extent to which the items on a multi-item test of instrument succeed in measuring the various aspects of the same characteristic and nothing else. Examples are ICC, Cronbach Alpha Score, and Kappa

Test-Retest Reliability

Type of reliability that reflects the ability of an instrument to be consistent between tests. Examples are ICC, SEM, and Kappa

Alternate Forms

Type of reliability that measures the consistency of test scores across two different versions of the same assessment. Example is Pearson r.

Intra- and Inter-Rater Reliability

Types of reliability that measure the stability of data when recorded by the same rater across several trials, and when measured by different raters (ideally at the same time). Examples are ICC and Kappa.

Kappa

Measure of reliability among items or raters for categorical data. Can be Weighted, Fleiss (>2 raters), or Cohen (2 raters). Gives a strength of association from 0 (weakest) to 1 (strongest).

Validity

The extend to which an assessment measures what it is actually designed to measure

Face Validity

Type of validity that is the extent to which something appears to measure what it claims to on the surface. Useful to convince laypeople.

Content Validity

Type of validity that is the extent to which something comprehensively covers all facets of a given construct or domain. Useful in professional contexts to sway experts.

Construct Validity

Type of validity that is the degree to which something accurately measures what it is designed to measure. Important for ensuring variables have been operationalized correctly. Includes Convergent (correlation with other existing tests) and Divergent (low correlation with tests that measure different things).

Criterion Validity

Type of validity that is the extent to which the results of measurement relate to a concrete, external outcome. Useful for measuring job performance or disease outcomes compared to tests that measure them. Includes Concurrent (score and outcome measured at the same time) and Predictive (outcome measured in the future).

Randomized Controlled Trial

A study where participants are randomly placed into groups (such as an active treatment group versus a placebo or standard care group) to eliminate bias.

RCT with Treatment Control

RCT where measurements are taken both before and after the intervention. Verifies randomization and more precise, but also more expensive and can cause pretest sensitization.

Post-test only RCT

RCT where measurements are taken only after the intervention. Cheaper, no sensitization, and sometimes required, but no way to ensure randomization and less precise.

Multigroup RCT

RCT with three or more distinct arms such as placebo and multiple dosage levels.

Factorial Design

Experimental setup where researchers study the simultaneous effects of two or more independent variables (referred to as factors) on a single dependent variable. Ex. the effects of both watering frequency and sunlight on plant growth.

Repeated Measure Design

Experimental approach where the same participants are exposed to all conditions or levels of an independent variable, with outcomes measured multiple times per subject over time or across tasks.

Quasi-Experimental Design

Study design that evaluates the causal impact of an intervention or treatment on a target population without using random assignment.

Correlational Research

A non-experimental research design that measures the statistical relationship between two or more variables without manipulating them.

Explanatory Design

Type of correlational study used to clarify or understand the relationships between two or more variables at a single point in time, ex. between daily sleep hours and test performance.

Predictive Design

Type of correlational study used to forecast or predict an outcome (criterion variable) based on one or more predictor variables measured earlier, such as SAT score predicting first year college performance.

Correlation Coefficient (r)

Measure of the linear strength and direction between continuous variables. Ranges from -1 (completely negative) to 1 (completely positive). The further it is from 0, the stronger the correlation (.1-.29 = weak, .3-.49 = moderate, .5-1 = strong)

Advantages of Correlation Studies

High real-world applicability, useful when manipulation is impossible, quick and cost effective

Limitations of Correlation Research

No way to determine causality, vulnerable to confounders and sample bias

Descriptive Research

A mainly quantitative research design aimed at accurately describing the characteristics, behaviors, or conditions of a population or phenomenon without manipulating any variables. Includes observational, case, and cross-sectional studies.

P-Value

Measure of whether or not the findings of a correlational study are statistically significant and not likely to have occurred by chance. Less than 0.05 is typically considered statistically significant.

Probability Sampling

Employing random, stratified, or cluster sampling to ensure equal representation of the target population.

Non-Probability Sampling

Employs non-random sampling methods such as convenience or judgment, making it faster and cheaper but less able to calculate error or generalize.

Type 1 Error

“False Positive,” rejecting a null hypothesis and then later finding out that it should not have been.

Type 2 Error

“False Negative,” failing to reject a null hypothesis and then later finding out you should have.