Reliability, Validity, and Hypothesis Testing in Behavioral Statistics
Fundamentals of Reliability and Validity
Purpose of Measurement Tools:
- In scientific research, specialized tools measure psychological constructs such as intelligence, attention, or depression.
- Research results are rendered inconclusive if measurement tools—including scales, tests, and questionnaires—are faulty.
- Analogy: Using faulty measurement tools is equivalent to constructing a house using tape measures that are off by varying degrees, levels that are inaccurate, and scales that fail to function consistently.
- Dependability and accuracy must be empirically verified for all instruments.
Core Definitions:
- Reliability: The consistency of a measurement tool or instrument.
- Validity: The accuracy of a measurement tool or instrument.
Classical Test Theory and Components of Measurement
Error in Measurement:
- All empirical measures inherently contain sources of error.
- Any Observed Score (such as performance on an I.Q. test, a reading from a bathroom scale, or an exam score) comprises exactly components:
- True Score: The theoretical component that genuinely reflects the construct of interest, such as actual intellectual capability, true physical weight, or actual statistical knowledge.
- Error Score: The component of the score introduced by extraneous factors, such as test bias, calibration imbalance on a scale, or distracting testing environments (e.g., an air conditioning breakdown causing excessive heat).
Characteristics of a Reliable Measure:
- A reliable tool minimizes the error score as much as possible, demonstrating consistent measurement across trials.
- High-Error Demonstration in Testing: Scoring far below average on Part A of an I.Q. test and scoring in the certified genius range on Part B indicates substantial measurement error.
- High-Error Demonstration in Physical Measurement: Stepping on a scale to find a target weight goal, then stepping on it immediately afterward to find an increase of , demonstrates that the instrument is prone to error rather than reflecting a true weight shift.
Forms and Methodologies of Reliability
Test–Retest Reliability:
- Involves measuring the exact same individuals at distinct points in time.
- Example: Administering an I.Q. test on Tuesday and readministering it on Friday to the same subjects. Inconsistent scores across administrations indicate low reliability.
Internal Consistency:
- Evaluates whether all items on a single test measure the same construct during a single administration.
- Split-Half Reliability:
- Assesses reliability by splitting a test in half and examining the relationship between the halves.
- Can be evaluated by comparing Part A to Part B of a measure (such as an Introversion scale), or by correlating scores on even-numbered items with scores on odd-numbered items.
- Cronbach's Alpha:
- Mathematically correlates every item with every other item across the test.
- Demonstrates whether all items designed to measure a specific psychological construct (such as introversion) show high inter-item correlation.
Interrater Reliability:
- The degree of agreement or consistency among independent observers or raters.
- Example: If an instructor and a teaching assistant grade the same student essay examinations but consistently disagree on scoring, interrater reliability is excessively low, rendering the grading protocol unreliable.
Forms and Methodologies of Validity
Core Concept:
- Validity confirms whether an instrument measures what it is supposed to measure.
- High reliability does not ensure validity. For instance, a scale reading consistently across every single trial is perfectly reliable, yet invalid. Similarly, consistently scoring on an I.Q. test on repeated trials indicates reliability, but fails to provide a valid measure of intelligence.
Construct Validity:
- Assesses whether the instrument accurately captures the underlying theoretical construct.
- Measured by correlating the instrument with other validated measures designed to assess the same construct.
- Example: Demonstrating construct validity for the GRE by establishing correlations with the SAT and standardized Reading Comprehension tests.
Content Validity:
- Evaluates whether the operationalized test content is fully representative of the specified behavioral or knowledge domain.
- Example of Poor Content Validity: An introductory statistics examination consisting exclusively of correlation problems, omitting means, standard deviations, and ANOVAs. This exam measures correlation proficiency rather than the broader domain of statistics.
- Subtypes Related to Construct Differentiation:
- Convergent Validity: Two separate instruments measuring the same construct should produce highly correlated, similar scores.
- Example: Comparing scores on the Introversion sub-scale of the Five Factor Inventory (FFI) with the Introversion sub-scale of the Myers-Briggs Type Indicator.
- Discriminant Validity: Two separate instruments measuring conceptually distinct constructs should produce divergent, dissimilar scores.
- Example: Because introversion is theoretically distinct from shyness, scores on the Introversion sub-scale of the Five Factor Inventory (FFI) should differ substantially from scores on the Wellesley Shyness Test.
Criterion Validity:
- Evaluates how effectively test scores relate to an external benchmark or standard, either in the present or in the future.
- Concurrent Validity:
- Assesses whether test performance corresponds to a person's current ability.
- Example: Determining whether current GRE scores accurately reflect concurrent academic performance in graduate school.
- Predictive Validity:
- Assesses whether test performance forecasts ability or outcomes at a future point in time.
- Example: Determining whether GRE scores successfully predict graduate program completion rates and career employment outcomes.
Quantitative Measurement of Reliability and Validity
Statistical Indexing:
- Both reliability and validity are typically quantified using correlation coefficients or related statistical indices (such as Cronbach's alpha), yielding coefficients between and .
- Values near signify very low reliability or validity.
- Values near signify high reliability or validity.
Evaluative Scenarios and Practice Classifications:
- Scenario 1: Blanca Evangelista administers an exam to participants and correlates scores on the even-numbered items with odd-numbered items, yielding a correlation of .
- Type: Split-Half Reliability.
- Magnitude: High reliability.
- Scenario 2: Electra Abundance administers a fashion quiz to models to forecast runway performance the subsequent week; the quiz correlates with show placement at .
- Type: Predictive Validity (Criterion Validity).
- Magnitude: Low validity.
- Scenario 3: Dr. Pray Tell calculates the rate of scoring agreement among judges at a talent show, resulting in a coefficient of .
- Type: Interrater Reliability.
- Magnitude: Moderate reliability.
- Scenario 4: Choreographer Damon Richards considers poise distinct from grace. He administers scales measuring both poise and grace to dancers, yielding a correlation of .
- Type: Discriminant Validity.
- Magnitude: High discriminant validity (a low correlation confirms that the two constructs are distinct).
Foundations of Hypothesis Testing
Overview of Hypothesis Formulation:
- Research in psychology progresses from foundational research questions to formal, testable hypotheses.
- Hypothesis: A specific, testable statement derived from a research question, informed by formal theory and systematic observation; an educated prediction.
- Hypotheses are developed following literature reviews of peer-reviewed articles addressing related questions.
- Example:
- Research Question: Do some personality types enjoy statistics more than others?
- Hypothesis: People who score below the median on extroversion on the Five Factor Inventory (FFI) will rate the enjoyment of a statistics course higher than those who score above the median.
Populations Versus Samples:
- Population: The entirety of the group about which a researcher intends to draw empirical conclusions.
- Example: Every adult female science fiction fan in the world.
- Sample: The specific subset of individuals observed and assessed to gather data.
- Example: A sample of friends responding to a Facebook advertisement to complete a survey evaluating perceptions of misogyny in science fiction.
- The numerical size of a sample is inherently smaller than the total size of the population.
Sampling Error:
- The extent to which a sample fails to accurately represent its target population.
- Can occur through systematic sample biases (e.g., all survey respondents having viewed exclusively older science fiction films, or all respondents residing in a region with atypical cultural norms regarding sexism).
- Researchers minimize sampling error primarily through random sampling procedures.
Statistical Significance and Chance:
- In a sample of individuals, extroverts might score points higher than introverts on a -point Likert scale measuring course enjoyment.
- Hypothesis testing and tests of statistical significance distinguish whether an observed difference is a genuine effect or merely an artifact of random sampling chance.
The Null Hypothesis Versus the Research Hypothesis
The Null Hypothesis ():
- Asserts that there is no difference or no effect among the groups under evaluation.
- Represents the statistical baseline indicating that any observed sample difference falls within the range expected by chance alone.
- Formulated at the theoretical level and expressed using population parameters (Greek letters, such as ).
- Examples:
- Individuals scoring below the median on extroversion on the FFI will rate their enjoyment of a statistics course equal to those scoring above the median.
- A group of depressed clients administered an SSRI will demonstrate equal improvement compared to depressed clients receiving a placebo.
The Research Hypothesis ( / / Alternative Hypothesis):
- Asserts an actual difference among groups that exceeds expectations based on chance.
- Represents a statement of inequality.
- Evaluated based on empirical observation and formulated using sample statistics (Roman alphabet symbols, such as or ).
- Examples:
- Introverts like statistics more than extroverts (\text{I} > \text{E}$).\n * Extroverts like statistics more than introverts (\text{E} > \text{I}$).
- Extroverts will score differently than introverts (\text{I} \neq \text{E}$).\n\n* **Comparative Summary:**\n * H_0H_1 states that group inequalities exceed chance.\n * H_0H_1 always applies to the empirical sample.\n\n# Types of Research Hypotheses\n\n* **Nondirectional Research Hypotheses:**\n * State that a difference exists between the comparison groups, but do not predict or specify which group will score higher.\n * *Example:* Extroverts will score differently than introverts on statistics enjoyment (\text{I} eq \text{E}).\n\n* **Directional Research Hypotheses:**\n * Explicitly specify the predicted direction of the difference, defining which group is anticipated to have higher or larger scores.\n * *Examples:*\n * Introverts enjoy statistics more than extroverts (\text{I} > \text{E}).\n * Extroverts enjoy statistics more than introverts (\text{E} > \text{I}).\n\n* **Applied Derivation Exercise:**\n * *Baseline Null Hypothesis (H_0):* The group of depressed clients who take an SSRI will show equal improvement on a pain scale compared with depressed clients who take a placebo.\n * *Derived Nondirectional Hypothesis (H_1\mu_{\text{SSRI}} eq \mu_{\text{Placebo}}).\n * *Derived Directional Hypothesis (H_1\mu_{\text{SSRI}} > \mu_{\text{Placebo}}).\n\n# Statistical Notation and Formal Symbolic Statements\n\n* **Standard Statistical Symbols:**\n * \mu: Population Mean\n * \bar{X}X: Sample Mean\n * \bar{X}_AX_A: Sample Mean for Group A\n * \bar{X}_BX_B: Sample Mean for Group B\n * H_1H_i: Research (Alternative) Hypothesis\n * H_0H_o: Null Hypothesis\n * eq: Does not equal\n * >: Greater than\n *