Midterm Review: Measurement, Validity, Intelligence, and Classical True Score Theory

Module 1: Introduction to Measurement

• Core Concepts and Definitions:

  • Measurement: Assigning numbers to individuals systematically as a means of representing properties of the individual.
  • Test: A device for obtaining a sample of an individual's behavior.
  • Testing: The process of measuring variables by means of a device. It requires technician-like skills, meaning that one technician can be substituted for another without altering the process.
  • Assessment: Gathering and integrating data to make a psycho-educational evaluation. It requires the educated selection of tools and the systematic organization of data.
  • Evaluation: Integrating data to make data-based decisions.

• Types of Assessment Models and Tests:

  • Norm-Referenced Test:
    • Uses the test performance of others taking the same test as a comparative standard.
    • Purpose: Intended for classification and selection.
    • Examples: Standardized admissions examinations such as the SAT and ACT.
  • Criterion-Referenced Test:
    • Measures performance with respect to predefined content standards.
    • Purpose: Intended for instructional planning and competency evaluation.
    • Examples: State tests, driving tests, and related standardized proficiency benchmarks.
  • Curriculum-Based Measurement (CBM):
    • Assessment of academic skill fluency.
    • Characteristics: Direct, low-inference assessment method.
    • Application: Frequently used for progress monitoring.
    • Examples: Acadience benchmark tests.

Module 3: Validity

• Definition of Validity:

  • Validity: The degree to which a test measures what it purports to measure.

• Major Types of Validity:

  • Content Validity: The degree to which a test's contents contain an appropriate and representative sample of the content contained in the instructional objectives.
  • Criterion-Related Validity: The degree to which an individual's score or performance on a test predicts their performance in another area.
    • Predictive Validity: Predicts performance at a future point in time.
    • Concurrent Validity: Assesses relationship to a criterion within the same window of time.
  • Construct Validity: The degree to which a test measures the theoretical concept or trait that it was designed to measure.
    • Convergent Validity: How well performance on different measures of the same construct correlate positively.
      • Example: Two intelligence tests producing similar scores for the same individual.
    • Divergent Validity: The degree to which tests designed to measure unrelated constructs do not correlate.
      • Example: The WISC (Wechsler Intelligence Scale for Children) should not correlate with Beck Depression Inventory scores for an individual.

• Messick's Validity Matrix:

  • Core Theoretical Foundations:
    • Valid judgments are value judgments, connecting score meaning directly to consequences.
    • Decision validity represents the validity found in the meaning of test scores.
  • Matrix Dimensions and Quadrants:
    • Evidential Basis / Test Interpretation: Construct Validity ($\text{CV}$) — Establishes that the test measures what it purports to measure.
    • Evidential Basis / Test Use: Construct Validity + Relevance/Utility ($\text{CV} + \text{R/U}$) — Confirms that when used in a given situation, a test will still measure what it purports to measure.
    • Consequential Basis / Test Interpretation: Construct Validity + Value Implications ($\text{CV} + \text{VI}$) — Addresses concern for value-laden labels that will be used to describe the test-taker.
    • Consequential Basis / Test Use: Construct Validity + Relevance/Utility + Value Implications + Social Consequences ($\text{CV} + \text{R/U} + \text{VI} + \text{Social Consequences}$) — Involves considering the social consequences of test use or the direct application of test inferences.

• Diagnostic and Decision Errors:

  • False Positive: Predicting that a student would be at risk when they are not.
    • Example: A student is predicted to be below benchmark, but performs at or above benchmark.
  • False Negative: Predicting a student is not at risk when they actually are.
    • Example: Predicting a student to be at or above benchmark, but they perform below benchmark.

• Decision Accuracy Metrics:

  • Base Rate: The proportion of individuals with the most frequently occurring outcome.
  • Good Hit Rate: Depends on the base rate and must improve on it.

Module 4: Validity and the Construct of Intelligence

• Intelligence as a Common Factor / Latent Variable:

  • Reasoning: Intelligence cannot be measured directly.
  • Indirect Measurement: Intelligence can only be assessed indirectly through effect indicators:
    • Vocabulary
    • Short-term memory
    • Problem solving
    • Pattern recognition
    • Nonverbal reasoning

• Long-Term Trends in Intelligence Scores:

  • Flynn Effect: The substantial and long-sustained increase in both crystallized and fluid intelligence test scores over time.

Module 6: Classical True Score Theory and Reliability

• Classical True Score Theory Equation:

  • Mathematical formula:         $X = T + E$
  • Variables:
    • $X$: Observed score
    • $T$: True score
    • $E$: Error of measurement (random)

• Theoretical Attributes of the True Score ($T$):

  • Why $T$ is defined by the test rather than the trait being measured: $T$ is a theoretical construct because one cannot take infinite samples.
  • Why it is impossible to actually observe a true score: $T$ is fixed but unknown because one cannot take infinite samples.

• Interpretation of Confidence Intervals:

  • Standard interpretation framework: "We can say with $\text{\_\_\%}$ confidence that the true score falls between $\text{\_\_\_}$ and $\text{\_\_\_}$."