CI & FT

Spring 2025
Instructor: W. David Scales

Analyses have become more sophisticated in response to the shortcomings of past assumptions.
With appropriate tools, virtually any question can be answered.
- Requirements include:
- Strong assumptions
- Good theoretical backing
- Empirical evidence to support claims

Exploration has only scratched the surface of analytical capabilities.
Underlying assumptions are fundamental to all further analyses.
Differences in analyses are characterized by:
- Calculational complexity
- Interpretation
Acknowledgment of fundamental flaws in previous methodologies.

Early Validity Types
- Criterion validity
- Content validity
- Initially simple and practical, these forms proved insufficiently complex.
Consequential Validity (Messick, 1989)
- Currently-dominant form of validity.
- Approaches validity as a continuous process rather than a fixed property of an instrument, test, or survey.
Challenge
- Establishing validity is time-consuming and expensive.
- The validation process is infinite, complicating definitive assessments.

Calls for separation of theory and practice to facilitate easier research.
Utilization of advanced psychometric techniques to accurately define constructs.
Kane (2004)
- Proposed an argument-based approach to provide validity evidence comparable to legal arguments.
Borsboom et al. (2004)
- Advocated for a causality-based approach to validity.
- Test validity criteria:
  - (a) The attribute must exist.
  - (b) Changes in the attribute must lead to changes in test responses.

Most statistics learned are based under Classical Testing Theory.
Most calculations have been mean score-based, raising concerns about accuracy at extreme score levels.
Key Issues:
- Accuracy of statistics generally registers best around the mean.
- Extreme scores lead to less reliable estimations due to limited information.

As scores diverge from the mean, confidence in their accuracy diminishes due to less common occurrences.
Explanation of p-value:
- Indicates the likelihood of obtaining a result if the null hypothesis (H0) is true.
Fundamental equation: $X = T + E$
- Where:
- $X$ = observed score
- $T$ = truth or accuracy
- $E$ = error
- Importance of understanding truth and error for result confidence.

P-values are sample-based statistics, limiting their applicability to broader populations.
Concerns arise regarding test biases:
- Who takes the tests?
- General Population vs. Specific Subsets.
- Sample composition impacts the perceived difficulty or easiness of test items.

Recognizing difficulty in making generalizations from potentially inaccurate results.
Key questions:
- How to ensure items on a test or survey effectively discriminate?
- Determining appropriateness of survey items for intended populations.
Challenges for measurements in accuracy and specificity among psychometricians.

Call for advanced analytical tools to enhance research capability.
Legacy advanced analyses offer a framework for current data sets and technologies.
More techniques are emerging for the analysis of specific data types or research issues.

Utilizes multiple independent variables (IVs) to predict a single dependent variable (DV).
Example: Predicting quality-of-life score based on temperature, population, income, and per-capita spending on social services.

Addresses limitations of mean-based statistics by not relying on means for calculations.
IRT allows analysis of items across ability levels confidently.
Matching items to examinees’ ability levels enhances accuracy.
Key components of IRT:
- Examinee’s ability ( $θ_j$ ).
- Item properties:
  - Discrimination ( $a_i$ ).
  - Difficulty ( $b_i$ ).
  - Probability of guessing correctly ( $c_i$ ).

Technique aimed at reducing data complexity by identifying simplified explanations for data clusters.
Two main types:
- Exploratory Factor Analysis (EFA): Used in pilot testing phases.
- Confirmatory Factor Analysis (CFA): Checks if items fit the intended model efficiently.
Importance of sampling: Changes in the population can drastically change factor analysis results.

Combines capabilities of factor analysis with advanced regression techniques to analyze relationships between latent variables.
Advantageous for visualizing hypothesized relationships and testing against data.

Models used for diagnosing latent traits and suggesting treatments or evaluations.
Generates probability profiles of meeting diagnostic criteria, linking unobservable traits to observable behaviors.

Allows for multi-level regression analyses, incorporating variations at individual student, classroom, school, and district levels.

A wide array of advanced techniques exists for complex research questions:
- Multidimensional IRT
- Multivariate statistics
- Test Equating
- Second-Language Testing & Development
- Program Evaluation
- Standard-setting Techniques
Comprehensive techniques are crucial for addressing intricate analytical inquiries.

Borsboom, D., Mellenbergh, G.J., & van Heerden, D. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071.
Cronbach, L.J., & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.
Cureton, E.E. (1951). Validity. In E.F. Lindquist (Ed.), Educational Measurement (pp. 621-694). Washington, DC: American Council on Education.
Kane, M.T. (2004). Certification testing as an illustration of argument-based validation. Measurement, 2, 135-170.
Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational Measurement (3rd ed.). New York, NY: