Data analysis Exam 3

What is validity?

The degree to which a test measures what it is supposed to measure (accuracy).

What is reliability?

The consistency or repeatability of a measure.

Can a test be reliable but not valid?

Yes (consistent but wrong, like a broken scale).

Can a test be valid but not reliable?

No—valid tests must also be reliable

What is an observed score?

The score obtained = true score + error score.

What is a true score?

he real ability without error (never directly known).

What is an error score?

The part of a score caused by measurement error.

What is logical validity?

When a test obviously measures what it should.

What is content validity?

When a test covers all relevant material (e.g., course topics).

What is criterion validity?

When test results relate to a standard/criterion.

What are the two types of criterion validity?

Concurrent and predictive.

What is construct validity?

Measures a theoretical construct (e.g., anxiety, intelligence).

What is intertester reliability?

Different testers getting the same results.

What is test-retest reliability?

Same test given twice → results compared.

What is intraclass correlation (ICC)?

A method (ANOVA-based) to estimate reliability.

What is interclass correlation?

Pearson r used to express validity

What are the 4 sources of measurement error?

• Test

• Instrument

• Scoring

• Participant

What is a z-score?

A score expressed in standard deviations from the mean

When should you use a z-score?

When comparing scores from different scales.

When should you use a z-score?

When comparing scores from different scales.

What is a meta-analysis?

A statistical method that combines results using effect sizes.

What is a systematic review?

A structured review analyzing study quality and methods.

Meta-analysis vs systematic review?

• Meta-analysis = statistical

• Systematic review = descriptive + structured

How are studies compared in meta-analysis?

Using effect sizes.

Why are large sample sizes better?

More accurate and less biased effect sizes.

What does homogenous mean?

Variances are similar.

What does heterogeneous mean?

Variances are different.

What tests homogeneity in meta-analysis?

H statistic.

What variables should be coded?

Variables that influence effect sizes (including moderators)

What is the file drawer effect?

Unpublished studies with non-significant results.

Are regular literature reviews publishable?

no

Are systematic reviews publishable?

Yes

What are the 4 steps of a systematic review?

• Identification

• Screening

• Eligibility

• Inclusion

What is a survey?

A descriptive research method using questionnaires/interviews.

What is more important: sample size or representativeness?

Representativeness

Why is sample size still important?

• Better representation

• Time/cost considerations

What are open-ended questions?

Free response, flexible but hard to analyse.

What are closed-ended questions?

Fixed responses (e.g., multiple choice).

What are the types of closed-ended questions?

• Ranking

• Scaled (Likert)

• Categorical (yes/no)

Give 3 rules for good questions.

• Clear wording

• Avoid bias

• Avoid double questions

What is a pilot study?

Trial run to test clarity and effectiveness.

How long should you wait before resending a survey?

2 weeks.

What is the Delphi method?

Multiple survey rounds with experts to reach consensus

What are the 4 interview question types?

• Hypothetical

• Devil’s advocate

• Idea

• Interpretive