MASTER QUIZLET – PART 3

Reliability, Validity, Sampling, Methods Section & Statistics

Reliability - What is reliability?

The consistency of test outcomes or scores when measuring a dependent variable.

Reliability - What question does reliability answer?

Can we consistently obtain the same measurement each time we test?

Reliability - What is test-retest reliability?

The consistency of scores when the same test is repeated over several days or weeks.

Reliability - Why is reliability important?

It ensures that the data accurately represent the true ability or characteristic of the participants.

Reliability - What are the threats to reliability?

• Learning effects

• Motivation

• Imprecise testing procedures

Reliability - What are learning effects?

Participants improve simply because they become more familiar with the test.

Reliability - How can motivation threaten reliability?

Participants may not put forth the same effort each time they are tested.

Reliability - How do imprecise testing procedures threaten reliability?

Inconsistent procedures can produce different results even when nothing has changed.

Validity - What is validity?

The extent to which a measure truly measures what it is intended to measure.

Validity - What must a measure be before it can be considered valid?

Reliable.

Validity - Why is validity important?

Researchers must choose measures that accurately assess what they intend to measure.

Validity - What is internal validity?

The degree to which changes in the dependent variable can be attributed to the independent variable rather than another factor.

Validity - What is external validity?

The degree to which study findings can be generalized to other populations, settings, or treatments.

Validity - What are the threats to validity?

• History

• Maturation

• Testing

• Instrumentation

• Statistical regression

• Selection

Threats to Validity - What is history?

Events occurring outside the study that may influence participant outcomes.

Threats to Validity - What is maturation?

Natural changes that occur over time, such as improvements in fitness or learning.

Threats to Validity - What is testing?

A learning effect caused by repeated testing.

Threats to Validity - What is instrumentation?

Changes in equipment, calibration, or testing instructions that affect results.

Threats to Validity - What is statistical regression?

The tendency for extremely high or low scores to move closer to the group average when tested again.

Threats to Validity - What is selection?

Bias introduced by how participants are chosen, limiting generalizability.

Sampling - What is a population (N)?

The entire group of people sharing one or more common characteristics.

Sampling - What is a sample (n)?

A subgroup selected from the population.

Sampling - Why should a sample be representative of the population?

To improve external validity and generalizability.

Sampling - What is random sampling?

Randomly selecting participants from the population so everyone has an equal chance of being chosen.

Sampling - What is random assignment?

Randomly placing selected participants into study groups.

Sampling - Why are random sampling and random assignment important?

Random sampling improves generalizability, while random assignment improves internal and external validity.

Sampling - What are the steps in the sampling process?

1. Identify the target population.

2. Identify the accessible population.

3. Determine the desired sample size.

4. Select a sampling technique.

5. Implement the sampling plan.

Sampling Methods - What is probability sampling?

A sampling method in which the probability of selecting each participant is known and based on random processes.

Sampling Methods - What is nonprobability sampling?

A sampling method that does not use random selection.

Probability Sampling - What is simple random sampling?

Every individual has an equal chance of being selected.

Probability Sampling - What is stratified random sampling?

The population is divided into groups, and random sampling occurs within each group.

Probability Sampling - What is systematic sampling?

Selecting every kth member of the population.

Probability Sampling - What is cluster sampling?

Selecting entire clusters or groups from the population.

Nonprobability Sampling - What is convenience sampling?

Selecting participants who are easiest to access.

Nonprobability Sampling - What is purposive sampling?

Selecting participants with specific characteristics needed for the study.

Nonprobability Sampling - What is snowball sampling?

Participants recruit additional participants for the study.

Sampling - What is sampling error?

Variation due to chance between the population parameter and the sample statistic.

Sampling - What is sampling bias?

Bias resulting from the way participants were selected.

Sample Size - How is sampling error related to sample size?

Sampling error is inversely related to sample size.

Sample Size - Why do descriptive and correlational studies usually require larger samples?

To better represent the population and improve generalizability.

Sample Size - Why do heterogeneous populations require larger samples?

Because greater variability exists within the population.

Sample Size - What is the minimum sample size rule of thumb per group?

Approximately 30 participants per group (or fewer if the measure is very sensitive).

Sample Size - According to the lecture, should a sample exceed 50% of the population?

No.

General Rules - What is the minimum sample size for correlational research?

30 subjects.

General Rules - What is the minimum sample size for experimental research?

15 subjects per group.

General Rules - What is the minimum sample size for survey research?

At least 100 subjects in each major subgroup.

Sample Size - When might a larger sample be needed?

• Many uncontrolled variables

• Small effect sizes

• Subgroup analyses

• High attrition

• High statistical significance/power required

• Heterogeneous population

• Unknown DV reliability

Sampling Mistakes - What are common sampling mistakes?

• Using only available participants

• Choosing participants outside the target population

• Selecting control and experimental groups from different populations

• Failing to define the target population

• Using samples too small for subgroup analyses

• Ignoring volunteer bias

• Changing the sampling procedure for convenience

• Failing to account for attrition

Methods Section - What is the goal of the methods section?

To describe the study in enough detail that another researcher could exactly replicate it.

Methods Section - What should be described in the methods section?

• Sample

• Sample size

• Sampling technique

• Participant characteristics

• Experimental design

• Measures and instruments

• Instructions to participants

• Statistical tests

Methods Section - How should the methods section be written?

Using precise wording and detailed descriptions.

Statistics - What is statistics?

A language used to organize, analyze, and interpret numerical data.

Statistics - What are the four types of numerical data?

Nominal, Ordinal, Interval, and Ratio.

Statistics - What is nominal data?

Categories with no meaningful order (such as gender or favorite team).

Statistics - What is ordinal data?

Ordered categories without equal intervals.

Statistics - What is interval data?

Numerical data with equal intervals but no true zero.

Statistics - What is ratio data?

Numerical data with equal intervals and a true zero.

Statistics - What is descriptive statistics?

Statistics used to describe a sample or an individual's relationship to a group.

Statistics - What is inferential statistics?

Statistics used to generalize findings from a sample back to the population.

Statistics - What is a parametric test?

A statistical test used with interval or ratio data that are normally distributed.

Statistics - What is a nonparametric test?

A statistical test used with nominal or ordinal data or data that are not normally distributed.

Statistics - What is a normal distribution?

A symmetrical distribution in which the mean, median, and mode are equal.

Statistics - In a positively skewed distribution, how do the mean and median compare?

The mean is greater than the median.

Statistics - In a negatively skewed distribution, how do the mean and median compare?

The mean is less than the median.

Statistics - What is the mode?

The most frequently occurring score.

Statistics - What is the median?

The middle score in an ordered data set.

Statistics - What is the mean?

The arithmetic average.

Statistics - When should the median be reported instead of the mean?

When the data are skewed.

Statistics - What is range?

The highest score minus the lowest score.

Statistics - What is standard deviation?

A measure of how much scores differ from the mean.

Statistics - What is variance?

The square of the standard deviation.

Correlation - What does a correlation coefficient (r) measure?

The degree and direction of the linear relationship between two variables.

Correlation - What values can r range from?

-1.0 to +1.0.

Correlation - Does correlation show cause and effect?

No.

Correlation - Which correlation coefficient is used for interval or ratio data?

Pearson correlation (r).

Correlation - Which correlation coefficient is used for ordinal data?

Spearman rank-order correlation (rs).

Correlation - What does the coefficient of determination (r²) represent?

The amount of variance in one variable explained by the other variable.

Statistical Significance - What are the basic steps in testing statistical significance?

• State the null hypothesis.

• Select the alpha level.

• Choose a one- or two-tailed test.

• Conduct the statistical test.

• Make the appropriate decision.

Statistical Significance - When is the null hypothesis rejected?

When the p-value is less than the alpha level.

Statistical Significance - What is a Type I error?

Rejecting a true null hypothesis.

Statistical Significance - What is a Type II error?

Failing to reject a false null hypothesis.

Statistics - What are common parametric tests discussed in class?

One-group t-test, independent t-test, dependent t-test, and ANOVA.

Statistics - What is the purpose of a Chi-square test?

To compare observed frequencies with expected frequencies.

Reliability vs. Validity - What is the difference?

Reliability refers to consistency of measurements, while validity refers to whether the measurement actually measures what it is supposed to measure.

Random Sampling vs. Random Assignment - What is the difference?

Random sampling selects participants from the population, while random assignment places selected participants into study groups.

Descriptive vs. Inferential Statistics - What is the difference?

Descriptive statistics summarize a sample, while inferential statistics generalize findings from a sample to the population.

Parametric vs. Nonparametric Tests - What is the difference?

Parametric tests require interval/ratio data and a normal distribution, while nonparametric tests are used with nominal/ordinal data or non-normal distributions.