Nonexperimental Methods and Statistics Midterm

Nonexperimental research: basic definition

Researchers measure variables as they naturally occur rather than manipulating them.

Predictor variables v. outcome variables

Predictor variables predict the outcome of the research
Outcome variables are what you’re trying to explain

Example of predictor v. outcome variables:
Imagine a researcher studies whether students who sleep more tend to earn higher grades.

Predictor variable: student amount of sleep
Outcome variable: grades

Control variables

Variables researchers account for statistically to reduce alternative explanations.

Anecdote

One person’s experience; weak evidence by itself.

Confirmation Bias

Favoring evidence that supports what you already believe while overlooking contradictory evidence.

AND

Database operator that finds results containing both concepts.

Ex: stress AND college students

OR

Database operator that finds results containing either term; useful for synonyms.

Ex: teenagers OR adolescents

NOT

Database operator that excludes results.

Ex: stress NOT workplace

Results are…

What the data actually shows.

Ex: “The group with eight hours of sleep had higher average test scores.”

Interpretations are…

What the results mean and what conclusions might be drawn.

Ex: “Adequate sleep may support academic performance.”

The first principle of the Belmont Report

Respect for Persons

Respect autonomy and protect people whose autonomy is limited.

Ex: Participants receive enough information to make an informed, voluntary decision about participation in the study.

The second principle of the Belmont Report

Beneficence

Minimize harm and maximize possible benefits.

Ex: A researcher reduces the risk of distress when asking participants about traumatic experiences.

The third principle of the Belmont Report

Justice

Distribute research burdens and benefits fairly.

Ex: Researchers avoid selecting a disadvantaged group simply because its members are easy to recruit, while others receive the benefits.

Three principles of the Belmont Report

Respect: people’s choices

Beneficence: people’s well-being

Justice: fairness

Coercion is…

Pressuring or threats to make someone feel like they cannot refuse.

Professors recruiting their own students for study participation should have a _______ ________ to participation.

Reasonable alternative

Deception is…

When researchers intentionally withhold or misrepresent information. Must be justified under appropriate ethical conditions.

Fabrication is…

Making up data or results that were never obtained.

Falsification is…

Manipulating research materials, procedures, or data, or omitting data, so the research record is inaccurate.

Theoretical construct

The abstract concept the researcher wants to study.

Conceptual Definition

A theoretical explanation of something; providing a basic definition.

Operational definition

How the thing the researcher wants to study is going to be measured.

A researcher wants to investigate whether stress is associated with sleep quality.

Identify the theoretical construct, conceptual definition, and operational definition.

Theoretical Construct: Stress

Conceptual Definition: The definition of stress (a person’s psychological response to demands perceived as overwhelming)

Operational Definition: Participants stress is measured through a questionnaire.

Construct:

Conceptual Definition:

Operational Definition:

What are we studying?

What does it mean?

How do we measure it?

Validity…

Asks whether a measure, finding, or conclusion is actually appropriate for its intended purpose. Six types.

Construct validity…

Does the measure represent the construct?

Example: A researcher wants to measure depression but uses only one question about how tired someone feels.
Problem: Does tiredness alone adequately represent depression?

Criterion validity…

Does it relate to a relevant criterion?

Example: A new aptitude test is compared with later job performance.
Problem: Does the test relate appropriately to the independent criterion?

Statistical validity…

Are the statistical conclusions supported?

Example: A researcher uses an inappropriate statistical test or draws conclusions that the results don’t justify.

Population validity…

Can findings generalize to the population?

Example: A researcher studies only students at one university, then claims the results apply to all adults.

Ecological validity…

Do findings generalize to realistic settings?

Example: People will behave one way in an artificial laboratory task, but the researcher assumes they will behave the same way in everyday life.

Can a result be statistically significant even when the measure is poor?

Yes

A bad loneliness questionnaire might consistently produce scores associated with another variable. That significant relationship does not prove the questionnaire accurately measures loneliness.

Reliability…

Consistency of measurement. Three types of reliability.

Interrater reliability

Do two different observers of a phenomenon agree?

Example: Two psychologists independently rate the same interview and give similar ratings.

Test-retest reliability

Are scores consistent across time?

Example: The same person takes the same personality questionnaire twice and receives similar scores.

Internal reliability/consistency

Do items intended to measure the same construct produce consistent responses?

Example: Several questions designed to measure anxiety work together consistently.

Double-barreled questions

One question asks about two different things at once.

Example: “Are you satisfied with your professor’s teaching and grading?”

Leading or biased questions

The wording of question pushes respondents toward a particular answer.

Example: “Don’t you agree that our excellent professor deserves a raise?”

Open-ended versus closed-ended questions

Open-ended: Allow respondents to answer in their own words.
Example: “What causes you stress?”

Closed-ended: Provides set response options.
Example: How often do you feel stressed? Never, rarely, sometimes, often, or always?

Neutral wording

Questions should avoid suggesting that one answer is more desirable or correct than another.

Example: Instead of “How much do you enjoy this helpful program?” ask “How would you rate this program?”

Response fatigue

Respondents become tired or lose concentration when a survey is too long or repetitive. They may start rushing or giving careless answers.

Mutually exclusive response categories

Response options should not overlap.

Example:
Bad: 18-25, 25-35, 35-45
Good: 18-24, 25-34, 35-44

Measuring the intended construct

Questions must actually measure the concept the researcher wants to study.

How many levels of measurement are there?

Four: Nominal, Ordinal, Interval, Ratio

Nominal Measurement

Categories with no meaningful order. No one category is greater than the other.

Example: Psychology major v. biology major, eye color, type of residence.

Ordinal Measurement

Categories have a meaningful order, but distances between them aren’t equal.

Example: Class rank, low/medium/high satisfaction. You know which is higher/lower but not by how much.

Interval Measurement

Numerical values with equal intervals but no true zero.

Example: The difference between 10C and 20C equals the difference between 20C and 30C, but 20C is not twice as hot as 10C.

Ratio Measurement

Numerical values with equal intervals and a meaningful zero.

Example: Age, number of absences, hours worked.

Simple random sampling

Every individual in a defined population has a chance of being randomly selected.

Ex: A researcher randomly selects 100 student IDs from a complete list of enrolled students.

Stratified random sampling

The researcher divides the population into meaningful subgroups, called strata, and randomly selects people from each subgroup.

Ex: Divide students by class year, then randomly select students from every year.

Proportionate stratified random sampling

The researcher randomly selects from each subgroup in proportion to that subgroup’s representation in the population.

Ex: If 40% of the population are freshman, approximately 40% of the sample are freshman.

Cluster sampling

The researcher randomly selects naturally occurring groups and studies individuals within the selected groups.

Ex: Randomly select five classrooms and survey students in those classrooms.

Systematic sampling

Researcher selects participants at a predetermined level

Ex: Select every tenth person on a list

Convenience sampling

Researcher selects participants who are readily available

Ex: Survey students who happen to be sitting in the library

Quota sampling

Researcher recruits until predetermined numbers from certain categories are reached, without random selection

Ex: Recruit 50 men and 50 women by approaching volunteers until each target is filled.

Snowball sampling

Existing participants help create new ones.

Ex: Researchers studying hard-to-reach population ask current participants to refer others.

Research question: Is there a relationship between hours of sleep and academic performance?

What is the Null Hypothesis?

There is no relationship between hours of sleep and academic performance.

Research question: Is there a relationship between hours of sleep and academic performance?

What is the Atlernitive Hypothesis?

There is a relationship between hours of sleep and academic perofrmance.

Nondirectional

Predicts that a relationship exists but doesn’t specify its direction.

Ex: Sleep and grades are related

Directional

Predicts the direction of the relationship.

Ex: More sleep is associated with higher grades.

p-values

Describes how unusual the observed result, or a result at least as extreme, would be if the null hypothesis and statistical test’s assumptions were true.

Does not tell you probability that the null hypothesis is true.

Ex: A study reports p = .03. If the null hypothesis and test assumptions hold, a result at least as extreme as the observed one would occur with probability 3%.

α (alpha)

The significance chosen for the test. Compare with p to make statistical decision.

Ex:

p <= α, reject null hypothesis

p > α, Fail to reject null hypothesis

Type I Error

False Positive - Reject a null hypothesis that is actually true

You make an association that isn’t there

Type II Error

You fail to reject a null hypothesis that is actually false.

You don’t make an association even though there actually is one.

Low alpha means _____ risk of Type I error

low

Low Power means _____ risk of type II error

Greater

Power is…

The probability of detecting an effect when that effect actually exists

Power increases with

Large sample size

Power increases with

larger population effects

Power increases with

Less irrelevant measurement variability

Power increases with

A less restrictive alpha level

The median is best for

Distributions with strong skews or extreme outliers.

n-1 is used in the denominator

To correct the tendency to underestimate population variance.