Research Methods Exam 2

Convergent validity

(Empirical) the amount to which the researchers assessment measures up to other standardized measurements (should be close, but not exact); moderate-strong r value

Discriminant validity

(Empirical) Degree to which the assessment is distinct from unrelated concepts; should be very small r value. Does it not associate with what you think it shouldn’t?

Criterion validity

(empirical) Extent to which measure is related to other relevant behavioral outcomes in the real world

Face validity

(subjective) Extent to which measure appears to measure what it claims. Can you easily tell the concept the researcher is measuring by the assessment?

Content validity

(subjective) Is the researchers mode of assessment covering every part of the concept?

Name the 5 construct validities

1. Convergent validity

2. Discriminant validity

3. Criterion validity

4. Content validity

5. Face Validity

Within subject vs Between subject

Within: applying all levels of IV within one participant

Between: applying one level of IV and comparing between participants

Random assignment vs Random sampling

Random assignment: randomly assigning people to a group in the experiment (important for internal validity; confounding variables, causal claims, temporal precedence)

Random sampling: every person has an equal chance of being selected for the experiment (important to external validity, but very difficult and sometimes impossible)

Fence sitting

(response sets) picking the neutral answer every time. To combat, remove the middle option

Acquiescence

(response sets) picking “yes”/”agree” every time. To combat, use reverse coded items

Straight-lining

(response sets) picking the same option for every question

Socially desirable responding

(response set) respondents hiding an unpopular stance/faking good

What is a response set?

The way people respond to surveys without actually engaging; only answering one response for every question

What is the jingle-jangle fallacy?

Jingle: falsely assuming two tasks measure the same construct because they have the same name (construct definition changes)

Jangle: falsely assuming two tasks measure different constructs because they have different names (like grit and conscientiousness)

What are the 4 scales of measurement?

-Nominal: categorical sorting where numbers have no meaning

-Ordinal: relative differences in a ranked order (1st, 2nd, 3rd)

-Interval: numerals represent equal intervals with no true zero (Celsius and Fahrenheit)

-Ratio: equal sized differences in rank/ amount with a true zero (kelvin)

These help dictate what you’re able to do with the data

What are the 3 types of reliability?

-Test-retest: strong correlation (>.50) between scores administered on different occasions (weeks, months, years); consistency over time (not useful if we expect things to vary)

-Interrater: the extent to which 2+ observers see the same thing; consistency across raters

-Internal consistency: extent to which items on a scale measure the same construct; if study participants give the same answer across multiple items no matter how the question is phrased

What things should you avoid having on a survey?

-Double-barreled questions: multiple questions in one

-Negative wording: confusing double negative wording leading to inaccurate responses

-Leading question: a statement or question that suggest there is a right answer

-Also consider question/item order: the order items are in can prime/condition people into answering a certain way

What are the 4 types of survey questions?

-Likert or Likert type: rates agreement through anchors 1-5/strongly disagree-strongly agree (a version of forced choice)

-Forced choice: picking best of 2+ options (easy to code, quick to complete, but reduces nuance and may not match true thoughts)

-Semantic differential: place a target on 2 descriptive adjective dimensions

-Open ended: allows for more nuanced/rich information (that’s difficult to code, need interrater reliability)

Observer bias vs observer effects

-Observer bias is when an observers expectation can influence the interpretation of the results

-Observer effects is when an observer can unintentionally effect how participants act in a study (reactivity)

What makes an observational measure good, potentially problematic, and what are its properties?

Observation can uncover unknown or socially abnormal responses people wouldn’t share in a survey

-A good observational measure’s observed behavior reflects conceptual behavior, is not up to interpretation, is not affected by reactivity, and is not affected by observer biases

-Problems: *Observer bias is when an observers expectation can influence the interpretation of the results. This can be balanced with masked/blind design.

*Observer effects is when an observer can unintentionally effect how participants act in a study (reactivity). To fix this, hide or habituate observers, or measure trace products of behavior instead of actual behavior.

Population vs sample

-Population is the large group you want to draw conclusions about; must be defined before drawing sample

-Sample is a sub-set of that population that you can actually measure for your study

Generalizability does not always matter

Probability sampling vs non-probability sampling

-Probability: everyone has an equal chance at being selected for the sample; gold standard

-Non-probability: chances are unknown or unequal for being selected for the sample; most popular for psychologists

Convenience sample

Selecting participants based on easy access or availability (how most psych research is done) [NP]

Quota sample

Determine number of subjects you want from a subset of population (keep sampling until you get to 30 women, 40 men, etc); like convenience for different subsets [NP]

Purposive sample

Recruiting only a certain type of participant in a non-random way for a special population [NP]

Snowball sample

Asking participants to recruit others in their community (good for hard to reach populations, but can cause homogenous samples) [NP]

Simple random sampling

Every person in the population has an equal chance at being included in the sample, chosen by a randomly selected algorithm [P]

Systematic sampling

Sampling every Kth person in a population (following a pattern could accidentally introduce biases) [P]

Cluster sampling

Randomly sampling naturally occurring groups in frame population and within naturally occurring clusters [P]

Stratified random sampling

Population divided into meaningful strata, and random samples are taken from strata mimicking % of population [P]

Oversampling

Going beyond what population reflects of smaller subgroups to ensure we can make meaningful conclusions about a group by having a large enough sample [P]

WEIRD acronym

Western

Educated

Industrialized

Rich

Democratic

^what most surveys consist of; make us ask “universal for WHO?”

Correlation is…

A mathematical measure of a relationship between two variables determined by r to assess statistical validity

What are the 4 challenges for correlations?

-Directionality problem/temporal precedence (dependent variable caused by the independent variable)

-Third variable problem: is there a confounding variable we aren’t considering?

-Selection bias: is everyone who should be in the sample actually in the sample?

-Spurious correlations: purely coincidental relationships which can be caused by chance, but doesn’t mean anything

Construct vs Statistical validity

-Construct: for correlational research, just evaluating the measures.

-Statistical: looks at strength (#), direction (±), precision

What are the 3 requirements to make a causal claim?

1. Temporal precedence

2. An association exists

3. Control of confounds

What are non-linear effects and how can they effect correlations?

Pearson’s r can only assess linear relationships, but other (curvilinear) results are still statistically significant. r does not represent association.

What is restriction of range and how can it effect correlations?

Since correlation requires variability, when range is restricted, only similar scores appear. r does not represent association.

What is a moderator?

A 3rd variable that changes the relationship between two other variables. Important for external validity

Treatment group vs control group vs constant

-Treatment: group where the condition is applied; designed to change level of IV

-Control: baseline/comparison group where IV is not changed

-Constant: something that stays the same for all groups (anything not the IV or DV)

What does it mean to have experimental control?

Participants experiencing the exact same thing except for manipulation to rule out confounds, which leads to high internal validity

Correlational study vs experiment

-Correlational: all variables are measures, which leads to poor internal validity

-Experiments: at least one variable is manipulated, and people are usually assigned to random levels. IV causes the difference, DV is caused by IV.

Issues for Within-Subjects experiments

-Carryover effect: manipulation persists and taints other results

-Practice effect: performance improves with repeated practice/exposure

-Fatigue effect: performance gets worse with exhaustion

These can be counteracted with counterbalancing; presenting sequences of effects in different orders

Selection bias vs selection effects

-Selection bias: when the way participants are selected skew the results

-Selection effects: characteristics are different and result in non-random assignment

What is covariance?

A measure of how two variables change together

A pattern of associations…

can lead to causations by checking qualifications through multiple studies (like smoking and cancer)

Conceptual vs operational definition

-Conceptual definition is the definition of the variable at a theoretical level

-Operational definition is the researchers specific way to measure/manipulate the conceptual variable

3 types of measures

-Self-report: operationalizes a variable via questionnaire/interview responses

-Observation: or behavioral operationalizes according to observable/physical behaviors

-Physiological: operationalize a variable by recording biological data requiring specific equipment

Reliability is…

A pre-requisite for validity