final research methods

replication

“blank “of findings is absolutely critical in science

no single study is perfect/definitive, some reasons are?

* always some chance of type one and type two errors
* always come possibility if confounding factors

three types of replication

* direct
* conceptual
* replication-plus-extension

direct replication

* try to replicate a study as closely as possible
* can enhance statistical validity (e.g, estimating effect size) and to some extent external validity
* not internal or construct validity

conceptual replication

* same conceptual variables; different operational definitions

replication + extension

* replication and new levels or variables to test additional questions
* sometimes a new independent variable could be added in the replication (rather that a new level of the original independent level)

meta-analysis

mathematical/statistical summary of scientific literature on a specific topic (I.e. the same conceptual variable)

strengths: of a meta-analysis

* validity (effect size and replication)
* external validity (identify moderators)
* generate new questions to research

potential limitations: of a meta analysis

publication bias journals tend to only publish significant findings.

called “file-drawer problems”

when null findings do not get reported and so are not included in the meta-analysis

muremberg code

* ethical yardstick aginst which defendants were judged
* informed consent
* risk and benifit
* subject can terminate his or her involvment
* experiment should be based upon proir animal studies
* only scientifically qualified individuals should conduct human experimentation
* physical and mental suffering and injury should be avoided
* there should be no expectation that death or disabling injury will occur from the experiment

APA code

* based on the Nuremberg code
* two additional elements were included in the guidelines for research
* must reduce harm due to any deception.
* why deceive
* demand characteristic
* prevents confounds
* must ensure confidentiality

Belmont Report 1979

Beneficence

*  Beneficence and nonmaleficence
* Fidelity and responsibility
*  Integrity

Justice

Respect for persons

**standards**

* informed consent
* deception - allowed under certain circumstances.
* debriefing - required whenever deception is used.
* institutional review board (IRB) for human research
* reviews ethical standards of research proposal
* IACUC (for animal research)

misconduct

\
* data fabrication
* data falsification
* “harking”
* p-hacking

“harking”

* modifying hypotheses of a study after analysis.
* adjusting hypothesis to fit the data
* how to prevent preregistered.

p-hacking

analyzing data in various ways to find a significant effect

measuring more than two variables

* controls for = hold constant
* e.g. let’s hold age constant.

multiple regression

using beta to test for third variables

statistical test to assess thrid variable

this is the one you are most interested in predicting

criterion variable and predictor variables

these are the ones you will test to see if they help predict criterion scores

beta indicates

the strength and direction (+/-) of an association between a predictor variable and criterion while holding all other predictor variables constant

if beta =0

no relationship

beta similar to r in that:

* sign indicates direction
* bigger numbers → bigger effect
* near zearo → no effect

betta differs from r in that :

* beta expressed in terms of standard deviation
* the estimated SD change in the criterion variable associated with a one SD change in a predictor variable while holding other variables constant.

beta can be

greater than one

b is

* unstandardized (the values are expressed in the original units of measurements, not in terms of standard deviation)

cannot compare

b-value in a single table

can compare

beta values in a single table

multiple regression can

identify moderators

multiple regression in popular press articles

* “Controlled for”
* “Taking into account”
* “Correcting for” or “adjusting for.”

external validity when sampling from populations

* generalizing from samples to populations
* to the same population or to others
* generalizing across settings and materials

population or interest

entire set

census

measure all people in a population.

sample

* measure a subset of people from a population.

unrepresentative (biased) sample

* some types of people are over -or under-represented.
* cannot make inferences about a population.

representative (“unbiased”) sample

* can make inferences about the population.
* required for frequency claims.

ways to get representative samples.

* all involve some form of probability sampling (aka random selection)
* all members of a population have an equal chance of being selected.
* cannot be used if you cannot identify all people in the population.
* e.g., depressed adults; obese children

* \

1. simple random sample

* good if the population is small (or names are easy to get)
* e.g., BC students or VA registered voters
* doesn’t always work.
* sample size matters

cluster sampling

can be used if the population is already divided into arbitrary groups.

multistage sampling

same as cluster sampling but you select one or two from each population (ex: school)

stratified random sampling

identifying specific demographic categories (“strata”), then randomly select individuals within the category (remains proportionate)

biased sample

can be a big problem in frequency claims.

convenience sampling

* using people that are easily available.
* sampling only volunteers
* self-selection
* e.g., online (or mail) survey
* biased

quota sampling

* create strata.
* sample non-randomly within the strata
* e.g., use the first people you find to fill out the survey.
* biased

for external validity,

* bigger is not necessarily better.
* smaller, unbiased samples generalize better than a large, biased sample.
* bigger is better in unbiased samples.

for the statistical validity of frequency claims

yes, bigger is better (up to a point)

correlational studies

\
* studies that only have measured variables.
* notes:
* ant correlational study may have many measured variables.
* if the focus is on any two variables → bivariate association

construct validity when interrogating association claims

how well was each variable measured?

statistical validity when interrogating association claims

* effective size = strength of the relationship
* usually, very small effect sizes are less important.
* exception
* cheap additive associated with increased crop yield.
* on a large scale, this could have a big effect.
* sometimes very small correlations can combine to be a big difference.

statistical significance when interrogating association claims

* what is the likelihood a correlation was found when none existed?
* always some probability this can happen (sampling error)
* inferential statistics → how likely was the correlation found due to chance (aka “sampling error)

statistical validity

3\. outliers


4. range restriction
5. curvilinear associations

* r=0

spurious correlation

correlations really due to third variable

external validity

* will the findings generalize to other people and places?
* are there moderating variables
* when the association between two variables depends on the level of a different variable (subgroup)

bivariate correlation

linear regression (r)

self-report measures (survey, interviews

\
* how to construct?
* question formats
* open-ended or forced-choice

ratings of adjected

ex: semantic differential scale

scales of agreement

Likert scale

wording of questions to avoid when constructing surveys

* avoid ambiguous words (including jargon)
* avoid negative phrasing
* avoid leading question
* avoid double-barreled questions.

ambiguous words

* using “usually” or “normally”

leading question

* worded in a way to bias people’s responses

response sets

responding in a consistent way regardless of the question

acquiescence

yeah/no saying

fence sit

only choosing middle response

faking it

answering to make themselves look good or bad

poor memory

not being able to remember correctly

observational measures

can be more accurate than self-report.

observer bias

* when an observer’s expectations influence the interpretation

case of clever Hans (1911)

had observer effects

observer effects

observer unintentionally behaves in ways that change participant’s behavior.

how can we deal with these threats to validity?

use a masked design

have clear rating criteria for the behaviors (codebooks)

train observers

masked design

observers do not know the purpose or conditions of the study.

how train observers

* practice scoring
* create a script for what to say/do
* have multiple, blind observers.

problems Assoc. w/participant

reactivity

variable

* is something that varies between people and conditions.
* at least two levels of values in a study

measured

only documented.

manipulated

something the researcher varies at the start of a study.

variables that cannot be manipulated at all.

age/sex/ethnicity/height

variables that cannot be manipulated for ethical reasons

abuse vs. no abuse, poor vs. healthy diet, schooling vs. no schooling, social isolation

conceptional

* (aka “construct')
* abstract/generalized usage (dictionary-like)
* theory → always conceptional variables
* hypothesis → sometimes conceptional variable

operational

* how the construct is defined in a specific study
* operational definition→ the specific methods used to manipulate or measure your constructs.
* hypothesis → most of the time operational variable

frequency claims

* how common is something?
* one measured variable

association claims

* involves documenting 2 (or more) measured variables.
* and then searching for a correlation
* correlation → when the level of one variable is associated (“covaries”) with the leaves of another variable.
* allow for predictions.

positive associations:

when the variable changes/Convery in the same direction

negative associations

variables change in opposite directions (inverse association)

zero association

no association between variables

causal claims

\
* a change in one variable is responsible for changing the other variable.

validity

the appropriateness of a conclusion

construct validity when evaluating frequency claims

* how well a conceptual variable is operationalized by external validity.
* are these valid measures?

statistical validity when evaluating frequency claims

\
* are the numbers accurate?
* most of the time, we study samples from a large population.
* samples provide estimates of the true value in the population.
* in frequency claims→ point estimate
* difference b/w sample and population called “sampling error.”
* always a margin of error (confidence interval)
* has the effect been replicated?
* are the estimates similar?

external validity when evaluating frequency claims

extent to which findings generalize to people or situations outside the study.

does your sample represent the population you intend to make a claim about?

will results generalize if you changed operational definitions of the variables

type one error

false positive

type two error

claiming there is no association when there really is one.

association studies

* shows covariance.
* sometimes shows temporal precedence.
* dose not show internal validity.

experiment

* a study that involves at least one manipulated variable (independent variable) and one measured variable (dependent variable)
* manipulate = assign participants to levels of the I.V. (group/conditions)

how to assign?

* random assignment to conditions


* counterbalances (“controls for”) all potential preexisting group differences

how to rule out other confounds

* counterbalance all other conditions.
* groups must be treated identically

internal validity

confounds

external

* generalization

construct

* how was the dependent variable measured?
* how well was the independent manipulated?