Exam 2 Notes Research Methods

to describe a sample, do we use descriptive or inferential statistics?

descriptive, because we are describing what we know about a sample

to describe a population, do we use descriptive or inferential statistics?

inferential, because we are making predictions about an unknown population based on known sample data

what type of questions would you ask about a study’s construct validity?

ask questions about how the individual variables were manipulated

did the independent variable definitely have the intended effect?

is the dependent variable measuring what it was intended to measure?

what are some things statistical validity address?

how well does the data support the claim?

what is the effect size?

is the sample size sufficient enough to detect an effect?

how strong is the effect?

how much variation is there in the data?

why don’t researchers usually aim to achieve all four validities?

no researcher has sufficient resources or time to ensure that all four validities are completely satisfied, so they must make sacrifices depending on what they are prioritizing. in a lab study focusing on theory testing, researchers prioritize internal validity and often sacrifice external validity. in a field study, researchers prioritize external validity and sacrifice internal validity

how does random assignment prevent selection effects?

random assignment ensures that all participants have an equal chance of being placed in any group. it makes sure that no one group has more of a specific type of participant, and that all differences/similarities between participants are equally distributed

how does using matched groups prevent selection effects?

matched groups prevent selection effects because they ensure that participants with certain similarities are evenly distributed across groups and are not more likely to be in one group than another. since there will be an equal amount of the same type of participant in each group, the shared trait among these participants will not impact the results or serve as a confound

describe how counterbalancing improves the internal validity of a within-subjects design

counterbalancing ensures that an equal number of participants are exposed to each combination of levels of the independent variable, so we can be sure that the order levels are shown is not influencing the results

what are the three things you should evaluate about the construct validity of a dependent variable?

convergent validity, discriminant validity, and criterion validity

how do manipulation checks provide evidence for the construct validity of an experiment’s independent variable

manipulation checks ensure that the manipulation of the independent variable is having the desired effect, so it ensures that a single variable is manipulated in a way that matches the construct

besides generalization to other people, what other aspect of generalization does external validity address?

generalizations to other situations, times, locations, contexts, etc.

what does it mean when an effect size is large in an experiment?

it means that the independent variable has a large impact on the dependent variable and tells you that the result would likely be relevant in the real-world

what are the 6 threats to internal validity?

selection effects, order effects, design confounds, observer bias, demand characteristics, and placebo effect

what questions would you ask to address internal validity?

ask about the relationship between variables instead of focusing on individual variables by themselves. ask if any confounding variables could be influencing the results

what is the difference between systematic and unsystematic variability?

systematic variability occurs when an external variable varies together with the independent variable, creating potential confounds and posing a threat to internal validity

unsystematic variability is random variation in a study that does not vary together with a variable, but is a threat to statistical validity because it creates overlap between groups and obscures the relationship between variables

in what four ways can a study maximize variability between independent variable groups?

strong/consistent manipulation: this makes the levels of independent variables as different as possible, reducing overlap

manipulation check: makes sure that the independent variable manipulation changed what it was intended to change, ensuring that the groups are actually distinct

higher scale of measurement: provides more options to the participants, increasing variability to capture more distinct differences between groups

avoid ceiling/floor effects: if there is a ceiling or floor effect, the results are all clustered, so avoiding these increases variability

what three ways can a study minimize variability within groups?

reduce measurement error by using reliable, precise tools and measuring more instances/samples

reduce individual differences by using a within-subjects design instead of a between-subjects design, or adding more participants so that the more you measure, the less impact one participant has

reduce situation noise by controlling the surroundings of an experiment

why is a null effect considered to be an ambiguous result?

a null effect is considered ambiguous because it provides evidence in both directions. when there is a null effect, the range of population values overlaps zero, so it has both positive and negative values that suggest results in both directions. this means sometimes the results differ from what the original results found, so there is not enough evidence to conclude what is true

when might we conclude that there really is no difference in the population?

it’s not really possible to conclude that there is no true difference in the population, but when replication and high statistical power repeatedly suggest null results, we can conclude there is probably no real difference

what is needed for a study to have high power?

narrow confidence interval, large sample size, less within-group variability

why might researchers who are doing theory testing not use a random sample?

they are prioritizing internal validity, and random sampling can be costly and time-consuming, so they cannot worry about external validity

when an experiment tests hypotheses in a lab, it does not necessarily mean the results would not apply to the real world- why?

many laboratory experiments are high in experimental realism- they create settings in which people experience authentic emotions, motivations, and behaviors. also, further replication studies will prioritize external validity and generalizability

what is a meta-analysis and why is it important?

in a meta-analysis, researchers average all the effect sizes to find an overall effect size, sort the studies into categories, and find the average effect for each. meta-analyses detect new patterns, test new questions, and assess the weight of evidence in a scientific literature. meta-analyses tell you whether there is a relationship between variables across studies and how strong it is. meta-analysis should include unpublished results and evaluate each study’s quality.

why is underreporting null findings bad?

underreporting null findings is bad because sometimes only one dependent variable shows a strong effect and researchers should not only report strong effects while ignoring weak ones. this leads readers to believe evidence for a theory is stronger than it really is

why is HARKing bad?

predictions that happen before data collection are more convincing than those made after, so HARKing misleads readers about the strength of evidence

why is p-hacking bad?

it leads to non-replicable results and misleads readers when they are not told about the different ways data was analyzed

what are the three core principles of the belmont report?

respect for persons, beneficence, and justice

how many people are on an irb?

at least 5

what requirements are there for the types of people on an irb?

some must come from specific backgrounds

some must be researchers and some must be community members with no ties

studies involving prisoners must have a prisoner advocate

how is an iacuc similar to an irb?

each institution has their own committee composed of specialists and community members that approve research protocols to ensure welfare and scientific justification and monitor and enforce compliance

what are the three Rs of animal research?

replacement, refinement, and reduction

also rehabilitation

describe the moral framework of balancing social benefit and animal welfare

valuable outcomes of the research must apply to both humans and animals

animal research must be the only way to study the question

the magnitude of the benefit must justify the potential harm

what were the independent variables and dependent variable in roediger and karpicke?

independent variables: study method (pure study vs. retrieval practice) and delay of memory test (5 minutes vs. 1 week)

dependent variable: percent of ideas recalled at memory test

what are two common reasons to use factorial designs?

to test whether an independent variable impacts different kinds of people or different situations the same way

to test whether an effect generalizes

how can you detect an interaction from a line graph?

if the lines are not parallel, crossing, or moving toward each other or away from each other

how can you detect an interaction from a bar graph?

if you compare the heights of the bars across groups, the difference between the pairs of bars should not be the same

how many independent variables are there and how many levels does each variable have in a 2×2 factorial design?

2 independent variables with 2 levels each

what indicates an interaction effect in a news article/popular press?

the phrases “it depends,” or “only when” or the mention of participant variables

what is the difference between descriptive and inferential statistics?

descriptive statistics summarize, analyze, and organize data on a known sample. Inferential statistics take that data and use it to make predictions and generalizations about an unknown population

what are the 3 main things we look at to evaluate statistical validity?

point estimate, precision, and replication

what do we look at to find point estimate?

direction and strength of effect in the sample

central tendency, variability, effect direction, and effect size

what do we look at to evaluate a study’s precision?

confidence interval

what does a histogram show?

a histogram shows how many of the cases in a batch of data scored each possible value, or range of values, on a variable

what measure of central tendency is best for quantitative data in general?

mean

what measure of central tendency is best for very skewed data?

median

what measure of central tendency is best for data with strong outliers?

median

what measure of central tendency is best for categorical data?

mode

what measure of central tendency is best for ordinal data?

mode

what measure of central tendency is best for bimodal data?

mode

why is it important to look at variability and not just central tendency?

variability provides the bigger picture. central tendency only tells you what the center of the data is, but variance tells you how spread out the data is from that central tendency. it tells you about extreme values like outliers that may skew the data and helps compare variability across data sets

describe the different ways we can measure variability

range, interquartile range, variance, and standard deviation

how do you evaluate whether the effect reported in an experiment is in the expected direction? explain

correlation coefficient r captures the direction and strength of an association on a scale from -1.0 to 1.0. if the r is negative, the relationship is negative, meaning high values go with low and vice versa. if the r is positive, the relationship is positive meaning, high values go with high and low values go with low.

what happens to cohen’s d as the distance between two means increases?

cohen’s d increases, indicating a larger effect size

what happens to cohen’s d as the distance between two means decreases?

cohen’s d decreases, indicating a smaller effect size

what happens to cohen’s d if the means are identical?

cohen’s d is zero and there is full overlap between the two groups

what happens to cohen’s d as standard deviation increases?

cohen’s d decreases because the data becomes more spread out

what happens to cohen’s d as standard deviation decreases?

cohen’s d increases because the data becomes less spread out

what is the difference between describing effect size with cohen’s d vs. the raw difference between means?

the raw difference between means calculates the difference between two means in their original units. cohen’s d takes this raw difference and divides it by the standard deviation

cohen’s d is useful for comparing effect size across studies, while the raw difference between means is useful for interpreting the results of a single study in its original units

what does confidence interval NOT mean?

95% confidence interval DOES NOT MEAN there is a 95% chance that the population value is within our specific confidence interval

what does confidence interval actually mean?

a 95% confidence interval means that, for 95% of infinite possible samples of the same size, the interval around the sample value will capture the true population value

what can and can’t confidence interval tell us?

it can give us a range of likely population values and provide a sense of precision of our estimate

however, it cannot guarantee that the interval contains the true population value

what components make up standard error?

standard error includes variability (standard deviation) and sample size

what is the relationship among standard error, margin of error, and confidence intervals?

as the standard error increases, margin of error increases. the margin of error determines the width of the confidence interval (small margin of error leads to more precise confidence interval)

do you have more precision with a narrower or wider confidence interval?

you have more precision with a narrower confidence interval

how does a larger sample size impact confidence interval?

larger sample size = more precise confidence interval

how does a smaller sample size impact confidence interval?

smaller sample size = less precise confidence interval

how does a larger standard deviation impact confidence interval?

larger standard deviation = more variability = less precise confidence interval

how does a smaller standard deviation impact confidence interval?

smaller standard deviation = less variability = more precise confidence interval

as the confidence % increases, does confidence interval become more or less precise?

more precise

what information does a p value NOT provide?

p value does NOT give a range of likely population values, effect size, precision, or a sense of what may happen in replications

what information does a p-value provide?

a p-value determines the statistical significance of results, operating under the null hypothesis

it is the likelihood that you would get a result as extreme as the p value across multiple studies if there is no effect in the population

what is the typical cut-off for p value?

p < 0.05; if the p-value is greater than that, it is deemed statistically insignificant

what are some things confidence intervals and p-values have in common?

both are inferential statistics about an unknown population

both assess statistical significance

the .05 p value correlates with a 95% confidence interval

in what ways are confidence intervals and p-values different?

confidence interval gives range of likely population values (of effect size), while p-value does not

confidence interval shows precision, p-value does not

confidence interval gives sense of what might happen in replications, p-value does not

p-value is a single value while confidence interval is a range

confidence intervals suggest direction and strength of an effect, while p-values don’t

why doesn’t statistical significance necessarily mean the finding is important?

even the smallest effect size can be statistically significant with a large enough sample, and statistical significance provides almost no information about the relationship between variables or effect size/magnitude, so effect size and other info is important

what are some reasons a study may have a null effect?

there may truly be no effect

or, there could be a true effect, but this study did not detect it because of an obscuring factor

there could be not enough variability between-groups

or there could be too much variability within-groups

what causes not enough between-groups variability?

weak manipulations

insensitive measures

ceiling/floor effects

what causes too much within-groups variability?

noise

situation noise

measurement error

describe the procedures that are in place to protect human participants in research.

belmont report ethics principles

irb

informed consent

privacy and confidentiality

risk minimization and safety monitoring

training for researchers

describe an interaction effect in everyday terms

two or more variables work together to produce an outcome that cannot be explained by just one of them alone. the effect of one factor depends on the value of another.

describe an interaction effect in arithmetic terms

a difference in differences

what do effect size and confidence intervals mean for statistical validity?

effect size measures the magnitude of a research result’s practical significance, while its confidence interval indicates the precision of this estimate. together, they assess the precision, magnitude, and strength of a study’s results