Research Methods Exam 2

When talking about generalizability, we’re focused on…

external validity and random selection of subjects from the population

When talking about how similar groups are when comparing within the study, we’re focused on…

internal validity and random assignment of subjects

Step 1 of how to get our sample

identify target or theoretical population (all the people of interest relative to our research question)

Population

rarely referring to everyone in the world; rather the segment of people we are interested in generalizing to

Step 2 of how to get our sample

Accessible sample (sampling frame): we think about who within that target population do we have access to

Step 3 of how to get our sample

reach out to select number of individuals in target population: selected sample

Step 4 of how to get our sample

actual sample: individuals we reach out to and who have agreed to participate

External Validity has two subtypes which are:

Population validity and ecological validity

Population Validity

Asks “are the people in our study similar to the people we want to generalize to in ways that matter”. Attention to who is being studied

Ecological Validity

Asks “do the conditions of the study resemble the real-world environments in which the findings are meant to apply”Do we think that to account for confounding variables, we’ve controlled our study so much that it no longer mimics what happens in the real world? Attention to where and how the study takes place

Strong external validity means…

we have considered who our results apply to and under what conditions 

-Transparent and thoughtful about the limits of our conclusions

Inclusion and Exclusion criteria

play a foundational role in defining what a study’s results actually mean and, just as importantly, what they do not mean

-These criteria determine who is allowed to participate and by extension, the specific population to which the findings can reasonably be generalized

Probabilistic Sampling

non-biased sampling, meaning every person in a population under a study has an equal chance of being selected. Includes simple random sampling, systematic random sampling, stratified random sampling, and cluster sampling

Simple Random Sampling

uses a computer program or random number generator to randomly select individuals from the target population

Systematic Random Sampling

draws a random sample from the target population by selecting units at regular intervals starting from a random point

First, determine the proportion of the population we want to sample (incredibly important because it tells us the range from which we will randomly select the first participant and the interval we will use to select the rest of the participants)

Stratified Random Sampling (think groups or categories)

involves dividing the population into meaningful groups and then randomly sampling from within those groups

First, divide the population based on a characteristic of interest, such as race, gender, age, academic year, etc

Stratified random for EQUAL proportions

results in a different total number of people per group

This is because there will be more of one part of a category (ex. More males than females) in a population and then equal proportions (33% of each category) will give you different numbers in the sample

Stratified random for UNEQUAL proportions

has same number of people per group

Purposefully want each group in our sample to contain the same total number of people (known as OVERSAMPLING)

Cluster sampling

Identify naturally occurring divisions (clusters) and then randomly select entire subsections of clusters to sample

primarily used for efficiency, particularly when things like time, money, or logistical resources are limited

Non-probabilistic sampling

used when probabilistic sampling is not feasible or in some cases, especially in qualitative research, when probabilistic sampling is not appropriate or desirable

-often preferred and sometimes the only option (qualitative research)

Includes convenience sampling, quota sampling, purposeful sampling, and snowball sampling

Convenience Sampling

researchers recruit individuals who are available and willing to participate. No attempt to ensure that every member of the population has an equal chance of being included. Participation simply depends on access, availability, and willingness

Quota Sampling

researchers separate people into specific strata or groups they want to compare (men and women, age categories, etc) and then enroll anyone willing to participate in the study

Purposeful Sampling

researchers first deliberately seek out individuals who have expert knowledge, experiences, or characteristics that are directly relevant to research questions and then enroll participants willing to participate

Snowball Sampling

Researchers begin by identifying and recruiting one participant who meets the study criteria. After building trust and rapport, the researcher asks the participant to recommend or introduce others who share the same characteristics/experiences. Each new participant may then refer additional individuals

What are the 4 issues related to external validity that shape what conclusions can reasonably be drawn and how broadly findings can be applied?

Restricted/Homogenous Samples, Context Specificity, Self-Selection, and Underrepresented Subgroups

Restricted or Homogenous Samples

when samples lack diversity in age, socioeconomic status, education, culture, or lived experience, they fail to capture the natural variability present in the broader population

Context Specificity

context influences how people interpret questions, respond to interventions, and interact with researchers. Conclusions may not hold outside original setting

Self-Selection

When participation depends on individuals opting into a study. Volunteers often differ systematically from nonparticipants in ways directly related to research topics

Underrepresented Subgroups

Although exclusion criteria can reduce heterogeneity and strengthen internal validity, they also narrow the population to which findings apply

True Experimental Designs

Randomized controlled trials

Quasi-Experimental Designs

randomization isn’t feasible

8 Different Strategies to Help Insulate Study from Confounding Variables

Pre-Test/Baseline Test, Control group, Random assignment of subjects, matching/matched pairs, blinding/double blinding, between-subjects designs/within-subjects designs, Quality assurance, Timing

Pre-test/Baseline Test

Measurement taken before starting an experiment/intervention to assess where things stand at the beginning to see if anything changes later on/compare results later. Confirms whether changes are due to intervention instead of the possibility of confounding variables affecting results. Allows researchers to identify differences in participants at the start of the study

Control Group

The participants who don’t receive the treatment or intervention being tested, or they receive the current gold standard level of treatment, or they receive a placebo. Serve as the comparison group to compare outcomes to treatment group so researchers can determine whether the treatment/intervention actually works

Random Assignment of Subjects

Helps ensure groups are equal before study begins. ONLY APPLIES TO TRUE EXPERIMENTAL. Doesn’t guarantee perfect equality among groups, especially when groups are small. Great way to minimize bias

1. Can’t always do random assignment because of ethical, logistical, or feasibility concerns

Matching/Matched Pairs

Makes random assignment of subjects more perfect by matching subjects on relevant characteristics, then randomly assigning them to different levels of independent variable. Most important to match subjects on something that relates to dependent variable

Blinding/Double Blinding

Blinding: Participants do not know whether they are in treatment group or control group to prevent expectations from influencing behavior and ultimately results

Double Blinding: Participants AND researchers don’t know who is in which group 

Between-Subjects Designs

comparing one group of subjects against a completely different group of subjects 

1. One group is assigned to the experimental phase which gets intervention, and the other is assigned to the control group. Allows researchers to compare outcomes to determine effect of independent variable.

Within-Subjects Designs

using one group of subjects and comparing them against themselves. Expose the same participants to multiple conditions or treatments during the course of the experiment. Serve as both experimental and control, just not at the same time 

Longitudinal Studies (capitalize on within-subjects designs)

where researchers can study changes over time. Researchers will follow the same subjects or participants over a long period. Challenge: Maintaining participant engagement over time 

Crossover Design (capitalize on within-subjects designs)

Involve participants experiencing multiple experimental conditions in a particular sequence. Each subject is first randomly assigned to either the treatment or control. Receive the pretest, then given intervention, then take the post test. There is then a wash out period where no treatment or intervention is given to reset the subjects. After the washout period, the subjects switch. Those who started in treatment switch to control and vice versa

Quality Assurance

Refers to strategies and policies for ensuring quality, integrity, and consistency at every stage of the project. Includes strategies for establishing procedures or protocols for how aspects of the study are completed

Timing

When the different elements of the study are administered

Threats to Internal Validity

Regression to the Mean, Attrition Bias, Allocation (selection) bias, Testing, Maturation, History, Performance Bias, Detection Bias

Regression to the Mean (threat)

The natural tendency for subjects who happen to score extremely low or high on an initial test because of luck or chance to score closer to the group average on subsequent tests regardless of the intervention

Attrition Bias

When there are systematic differences between the groups because subjects from one group have disproportionately dropped out or have been excluded during follow-up. Requires that there is a significant dropout rate difference

Allocation (Selection) Bias

When there are systematic differences in how subjects are assigned to the treatment and comparison/control groups; when people in treatment and control are not truly comparable at start of study

Testing

happen when subjects take the same or similar measurements multiple times within a relatively short period of time. Worn out from repeated assessment 

Maturation

1. Typically occurs when too much time passes between measurements 

2. When subjects naturally change as a function of time rather than due to the IV

3. Problematic over extremely long studies or short studies where groups can change quickly (young kids or elderly)

History

When something external happens right before the study or between the pre-test and post-test that alters how a subject responds to the assessment

Performance Bias

When subjects change their behavior after realizing they are or are not getting the intervention, or when researchers provide different care to the groups because they know which group they belong to

Detection Bias

1. When researchers inconsistently assess the outcome because they expect a certain outcome from the subject 

2. Researchers only

Observational Studies: Cohort, Case-Control, Cross Sectional, and Case Report/Case Series

The use of an attribute variable differentiates observational designs- researchers don’t decide which subjects receive the exposure, rather they study people who already are exposed/not exposed and then study the differences

Cohort

Can be Prospective or Retrospective. Researchers begin by organizing participants based on whether or not they’ve been exposed to something. Then, they examine who develops the outcome over a defined period of time

Prospective Cohort Study

researchers are part of the timeline from the beginning and therefore can collect data in real time. Identifies people who are or are not exposed to something in the present, then follows those people over time to determine disease rates for each group in the future

Retrospective Cohort Study

Identifies people who are or are not exposed to something in the past, then follows up in the present to determine disease rates for each group. Researchers are not part of the original years-long process. Only involved in present day 

Case-Control

Start by separating people on outcome. Those with the disease (cases) and those without the disease (controls). Researchers then look back to see whether these groups differed on exposure

Cross-Sectional

Exposures and outcomes are collected at a single point in time in a defined population which allows researchers to describe the distribution of variables as they exist in the moment. Good for estimating prevalence

Relative Risk is best for…

Cohort

Odds Ratios are best for…

Case-Control but are also good for cross-sectional studies

Prevalence Ratios are best for…

Cross-Sectional

Relative Risk (RR) Definition

the ratio of the probability of a disease or outcome occurring in an exposed group compared to the probability of the disease or outcome occurring in a non-exposed group; requires incidence rate-new cases

Odds Ratio (OR) Definition

the ratio of the odds of exposure for people with the disease compared to the odds of exposure for people without the disease

Prevalence Ratio (PR) Definition

measure of association that compares the prevalence of an outcome in an exposed group to the prevalence of the same outcome in an unexposed (or reference) group at one point in time 

-Calculated the same as relative risk, but interpreted differently 

In Quantitative research, data falls in 1 of 4 groups:

Nominal, Ordinal, Interval, or Ratio

Nominal

Simplest form of data whereby options are only categorized (labeled) and therefore have no intrinsic numeric value and have no greater value than each other (ethnicity, sex, race). Order doesn’t matter

Ordinal

Options can be ranked in some order but the interval from one option to the next varies and therefore mathematical principles cannot be applied. Hierarchy from less to more (not confident at all, a little confident, pretty confident, etc) Gaps between answer options are ambiguous and unequal

Interval

Options are not only rank ordered but also have equal intervals between each category (equidistant)

1. Temperature in degrees is an example 

2. Missing a true zero that means nothing

Ratio

Options are ranked ordered, equidistant, and utilize a true zero (i.e. 0 actually means none or nothing of whatever you’re measuring)

1. Car speed in mph is an example

Discrete Data

data that you cannot perform any of the four basic mathematical operations to; addition, subtraction, multiplication, division 

-Nominal and Ordinal data are considered discrete

Continuous Data

allows for at least some of the four mathematical operations to apply

-Interval and Ratio data are considered continuous 

-Interval and Ratio can use add/subtract but ONLY ratio can use multiply/divide

Nominal and Ordinal data require the use of non-parametric statistics, which are:

distribution-free methods used when data lacks any equal intervals or follows an unknown distribution (Chi-squared, Mann-Whitney U, etc)

Interval and Ratio data does allow for the use of parametric statistics, which are:

more powerful and assume a normal distribution and equal variance (t-tests, linear regression etc)

Reliability

Refers to the consistency of the test or measure

-If you repeatedly measure the same construct, reliable measurements should produce similar results

-Reliability is primarily concerned with reducing random error

Validity

Refers to the accuracy of the test or measure

-Is the measure accurate

Relationship between reliability and validity

Reliability is necessary but not sufficient for validity. If a measure is inconsistent, then it cannot produce valid interpretations. However, a measure can be extremely reliable and still measure the wrong things

Inter-Rater Reliability

How consistent raters evaluate or judge an IV

-When there is multiple raters or observers, you examine interrater reliability to determine the degree to which those observers agree in their evaluations

Test-Retest Reliability

focuses on the stability of the measurement over time

-refers to the extent to which a measurement produces similar results when administered multiple times 

-Should always be interpreted within the context of the construct being studied

Internal Consistency

focuses on how well the items on the measure work together to produce similar scores 

-Key Question: Are our measurements producing consistent results?

Ways to Increase Reliability

-Ensure that observers have proper training (inter-rater reliability) to ensure they all evaluate participants in the same way

-Have a sufficient number of questions (test stability and internal consistency)

-Ensure questions are easy to understand (test stability and internal consistency)

-Increase your sample size for variability (test stability and internal consistency)

Foundational Validity is divided into

Face Validity and Content Validity

Face Validity

you look at the items and see whether “on its face” (appears) it seems like a good translation of the construct 

Content Validity

addresses how well test questions match the content or subject area they are intended to assess

-Experts in that field generally judge content validity

Prognostic Validity Definition

evaluate whether scores relate to other variables in ways that theory predicts

Prognostic Validity is divided into

Predictive Validity, Convergent Validity, and Discriminant Validity

Predictive Validity  

 tells you how well a certain measure can predict future behavior or performance

Convergent Validity

the degree to which a construct measure “converges” with other measures that should be measuring the same thing

Discriminant Validity

the degree to which a construct measure “diverges” from other measures that purport to be measuring something different 

-Ex. A measure of depression should not highly correlate with measures of mathematical ability because those constructs are conceptually unrelated 

Validity and Reliability Relationship

Reliable measurements are necessary for validity, but reliability alone is never enough