Research Methods Midterm

Scientific Method

A systematic approach to inquiry that involves formulating hypotheses, conducting experiments, and analyzing data to draw conclusions. The scientific method ensures that research is objective, replicable, and based on empirical evidence.

Theory

Formulation based on observations and insights, consisting of tentative premises

Steps of Testing a Theory

theory/idea → conceptual hypothesis → research hypothesis → operations

Research Question

non-directional statement about specific relationships in a study that ends with a question mark

Conceptual Hypothesis

a prediction about relationships involving the theoretical constructs, and therefore guides the purpose of a research study. It may be viewed as a prediction about what should happen in a research study

Research Hypothesis

a testable directional prediction about specific relationships in a study

Operationalization

Specify the variables that are critical in empirically assessing the phenomenon of interest

Multiple Operationalization

No single operation or measurement provides enough information to adequately define a concept, a theoretical concept, so the construct is measured through several techniques or operations

Triangulation

practice of using multiple methods, data sources, investigators, or theoretical frameworks to examine the same phenomenon

3 types of research

experimental

quasi-experimental

nonexperimental

Experimental Design

Static, controlled, not generalizable, good for looking for causality

Quasi-Experimental Design

Why? Not enough people/things to assign randomly, for more external validity

Nonexperimental Design

observational, exploratory, coming up with hypotheses

What is the key difference between experimental and quasi-experimental?

Randomization

What is the difference between nonexperimental and experimental?

Manipulation

3 Types of Causation

undirectional, bidirectional, non-causal covariation

Unidirectional Causation

causation goes in one direction

Bidirectional Causation

causation goes back and forth/Both variables change with each other

e.g. parent-child relationships

Non-causal Covariation

when two variables change together in a predictable pattern, but neither variable causes the other to happen

e.g. marriage rate decline followed by an increase in suicides, both are effects of economic depression

Spurious Relationships

when an observed relationship is found to have been mistakenly interpreted as a causal relationship of variable 1 or variable 2 when it is actually a case of an extraneous/unmeasured 3rd variable

Independent Variable

the cause or the factor that you change or control in an experiment

Dependent Variable

is the effect or the outcome that you measure to see if it changes

Internal Validity

how confident are we that A causes B?

External Validity

is it generalizable to other groups?

Can you have internal validity without external validity?

Yes

Can you have external validity without internal validity?

No

Confounds

anything that can serve as a rival hypothesis

Reliability

How consistent is that measure?

Validity

Is it measuring what it is supposed to measure?

Moderators

• tells us when effects will happen

  • Influences strength of relationship

  • It should not be associated or related

  • Cannot have multiple

Mediators

•  tell us how or why something happens

  • Is necessary to complete the causal relationship

  • You can have multiple 

Inferential Stats

assesses the likelihood that an obtained effect could have occurred by chance, given the null is true

Statistical Significance

• possibility you get differences randomly

  • Achieved if the probability of obtaining the observed effect by chance is so low as to render the chance explanation implausible

Statistical Power

the probability of obtaining a statistically significant effect, if indeed that effect truly exists.

• Use GPower: you put in effect size and analysis and it tells you how many people you need or vice versa

Type I Error

You reject a null hypothesis that is actually true. You mistakenly conclude that an effect or difference exists when it does not.You reject a null hypothesis that is actually true. You mistakenly conclude that an effect or difference exists when it does not.

Type II Error

You fail to reject a null hypothesis that is actually false. You miss a real effect or difference that does exist.

Nominal/Categorical Scale

• categorical variables

  • E.g. types of wood

Ordinal Scale

• rank order, not mathematically meaningful

  • E.g. first place, second place, third place

Interval Scale

• a numerical measurement where the data are ordered and have equal distances between values, but no true zero point

  • E.g. temperature (Celsius)

Ratio

• a quantitative measurement with equal intervals between values and a true, absolute zero point

  • E.g. speed (mph)

QRP

Questionable Research Practices: ways of producing, maintaining, sharing, analyzing, or interpreting data that are likely to produce misleading conclusions, typically in the interest of the researcher

HARKing

hypothesis after results are known

PARKing

preregistering after the data is known

Why is Open Science Framework good?

forces you to really think everything through and catch mistakes

true/false: unethical research is wasteful

 true (rosenthal)

Ethical considerations need to be considered in all parts of the research process, specifically in 5 things:

• Study Design

• Data Collection

• Data Analysis (p-hacking)

• Reporting Results

• The harm of not conducting a study: you know a study that could help people but choose not to do it or have it done

Study Designs

• The design must adequately match the research question

• Constructs must be operationalized (operational definitions)

• Study should use scales that are appropriate for the constructs of interest

• Hypothesis must be testable

  • People’s souls influence behavior → can we define souls?

  • We can test whether people who believe in souls behave differently from people who do not

Data Collection (4 things are included)

Informed Consent

Deception

Debriefing

Protecting Privacy

Informed Consent

a process in which potential participants are informed about what participation in the study will involve so that they can make an informed decision about whether or not to engage in the research

Deception

• research procedures might be presented and justified in the guise of an elaborate “cover story” designed to manage the participant’s perception of the research setting and conceal the true purpose of the study

  • These are less common nowadays

Debriefing

• a session following participation in a study in which the participants are informed of the true purpose of the research

Data Analysis No-Nos (6 things)

• Fabrication

• Deleting data/Omitting

• Outlier Rejection

• Subject Selection

• Data Mining

• Misrepresentation of Findings

Fabrication

making up data

Omitting Data

getting rid of data or deleting it WITHOUT disclosing it

Outlier Rejection

removing outliers WITHOUT disclosing it

Subject Selection

 purposely choosing who will be in the study

Data Mining

• finding patterns in data to predict outcomes

  • People go through their data without a hypothesis

Reporting Results (5 things)

• Censorship

• Authorship

• Plagiarism

• Hyperclaiming

• Causism

Censorship

protect your ideas

Authorship

be clear about what you’re asking for; communicate to avoid authorship battles

Plagiarism

copying work without credit

Hyperclaiming

don't claim causality just to do it. Dont cry wolf, think about your career

Causism

the methodological error of overstating causal relationships from non-experimental or correlational data

Measurement

a form of operationalization in which abstract concepts are translated into observable and quantifiable information or data

Essential characteristics of experiments

• Extreme control

• Restricted focus

• Restricted sample

3 Problems with experiments (ACE)

• Artificial

• Constricted (restricted)

• Experimental Demand

  • Participants can change behavior because they are in an experiment

Experimenter Expectancy and Bias

when a researcher's unconscious or conscious expectations subtly influence study participants, data collection, or result interpretation, turning hypotheses into a self-fulfilling prophecy

Demand Characteristic

The totality of all social cues communicated in the lab not attributable to the manipulation

Placebo Effect

when participants’ belief in the efficacy of the treatment, rather than the actual effects of treatment, is responsible for the results found in an experiment

Solutions to experimenter expectancy and bias

• Monitoring

• Blind procedures

  • Single blind

  • Double blind

  • Triple blind

• Mechanized procedure

Intention-to-treat Analysis

all participants are evaluated regardless of whether they received/completed the full manipulation

Threats to Internal Validity

MR. SMITHI

• Maturation

• Regression to the mean

• Selection

• Mortality

• Instrumentation

• Testing

• History

• Interaction (selection-based)

Maturation

 things that happen naturally (growth/experience)

Regression to the Mean

random error all lands one way → re-test, they go back up a little to the mean

Selection

• how did they get into the study in the first place?

  • any one group gets exposed to threats of internal validity

Mortality

participants dropping throughout the study

Instrumentation

• not measuring correctly, tool = broken

Testing

 pretest sensitization, interventions become more salient

History

anything that happens between pretest and posttest outside of design

Interaction

If more than one of these things happens, this makes unequal effects across groups

O = T + E

Observed = True + Error

What types of error can be found in O = T + E

Systematic and Random

Systematic Error

• can be conceived as systematic measurement bias, does not cancel out.

Random Error

due to chance events which tend to increase the variability of scores in nonsystematic ways. Theoretically, random errors cancel each other out.

How does systematic and random errors relate to Type I and II errors?

• Systematic can affect both

• Random is more likely to affect Type II

How does systematic and random errors relate to reliability and validity?

• Random error affects reliability (and therefore validity)

• Systematic error affects validity

What is the difference between measurement validity and internal validity?

what you are measuring versus how well you can prove cause and effect

What are the 3 types of reliability?

• Temporal Stability

• Internal Consistency

• Inter-rate Reliability

Temporal Stability

• consistent over time - will the second test give you the same scores? Will I get the same patterns?

How is temporal stability measured?

Test-retest - but many people may remember responses (wait 2 months and you're good)

parallel forms

Parallel Forms

devising two separate item sets intended to assess the same underlying construct, administered to the same participants at the same time, with degree of relatedness calculated.

Internal Consistency

 are all the items measuring the same construct

How is internal consistency measured?

split half or cronbach’s alpha

Split Half

• assessed by randomly dividing a scale into two sets containing an equal number of items, both administered to the same respondents, with a test of relatedness calculated between these two summed scores

Cronbach’s Alpha

• Range of alpha?

  • .5-1

• Ideal alpha?

  • .7/.75 (.7 in txtbook)

• Is it possible to have “too high” an alpha?

  • Yes

• When items increase, the alpha should also go up with every item.

Inter-rater Reliability

• researchers compare codes, are they understood the same? Need .70

Contemporary Test Theory

 the search for a latent factor (a characteristic or construct that is not directly measured or observed but that underlies responses on a measurement scale)

Item Response Theory

 adjust questions as they’re being taken?

Factor Analysis

do items of a scale go after a single construct?

Exploratory Factor Analysis

a technique used to determine the underlying factor structure if the researcher does not have hypotheses regarding the number of underlying factors and the items that might constitute each factor