Research Methods

Theory

Why? How?

Explanations and predictions - based on what happens in the past

Naive science

• personal experience

• Intuition (easy way out)

• Authority (rely on what experts say)

• Appeals to traditions, customs and faith

• Magic, superstitions and mysticism

Insufficient/incomplete data

Either no or biased inquiry

Scientific method

• systematic process→ increase chance conclusion is correct

• Falsifiable theory

• Replication

• Reflective and self-critical approach

• Cumulative and self-correcting process

• Cyclical process

Scientific research process

1. Question / puzzle

2. Conceptualization

3. Operationalization

4. Research design

5. Observation

6. Data analysis

7. Interpretation

Why research methods?

• From intuitions and anecdotes to systematic evidence

• Question and problem driven

  • Good research question is important

• Research methods are useful tools

  • provide transparency and replicability

• Methodological pluralism and diversity

• Methods as constraints vs opportunities

  • Limit due to very clear guide lines

  • Advantage gives credibility through use

Ontology

What is the nature of the social world

Epistemology

What can we know about social phenomena

Methodology

How do we gain/ obtain knowledge

Positivism

Developed by French philosopher August Comte

search for the truth though systemic collection of observable facts

Sociology

Scientific study of the social world

What are the 3 positions of positivism?

• classical

• Logical

• Falsification

Classic positivism: basic tenets

• naturalism

• Empiricism

• Laws

  • Establish social laws that allow us to predict people behavior

• Induction (observation → theory)

• Cause and effect relationship (observable ‘constant conjunction’ David Hume)

• Science is objective and value free → establish facts

Naturalism

Social science = natural science

• approach them in the same way

• Observation is KEY

Empiricism

Knowledge of the wold is limited to what can be observed and measured (sensory experience)

Laws

Social world is subject to regular and systematic processes; laws are explanatory and predictive

Logic positivism: basic tenets

• empiricism (observation) + logical reasoning

• Deduction (theory → observation)

• Retroduction (observation ←→ theory)

• Verification (establishing truth claims) → find support for their reasoning

Induction & deduction & retroduction

Critique of logical positivism

Karl Popper

• Rejection of induction: not a good way of creating/ testing a theory

  • Particular experience /→ general knowledge

  • One counter-observation and law is falsified → swan example

• Rejection of verifiability: develop our theories with logical resonating , key is more to look for evidence that contradicts the theory

  • Verifying theory is pointless

  • Goal must be falsification

Deductive-Nomological model

Carl Gustav Hempel - logical positivist

• observed phenomenon is explained if it can be deduced from a universal law-like generalization

• Laws express necessary connections between properties, accidental generalization does not exist

Hypothetical-deductive model

• test ability of law to predict events

• Law → hypothesis → explicit prediction

  • Prediction correct→ hypothesis corroborated / supported

  • Prediction incorrect → hypothesis falsified/ not supported

Challenge 1 to positivism: Scientific Realism

Response: there is a lot going on that we can not observe objectively

Similarities to positivism:

• social and natural worlds (sciences) are similar

• Realism: ‘objective’ reality exists

Key differences:

• Realist can consist on unobservable elements as well

  • Ex. Structural relationships, legitimacy

• Assessment by observable consequences

  • Ex. Consider the gov legitimate - more likely to vote

• Causal mechanisms instead of law-like generalizations

  • Don’t like to talk about laws- there are always exceptions

  • If a cause is present it makes the theory more likely 

• ‘Best’ theory is the one that explains phenomena the ‘best’

  • Not correct but most likely (highest explanatory value)

Example scientific realism

Mechanism (Tilly)

• Environmental: externally generated influences on conditions affecting social life

  • Contextual factors impact how political actors act

• Cognitive: operate through alterations of individual and collective perception

  • Protestant mindset worked well with capitalism

  • Stereotypes

• Relational: alter connections among people, groups and interpersonal networks

Individualism vs Holism

• Individualism - break everything down to smallest unit of analysis

  • Individuals make up society, we have to study them to understand

• Holism: states can make decisions —> more than just the individual that make up the state but as its own entity

  • More than the sum of its parts —> becomes the actor itself

Coleman’s ‘Bathtub’

• Start with general explanations, then go very specific, put all the individual decisions/outcomes together you come back to general outcomes

  • General= macro

  • Specific = micro

• Ex. Democratic peace theory

Challenge 2 to positivism: Interpretivism

• Fundamentally different from positivism

  • Social world and natural world are fundamentally different -> different methods needed

• Social world

  • Subjectively created -> the only thing that matters is from the perspective of the research - how they see it

  • Understanding human behavior by interpretation of meaning of social behavior

  • Example of approaches: hermeneutical (reader perspective), critical theory, constructivism, post-colonialism, feminism

  • But: some similarities in collecting evidence and establishing causal relations (?)

Researchers have values→ source of bias?

• Critical theory: yes, can’t be separated!

• Positivism: well, let’s distinguish:

  • Normative theory (what ought or should be)

  • Empirical theory (what is)

• Robert Cox: all theory is normative

• Max Weber: distinguish, yes, but values cannot be ignored (what is seen as relevant, point of view)

How to separate facts and values

• Transparency

  • Self-aware/disclosure

  • Critical examination by larger scientific community 

    • If you are not aware of your bias, other people will be

Researchers expectations are hard to escape

• Expectations shape perceptions

  • Image of bunny/duck

• Observation can change (social) phenomena

  • Hawthorn effect

• Everyone should be aware of how humans operate and take this into account during research

Hawthorn effect

if you know you are being observed you change your behavior

Thomas Kuhn - the Structure of Scientific Revolutions (1962)

• Science is a social institutions

• Scientific community subscribes to a common view, paradigm or conceptual scheme (=‘normal science’)

• Defines objects, norms and methods of investigation

• ‘Truth’ is based on consensus in scientific community

• Paradigm shifts: ‘Revolutionary’ change in paradigms

Scientific Revolution

People like to stick to what they believe- only when they have to admit that they are wrong is when new science can come about

Scientific research programs

Imre Lakatos - Falsification and the methodology of scientific Research Programs (1970)

• Scientific research programs= incremental, cumulative, progressive articulation of scientific research programs lead to the growth of scientific knowledge

• Hard core with a protective belt of auxiliary hypothesis

• Novel facts: progressive (problem shifts) or degenerating research programs 

Research question

Likes a puzzle

Problem, topic, puzzle → general RQ → specific RQ → scientific inquiry

• Process is like a funnel- start wide then you narrow it down

Research process vs presentation

Process = narrow it down, find question

Presentation = journal, article

What makes a research question relevant?

Two criteria: does the answer to the question have…

1. Scientific relevance /importance

2. Social (real world) relevance/importance

What makes a RQ useful?

• Should guide and structure the whole research process (’wheel of science’)

• Be researchable (possible to answer)

• New (not answered before)

Tension with relevance of RQ

Finding a good RQ?

• Real world events and problems

  • Recent (& historic) events are good choices, but ongoing events are very risky (better avoided)

• Existing literature

  • ‘gaps’ and ‘controversies’

  • Caution: blind acceptance of ‘normal science’ might limit search for better understanding

• “Puzzle” : unexpected contradictions

• Replication (but not fully accepted yet)

Typical steps to research process

1. General RQ/ working hypothesis

2. Literature review

   • What do we already know?

   • What do we not know?

3. Theory/theoretical framework

   • Relevant concepts and factors

   • Expectations and hypotheses

4. Research design (data and sources)

Types of research questions

• descriptive

• Explanatory

• Predictive

• Prescriptive

• Normative

Descriptive RQ

• open ended

• Collect information

• Ex. Two factors, are they related to each other

Explanatory RQ

Make claim about causal relationship

Predictive RQ

If we do this, what are the implications in x years

Ex. Policy implemented to try and lower emissions, what will happen or emissions look like in x years

Prescriptive RQ

Goal oriented

Ex. We want to lower emissions→ what policy is needed to reach this

Normative RQ

Philosophical questions

Types of RQ preferred by academic audiences

Descriptive, explanatory, normative

Types of RQ preferred by applied research and consulting

Predictive and prescriptive

Literature search: sources

• Core books in library (open stacks?)

• Databases

  • Via library

  • Google scholar

• Reviews/state-of-the-art articles

  • Handbooks/encyclopedias

  • Annual Review of Political Science

  • Then following references (”snowball sampling”)

    • (Wikipedia can be a good starting point)

    • ChatGPT

Analytical review

• Summarize: outline the relevant existing research/knowledge/theories/methods/evidence (topical/thematic review)

• Evaluate: identify the contributions (strengths) and limitations (weaknesses) of existing research

• Conceptualist: use it to define key concepts

Literature review

• not annotated bibliography or personal diary

• Analytical review

• Develop general into specific RQ/hypothesis

Theory definition

• Simplified model of reality

• Identifies key concepts/factors and their relationship

• “A proposition which has been elaborated and/or has withstood repeated testing”

• A theory is “a set of interrelated constructions, definitions, and propositions that present a systematic view of phenomena by specifying relations among variables, with the purpose of explaining and predicting the phenomena ”

Types of theories

• Scope/level: grand theory vs middle-range theory (Merton 1968)

• Process: inductive vs deductive

• Nature of question: empirical vs normative

‘Ground theory’

• Sounds like a theory but is actually a research method -> qualitative

  • Inductive and qualitative approach: method of inquiry and theory building

• Process

  • Coding: close (tentative) coding of collected data

  • Sorting: compare, sort and synthesize the codes

  • Memo writing: were memos outlining/describing codes

How can theories be used?

• Apply existing theory to new cases/data

• Revise theory and test with old and new cases/data

• Identify and define concepts/factors and their relationships, usually but not always in the form of hypothesis

Hypothesis

•  a proposed explanation for a phenomena

  • Usually by stating some kind of (testable) cause and effect relationship

Relationship

• specify relationship between factors→ link two or more variables

  • Cause: explanatory factor/ independent variable

  • Outcome: dependent variable

Null relationship

2 concepts not related

Covariance relationship (positive relations)

Concept A present, B present as well (occur at the same time)

Causal relationship

A causes B

Can be reciprocal causation

Relationships ex Democratic Peace Theory

An empirical regularity - democracies don’t go to war with each other (macro level)

• Hawks and Doves - tested the theory on a individual level

• Explanations

  • Liberal norms (based on reason or culture/norms) 

  • Institutional explanations

    • Institutional constraints (checks and balances)

    • Audience costs (casualties and re-election)

  • System-level explanations

    • Historical context (ex. Cold War)

    • Geographical proximity

    • Economic links/trade

  • Decision-makers (micro foundations)

    • Beliefs, interests, leadership style, etc

causality definition

Necessary conditions (after Stuart Mill)

• Covariance (correlation)

• Temporal Ordering (time order)

• Spatial and Temporal contiguity/link (process)

• Nonspurious connection (no confounds)

Important: covariance is a necessary but not sufficient condition for causality

Covariance (correlation)

X & Y

cause and effect happen at the same time = + go up at same time - one goes up and one goes down

• Does not say anything about being linked only that they occur

Temporal ordering (time order)

Show that the cause changes before the effect takes place

have to prove that X moves first and Y just responds to changes in X

• Crucial to establish causality

Spatial & Temporal contiguity/link (process)

Why and how the cause is linked to the effect

• Spatial link- they can effect each other

• Temporal link- makes sense that they can effect each other

Nonspurious connection (no confounds)

Rule out alternative explanations and 3rd factors

Causality limitations

• Deterministic vs Probabilistic relationships

  • Natural laws- deterministic

    • Rarely happens in social sciences

• Use caution with nomothetic causality (esp. in social sciences)

  • Complete causation -> unlikely

  • Exceptional cases -> possible

  • Majority of cases -> not required

    • Tends to speak very broadly (ex states/democracy) but might only apply to a small set of cases (ex. Western democracy)

• Key question: intervening and/or anteceding factors?

Relationship(s) between factors: key terms

• Cause: independent variable

• Outcome: dependent variable

• Intervening factors: moderation mediator

• Antecedent factors: confound control 

Zero-order relationship

independent variable → dependent variable

+ Example

Mediator

Independent variable→ mediator→ dependent variable

IV and DV can have a partial relationship (red line)

+example

Moderator

Independent variable→ moderator→ dependent variable

Moderator: can reinforce (+) or suppress(-) relationships, even extreme cases distort(x) them

+example

Confound

independent variable ← confound → dependent variable

Confound: completely explains relationship

+examples

Research design definition

A strategy (blueprint) for investigating the research question/hypothesis in a coherent and logical way, including what kind of data is needed, how the data is collected, and what methods of analysis will be used

• Type of research

• Evidence or data needed (for test, not confirmation)

• Method of data collection

• Modes of analysis (logic to draw inferences)

• How threats to internal and external validity and measurement reliability and validity are minimized

Research designs major types

• experimental

• Cross-sectional and longitudinal

• Comparative

• Historical (paired with comparative)

Experimental design

• Randomized intervention/treatment

  • Causal factor is randomly assigned (med experiments (placebo vs real))

  • No confounding variable

• Lab, field & survey experiments (3 types)

  • Social science - traditionally field experiments

• (Natural experiments) -> NOT an experiment 

  • Does not include a randomized treatment, does not treat/manipulte anything

Cross-sectional and longitudinal designs

• Observation

  • Cross-sectional - different cases

  • Longitudinal - over time

• Large n research (counties, conflicts, individuals, etc)

• Panel & Cohort Designs

  • Panel- select people/states and follow them over time

  • Cohort- use same survey (ex) but new people/sample taking it each time

Comparative design

• Single n case study - in-depth process tracing (find out step by step of what happened)

• Small n (2-4) and large n case study

Methods of data collection

• Requires the definition of clear boundaries:

  • Time | place/space | units/actors | factors/variables

• Data sources & quality: availability, reliability, validity

Common methods of data collection

• Questionnaires/surveys

• Interviews/focus groups

• Participation observation/ ethnographic research (exception, does not happen often)

• Textual/content/discourse analysis (existing documents)

• Statistical data

Multi-method research

Combination of different designs & methods can be useful

Stanley Milgram - Obedience to Authority 1974

• study occurred in 60s

• Hypothesis: individuals will obey requests by authority even if requests is considered to be unethical

• Cover story: ‘learning by punishment’

• Participants in experiment:

  • experimenter (knew)

  • Teacher (volunteer)

  • Learner (knew)

• Experiment: volunteers (teachers) were brought in to ask questions to learner, every time they got a question wrong the teacher would give them an electric shock (15-450 volts)

  • The experiment was fake and was actually to see how long people would go doing immoral activity when pressured from authority (experimenter)

• 2/3 went to the max voltage

Stanley Milgram - Obedience to Authority 1974 Background

Holocaust - Nazi officers after the war was over often used the excuse ‘i was just following orders’

Stanley Milgram - Obedience to Authority 1974: Procedure for learner

• verbal protest

• Knocking on wall

• Refusal/silence

Stanley Milgram - Obedience to Authority 1974: procedure for experimenter

• ‘Please continue’

• ‘The experiment requires you to continue’

• ‘You have no other choice, you must go on’

Stanley Milgram - Obedience to Authority 1974 After experiment was over

• Would have sever psychological damage-> after study completed they would get to know what was actually happening (full debriefing)

  • Followed up with them a year later -> claimed no problems had occurred

Philip Zimbardo - Stanford Prison experiment (1971) background

Personality traits of prisoners & guards are key to understanding abusive prison situations (situation vs personality)

Philip Zimbardo - Stanford Prison experiment (1971) procedure

• 2 week ‘prison simulation’ (planned)

• 24 participants (pre-screened)

• Random assignment as “prisoner” or “guard” (in uniforms)

Philip Zimbardo - Stanford Prison experiment (1971) Results

• Quickly spun out of control

• Early termination after 6 days

• 1/3 of “guards” exhibit sadistic tendencies

  • Can be seen with peacekeepers today

Fraud with fake data Michael LaCour (2014)

• Conduct study on effect of canvassing before referendum of legalization of same sex marriage -> large difference in results

• Another group wanted to replicate-> did not work

• LaCour had taken real data and used it with fake data to show referendum differences

Fraud with fake data Diederik Stapel (2011)

• By 2015, 58 (co-) authored papers retracted 

• Went out to collect data alone-> came back with data and published together with others (co-authors did not know the data was fake)

Plagiarism (politician edition)

• Plagiarized master thesis or PhD dissertation

• Germany (2011): Karl-Theodor zu Gutenberg (defense Secretary)

• Romania (2012): Victor Ponta (Prime Minister)

• Germany (2013): Annette Schawan (Education Minister)

• Spain (2018): Carmen Montón (Health Minister)

• Norway (2024): Sandra Borch (Higher Education Minister)

When is research ethics relevant?

All stages of research

• Data collection & storage: risks, privacy & fraud prevention

• Data analysis: transparency & replication

• Academic writing: plagiarism

3 (4) basic principles of research ethics

1. Do no harm: benefits much outweigh risks

   1. Ex. Vaccines

2. Informed consent: voluntary participation

3. Protection of privacy/confidentiality

4. Transparency and documentation (good research practice)

Informed (or implied) consent: content

• Topic and nature of questions (sensitive/intrusive)

• Purpose/goal (incl. disclosure of financial interests/sponsors)

• Use of information

• Participation requirements/expectoration

  • Voluntary participation

  • Freedom to stop

  • Permission to record (audio/video) & to quote responses

  • Permission to use data

• Risks involved

Informed (or implied) consent: further considerations

• Competition & comprehension (marginalized/vulnerable population)

• Incentives for participation?

• (Unobtrusive) observation

  • Consent after observation?

• Experimental manipulation

  • Concealment vs deception

    • Both of these cases need debriefing (explaining the truth)

Privacy/confidentiality: protection of participants/sources

• Public information (’on record’) (rules also apply)

• Confidential information (not always possible/in your control)

• Anonymous information (ideal, but rarely possible)

• In practice:

  • don’t make promises you can’t keep

  • When in doubt: make advance & explicit agreements

Ethical behavior of researchers

• Avoiding bias

• No incorrect reporting

• No inappropriate use of information

Ethical behavior of sponsor

• No restrictions imposed by the sponsor

• No misuse of information

Formal procedures

• Formal review of research projects:

  • USA: Institutional Review Board (IRB)

  • Leiden: Ethics Committee

• More formal rules for documentation and archiving:

  • Pre-registration

    • Publicly committing to your research

  • EU General Data Protection Regulation (GDPR)

  • Data Access & Research Transparency (DART)

  • Data Management plans (Leiden)

Plagiarism

presenting, intentionally or otherwise, someone else’s words, thoughts, analysis, argumentation, pictures, techniques, computer programs, etc. as your own work

Internal validity

• ability to draw causal inferences

  • Confidence in (observed) causal relationship

  • Ability to reuse out alternative explanations (confounds)

External validity

generalizability of research findings