Research Design

A broad structure that guides th collection and analysis of data- involves figuring out what you want to accomplish in the study (causality, changes in phenomena. )

Nomothetic explanations must satisfy 3 criterias:

1. Correlation

2. Time order

3. Non-Spuriousness

Correlation

• The proposed cause and effect mustt occur together and be observable

Time Order

• The cause must come before the effect

Non-spuriousness

• Alternative explanations for a correlation must be ruled out.

• “spurious” means false or illegitimate

Experimental Desgin

• True experiments are often rare in sociology.

• Experiments involve a systematic comparison of what happens between 2 sets of participants (treatment vs control group)

• They are also good at establishing causation or internal validity

Manipulation

• An experiment manipulates an independent variable (cause) to determine its influence on the dependent variable (effect).

• Often, the independent variable is manipulated while the dependent variable is then observed and measured.

Problems with Manipulation

• Problem: Many variables of concern to researchers cannot be manipulated.

• Ethical Issues: You can’t do anything inhumane

• Another Issue:

  • Lack of simulation: Many phenomena of interest are complex and long-term, which cannot be simulated in experiments (gender roles, political preferences)

  • Not in depth- When causal variables are identified, the perceptions of participants are usually ignored

  • Laboratory vs Field Experiments: The former takes place in artificial settings whereas the latter occurs in real-life surroundings (classrooms, factories)

Classical Experimental Design

• Rosenthal Jabobson (1968) tried to determine teacher expectations about academic performance - they used a classical experimental design.

  • The researchers labelled some students as “sputters” (those likely to have significant academic growth). However, they lied to the teachers about who was a spurt. Eventually, the students whom the teachers thought were spurters did do better.

  • The experiment exhibited most features of a classical experimental design - The students were randomly assigned to 2 groups, there was manipulation on the experimental group and the other group got no treatment (control group).

    • Made sure that the dependent variable (performance) was measured before manipulation to ensure the 2 groups are equal.

Notation used for classical experimental design

Obs: An observation made of the dependent variable

Exp: The experimental treatment (independent variable)

No Exp: Refers to the absence of an experimental treatment (control group)

T: The timing of the observations made in relation to the dependent variable

The Laboratory Experiment

• One main advantage of lab over field research is greater control of the environment, which enhances the ability to establish nomothetic causation, ex, a study where women were told to perform worse on a test than men (they did) vs when told it was comparing Canadians vs Americans (no difference)

Quasi - Experiments

• Help study real-world programs when true experiments aren’t possible

• No random Assignment.

• These have some features of the experimental model, but lack some features that help establish causation

• There are several types of these experiments

• Compare “Before” and “After” groups when randomization isn’t possible.

Natural Experiments

• Experimental-like conditions are produced by naturally occurring phenomena or changes brought about by people not doing research.
  
  (earthquake example)

Cross Sectional Design

• “snapshot” study → Data collected at one point in time.

• No before-and-after study,

• No manipulation of the independent variable

• Used to examine variation across multiple cases (people, families, nations, etc.)

• Commonly use questionnaires and structured interviews.

• Larger samples are preferred.

Ambiguity of Casual Influence

• Term by Blaxter (1990) - refers to uncertainty about which variable causes which.

• Cross-sectional studies show association only, not direction of causation.

Replicability (Cross-sectional Research: Key Evaluation.)

• High reliability if the researcher clearly outlines:

  • Sampling procedures (how respondents are chosen)

  • Data collection methods (e.g., questionnaire, structured interviews)

  • Data analysis steps

  • Because methods are standardized and can be repeated, others can replicate the study.

External Validity (Generalizability) (Cross-sectional Research: Key Evaluation.)

• Strong external Validity when the sample is random, allowing genralization to the larger population

• Weaker external validity when non-random sampling is used - results may not represent everyone

Cross-sectional studies.

• Are replicable and often externally valid (with random samples

• Have limited causal inference ability.

• They are often used because many important variables cannot be manipulated

• Allow for causation causal reasoning whne variables have an assumed time order (e.g. age → income)

Longitudinal Design (s)

• examines the same cases or groups over time - for example

  • Time 1 (T1), Time 2 (T2) , Time 3 (T3), etc)

  • No manipulation of independent variables (unlike experiments)

  • Allows researchers to track change over time and determine the time order of variables.

  • Helps to identify whether changes in one variable precede changes in another (supports casual inference)

  • Useful for studying social change, development and longterm trends.

Types of Logitudinal Designs:

1. Panel Study

2. Cohort Study

Panel Study

• The same individuals or groups are studied repeatedly over time

• Must address entries and exits (e.g. Marriage, moving out, death)

Cohort Study

• Follows different individuals who share a common experience (e.g, birth year, graduation year)

• They don’t track the same people each time, but compare similar groups.

Case Study Design

• Involves a detailed and intensive analysis of a single case. The goal is to gain a deep understanding of that case’s features, context, and dynamics. The case itself is the object of interest, not just a source of data for testing general theories.

When equalization the approach is typically?

Inductive (theory comes from the data)

When quanititative, the appraoch is often?

Deductive - Theory guides data collection and hypothesis testing.

TCPS2

• Canada’s Ethical Framework

• Ensures that human dignity is fundamental in research.

• Have 3 Core Principles

  • Respect for Persons

  • Concerns for Welfare

  • Justice

Respect for Persons

• Treats people as autonomous individuals, not as objects

• Requires free, informed, and ongoing consent

  • Participants must:

    • Know what the study is about

    • Understand risks and benefits

    • Be able to withdraw at any time with no penalty

  • Extra protection for those unable to consent (children, cognitive impairment → Use guardians)

Challenges:

• Full Disclosure is sometimes impossible (e,g. experiments —> risk of influencing behaviour)

• In ethnography, it’s impossible to get consent from every person in public spaces.

Concern for Welfare

• All aspects of well well-being of a person/group/community

• Welfare includes physical, mental, emotional, economic, spiritual, and social well-being

  • Research must:

    • Minimize harm

    • Maximize Benefits

    • Balance risks vs. Benefits.

• First, do no harm -→ Welfare takes priority over gaining knowledge.

Justice:

• Treat people fairly and equitably

  • No group should

    • Experience unfair risk

    • Be excluded from the benefits of research

  • Avoid exploitation of marginalized groups.

REB (Research Ethics Boards

• Must approve all research involving humans BEFORE it starts

  • Review, approve, request changes, or reject studies

  • Require annual reports for long term studies

  • Must avoid conflicts of interest (e.g. Financial stake

  • Apply TCPS2 Guidlines

    • Qualitative research often struggles with REB approval

      • It’s less structured, hard to control who's being observed. (ethnography)

General Ethical Principals

1. Voluntary Participation

2. Informed consent

3. No harm to participants

4. Anonymity and Confidentiality

5. No or Minimal Deception

Problems with Over or Under Informing Participants

• Too much detail may lead to biased behaviour

• Too little violates respect for persons

Anonymity in Sensitive Research - Research Method?

• Randomized Response Technique

  • Used when asking about illegal/embarrassing behaviour (e.g., drug use)

Procedure:

• Participant flips a coin in private.

• If heads → must answer “yes” regardless of truth

• If tails → answer truthfully

• The researcher only sees yes/no, not the coin outcome.

  • Protects:

    • Participant anonymity

    • A researcher is less at risk of being forced to testify

Covert Research

• Research where people are not told they’re being studied.

Main Steps of Quantitative Research

1. Theory

2. Hypothesis

3. Research Design (experiment, survey, cross-sectional, etc.)

4. Operationalization → turning concepts into measurable indicators

5. Select research site

6. Ethics review

7. Select participants (sampling)

8. Collect data (survey, experiment, structured observation)

9. Process data (coding, entering data)

10. Analyze data (statistics)

11. Interpret findings

12. Write report
    

The Hairy Rat Only Shaves Every Select Crevice Properly And In Wrath

Likert Scales

• Used for attitudes

• Features:

  • Statements (not questions)

  • 5-7 point scale (Strongly agree → Strongly Disagree

Coding Data

• Used when answers are unstructured (open-ended responses, documents)

• Read all data, identify themes

• Create Categories

• Assign numbers to categories

• Re-read and code consistently.

Principles of Coding

1. Categories cannot overlap

2. Must be exhaustive ( have “Other”)

3. Must have clear rules and examples.

Reliability

• consistency of a measure (does it give similar results under the same conditions)

Cronbach’s Alpha (a)

• A Statistic from 0 → 1

• Higher = better internal reliability

• Common rule:

  • Around 0.8+ = good (some accept ~ 0.7, exploratory, sometimes lower)

• If alpha is low → items might be measuring different

Split half Method

• Split the items into two halves (e.g., odd vs even questions)

• Correlate the total score of half 1 with half 2

• High correlation = good internal consistency

Measurment Validity

• Validity: are you actually measuring the concept you claim to measure

  • If it’s not reliable, it cannot be valid

Face Validity

• Does the measure look like it measures the concept

• Very basic/intuitive check

• Often judged by experts or the researcher themselves

Face Validity

• Assesses whether the measures seem to be measuring the intended concept at first glance

• Often judged intuitively by experts or the researcher

• A job satisfaction questionnaire appears to evaluate job happiness based on it’s questions

Concurrent Validity

• Compares the measure to another realted outcome assessed at the same time to check for agreement

• A new job satisfaction scale is evaluated by checking if individuals with high satisfaction scores also show lower absenteeism rates. If the patterns align with expectations, it indicates good concurrent validity.

Construct Validity

• Determines if the measure behaves as predicted by theory and is related to other variables consistently

• The theory states that routine jobs lead to lower job satisfaction. If a job satisfaction measure shows lower scores for those in routine jobs, it supports construct validity. If expected relationships are not found, it raises questions about the measure’s accuracy, the theory's validity, or the reasoning connecting them.

Convergent Validity

• Assesses whether the measure aligns with another measure of the same concept using a different method

• Comparing self-reported time management by managers with direct observations of their activities, Agreement the two indicates convergent validity, discrepancies suggest one or both measures might be flawed.

Reliability vs Validity (quick rule)

• A measure can be:

  • Reliable but not valid (consistently wrong)

• So: no reliability = no validity

Main Goals of Quantitative Research

• Measurement

• Establishing Casuality (Internal Validity)

• Generalization (External Validity)

• Replication

Critisism of Quantitative Reseach

1. Treats people like objects/nature

2. False sense of precision (the numbers, people interpret things differently)

3. Artificial instruments and disconnection from real life (unnatural situations)

4. Focus on Variables, not lived processes

5. Explains findings without participants’ perspectives

6. Objectivist view of reality

Measurement Variation

• Measured variation = True Variation + Error Variation

  • Goal = minimize error to increase validity

• Measurement variation is the total amount of differences you see in your data - the final numbers you collect. It includes both the real differences between people (true variation) and the mistakes or noise in measurement (error variation).

Personal Factual Question in Surveys

• Age, Income, Marital Status

• Behaviour frequency (church attendance, moving-going)

Factual About an Entity/Event Question in Surveys

• Asking what someone witnessed

• Problem: people are not accurate observers

Attitude Question in Surveys

• Often measured using Likert scales

  • Strongly Agree → Strongly Disagree

Belief Questions in Surveys

• Example: “It is up to the individual to decide right/wrong”

Knowledge Questions in Surveys

• Test knowledge on a topic (e.g., historical facts)

Questions about others Survey questions

• Second-hand reports are often inaccurate

• Used when:

  • checking validity of someone’s report

  • Person cannot self-report

3 Golden rules for designing research questions

1. Keep research questions in mind

2. Be specific

3. Put yourself in the respondent’s shoes

Question Order - Why it matters

• Question order can change responses (context effects)

• Example:

  • Crime questions before rating police officers → might make people rate police differently

Response set

• When people answer based on a pattern, not rel attitudes

Vignette Questions

• A short story/hypothetical scenario describing a situation.

• Respondents are asked: What should they do? What would you think?

Computer- Assisted Interviewing (CAPI and CATI)

CAPI: Computer-Assisted Personal Interviewing (face-to-face with laptop/tablet)

CATI: Computer-Assisted Telephone Interviewing.

Rapport

• A positive and harmonious relationship between individuals, characterized by mutual respect.

• Promotes a productive interview/survey

Promoting

• Last resort only

• The interviewer suggests an answer

• There is a loss of validity

Objectivism

• the idea that there is a reality out there regardless

Positivism

Reality is what we percieve it to be

Quasi Experiment

• No random assignment

• Has before and after comparison

• Assess the effectiveness of internetion

• Uses pre-existing, naturally occurring groups

• Example: School Program Comparison

Cross-Sectional

• No random assignment (data at one point)

• Done in a single moment (data collection

• Example: Mental Health Survey

Longitudinal Study

• Same Participants over time

• Data is collected in multiple moments

• Tracks changes over time

• Example Birth Cohort Study.

• Two kinds: Cohort and Panel Study

Cohort Study

• Form of Longitudinal study

• Group of individuals with similar characteristics

• Involves long-term observation of a group of participants by the researcher

• Usually, the researcher has to maintain contact with the cohort members

• Focuses on shared experiences. (all smokers, all nurses)

Panel Study

• A longitudinal study

• The same individuals or groups are studied repeatedly over time.

Quasi Experiments Example:

• Research designs that evaluate the effect of an intervention or treatment without random assignment to groups. Participants are assigned based on predefined criteria.

• EXAMPLE:

  • University Degree → higher incomes

    • It’s unethical to hold people back for 4 years

    • Another way is to compare people who already have degrees vs who don’t, This way you don’t manipulate the independent variable

3 Criteria of nomothetic causation:

1. Correlation

2. Time Order

3. Non-spurious

2 Types of correlation:

• Positive vs Negative

Positive correlation

If one variable is increased, the other increases; if one decreases, the other decreases.

Example: Watching violent movies causes people to be more violent

Negative Correlation

• Exists if higher scores on one variable are found with Lowe scoring on the other

  • As one goes up (increases), the other goes down (decreases)

Example: the correlation between hours studied and the # of F’s a person receives.

Correlation coefficients:

• The most commonly used correlation coeffciant is pearson’s r

Pearson’s R

• the range of pearson’s r is -1 to +1

• Positive correlations have a value of 0, but the closer to 1, the more perfect.

Pearson’s r=+1

• Perfect correlation

  

Pearson’s r = 0.5

• It’s correlated positively but not perfectly

• Example: SAT SCORES

  

Person''s r = 0.0

• No Correlation

  

Perfect Negative Correlation

r= -1

E.g. Possiblilty to concentrate vs Sleep deprivation

A correlation of -.5

• Negative but not perfect
  

  

Time order

• The cause has to come before the supposed effect

• Example: Smoking causing cancer

Fallacy

• If something is fallacious, it means that it’s raw and incorrect, so a fallacy is something that’s not true. (invalid argument)

Post hoc:

• Means after this

Ergo

• means therefore

Propter Hoc

• means because of this.

Non - spuriousness

• Spurious means false or illegitimate

• Example: ice cream and air conditioning sales.

  • The relation between those is spurious

    • If two variables are correlated but not causally related, the relationship is spurious.

  • For it to be non-spurious it has to cause the other not just be correlated)

Antecedent

• means it happens before, before in time

How do we test for spuriousness?

• We control the third factor.

• For example, the shoe size = reading ability.

  • We would hold the age constant.

    • (Only look at 10-year-olds and see if the correlation still exists.

Independent Variable

The proposed cause

Dependent Variable

The proposed cause

Unit of analysis:

is the entity that the researcher is studying,

Ecological Fallacy:

• If the unit of analysis of the data we have is not the same as the unit of analysis of the conclusion that’s an ecological fallacy

• An error in reasoning that may occur if the unit of analysis of the data is not the same as the unit of analysis of the conclusions

• Example: Suppose a study finds that cities with higher average incomes have fewer people living in poverty. If someone concludes that all individuals in wealthy cities are not poor, that's an ecological fallacy. There could still be many poor individuals living in those cities, despite the high average income.

Research Ethics:

1. What scientific knowledge did this study produce

2. Can the knowledge generated by this study be used to explain real life situations?

3. Was this study ethical

General Ethical Principals:

1. Voluntary Participation

2. Informed Consent

3. No Harm to participants

4. Anonymity and Confidentiality

5. No or Minimal Deception

Voluntary Participation

• Participants should be able to leave the study at any time.

• No threats or blackmail to participate

Informed Consent

• Before people consent to the study they need to be informed about the risks involved with the study and a basic idea of what the study is about.

No Harm to participants

• People should not be harmed by their participation in the study.

• Sometimes it might be necessary to inflict mild harm for the betterment of humanity.

4. Anonyminity and Confidentiality

• Needed to precent the harm that could arise if private information becomes public