POLS 2650 Lecture Notes - Exam

Empirical research

• Research based on actual, 'objective' observation of a phenomenon.

• “a methodology that requires scholars to clearly state hypotheses or propositions that can be evaluated with actual “objective” observation of political phenomena”

• Research process:

  • Identification of an idea or problem to solve

    • “Is [voter] turnout among evangelicals [Christians] higher in elections where there are distance differences between candidates on moral issues than in elections where the differences are small?” (e.g., moral issues – same sex marriage, abortion, etc.)

  • Hypothesis formation

    • “Evangelical christian’s are more likely than others to base their votes on candidate’s positions on moral issues”

  • Data collection

    • “We need to clearly define the concepts of moral values and evangelical Christians”

    • Create and administer a survey or poll that contains relevant questions

  • Interpretation and decision

    • “determine whether or not the observed results are consistent with the hypotheses”

    • “judging how well data support scientific hypotheses is usually not an easy matter”

  • Modification and extension

    • “depending on the outcome of the test, one can tentatively accept, and and on, or modify the hypothesis”

    • Rejection of a hypothesis can be “both interesting and beneficial”

      
      

Hypothesis

• An educated guess or a proposed explanation for a phenomenon. It is a statement that can be tested through experimentation or observation 

• are a specific research expectation

• derives from a theory

• can be descriptive

• most often explanatory - formulates an expectation regarding the relationship between two or more variables

• A good one had 4 characteristics:

  • falsifiability

    • A falsifiable theory can state beforehand which observations or research results it would not expect

    • If there is no imaginable evidence that would prove the theory wrong, it cannot be subjected to empirical testing

  • generality

    • A general theory has a large domain; that is, it applies to different cases and contexts

    • If the theory only explains a single event or instance, it offers little more than a description

  • parsimony

    • A parsimonious theory is simple; that is, it does not include more variables or details than necessary

    • If a theory is as complex as the process or event it explains, it does not simplify reality

  • coherence

    • A coherent theory makes logical sense, is in line with existing knowledge, and provides a credible and probable account of reality

      
      

Proposition

• a statement that expresses a judgment or an option. It is a claim that can be either true or false, but it is not necessarily testable in the same way that a hypothesis is 

• (differ from hypotheses): can be true or false, a statement that expresses a judgment or an opinion 

Scientific inquiry

• involves the application of the scientific method 

  • the scientific method “involves asking a question, research the question, making and testing a hypothesis, analyzing data, and documenting results

  • scientific method is systematic 

• central attention to causation

• Involves:

  • scientific method 

  • systematic in its approach 

  • testable 

  • rigorous (doing something to a high standard)  

• Limitations:

  • conducting research is not a simple process 

  • the information a researcher chooses to use (or collect), the method that they follow to investigate a research question, and the statistics (or analyses) used to report research findings may affect the conclusions drawn

    • data quality issues depending on the source 

    • experimental issues (eg. some phenomena are not conducive to experimentation) 

    • This is why research must be critically evaluated 

Causation

• The relationship between cause and effect; determining how one variable influences another.

• Two types of relationships:

  • Deterministic

    • Necessary conditions: B never occurs without A

    • Sufficient conditions: whenever A occurs, B will follow

  • Probabilistic

    • Whenever A occurs, the chance of B increases

• does not mean:

  • Complete explanation about the causal relationship of interest

  • Absence of outliers (i.e., cases that do not exhibit the same pattern as other cases–they may fall at the extremes)

  • The causal effect is observable in the majority of cases

• Must demonstrate 4 things

  1. Covariation

     • Two variables (i.e., independent and dependent variables) covary when certain values on the independent variable as associated with certain values on the dependent variable

     • Example: Amount of time spent studying and grade on final exam

  2. Time order

     • The cause is preceded by the effect and not the other way around

     • Example: Amount of time spent studying and grade on final exam

  3. Non-spuriousness

     • The covariation observed is not produced by a third variable (e.g., causal relationship between urbanization and birth rate is NOT produced by the number of storks)

  4. Theoretical support

     • There is a plausible and logical explanation that connects cause and effect

     • Example: Amount of time spent studying and grade on final exam

Normative inquiry

• Inquiry based on values and opinions, which is subjective and speculative.

  • Values, opinion based

  • Subjective (describe personal engagement with reality) 

  • Metaphysical (speculative, supernatural, imperceptible)

  • ”This table is ugly” 

  • “Canada should not have a senate”

Non-normative inquiry

• Inquiry based on objective facts and reality.

  • facts

  • objective (describes reality) 

  • real 

  • “this table has four legs” 

  • “Canada has a senate” 

• What science should be

Verification

The process of establishing the truth or accuracy of a statement or hypothesis.

Falsification

The process of proving a statement or hypothesis false based on empirical evidence.

Empiricism

• ”relying on observation to verify (or refute) propositions (or scientific hypotheses)” 

• ”modern political science relies heavily on one kind of knowledge: knowledge obtained through objective observation, experimentation and logical reasoning”

• ”The ultimate goal of scientific research, which is to always attained, is to use its results to construct theories that explain a political phenomena”

• assume that every study builds on existing research and, because of this, science progresses

• Observation is the key source of knowledge through verification and falsification

• They insist that observation is the source of all knowledge

• Key Assumptions:

  1. There is a reality out there, governed by law-like patterns

  2. Explanation is the identification of general and parsimonious causal relationships

  3. Observation is the key source of knowledge through verification and falsification

  4. Science should focus on facts and ignore ‘metaphysical’ values

  5. Findings must be transmissible and science will be cumulative

• Critiques:

  • “Is there really such a thing as the truth?”

    • They would say yes

    • “Most political scientists take reality pretty much as a given. That is, they posit that the objects they study – elections, wars, constitutions, government agencies – have an existence independent of observers and can be studied more or less objectively.”

Independent variable

A variable that is manipulated or changed to observe its effect on the dependent variable.

Dependent variable

A variable that is tested and measured in an experiment; it is expected to change when the independent variable is altered.

Operationalization

• The process of defining the measurement of a phenomenon that is not directly measurable.

• Specifying how a concept will be measured - developing a definition

• the last step from a hypothesis

• It involves at least 3 sets of choices:

  1. Which indicators (variables), and how many?

  2. Which values (attributes of variables), and how many?

  3. Will the approach be quantitative or qualitative?

• Sometimes, we may not be able to measure these complex terms in an objective and complete manner

  • we may, instead, opt to use different proxy measures, or measures that are a substitute for what we really want to measure but do not capture a complex concept completely, adds subjectivity to what we are studying

Validity

• Different from reliability

• corresponds to ‘true’ values

• Face validity: “When a measure appears to accurately measure the concept it is supposed to measure....a matter of judgement” (Johnson et al., 2020, p. 353)

• Content validity: “Involves determining the full domain or meaning of a particular concept and then making sure that all components of the meaning are included in the measure.” (Johnson et al., 2020, p. 351)

• Predictive validity: “ability of a test or other measurement to predict a future outcome.” (Scribbr, 2022, para 1)

  
  

  
  

Reliability

• refers to the stability of the measurement

• Test-re-test

• Increase number of observers

External validity

• The extent to which research findings can be generalized to settings outside the original study.

• “The ability to generalize from one set of research findings to other situations.” (p. 353)

• “In short, the results of a study have “high” external validity if they hold for the world outside of the experimental situation; they have low validity if they only apply to the laboratory.” (p. 129)

• Example: participants in a clinical (or laboratory) study are given a blood pressure medication to reduce their blood pressure. The results that occur in the laboratory also occur in non-laboratory conditions (high external validity)

  
  

Internal validity

• The degree to which a study accurately establishes a causal relationship between variables.

• “The ability to show that manipulation or variation of the independent [or causal variable] actually causes the dependent [or outcome variable] to change.” (p. 354)

• Example: participants in a clinical (or laboratory) study are given a blood pressure medication to reduce their blood pressure. The blood pressure medication given to participants (as opposed to any intervention) is shown to cause a reduction in the participant’s blood pressure. (high internal validity)

  
  

Longitudinal study

• Research that collects data from the same subjects repeatedly over a period of time.

• Comparisons across time

• every point in time is considered a case

• These designs “are characterized by the availability of measures of variables at different points in time.”

  • Example: effect of Pierre Poilievre’s comments to the media on voter’s perception of his [Poilievre’s] competence as a leader over time

  • Other examples that you can think of?

• Advantages:

  • “change in the level of variables or conditions can be measured and modeled.”

  • “it is sometimes easier to decide time order or which comes first, X or Y”

  • “they can in principle estimate three kinds of effects: age, period (history), and cohort.”

• Disadvantages:

  • “the researcher does not control the introduction of the independent variable(s)”

  • It is difficult, if not impossible, to collect new data about the past

  • No control group (difficult to isolate the effect of one IV)

  • Reliability is likely a challenge, meaning of indicators may change over time.

Cross-sectional study

• Research that collects data at a single point in time from multiple subjects.

• Comparisons across place at a single point in time

• “Perhaps the most common observation research design is cross-sectional analysis”

• “measurements of the independent variable are taken all at the same time or approximately the same time.” (snapshot)

• “the researcher does not control or manipulate

  • the independent variable,

  • the assignment of subjects to treatment or control groups, or

  • the conditions under which the independent variable is experienced.”

• “If the units of analysis are individuals, the study is often called a survey or poll.”

• “if the subjects are geographical entities, such as states or nations or other groupings of units, the term aggregate analysis is frequently applied”

• “attributes of the units are measured or observed, and the data recorded.”

  • What are attributes?

    
    

Case study

• In-depth analysis of a single case or multiple cases within a specific context.

• May or may not be comparisons across cases

• “the detailed examination of a single example of a class of phenomena” (Flyvberg, 2006, p. 220)

• Purposes of case studies (Johnson et al., 2020, p. 137)

  • Idiographic

    • “aim to describe, explain, or interpret a singular historical episode with no intention of generalizing beyond the case.”

    • What kinds of singular historical episodes could you chose to study?

    • Inductive: “lack an explicit theoretical perspective and simply have the purpose of describing all aspects of the case” (descriptive)

    • Theory-guided: “are explicitly structured by a well-developed conceptual framework”

    • Example: application of “Kingdoms “three streams” model of policy making to structure a description of the politics of a particular policy” (e.g., problem stream, policy stream, political stream, policy window)

  • Hypothesis generating

    • “examine or more cases for the purpose of developing more general more general theoretical propositions” that can be tested in future research.”

    • Example: “researchers might study several cases of conflicts between nations to identify the key factors that seem to have led either to the outbreak of war or to peaceful resolutions of the conflict.”

    • What other topics could you examine using hypothesis-generating case studies?

  • Hypothesis testing

    • “entail testing hypothesized empirical relationships.”

    • “These types of case studies include investigations of causal mechanisms...”

    • What topics could you examine using a hypothesis-testing case study?

• Non-explanatory:

  • Exploratory - can look at one case to generate a new hypothesis

  • Descriptive - can look at one case to get a “thicker” (or more in depth) description than what is possible in studies of multiple cases

  • Critiques:

    • Many scholars doubt the value or even possibility of detecting law-like patterns of reality from case studies

    • Response to these critiques:

      • Flyvberg (2006, p. 226): “generalization...is considerably overrated as the main source of scientific progress

      • In this perspective, narratives (or in-depth description, or account, of a single phenomenon) are a strength, not a weakness

• Advantages:

  • High internal validity (very accurate measurement of the case itself)

  • Context-dependent knowledge

• Disadvantages:

  • Low external validity (difficult to extrapolate findings to other contexts)

  • Replication is difficult

  • Danger of personal investment (blinders)

  • Comparisons are necessary for explanatory cases

    
    

Field research

• Research conducted in a natural setting where subjects reside, emphasizing observation.

• Mimics classic experiment outside the laboratory

• It randomly administers an experimental stimulus to similar groups in the real world

• Example field experiment research question: “Which method of informing voters about an upcoming election has the largest effect on their likelihood to come out and vote?”

• Key features:

  • In field research, researchers are physically present in the social setting they aim to understand

    • house of commons, cabinet meetings

  • Very common in descriptive, exploratory and explanatory research

  • Especially common in case studies or comparative case studies

• Advantages

  • Compared to interviewing and unobtrusive techniques, field research has several advantages:

    • Measures behavior more directly

    • Avoids reported accounts, of which the validity might be difficult to assess

    • ‘Being there’ enhances understanding and exposes more information

• Disadvantages

  • Limited to observable behavior (Why?)

  • Raises its own problems regarding reliability, validity, and ethics (How?)

  • Comparatively time-intensive (Why?)

• Subtypes

  • Any project that involves a researcher traveling to the context of study can be considered field research

  • Some specific subtypes:

    • Structured observation: quantitative approach aimed at reliable and systematic measurement (e.g. sports statistics)

    • Ethnography: qualitative approach, aimed at understanding ‘culture’ from insiders’ point of view (common in anthropology)

• Setting and identity of researcher

  • Setting: Is it publicly accessible (open) or not (closed)?

  • Researcher Identity: Does everyone know they are being researched (overt observation) or not (covert observation)?

• Overt and covert observations

  • These characteristics are central to a common trade-off between validity and ethics in field research

  • Overt observation is likely to produce measurement error

    • Reactivity

    • Direction and scope of measurement error can differ from one researcher to another

  • Covert observation overcomes these problems but raises ethical concerns

    • Unable to ensure voluntary and informed consent

    • Particularly problematic in closed settings

• Role of researcher

  • Most ethnographers are participant observers: they take part in the social processes they are trying to understand

  • Other field researchers maintain the role of complete observer: they strictly observe from the sidelines

  • The choice is partially philosophical–standpoint theorists tend to prefer participant observation

  • The choice has consequences for objectivity and reactivity

• Data collection

  • Most ethnographers attempt to write as comprehensive notes as possible

  • Other field researchers make more targeted and limited notes

    • Comprehensive notes might produce more unexpected findings

    • Targeted notes more feasible in deductive research

• Trade-Offs between reliability and validity

  • Decisions about the way we make observations (participant vs. complete observer, ‘comprehensive’ vs. targeted notes) raise another common trade-off: between reliability and validity

    • In-depth observation of complex social settings is very difficult to conduct reliably

    • Selective and incomplete observation

    • More partial and distant observation addresses these problems but raises concerns about measurement validity

• Alternative to the classic experiment

Qualitative research

• Research that explores subjective experiences and meanings through non-numerical data. (use words to measure concepts)

• Elements:

  • Outsider perspective

  • hypothesis-testing

  • external validity

  • transparency and replicability

  • danger of false sense of precision and accuracy

  • effects-of-causes approach

    • “starts with a potential cause and works forward to measure its impact on the outcome”

Quantitative research

• Research that involves numerical data and statistical analysis to identify patterns. (use numbers to measure concepts)

• Elements:

  • Insider perspective

  • hypothesis-generating

  • internal validity

  • credibility

  • danger of authority arguments and cherry-picking of evidence

  • causes-of-effects approach

    • “starts with an outcome (e.g., war, election results, passage of a major piece of legislation) and works backward toward the causes.”

      
      

Non-scientific inquiry

• Knowledge may be based on agreement, tradition or authority, personal experience even though it may not be true (partially, fully, or universally true)

• typically not systematic or rigourous

• Limitations:

  • misinformation (Trump)

  • selective attention to information or bias

  • historically, some claims based in traditions, or made by authorities, have been false

  • illogical reasoning

    
    

Survey research

• Method of data collection that consists of asking the same questions to many individuals in the exact same order

• It’s good to use when dealing with a large population. However it is harder to do more in-depth research because you can’t ask why someone wrote the answer they did. It is also difficult to ensure that the answers are honest and truthful.

• The quantitative approach to interviewing is survey research, which is ideal for gauging the views and beliefs of large groups of respondents

• Response patterns are heavily influenced by the nature of the survey

• When evaluating survey results, we should always investigate:

  • Sampling technique

  • Response rate

  • Question ordering

  • Question formulation

  • Survey mode(s)

• 4 common types (can be combined):

  1. In-person

  1. Mail

  2. Phone

  3. Web

• Advantages:

  • Best method to gauge attitudes and perceptions of a large population

• Disadvantages:

  • Social desirability bias

  • non-attitudes (when participants don’t care about the issue under study in the survey)

  • Improper reading of questions or careless completion of the survey questions

  • Subject to survey design issues (poor design = poor data)

  • Don’t provide any insights as to the reason so for the responses (superficial)

    
    

    
    

Content analysis

A research method for systematically analyzing texts and artifacts to identify patterns.

Triangulation

Combining multiple methods or data sources to enhance the credibility of research findings.

Critical theory

• An approach that aims to critique and change society rather than merely understanding it.

• the focus should be on improving society rather than just trying to understand (or explain) what is going on

• “the belief that a proper goal of social science is to critique to improve society (by making it more just and humane) rather than merely understand or explain what is going on

• Focuses on questions about “what out to be” rather than “what is”

• Alternative to empiricism

Critique of law-like patterns

• “Like any other organisms, humans are aware of their surroundings. They have the additional ability to empathize with others and frequently attempt to reach others’ minds”

• “human beings - individually, but especially jointly - are self-interpreting and reflective, capable of assigning meanings to their actions and revising these meanings recursively” (eg. through a process of repetition and extension or building)

Social desirability bias

The tendency of respondents to answer questions in a manner that will be viewed favorably by others.

Manipulation

Deliberate alteration of variables in experimental research to detect effects on the dependent variable.

Sampling

• Example:

  • Population: all university students in the province of Ontario (set of cases we want to make claims about) –1 million university students (hypothetical number)

  • Sampling frame: create a list of the names and contacts of all university students in Ontario

  • Sample: select 500 university students from the list of names and contacts (set of cases we end up analyzing)

• Goal of sampling

  • Our goal is to generalize findings from our sample to the population

    • e.g., what we find in our study sample will be found in the study population)

  • Sampling error will introduce bias (or an over-representation of a characteristic) into your study (e.g., an over-representation of fourth year university students in your sample)

    • Why is having an over-representation of fourth year university students in a study about voting behaviour a problem?

    • If your sample is biased, your research conclusions will be biased!

  • Important: Need to take steps to reduce sampling error because you want to reduce the likelihood of bias in your research conclusions

• Population sample: studying all cases in the population when feasible

• Probability (random) and non-probability (non-random) sampling

  • Probability (random samples):

    • Used when population is large and known

    • more common in qualitative research

    • generalization through inferential statistics

    • Probability (random) samples are preferred in descriptive and explanatory research of large populations

    • Avoids selection bias (e.g., of over-representation of 4th year university students in a study about the voting behaviour of all university students)

    • Allows for inferential statistics (POLS3650)

    • Statistics that use data from a study population to make conclusions about the data of a study population

    • Simple random sample: each case has an equal chance of being selected (create a numbered list of all cases and use a random number generator to select cases from that list)

    • Stratified sample: “elements [or cases] sharing one or more characteristics [(e.g., gender, level of education, income, year in university, etc.)] are grouped and elements [or cases] are selected from each group in proportion to the group’s representation in the total population.” (p. 357)

      • Example: study population of 1 million university students has 35% of students in year 1, 25% students in year 2, 20 % in year 3, and 20% in year 4. My study sample of 500 university students will have 35% of students in year 1, 25% of students in year 2, 20% of students in year 3, and 20% of students in year 4.

    • Cluster sample: “used when no list of elements exists.” (p. 351)

      • Select cluster at random

      • Then select cluster within selected cluster at random

      • Finally, select case within smallest cluster at random

      • Repeat

  • Non-probability (non-random samples):

    • Used when population is small or known

    • more common in qualitative research

    • generalization (if at all) through focused comparisons

    • Convenience sample: Study cases that are easily accessible

      • Example: I stand at the entrance of the library and ask students as they enter if they voted in the last Ontario provincial election

    • Purposive (or purposeful) sample: study cases that allow for meaningful comparisons (see next lectures)

      • Common in qualitative research because you purposefully want to study a set of cases that posses specific characteristics relevant to your research question

    • Snowball sample: each case suggests new cases to investigate

      • Example: I ask every student that participates in my study to provide me with the name and contact information of other students they know

    • Quota sample: The researcher sets quotas (proportions) based on key variables (e.g., age, gender, income level, education) to ensure these traits are represented in the sample.

      • Using known characteristics of the study population (e.g., 1 million university students in Ontario), the research establishes what the study population should like (e.g., the total number, and percentage of 1st, 2nd, 3rd, and 4th year university students in Ontario)

      • Cases are then selected by a non-probability (non-random) technique to resemble population characteristics (e.g., smaller number, but similar percentages, of 1st, 2nd, 3rd, and 4th year university students in Ontario)

        
        

Sampling error

The error caused by observing a sample instead of the whole population.

Qualitative interviews

• In-depth, one-on-one discussions that explore the participant's perspective.

• Key features

  • one-on-one discussions between a researcher and research participant

  • Much more detailed than surveys, and therefore much fewer respondents

  • Especially useful to investigate:

    • Internal explanations

    • Historical accounts

    • Motivations

  • Five aspects that have major implications

    1. Type of interviewees

    2. Structure of interview

       • structured interviews: have set question ordering and wording, allow for no improvising, and the researcher takes the lead (pros and cons?), interviewer takes the lead

         • pro: good for looking for patterns

         • con: don't know their thoughts on the topic

       • Unstructured interviews: vary from each other in questions, allow for much improvisation, and the respondent takes the lead (pros and cons?), interviewee takes the lead

    • An interview guide that contains all of the questions that will be asked is typically required by the REB regardless of the structure of the interview

    • The choice to conduct structured vs. unstructured interviews balances concerns about reliability, flexibility, and artificiality.

    3. Method of communicating - interviews can take place in-person, over the phone, or in online chat room

       • choice about method balances concerns about reactivity, expected length, response rate, costs (what is your preference?)

    4. Length of interview

       • Long interviews will result in more data (more time needed to transcribe and analyze)

       • Short interviews likely increase response rate, completion rate, and quality of answers (less time needed to transcribe and analyze)

    5. Interview questions

       • question ordering and formulation have large consequences

       • Start with warm-up questions, move from abstract to specific

       • avoid social desirability bias

       • adjust language to participants

       • it's important to pilot test your interview guide and interview questions before data collection.

• Strengths:

  • Avoids superficiality of surveys

  • great corroboration technique

  • Perfect for studying internal explanations

  • widely applicable

• Weaknesses:

  • Limited reliability

  • more artificial than observation

  • reactivity

  • imperfect and selective memory of interviewees

    
    

Secondary analysis

• The use of existing data collected by other researchers for new analyses.

• A method of unobtrusive research (the analysis of already existing data)

• Quantitative data collection

• Major advantages:

  • few concerns about reactivity, few ethical concerns (if collected ethically to start with), verity time efficient

• Major limitation:

  • researcher has no control over the nature and availability of data

• It is the ‘recycling’ of data compiled by others

  • Researchers perform new analyses on this data

  • Researchers need to understand the quality of the data compiled by others BEFORE performing their new analyses.

• Many data are available that you might expect - little point in collecting data that are already out there

  • Ethical implications to collecting already existing data unless there is a very good reason to do so

• Important considerations

  • What is the quality of the data?

    • What do we mean by this? Why determine this ahead of time?

  • Who compiled the data?

    • Does the person or organization collecting the data have a stake in the outcomes of the research?

      • People can create bias, measure incorrectly, word things specifically

      • How have the data been collected? How have the concepts been measured?

        • Why do you need to determine this ahead of time?

          • The way people measure their variables and define them can change the outcome

  • Are the data applicable to your research question?

    • Have all data been measured the same way? Why do you need to determine this ahead of time?

      
      

Research ethics

• The principles guiding conduct in research to protect participants and ensure integrity.

• “Ethics are moral obligations that guide us in determining whether a certain behavior is right or wrong”

• research subjects (or participants) have rights

• in the absence of ethics regulations, scientists have violated the rights of their subjects

• there can be obvious ethical violations with evil intent

• there can also be unintentional ethical violations

• today, research on human subjects is impossible without approval from ethics review boards

• technology advancement is the reason the legislation has to be revamped consistently

• In Canada, researchers that perform studies on humans have to complete the Tr-council policy statement: ethical conduct for research involving humans (TCPS2)

• Main ethical principles:

  • respect for persons

    • Individuals should be treated as autonomous agents (independently or without coercion)

    • Persons with diminished autonomy are entitled to protection

      • people with cognitive disabilities

        children

    • Consent

      • Informed consent - participant is provided with information about the study, including the risks and benefits

      • Implied consent – by completing an anonymous survey, a participant gives their consent to participate in the study

      • Debrief participants – inform participants of the real goal of the study, if deception was involved at the outset of the study when participants voluntarily agreed to take part in the study

        • tell participants after the fact, and they debrief you

        • the debriefing would then make you fully informed; participants are partially informed before going into it

    • Voluntary participation

      • not coerced

      • Active consent — typically a formal verbal or written agreement to participate

        • in the study after being informed about the risks and benefits of participation

      • Passive consent (e.g., a parent signing a form to allow their child to participate in research)

        • Exit option – ability to leave the study at any time without providing a reason

        • your consent can be withdrawn

  • beneficence

    • Do no harm

    • Persons are treated in an ethical manner not only be respecting their decisions and protecting them from harm, but also making efforts to secure their well-being

    • do no harm (i.e. no physical or mental risks, respect privacy when sharing findings, ensure confidentiality of responses and anonymity of participants

    • Maximize benefits and minimize harms

      • publish results to the news, media

      • keeping this anonymous

      • Don’t give too much demographic information

  • justice

    • instructs researchers to treat individuals fairly and equally” Ensure that the costs and benefits do not disproportionately affect one group of people over others

      • typical participants = white men

    • researchers should not be undertaken on vulnerable populations because they are convenient or because the researcher has easy access

      • persons of colour, children, etc.

    • “… research is about strive to involve participant groups who are likely to benefit from the findings, rather than focusing on groups unlikely to benefit .”

      • Henrietta, 1950s found the cure for polio and covid

• some researchers argue that observing ethical obligations may have methodological disadvantages

• Observing the principle of beneficence (do no harm) might make it difficult to study hurtful things

• observing the principle of voluntary participation results in studying only people that want to be studied

• Observing the principle of informed participation likely leads to reactivity, the phenomenon that people respond to being studied

• Usually research ethics boards (REBs) allow small deviations for methodological purposes if the research has the promise to yield large social benefits

• A different set of ethical principles that focus on maintaining professional (academic) integrity in the conduct of research

  • proper referencing, citation, and acknowledgement

  • disclosure or potential conflicts of interest.

  • Data accessibility and research transparency

  • Honest presentation of findings

    
    

Steps in a research question

1. Formulation of question

2. Theorizing and hypothesis formulation

3. Operationalization

4. Choice of research design and case selection

5. Data collection

6. Data analysis

7. DIssemination of findings

Literature Review

• Purpose:

  • “see what has and has not been investigated” (about what?)

  • “to develop general explanations for observed variations [or differences] in a behavior or phenomenon” (what differences?)

  • “to identify potential relationships between concepts and to identify researchable hypotheses”

  • Possible hypotheses for the research question?

  • “learn how others have defined and measured key concepts” (what concepts?)

  • “identify data sources that other researchers have used” (what data sources?)

  • “develop alternative research designs” (more information on this later in the course)

  • “discover how a research project is related to the work of others”

• Finding a subject:

  • Consult literature

  • Peer-reviewed sources. What does this mean?

  • Journal articles in social science citation index

  • Books of academic publishers

  • Editor-reviewed sources

  • Academic books and journals with an editorial committee

• Identify appropriate electronic scientific databases to search for high quality sources

  • Google Scholar–widely available

  • University of Guelph Library

  • Scientific articles: Web of Science (multi-disciplinary research) – journal database or JSTOR

  • Books: Use Omni search engine

• You do not need ethics approval to conduct this

• “Once you have identified references for possible inclusion in a literature review, the next step is to figure out how the references fit together in a way that…

  1. explains the base of knowledge, or what we know about a topic from previous work, with respect to the research question, and

  2. establishes how the current project is going to build on that knowledge.

Level and type of research related risks

• Different research subjects (or participants) are faced with different ethical risks

  • no risks when there are no human subjects involved (eg. examining party manifestos)

  • Large risks when human subjects are vulnerable or sensitive (eg. interviews with war victims)

  • Large risks when researchers use manipulation (eg. experimental research)

    • examining party manifestos

• large risks when human subjects are vulnerable or sensitive

  • interviews with war victims

  • Indigenous communities

  • children

  • LGBTQ+

  • People with disabilities

  • elderly

Reactivity

The phenomenon where participants alter their behavior because they are being observed.

Observable vs. Non-observable parts of reality

• “the distinction here (in scientific realism) between the observable and the unobservable reflects human sensory capabilities: the observable is that which can, under favorable conditions, be perceived using the unaided senses (eg. planets and platypuses)

• the unobservable is that which cannot be detected this way (eg. proteins and protons)

• scientific beliefs of observability “generally extend to things that are detectable using instruments”

• Scientific realists argue reality consists of both

  • observation cannot lead to unobservable parts of reality

    
    

Value-free observation

• What are facts: something that can be proven

  • something that is known to have happened or to exist, especially something for which proof exists, or about which there is information

• What are values: what we believe is right or wrong

  • the beliefs people have, especially about what is right and wrong and what is most important in life, that control their behavior

• Many doubt that science can really be value-free:

  • Researchers' values influence their research projects

    • many researchers believe theories cannot be conducted without some impact from personal values

• Researchers’ standpoints (position in society) influence their research projects. What is meant by position in society

  • Pro insider: You can only truly understand something that you have experienced. Do you agree or disagree with this statement

  • Pro outsider: it is easier to be objective about something in which you are not personally involved. Do you agree or disagree with this statement

  • Social standing create bias in observations

• Many doubt that political science should be value free

• Empiricists assume there is a fundamental distinction between facts and values, and insist that science can only make claims about facts

Overt observation

A research method where participants are aware they are being studied.

Covert observation

A research method where participants are unaware they are being studied.

Manifest content

• The literal meaning of communication elements, straightforward and easily quantified.

• the literal content of a message

Latent content

The underlying meaning of communication elements, requiring interpretive analysis.

Method of difference

A comparative research technique identifying what is different between cases to infer causation.

Process tracing

• a qualitative research method used to examine causal mechanisms in detail by analyzing the sequence of events or processes that link a cause to an observed outcome.

• It seeks to uncover how and why something happened by tracing the steps in a process.

• “refers to case studies that “explicitly unpack mechanisms and engage in detailed empirical tracing of them”

• “use deductive reasoning and ask, “If an explanation is true, what would be the specific process leading to the outcome?

• “often involve only one case because of the copious amount of information and detail that is required to trace a causal mechanism and to show that rival explanations do not account for an outcome.”

• “depends on logic and has been compared to a detective sifting through evidence in order to solve a mystery.”

  
  

Unethical experiments in the US

• Monster study: “Orphaned children with normal speech patterns were told they had poor speech, including a stutter” (…) “developed stutters and suffered negative psychological and behavioral effects”

• Milgram obedience experiments: “Study participants were told to shock a “learner” for incorrect answers; however, the study participants did not know that the learner was not a real person, but rather a recording. After each shock, the participant was instructed to increase the voltage of the next shock, despite the learner’s call for them to stop”

Positivism

A philosophical theory that asserts that knowledge is primarily derived from scientific evidence.

Research goals

• Explanation:

  • advancement of theory-based account or why a situation or event has occurred (cause and effect)

  • Aims to test existing ideas systematically

  • Examples (George & Merkus, 2023):

    • “Why do undergraduate students obtain higher average grades in the first semester than in the second semester?”

    • “How does marital status affect labor market participation?”

• Exploration

  • tentative investigation aimed at generating new ideas about a new and understudied subject

• Description

  • systematic and exhaustive record of one group of cases (or phenomenon) or one category of things

  • Examples (McCombes, 2023):

    • How has the Guelph “housing market changed over the past 20 years?”

    • “What are the most popular online news sources among under-18s?”

      
      

Non-observable phenomenon

Events or factors that cannot be directly seen or measured, such as beliefs or attitudes.

Delimitation

Defining the boundaries of a research study or project.

Scope of study

The extent of a research project including its limitations and coverage.

Hypothesis testing

The process of testing an existing hypothesis based on collected data.

Data analysis

The systematic application of statistical or logical techniques to describe, summarize, and compare data.

Scientific knowledge

• Verification and falsification 

  • “scientific knowledge depends on verification” 

    • “our acceptance or rejections of a statement regarding something “known” must be influenced by observation”

      • A causes B

    • Falsifiability is another “key characteristic of scientific claims, meaning that statements or hypotheses can in principle be rejected in the face of contravening empirical evidence” 

      • A does not (always) cause B

• Must be:

  • transmissible 

    • “Scientific knowledge must be transmissible - that is, the methods used in making scientific discoveries must be made transparent so that other can analyze and replicate findings” (Replicability) 

      • There is a debate among political scientists as to whether they should be compelled to share their data, analytic techniques, and how they drew their conclusions 

        • costs time, money, intellectual property, bias/unethical 

  • cumulative 

    • “substantive (or considerable) findings and research techniques are built upon those of prior studies” 

    • “The process of constantly testing and refining prior research produces an accumulated body of knowledge” 

  • explanatory

    • Scientific knowledge “provides a systematic, empirically verified understanding of why a phenomenon occurs” 

    • A conclusion can be derived (logically) from a set of general propositions and specific initial conditions 

    • The general propositions asser that when things of type X occur, they will be followed by things of type Y 

      • eg. US “states with easier voter registration systems have higher election turnout rates than do states with more difficult systems.”

      • X = easier voter registration systems, Y = higher election turnout rates

      • Important: explanations do not need to “explain or predict a phenomenon with 100 percent accuracy.”

  • parsimonious

    • parsimony = simplicity

    • Scientific explanations that rely on a few explanatory factors are generally preferred or those with a lot of explanatory factors

    • Ex. Why do some people trust and follow authoritarian leaders

      1. Immediate personal, social and economic conditions of the individual

      2. All of the factors in explanation 1 plus deep-seated psychological states stemming from traumatic childhood experiences

      
      

Cumulative knowledge

The building of knowledge upon previous findings and theories.

Transparency in research

The principle of sharing methodologies and data for verification and replicability.

Constructionism

• the truth is subjective because social reality is subjective

• reject the idea of an absolute truth, they insist all we can learn is how different people give meaning to the social world

• explanation involves the identification of:

  • Intersubjective meanings: understandings of social reality that are shared by multiple people and that, in turn, structure (or shape) social reality

  • Internal explanations: explanations of human behaviour that are offered by actors themselves

Intuition in research

Relying on one's gut feelings or instinct rather than systematic analysis.

Unit of analysis

The major entity that is being analyzed in a study, such as individuals, groups, or institutions.

Selection bias

The tendency for certain individuals to be selected for a study while others are not, leading to unrepresentative samples.

Theory development

The process of creating new theories based on empirical observations and existing knowledge.

Statistical significance

A statistical statement of how likely it is that an obtained result occurred by chance.

Primary data

Data collected firsthand for a specific research purpose.

Secondary data

Data that was collected previously by someone else and used for current research.

Random sampling

A sampling method where every individual in a population has an equal chance of being selected.

Non-random sampling

A sampling method that does not give every individual in a population an equal chance of selection.

Nominal scale

A measurement scale that indicates qualitative differences without order.

Ordinal scale

A measurement scale that indicates rank or order of items without specifying the distance between them.

Interval scale

A measurement scale that indicates not only order but also the exact differences between values.

Ratio scale

A measurement scale that possesses all the properties of an interval scale, and also has a true zero point.

Focus group

A qualitative data collection method involving guided group discussions.

Thematic analysis

A method for identifying, analyzing, and reporting patterns within qualitative data.

Intercoder reliability

The level of agreement among multiple coders analyzing the same data set.

Reflexivity in research

The awareness and consideration of the impact of the researcher's perspectives on the research process.

Document analysis

A research method involving the examination and interpretation of documents.

Simple random sample

• A probability sampling method

• each case has an equal chance of being selected (create a numbered list of all cases and use a random number generator to select cases from that list)

Contingency table

A type of table that displays the frequency distribution of variables.

Chi-square test

A statistical test used to determine if there is a significant association between categorical variables.

Stratified sample

• A probability sampling method

• “elements [or cases] sharing one or more characteristics [(e.g., gender, level of education, income, year in university, etc.)] are grouped and elements [or cases] are selected from each group in proportion to the group’s representation in the total population.” (p. 357)

  • Example: study population of 1 million university students has 35% of students in year 1, 25% students in year 2, 20 % in year 3, and 20% in year 4. My study sample of 500 university students will have 35% of students in year 1, 25% of students in year 2, 20% of students in year 3, and 20% of students in year 4.

Cluster sample

• a probability sampling method

• “used when no list of elements exists.” (p. 351)

  • Select cluster at random

  • Then select cluster within selected cluster at random

  • Finally, select case within smallest cluster at random

  • Repeat

Convenience sample

• A non-probability sampling method

• Study cases that are easily accessible

  • Example: I stand at the entrance of the library and ask students as they enter if they voted in the last Ontario provincial election

Purposive (or purposeful) sample

• A non-probability sampling method

• study cases that allow for meaningful comparisons (see next lectures)

  • Common in qualitative research because you purposefully want to study a set of cases that posses specific characteristics relevant to your research question

Snowball sample

• A non-probability sampling method

• each case suggests new cases to investigate

  • Example: I ask every student that participates in my study to provide me with the name and contact information of other students they know

Quota sample

• A non-probability sampling method

• The researcher sets quotas (proportions) based on key variables (e.g., age, gender, income level, education) to ensure these traits are represented in the sample.

  • Using knon characteristics of the study population (e.g., 1 million university students in Ontario), the research establishes what the study population should like (e.g., the total number, and percentage of 1st, 2nd, 3rd, and 4th year university students in Ontario)

  • Cases are then selected by a non-probability (non-random) technique to resemble population characteristics (e.g., smaller number, but similar percentages, of 1st, 2nd, 3rd, and 4th year university students in Ontario)

The classic experiment

• developed for testing causal claims

• It inspired most other research designs

  • It is not the most common, but functions as a model for explanatory research

• Comparisons

  • The experiment only makes sense if we can make reasonable comparisons between the 2 groups (i.e., experimental and control groups)

  • The 2 groups need to be similar in their:

    • initial value of the DV

    • expected reaction to the stimulus

  • To achieve this, we assign participants to groups by randomization

    • What is randomization?

• Causality

  • Establishes:

    • Covariation: compare experimental vs control group, pre-test vs post-test

    • Time order: pretest, stimulus, posttest

    • Non-spuriousness: manipulated comparisons ensure only difference is the stimulus

  • It is not important whether the total group of experimental subjects is representative of the total population!

    
    

    
    

Alternatives to the classic experiment

• Variations on experimental design

  • Field experiments in a natural design

  • Natural experiments

  • Quasi experiments

  • Observational studies

Quasi experiments

• Alternative to the classic experiment

• Like field experiments, these experiments rely on the logic of administering a stimulus to different groups outside of the laboratory

• In sharp contrast, however, the researchers cannot be sure that the groups are randomized

• Example: “Do television debates affect voting decisions?” Aalberg and Jenssen exposed some graduate students to a panel debate for the 2001 Norwegian elections and others to non-political entertainment.

  
  

observational studies

• “used to describe designs in which the researcher neither manipulates experimental variables nor randomly assigns subjects to treatment”

• The researcher “merely observes causal sequences and covariations.”

• “Cross-sectional designs and longitudinal designs are two frequently used observation research designs.”

• Challenges:

  • When we step out of the laboratory, our data are observational rather than experimental. This introduces 2 challenges

    • No control over the values of the independent variable (we do not administer the stimulus)

    • No control over the allocation of groups (we cannot ensure randomization)

  • The implications of this design are important

    • Inferring causality becomes more difficult (in particular, more difficult to establish time order and non-spuriousness)

    • Case selection becomes more important

  • When there are many cases, we can address these challenges with statistics

    • Probability (random) samples cancel out values on variables in which we are not interested

    • Multivariate analyses allow us to ‘control’ for (or hold constant) third variables (see POLS3650)

Age effects

• “can be considered a direct measure of (chronological) time and be assessed like other variables”

• Example: “an investigator may be interested in the effect of age on political predispositions or ideology. (It is commonly asserted that as people age, they become more politically conservative.)

Period (history) effect

• “a period (interval of time) may be thought of as an indicator of history during a period, and the consequences on individuals are period effects. It is the “history” that occurs during the period, not chronological age that matters.

• Example: 1960’s and 1970’s – “events such as Watergate and the Vietnam War adversely affected many citizens’ trust in government, whether they were young or old.”

Cohort effects

• “A cohort is defined as a group of people who all experience a significant event in roughly the same time.”

• “A birth cohort, for instance, consists of those born in a given year or period” (e.g., people born in the year 2002, or baby boomers)

• “an “event” cohort is those who shared a common experience, such as their first entry into the labor force at a particular time.”

• “It is often hypothesized that individuals in one cohort will, because of their shared background, behave differently than individuals in a different cohort.” For example, “people born in the years immediately after World War II (the baby boomers) may have different political attitudes and affiliations than those who were born in the 1980s.”

Comparative designs

• Often the term ‘case’ is used for investigations of a single country that is, in fact, more than one case

• Longitudinal studies

• Comparative case studies - method of difference

  • “the researcher selects cases in which the outcomes differ, compares the cases looking for the single factor that the cases do not have in common, and concludes that this factor is “the effect, or cause, or a necessary part of the cause, of the phenomenon.”

  • “applies to situations where the researcher is investigating outcomes that vary in degree (e.g., high, medium, and low levels of an outcome and identifies a factor that also varies in degree”

Ordering of questions

• How we order questions has major consequences for how participants respond to them

• Best practices:

  • Alternate direction of questions. Why?

  • Move from general to specific questions

    • Avoid priming certain conditions over others

    • priming: persuading the answer

• To avoid random measurement error:

  • Keep the questions and answers short and simple

  • Be precise (as opposed to vague) in terms of your wording

    • How would you rate the government’s current performance?

    • How would you rate the Ontario government’s current performance on the issue of climate change

  • Avoid double-barreled questions (what are these)

• To avoid systematic measurement error (bias):

  • Avoid the inclusion of authorities or experts in question

  • Avoid argumentative questions

    
    

Pilot testing surveys

• Involves administering the survey to a small group of individuals that fit your target population (2 to 3)

• These individuals complete the survey and provide important feedback to the researcher on the design and comprehension of the survey

• Occurs before the widespread deployment of the actual survey

• Allows the researcher to modify they survey before its wide-spread deployment to the sample

  
  

Content analysis

• Quantitative data collection

• Is the study of recorded communication

• Do not confuse it with a literature review

• Investigates primary material (I.e ., First hand material). for other researchers or studies

• Any record communication can be included (Written, Verbal, Non-verbal…)

• Political attitudes and beliefs

  • Considerations for assessing the validity of political attitudes and beliefs

  • who is the author and who is the intended audience

  • does the author have a reason to lie/withhold information or exaggerate/ embellish

• Discourse

  • discourse focuses on the structure of political or public communication

  • there is a link between language and the way we view the world, and that politicians manipulate this for their own ends

  • it is argued that control and domination of representations allows politicians to generate worldviews consistent with their goals and to downgrade negate or eliminate alternative representations

• the quantitative approach is to quantify words, phrases and or other elements can see how often words are being repeated

• words like illegal or phrases like democracy is at stake

• especially appropriate in a deductive study (i.e one that tests theory or hypotheses) that aims to maximize reliability of measurement

• Manifest vs Latent Content

  • Manifest (or surface level) content is easier to quantify than latent (hidden) content

    • “In manifest content analysis, context is derived from the visible and literal meaning of the words—taken at face value.” (Delve, n.d., para 13)

    • Question: Is the literal meaning of words always easy to derive or consistently derived across multiple persons?

    • “In latent content analysis, you apply a deeper, interpretive analysis that seeks to infer underlying meaning from the words or phrases you choose to analyze.” (Delve, n.d., para 13)

• Strengths

  • No reactivity

  • Not very costly in terms of time and money

  • Easy to replicate, especially, when analyzing public communication

  • Few ethical concerns; especially, when analyzing public communication

  • Well-suited for longitudinal research designs

• Weaknesses

  • No control over nature and availability of data

  • Can be difficult to distinguish truthful from untruthful statements

  • Measurements and analyses can be difficult

  • **also have to be concerned with AI- generated communications and edited or manipulated communications

    
    

How to decrease reactivity during qualitative interviews

• Ensure you are in a quiet and private location

• Consider your appearance: gender, race, attire

• Consider cultural conventions

• Emphasize how valuable respondent’s views are to you

• Use probing questions on short answers

• Employ the awkward silence

• Minimize interruptions (Don’t interrupt the interviewee while they are talking!)

• Stay neutral in terms of your verbal and non-verbal communication

  
  

Documenting qualitative interviews

• Taking notes (pros and cons?)

• Audio recording (pros and cons?)

• Video recording (pros and cons?)

• No obvious best technique: each has distinct implications for reactivity, the observation of non-verbal cues, and accuracy/comprehensiveness

• Which method of documentation do you prefer? Why?

• After each qualitative interview is complete

• Once the interview is over, write/generate transcripts: detailed (if possible complete) minutes of the interview. This can be very time consuming and costly.

  • Send transcript to respondent

  • Reduces ethical risks (guarantees informed participation)

  • Opportunity for additional validation

  • You may also want to document important ideas, concerns, or thoughts that come to you about the interview or data while generating transcripts.