POLS 2650 Lecture Notes - Exam
Lecture 2 - Sept. 11th
Why political scientists conduct research
To learn about important political phenomena
Some examples of important political phenomena provided in text:
“why women make up a larger percentage of legislators in some countries than in others” (Rwanda has the highest percentage of women legislators)
“how some nations exercise power over others” (US, China)
“how political institutions function”
eg. “does the use of non-profit service organizations to deliver public services change government control of and accountability of those services”
eg. home related care, some instances the UN
Empirical research is “a methodology that requires scholars to clearly state hypotheses or propositions that can be evaluated with actual “objective” observation of political phenomena”
Hypothesis: an educated guess or a proposed explanation for a phenomenon. It is a statement that can be tested through experimentation or observation
Proposition: a statement that expresses a judgment or an option. It is a claim that can be either true or false, but it is not necessarily testable in the same way that a hypothesis is
Scientific inquiry
involves the application of the scientific method
the scientific method “involves asking a question, research the question, making and testing a hypothesis, analyzing data, and documenting results
scientific method is systematic
central attention to causation
Scientific inquiry vs. non-scientific inquiry
Scientific inquiry:
scientific method
systematic in its approach
testable
rigourous (doing something to a high standard)
Non-scientific inquiry:
Knowledge may be based on agreement, tradition or authority, personal experience even though it may not be true (partially, fully, or universally true)
typically not systematic or rigourous
Limitations of non-scientific inquiry:
misinformation (Trump)
selective attention to information or bias
historically, some claims based in traditions, or made by authorities, have been false
illogical reasoning
Does science always lead to the truth
science does not always lead to the right answer or the truth, but it does rely on conscious strategies designed to reduce the dangers of relying on agreement and experience
Science is a method for acquiring knowledge
it involved the use of rigorous standards of observation and illogical reasoning to investigate a phenomenon in a systematic and objective manner - helps to reduce bias but may not eliminate bias completely
the difference between scientific and non-scientific inquiry is the use of research methods
Limitations of scientific inquiry
conducting research is not a simple process
the information a researcher chooses to use (or collect), the method that they follow to investigate a research question, and the statistics (or analyses) used to report research findings may affect the conclusions drawn
data quality issues depending on the source
experimental issues (eg. some phenomena are not conducive to experimentation)
This is why research must be critically evaluated
Lecture 3 - Sept. 16th
Empirical research: research based on actual, “objective” observation of a phenomenon
Empiricism
”relying on observation to verify (or refute) propositions (or scientific hypotheses)”
Propositions (differ from hypotheses): can be true or false, a statement that expresses a judgment or an opinion
Example:
“Positive (political) campaigns lead to higher voter turnout than do those that are characterized by mudslinging and name-calling”
Predictor variable: Whether the campaign is positive or negative
Outcome variable: Voter turnout
US “states with easier voter registration system have higher election turnout rates than do states with more difficult systems”
Predictor variable: Difficulty of voter registration systems
outcome variable: election turnout rates
On the EXAM we will have to identify the the independent and dependent variables
Elements of empiricism
”modern political science relies heavily on one kind of knowledge: knowledge obtained through objective observation, experimentation and logical reasoning”
”The ultimate goal of scientific research, which is to always attained, is to use its results to construct theories that explain a political phenomena”
Verification and falsification
“scientific knowledge depends on verification”
“our acceptance or rejections of a statement regarding something “known” must be influenced by observation”
A causes B
Falsifiability is another “key characteristic of scientific claims, meaning that statements or hypotheses can in principle be rejected in the face of contravening empirical evidence”
A does not (always) cause B
Normative vs. Non-normative - ON EXAM
Normative: (values, opinion based)
Subjective (describe personal engagement with reality)
Metaphysical (speculative, supernatural, imperceptible)
”This table is ugly”
“Canada should not have a senate”
Non-normative: (facts)
objective (describes reality)
real
“this table has four legs”
“Canada has a senate”
Most scientists would agree that science is, or should try to be, a non-normative entreprise
Scientific knowledge is transmissible
“Scientific knowledge must be transmissible - that is, the methods used in making scientific discoveries must be made transparent so that other can analyze and replicate findings” (Replicability)
There is a debate among political scientists as to whether they should be compelled to share their data, analytic techniques, and how they drew their conclusions
costs time, money, intellectual property, bias/unethical
Scientific knowledge is cumulative
“substantive (or considerable) findings and research techniques are built upon those of prior studies”
“The process of constantly testing and refining prior research produces an accumulated body of knowledge”
Scientific knowledge is explanatory
Scientific knowledge “provides a systematic, empirically verified understanding of why a phenomenon occurs”
A conclusion can be derived (logically) from a set of general propositions and specific initial conditions
The general propositions asser that when things of type X occur, they will be followed by things of type Y
eg. US “states with easier voter registration systems have higher election turnout rates than do states with more difficult systems.”
X = easier voter registration systems, Y = higher election turnout rates
Important: explanations do not need to “explain or predict a phenomenon with 100 percent accuracy.”
Scientific Knowledge is parsimonious
parsimony = simplicity
Scientific explanations that rely on a few explanatory factors are generally preferred or those with a lot of explanatory factors
Ex. Why do some people trust and follow authoritarian leaders
Immediate personal, social and economic conditions of the individual
All of the factors in explanation 1 plus deep-seated psychological states stemming from traumatic childhood experiences
Theory
Plays an important role in research
The accumulation of observed relationships sometimes leads to the creation of a theory that is, a body of statements that systemize knowledge of and explain relationships between phenomena. The accumulation of observed relationships sometimes leads to the creation of a theory - that is, a body of statements that systemize knowledge of and explain relationships between phenomena.
Overview of empirical research process
Identification of an idea or problem to solve
“Is [voter] turnout among evangelicals [Christians] higher in elections where there are distance differences between candidates on moral issues than in elections where the differences are small?” (e.g., moral issues – same sex marriage, abortion, etc.)
Hypothesis formation
“Evangelical christian’s are more likely than others to base their votes on candidate’s positions on moral issues”
Data collection
“We need to clearly define the concepts of moral values and evangelical Christians”
Create and administer a survey or poll that contains relevant questions
Interpretation and decision
“determine whether or not the observed results are consistent with the hypotheses”
“judging how well data support scientific hypotheses is usually not an easy matter”
Modification and extension
“depending on the outcome of the test, one can tentatively accept, and and on, or modify the hypothesis”
Rejection of a hypothesis can be “both interesting and beneficial”
Lecture 4 - Sept. 18
Assignment 1: personal pronouns are ok
Recall: Key Assumptions of Empiricism
There is a reality out there, governed by law-like patterns
Explanation is the identification of general and parsimonious causal relationships
Observation is the key source of knowledge through verification and falsification
Science should focus on facts and ignore ‘metaphysical’ values
Findings must be transmissible and science will be cumulative
Critiques of empiricism
“Is there really such a thing as the truth?”
Empiricists say yes
“Most political scientists take reality pretty much as a given. That is, they posit that the objects they study – elections, wars, constitutions, government agencies – have an existence independent of observers and can be studied more or less objectively.”
Constructionism - an alternate perspective to empiricism
“Humans do not simply discover knowledge of the real world through neutral processes, such as experimentation or unbiased observation; rather they create the reality they analyze” (Do we agree or disagree)
Example by Johnson et al.
“Consider the term Democratic Party”
citizens behave as if it exists
Instead of having an independent, material existence like an election or a strand of DNA, a political party exists only because citizens behave as if it exists”
“This means that two individuals that come from different social, historical, and cultural backgrounds may not comprehend and respond to the term in the same way”
In other words, reality is subjectively created
The problem of definition and measurement
In lecture 3, we sought to define the terms upper class, lower class, and political power. What happened?
We found a lack of general agreement as to how to best define these complex terms.
Measurement of these 3 terms can also be tricky
we may not be able to measure these complex terms in an objective and complete manner
we may, instead, opt to use different proxy measures, or measures that are a substitute for what we really want to measure but do not capture a complex concept completely, adds subjectivity to what we are studying
Critique of law-like patterns: self-reflection and individuality
“Like any other organisms, humans are aware of their surroundings. They have the additional ability to empathize with others and frequently attempt to reach others’ minds”
“human beings - individually, but especially jointly - are self-interpreting and reflective, capable of assigning meanings to their actions and revising these meanings recursively” (eg. through a process of repetition and extension or building)
examples of when and where meanings of our actions may morph?
The study of voting behavior
“Consider a political scientist who wants to investigate the effects of negative campaign advertising on attitudes”
“Suppose that Jane and Mary are subjects in a study”
“We cannot assume that they will react to experiment stimuli (eg. negative campaign advertising) exactly the sample way, even though they are the same age, gender, political persuasion, and so forth”
“Social scientists have to get around this problem by using groups or samples of individuals and then examining the average effect of the stimulus”
what do we know about averages
Outliers can greatly skew the average, which creates a problem on relying on averages
“The objects political scientists study are multifaceted and conscious beings with volition of their own who often change opinions and behaviors; thus, statements about them must necessarily be tentative, general, and time bound.”
Given the reliance on averages by political scientists, and the complexity and changing nature of human behavior, it begs the question: do law-like patterns really apply to human behavior?
Alternative to identifying patterns
Since constructionists reject the idea of an absolute truth, they insist all we can learn is how different people give meaning to the social world
From the constructionist perspective, explanation involves the identification of:
Intersubjective meanings: understandings of social reality that are shared by multiple people and that, in turn, structure (or shape) social reality
Internal explanations: explanations of human behavior that are offered by actors themselves
Can you think of an example?
Other scientists do not reject the existence of an absolute truth, but question where identifying patterns is the most worthwhile exercise
What if we are interested in things we cannot observe?
What if we are interested in things of which there are only a few instances?
In this perspective, explanation is much more demanding
Use inference to arrive at the best explanation: go back and forth between theory and observation to establish which explanation is best among available alternatives
Discover causal mechanisms: the process by which one things leads to another
Alternative to empiricism - Critical Theory
Critical theory - “the belief that a proper goal of social science is to critique to improve society (by making it more just and humane) rather than merely understand or explain what is going on
Focuses on questions about “what out to be” rather than “what is”
Approaches
Empiricism: truth is objective
Constructionism: the truth is subjective because social reality is subjective
Critical theory: the focus should be on improving society rather than just trying to understand (or explain) what is going on
Sept. 23rd
Recall, key assumptions of empiricism:
There is a reality out there, governed by law-like patterns
Explanation is the identification of general and parsimonious causal relationships
Observation is the key source of knowledge through verification and falsification
Science should focus on facts and ignore ‘metaphysical’ values
Findings must be transmissible and science will be cumulative
Assumption: Observation is the key source of knowledge through verification and falsification
Empiricists insist that observation is the source of all knowledge
Scientific realists argue that reality consists of observable and non-observable parts
observation cannot lead to knowledge about unobservable parts of reality
What are some non-observable parts of reality
Observable vs. Non-observable parts of reality
“the distinction here (in scientific realism) between the observable and the unobservable reflects human sensory capabilities: the observable is that which can, under favorable conditions, be perceived using the unaided senses (eg. planets and platypuses)
the unobservable is that which cannot be detected this way (eg. proteins and protons)
scientific beliefs of observability “generally extend to things that are detectable using instruments”
Assumption: Value-free Observation
Empiricists assume there is a fundamental distinction between facts and values, and insist that science can only make claims about facts
What are facts: something that can be proven
something that is known to have happened or to exist, especially something for which proof exists, or about which there is information
What are values: what we believe is right or wrong
the beliefs people have, especially about what is right and wrong and what is most important in life, that control their behavior
Many doubt that science can really be value-free:
Researchers' values influence their research projects? How?
many researchers believe theories cannot be conducted without some impact from personal values
Researchers’ standpoints (position in society) influence their research projects. What is meant by position in society
Pro insider: You can only truly understand something that you have experienced. Do you agree or disagree with this statement
Pro outsider: it is easier to be objective about something in which you are not personally involved. Do you agree or disagree with this statement
Social standing create bias in observations
Many doubt that political science should be value free
Assumption: Cumulative Science
Empiricists assume that every study builds on existing research and, because of this, science progresses
However, what researchers already (think they) know influences their research!
every observation we make we relate to previous knowledge and observations
Rosenthal effect: research expectations shape research findings
Do you agree or disagree with this statement?
Scientific revolutions
Kuhn (1962) argued that scientific research takes place within context-specific paradigms, which are abandoned during times of scientific revolutions
Paradigms are “a set of assumptions governing how we interact and interpret the world.... Paradigms need constant reinforcement to function. If events occur that cannot be explained by the current paradigm, a new one may be generated.” (Kestrana, 2022, para 3)
Can you think of an example when an existing paradigm was abandoned or a new paradigm created?
Collapse and creation of paradigms
“Scientific paradigms often endure a long time before they are replaced.”
“ For example, one of the earliest descriptions of the universe, Aristotle and Plato's "Two-sphere Universe," lasted about 550 years. This was followed by the Ptolemaic paradigm which lasted even longer before it was replaced by Copernicus, Kepler, and Newton's theories.”
“Since then, as modern technology facilitates greater and faster communication between scientists, paradigms appear and collapse at a faster rate.”
Research ethics
have you participated in a ethics board
Definition of ethics
“the discipline dealing with what is good and bad; a set of moral issues or aspects”
“Ethics are moral obligations that guide us in determining whether a certain behavior is right or wrong”
Why research ethics?
research subjects (or participants) have rights
in the absence of ethics regulations, scientists have violated the rights of their subjects
there can be obvious ethical violations with evil intent
there can also be unintentional (?) ethical violations
September 25
Unethical experiments in the US
Monster study: “Orphaned children with normal speech patterns were told they had poor speech, including a stutter” (…) “developed stutters and suffered negative psychological and behavioral effects”
Milgram obedience experiments: “Study participants were told to shock a “learner” for incorrect answers; however, the study participants did not know that the learner was not a real person, but rather a recording. After each shock, the participant was instructed to increase the voltage of the next shock, despite the learner’s call for them to stop”
Level and types of research-related risk
Different research subjects (or participants) are faced with different ethical risks
no risks when there are no human subjects involved (eg. examining party manifestos)
Large risks when human subjects are vulnerable or sensitive (eg. interviews with war victims) - other examples of subjects that are vulnerable or sensitive?
Large risks when researchers use manipulation (eg. experimental research)
examining party manifestos
large risks when human subjects are vulnerable or sensitive
interviews with war victims
Indigenous communities
children
LGBTQ+
People with disabilities
elderly
Large risks when researchers use manipulation
experimental research
Ethics legislation
Increasing Ethics legislation
today, research on human subjects is impossible without approval from ethics review boards
technology advancement is the reason the legislation has to be revamped consistently
Institutional research boards
In Canada, researchers that perform studies on humans have to complete the Tr-council policy statement: ethical conduct for research involving humans (TCPS2)
Main ethical principles
respect for persons
beneficence
justice
Respect for persons involves consent
Informed participation
Informed consent — participant is provided with information about the study, including the risks and benefits
Implied consent – by completing an anonymous survey, a participant gives their consent to participate in the study
Debrief participants – inform participants of the real goal of the study, if deception was involved at the outset of the study when participants voluntarily agreed to take part in the study
tell participants after the fact, and they debrief you
the debriefing would then make you fully informed; participants are partially informed before going into it
Respect for persons involves voluntary participation
Voluntary participation (not coerced)
Active consent — typically a formal verbal or written agreement to participate
in the study after being informed about the risks and benefits of participation
Passive consent (e.g., a parent signing a form to allow their child to
participate in research)
Exit option – ability to leave the study at any time without providing a reason
Why might a participant chose to exit a study?
your consent can be withdrawn
Respect for persons
Individuals should be treated as autonomous agents
autonomous means independently or without coercion
Persons with diminished autonomy are entitled to protection
who are persons with diminished autonomy?
people with cognitive disabilities
children
Beneficence ( Do no Harm)
Persons are treated in an ethical manner not only be respecting their decisions and protecting them from harm, but also making efforts to secure their well-being
do no harm (i.e. no physical or mental risks, respect privacy when sharing findings, ensure confidentiality of responses and anonymity of participants
what is the difference between confidentiality and anonymity?
Maximize benefits and minimize harms
publish results to the news, media
keeping this anonymous
dont give too much demographic information
Justice
instructs researchers to treat individuals fairly and equally” Ensure that the costs and benefits do not disproportionately affect one group of people over others
typical participants = white men
researchers should not be undertaken on vulnerable populations because they are convenient or because the researcher has easy access
persons of colour, children,etc.
“… research is about strive to involve participant groups who are likely to benefit from the findings, rather than focusing on groups unlikely to benefit .”
Henrietta, 1950s found the cure for polio and covid
Ethics before everything
some researchers argue that observing ethical obligations may have methodological disadvantages
Observing the principle of beneficence (do no harm) might make it difficult to study hurtful things
observing the principle of voluntary participation results in studying only people that want to be studied
Observing the principle of informed participation likely leads to reactivity, the phenomenon that people respond to being studied
Usually research ethics boards (REBs) allow small deviations for methodological purposes if the research has the promise to yield large social benefits
Academic Integrity
A different set of ethical principles that focus on maintaining professional integrity in the conduct of research
proper referencing, citation, and acknowledgement
disclosure or potential conflicts of interest.
Data accessibility and research transparency
Honest presentation of findings
Sept. 30th
Quiz: covers all material from chapters 1 and 2
15 mc, 30 mins to complete
Beginning the research process
Steps in a research question:
Formulation of question
Theorizing and hypothesis formulation
Operationalization
Choice of research design and case selection
Data collection
Data analysis
DIssemination of findings
Finding a subject
starts with personal interest or curiosity
Ideas may come from scholarly or non-scholarly sources (eg. your life, media, course-related materials, social and family networks)
Develop a research question about political phenomena (some examples from the textbook)
“Why is voter turnout for local elections higher in some cities than in others?”
“Do small nations sign more multilateral treaties than large nations?”
“Why do leaders in parliamentary systems call for snap-elections?”
Purposes of literature review
Example research question: “Why is voter turnout for local elections higher in some cities than in others?”
Conduct a literature to:
“see what has and has not been investigated” (about what?)
“to develop general explanations for observed variations [or differences] in a behavior or phenomenon” (what differences?)
“to identify potential relationships between concepts and to identify researchable hypotheses”
Possible hypotheses for the research question?
Conduct a literature to:
“learn how others have defined and measured key concepts” (what concepts?)
“identify data sources that other researchers have used” (what data sources?)
“develop alternative research designs” (more information on this later in the course)
“discover how a research project is related to the work of others”
Finding a subject: Literature review
Consult literature
Peer-reviewed sources. What does this mean?
Journal articles in social science citation index
Books of academic publishers
Editor-reviewed sources
Academic books and journals with an editorial committee
Anything else should be treated with great caution
Identify appropriate electronic scientific databases to search for high quality sources
Google Scholar–widely available
Who has used Google Scholar in the past?
University of Guelph Library
Scientific articles: Web of Science (multi-disciplinary research) – journal database or JSTOR
Books: Use Omni search engine
You do not need ethics approval to do a literature review (EXAM)
Will ask us to identify the phenomena
Select appropriate search terms and use operators in search engines
For example, it I wanted to search for articles on why is voter turnout for local elections higher in some cities than in others, I would use the following terms and operators:
Voter turnout AND “local elections” AND “cities”
Please try this in Google Scholar
What did you get?
Managing references (or citations)
Various types of reference management systems exist to manage and store your sources
BibTeX
EndNote
RefMan
RefWorks
Zotero (free at the University of Guelph)
Reviewing the literature
“Once you have identified references for possible inclusion in a literature review, the next step is to figure out how the references fit together in a way that…
(1) explains the base of knowledge, or what we know about a topic from previous work, with respect to the research question, and
(2) establishes how the current project is going to build on that knowledge.
Researching goal: Description
Description: systematic and exhaustive record of one group of cases (or phenomenon) or one category of things
Examples (McCombes, 2023):
How has the Guelph “housing market changed over the past 20 years?”
“What are the most popular online news sources among under-18s?”
(Other description examples in political science?)
Research goal: Explanation
Explanation: advancement of theory-based account or why a situation or event has occurred (cause and effect)
Aims to test existing ideas systematically
Examples (George & Merkus, 2023):
“Why do undergraduate students obtain higher average grades in the first semester than in the second semester?”
“How does marital status affect labor market participation?”
(Other exploration examples in political science?)
Exploration, Description, or Explanation (WILL BE ASKED TO IDENTIFY WHICH ON EXAM)
Why is voter turnout for local elections higher in some cities than in others?”
“Do small nations sign more multilateral treaties than large nations?”
“Why do leaders in parliamentary systems call for snap-elections?”
How are local elections held in the city of Manila?
Summary
A research project begins with identifying a subject of study, consulting the literature to find out what has already been done, and establishing a research goal
Most research projects pursue one or more of 3 goals:
Exploration: tentative investigation aimed at generating new ideas about a new and understudied subject
Description: exhaustive and systematic record of aspect of reality
Explanation: theory-based account of one thing leading to another (cause and effect)
Oct. 2nd
Research questions and theory
Steps in a research project:
Formulation of question
Theorizing and hypothesis formulation
Operationalization
Choice of research design and case selection
Data collection
Data analysis
Dissemination of findings
Last class we talked about: identifying a research question
What do you need to keep in mind when identifying a research question
Research questions
Once your literature review is complete, you can formulate a research question
This question is usually reformulated (or modified) over the course of the research project, BUT we should start with a question that is feasible
Why?
This question will inform all subsequent methodological choices and steps in a research project (e.g., qualitative, quantitative, mixed-methods)
We can only evaluate the quality of methodological decisions once we know the research question
Formulating a good research question is not easy
Good question have four characteristics:
Specificity
Boundness
Agnosticism
Empirical verifiability
Specificity
Try to be as unambiguous and precise as possible (Why?)
Avoid terms such as “important” and “effective” (normative connotations)
Try to establish what you are not interested in (Why?)
Ask others how they understand your question (Why?)
Specificity is most important for explanatory and descriptive research
Boundedness
Establish empirical domain: time and place covered by your investigation
A smaller domain is not necessarily better
Group activity - Is version A or B better
Version a: Why do some people decide not to vote?
Version b: Why do some citizens of Western democracies decide not to vote in parliamentary elections?
Version a: Do female candidates for parliament receive more negative news coverage than male candidates?
Version b: Do female candidates for parliament in Canada receive more negative news coverage than their male counterparts?
Version a: Does the electoral system have an effect on strategic voting in Canada?
Version b: Does the electoral system have an effect on strategic voting in Western democracies?
Agnosticism
Do not include your expectations in the question (Why?)
Version a: Why do Canadian newspapers have such a left-wing bias?
Version b: Do Canadian newspapers exhibit a specific ideological bias, and if so, why?
Version a: What are the main explanations for Trudeau’s failure to reduce the spread of COVID-19?
Version b: What explains cross-national differences in governments’ level of success in reducing the spread of COVID-19?
Empirical verifiability
Avoid normative questions and concepts
Answering the question should be theoretically possible
Answering the question should be practically feasible
Which version of the research question is better and why
Version a: Which political party in Canada has done the best job governing in Canada?
Version b: Which political party in Canada has brought about the largest reduction in the unemployment rate during its time in government?
Version a: What percentage of the labour force suffers from false consciousness?
Version b: What percentage of the population under the poverty line considers the economic system to be just?
Which version of the research question is better and why
Version a: How does the average West Wing staff member who served under Trump rate his competence as president?
Version b: How does the level of public criticism President Trump has received from his own staff compare to what was leveled against other presidents
Research Variables
Variables are a ‘category’ of characteristics we study
Variables can take on different forms or attributes
Variables can vary from one case to the next
Variables and theories
Theory:
identifies patterns and regularities in the world
makes sense of these patterns (either in whole, or in part)
Often posits the existence of a causal relationship between two or more variables
Independent variables (or predictors or causal variables): “a phenomenon that we think will help us explain political characteristics or behavior”
Dependent variable (or outcome variables): “thought to be caused, to depend upon, or to be a function of an independent variable” (eg. political characteristics or behavior)
Sometimes theory makes more complex predictions between variables
Antecedent variable: “A variable that occurs prior to all other variables and that may affect other independent variables”
Intervening variable: “A variable that occurs closer in time to the dependent variable and is itself affected by other independent variables”
Example 1:
Hypothesis 1: People “who favoured national health insurance [in the United States]” were “more likely to have voted for Barak Obama in 2008 than a person who did not favor such extensive coverage.”
Hypothesis 2: “...people who have inadequate medical insurance are more likely to favor national health insurance”
Independent (or predictor or causal) variable: ?
Dependent (or outcome) variable: ?
Antecedent variable (what might affect the independent variable)?
Example 2:
Hypothesis 1: “voters’ years of formal education affect their propensity to vote”
Hypothesis 2: “formal education creates or causes a sense of civic duty, which in turn encourages voter turnout”
Independent variable: ?
Dependent variable: ?
Intervening variable: ?
Theory and observation: induction and deduction
Induction: using empirical observations to develop a new theory
deduction: testing an existing theory with empirical observations
Purpose of theory:
Simplify reality
Explain a phenomenon
Predict a phenomenon
Summary
The research question guides every aspect of the research process
Good research questions are specific, bounded, agnostic, and empirically verifiable
Researchers make sense of reality by identifying variables (categories of characteristics) and values (the specific characteristics of each individual case)
Theories help us to understand the world around us by simplification and explanation
There is a constant back and forth between theory and observation: researchers develop new theories based on observations (induction), but also test existing theories with systematic observation (deduction)
Oct. 7th
Hypotheses
are a specific research expectation
derives from a theory
can be descriptive
most often explanatory - formulates an expectation regarding the realtionship between two or more variables
Hypotheses as a causal model
Hypotheses as complex causal models
Good hypotheses have 4 characteristics
Falsifiability
Generality
Parsimony
Coherence
Properties of a good hypothesis: Falsifiability
A falsifiable theory can state beforehand which observations or research results it would not expect
If there is no imaginable evidence that would prove the theory wrong, it cannot be subjected to empirical testing
Properties of a good hypothesis: Generality
A general theory has a large domain; that is, it applies to different cases and contexts
If the theory only explains a single event or instance, it offers little more than a description
Recall: Cases
What is under study
Also known as the unit of analysis
What are the cases and contexts in the following research questions (activity)
Do Canadian newspapers exhibit a specific ideological bias, and if so, why?
Why do some citizens of Western democracies decide not to vote in parliamentary elections?
Do female candidates for parliament in Canada receive more negative news coverage than their male counterparts?
Properties of a good hypothesis: Parsimony
A parsimonious theory is simple; that is, it does not include more variables or details than necessary
If a theory is as complex as the process or event it explains, it does not simplify reality
Properties of a good hypothesis: Coherence
A coherent theory makes logical sense, is in line with existing knowledge, and provides a credible and probable account of reality
Example Hypothesis: (Activity)
“In two-party systems, parties will move more and more to the political centre.” (Downs, 1957)
Please evaluate this hypothesis in terms of the following:
Falsifiable?
General?
Parsimonious?
Coherent
Conceptualization, operationalization, and measurement
Conceptualization
Involves defining all concepts in your hypotheses/research question - developing a conceptual definition
Operationalization
Involves specifying how you will measure these concepts - developing an operational definition
Measurement
Involves employing operational definition to record aspects of the real world
Example: moving from hypothesis to operationalization
Example 2: Moving from hypothesis to operationalization
“In two-party systems, parties will move more and more to the political centre” (Downs, 1957)
Conceptualization (How will you define main concepts?)
Operationalization (How will you measure main concepts?)
Conceptual definitions
Are essential to avoid confusion and achieve precision in communicating research findings
Dimensions of the concept are specified
Ideas around conceptualization
Conceptualization (and operationalization) is often contentious, political, or even arbitrary – Why?
When assessing the truth claims of others, always check the conceptual definitions
Conceptualization continues
Many of the concepts political scientists are interested in are not easily observable
Nevertheless, concepts need
empirical referents
Direct observables
Indirect observables (or proxies)
Constructs (i.e., working hypothesis or concept)
Remember: Case = Unit of Analysis
Refers to the events of reality that ‘possesses’ the concepts we aim to operationalize - also referred to as the case
Directly related to our hypothesis and concepts
Questions
Summary
Theoriues help us to understand the world around us by simplification and explanation
There is a constant back and forth between theory and observation: researchers develop new theories based on observations (induction), but also test existing theories with systematic observation (deduction)
Theories generate hypotheses
Many theories claim a causal relationship between two or more variables, which we can visualize in a causal model
Good theories are falsifiable, general, parsimonious and coherent
Oct. 9th
Operationalization is the last step from a hypothesis
Operationalization involves at least 3 sets of choices
Which indicators (variables), and how many?
Which values (attributes of variables), and how many?
Will the approach be quantitative or qualitative?
Recall the possible attributes (from picture of a table above)
Indicators (variables)
Single or multiple indicators?
The more abstract and complex the concept, the better the case for multiple indicators
Would you choose a single indicator or multiple indicators (or variables) for democracy?
Values - B
The set of values for each variable need to be mutually exclusive (eg. do not overlap conceptually) and collectively exhaustive (eg. all possible options are included in the set)
Each case is assigned one value and only one value
Precision of measurement
values can be measured at different levels of precision
Level of measurement has major consequences for data analysis (of limited relevance for this course)
Concept, unit of analysis, indicator, values
Quantitative and Qualitative methods: a narrow definition
In a narrow definition, the distinction between quantitative and qualitative methods is about operationalization
Quantitative research methods use numbers to measure concepts
Qualitative research methods use words to measure concepts
Quantitative and qualitative methods: a broad definitions
some scholars use the terms much more broadly to denote a general approach to the conduct of social science
Quantitative and Qualitative Methods: Strengths and Weaknesses
Quantitative and qualitative research methods are best seen as having distinct strengths and weaknesses
The choice of methods depends crucially on the research goal and question
Quantitative and qualitative methods can be combined fruitfully (see lecture 23)
Comparison
Measurement quality: Error
Measurement error: assigning an incorrect value
Two types of measurement error
Systematic: bias (i.e., the error in measurement is consistent across all cases)
A scale that is out by 1 kg is used to measure the weight of every individual (or case) in a research study (i.e., the incorrect measure is used consistently)
Random (i.e., the error in measurement randomly affects cases)
A scale that is out by 1 kg is randomly used to measure the weight of some individuals (or cases) in a research study (i.e., the incorrect measure is used randomly). A scale that measures weight correctly is used to measure the weight of the remaining individuals (or cases) in that same research study.
Why is this a problem?
Measurement: Validity vs. reliability
Validity: corresponds to ‘true’ values
Face validity
Content validity
Predictive validity
Reliability: refers to the stability of the measurement
Test-re-test
Increase number of observers
Measurement quality
Oct. 16th
Key term: external validity
“The ability to generalize from one set of research findings to other situations.” (p. 353)
“In short, the results of a study have “high” external validity if they hold for the world outside of the experimental situation; they have low validity if they only apply to the laboratory.” (p. 129)
Example: participants in a clinical (or laboratory) study are given a blood pressure medication to reduce their blood pressure. The results that occur in the laboratory also occur in non-laboratory conditions (high external validity)
Key term: Internal validity
“The ability to show that manipulation or variation of the independent [or causal variable] actually causes the dependent [or outcome variable] to change.” (p. 354)
Example: participants in a clinical (or laboratory) study are given a blood pressure medication to reduce their blood pressure. The blood pressure medication given to participants (as opposed to any intervention) is shown to cause a reduction in the participant’s blood pressure. (high internal validity)
Key term: effects-of-causes approach and causes-of-effects approach
Effects of causes appraoch: “starts with a potential cause and works forward to measure its impact on the outcome” (p. 130)
Causes-of-effects approach: “starts with an outcome (e.g., war, election results, passage of a major piece of legislation) and works backward toward the causes.”
Measurement quality: error
Measurement error: assigning an incorrect value
Two types of measurement error
Systematic: bias (i.e., the error in measurement is consistent across all cases)
A scale that is out by 1 kg is used to measure the weight of every individual (or case) in a research study (i.e., the incorrect measure is used consistently)
Random (i.e., the error in measurement randomly affects cases)
A scale that is out by 1 kg is randomly used to measure the weight of some individuals (or cases) in a research study (i.e., the incorrect measure is used randomly). A scale that measures weight correctly is used to measure the weight of the remaining individuals (or cases) in that same research study.
Why is this a problem?
Measurement: Validity vs. reliability
Validity: corresponds to ‘true’ values
Face validity
Content validity
Predictive validity
Reliability: refers to the stability of the measurement
Test-re-test
Increase number of observers
Key Terns: Different types of validity
Example: Diversity measured using ethnic group and religion that people adhere to.
Face validity: “When a measure appears to accurately measure the concept it is supposed to measure....a matter of judgement” (Johnson et al., 2020, p. 353)
Content validity: “Involves determining the full domain or meaning of a particular concept and then making sure that all components of the meaning are included in the measure.” (Johnson et al., 2020, p. 351)
Predictive validity: “ability of a test or other measurement to predict a future outcome.” (Scribbr, 2022, para 1)
Research design: Comparisons
Most research relies on comparisons:
more informative
reduce measurement error (identify anomalies or outliers in the data)
Necessary to advance causal claims
The research design of a project refers to its strategy of comparison
Case selection (this lecture)
Type of comparison (lectures 12-15)
Case selection and sampling
Part of developing a research design is determining which cases will be investigated or studied (eg. countries, individuals, parliament from slide 8)
This decision is straightforward if it is feasible to study all cases that are relevant to your research design -) simply select and study all cases
Often this is not possible and you must select a sample from the larger population that you are interested in
Case selection and sampling diagrammatically
How many university students voted in the last Ontario provincial election?
Population: all university students in the province of Ontario (set of cases we want to make claims about) –1 million university students (hypothetical number)
Sampling frame: create a list of the names and contacts of all university students in Ontario
Sample: select 500 university students from the list of names and contacts (set of cases we end up analyzing)
Goal of sampling
Our goal is to generalize findings from our sample to the population
e.g., what we find in our study sample will be found in the study population)
Sampling error will introduce bias (or an over-representation of a characteristic) into your study (e.g., an over-representation of fourth year university students in your sample)
Why is having an over-representation of fourth year university students in a study about voting behavior a problem?
If your sample is biased, your research conclusions will be biased!
Important: Need to take steps to reduce sampling error because you want to reduce the likelihood of bias in your research conclusions
Probability (random) and non-probability (non-random) sampling
Non-probability (non-random) samples
Convenience sample: Study cases that are easily accessible
Example: I stand at the entrance of the library and ask students as they enter if they voted in the last Ontario provincial election
Purposive (or purposeful) sample: study cases that allow for meaningful comparisons (see next lectures)
Common in qualitative research because you purposefully want to study a set of cases that posses specific characteristics relevant to your research question
Snowball sample: each case suggests new cases to investigate
Example: I ask every student that participates in my study to provide me with the name and contact information of other students they know
Quota sample:
Using known characteristics of the study population (e.g., 1 million university students in Ontario), the research establishes what the study population should like (e.g., the total number, and percentage of 1st, 2nd, 3rd, and 4th year university students in Ontario)
Cases are then selected by a non-probability (non-random) technique to resemble population characteristics (e.g., smaller number, but similar percentages, of 1st, 2nd, 3rd, and 4th year university students in Ontario)
Probability (random) samples
Probability (random) samples are preferred in descriptive and explanatory research of large populations
Avoids selection bias (e.g., of over-representation of 4th year university students in a study about the voting behaviour of all university students)
Allows for inferential statistics (see POLS3650)
Statistics that use data from a study population to make conclusions about the data of a study population
Probability (random) samples
Three well-known types:
Simple random sample: each case has an equal chance of being selected (create a numbered list of all cases and use a random number generator to select cases from that list)
Stratified sample: “elements [or cases] sharing one or more characteristics [(e.g., gender, level of education, income, year in university, etc.)] are grouped and elements [or cases] are selected from each group in proportion to the group’s representation in the total population.” (p. 357)
Example: study population of 1 million university students has 35% of students in year 1, 25% students in year 2, 20 % in year 3, and 20% in year 4. My study sample of 500 university students will have 35% of students in year 1, 25% of students in year 2, 20% of students in year 3, and 20% of students in year 4.
Cluster sample: “used when no list of elements exists.” (p. 351)
Select cluster at random
Then select cluster within selected cluster at random
Finally, select case within smallest cluster at random
Repeat (see diagram to right)
Sampling decision tree: population, probability, or non-probability
Oct. 21st
Causation and research design
Research design: strategy of comparison
The research design of a study refers to its strategy of comparison
Case selection
Types of comparisons
Experimental design (manipulated/controlled comparisons)
Cross-sectional design (comparisons across place at a single point in time)
Longitudinal design (comparisons across time)
Case study design (may or may not be comparisons across cases)
Causation: what is it
Two types of causal relationships:
Deterministic causation
Necessary conditions: B never occurs without A
Sufficient conditions: whenever A occurs, B will follow
Probabilistic causation:
Whenever A occurs, the chance of B increases
Causation does not mean:
Complete explanation about the causal relationship of interest
Absence of outliers (i.e., cases that do not exhibit the same pattern as other cases–they may fall at the extremes)
The causal effect is observable in the majority of cases
What is needed to establish causation
Causation: how do we observe it
To make a convincing case for a causal relationship, we need to demonstrate 4 things
Covariation
Time order
Non-spuriousness
Theoretical support
Causation: Covariation needed
Two variables (i.e., independent and dependent variables) covary when certain values on the independent variable as associated with certain values on the dependent variable
Example: Amount of time spent studying and grade on final exam
Causation: time order needed
The cause is preceded by the effect and not the other way around
Example: Amount of time spent studying and grade on final exam
Causation: non-spuriousness needed
The covariation observed is not produced by a third variable (e.g., causal relationship between urbanization and birth rate is NOT produced by the number of storks)
Causation: theoretical support needed
There is a plausible and logical explanation that connects cause and effect
Example: Amount of time spent studying and grade on final exam
The classic experiment
The classic experiment has been developed for testing causal claims
It inspired most other research designs
It is not the most common, but functions as a model for explanatory research
What does it look like?
The classic experiment: comparisons
The experiment only makes sense if we can make reasonable comparisons between the 2 groups (i.e., experimental and control groups)
The 2 groups need to be similar in their:
initial value of the DV
expected reaction to the stimulus
To achieve this, we assign participants to groups by randomization
What is randomization?
The classic experiment: Causality
The classic experiment allows for establishing causality
Covariation: compare experimental vs control group, pre-test vs post-test
Time order: pretest, stimulus, posttest
Non-spuriousness: manipulated comparisons ensure only difference is the stimulus
It is not important whether the total group of experimental subjects is representative of the total population!
Oct. 23rd
Causation and research design
Research design: decision tree
Recall: The two types of causal relationships
Deterministic causation
Necessary conditions: B never occurs without A
Sufficient conditions: whenever A occurs, B will follow
Probabilistic causation
Whenever A occurs, the chance of B increases
Alternatives to the classic experiment
Variations on experimental design
Field experiments in a natural design
Natural experiments
Quasi experiments
Observational studies (next class)
Field experiments
Mimics classic experiment outside the laboratory
It randomly administers an experimental stimulus to similar groups in the real world
Example field experiment research question: “Which method of informing voters about an upcoming election has the largest effect on their likelihood to come out and vote?”
Quasi experiments
Like field experiments, quasi experiments rely on the logic of administering a stimulus to different groups outside of the laboratory
In sharp contrast, however, in quasi experiments the researchers cannot be sure that the groups are randomized
Example: “Do television debates affect voting decisions?” Aalberg and Jenssen exposed some graduate students to a panel debate for the 2001 Norwegian elections and others to non-political entertainment.
Classic experiment and alternatives compared
Oct. 28th
Observational studies
Why base rational designs
Experiments do not only raise ethical and methodological problems, but they are also rarely suited for our questions
Ethical problems
Methodological problems
Sometimes field experiments, natural experiments, or quasi-experiments offer a solution
Most often the best thing at our disposal is observational design
Observational study
“used to describe designs in which the researcher neither manipulates experimental variables nor randomly assigns subjects to treatment”
The researcher “merely observes causal sequences and covariations.”
Examples?
“Cross-sectional designs and longitudinal designs are two frequently used observation research designs.”
Cross-sectional designs
“Perhaps the most common observation research design is cross-sectional analysis”
“measurements of the independent variable are taken all at the same time or approximately the same time.” (snapshot)
“the researcher does not control or manipulate
the independent variable,
the assignment of subjects to treatment or control groups, or
the conditions under which the independent variable is experienced.”
“If the units of analysis are individuals, the study is often called a survey or poll.”
“if the subjects are geographical entities, such as states or nations or other groupings of units, the term aggregate analysis is frequently applied”
“attributes of the units are measured or observed, and the data recorded.”
What are attributes?
Surveys (Are observational): Advantages and disadvantages
Advantages:
Controls for things that are similar across cases
Control in this instance means holding constant
Data collection is relatively straightforward
Disadvantage:
Not sensitive to time order
Longitudinal designs
These designs “are characterized by the availability of measures of variables at different points in time.”
Example: effect of Pierre Poilievre’s comments to the media on voter’s perception of his [Poilievre’s] competence as a leader over time
Other examples that you can think of?
Longitudinal design: advantages and disadvantages
Advantages:
“change in the level of variables or conditions can be measured and modeled.”
“it is sometimes easier to decide time order or which comes first, X or Y”
“they can in principle estimate three kinds of effects: age, period (history), and cohort.”
Disadvantages:
“the researcher does not control the introduction of the independent variable(s)”
It is difficult, if not impossible, to collect new data about the past
No control group (difficult to isolate the effect of one IV)
Reliability is likely a challenge, meaning of indicators may change over time.
Age and period effects
Age effects: “can be considered a direct measure of (chronological) time and be assessed like other variables”
Example: “an investigator may be interested in the effect of age on political predispositions or ideology. (It is commonly asserted that as people age, they become more politically conservative.)
Period (history) effects: “a period (interval of time) may be thought of as an indicator of history during a period, and the consequences on individuals are period effects. It is the “history” that occurs during the period, not chronological age that matters.”
Example: 1960’s and 1970’s – “events such as Watergate and the Vietnam War adversely affected many citizens’ trust in government, whether they were young or old.”
Cohort effects
“A cohort is defined as a group of people who all experience a significant event in roughly the same time.”
“A birth cohort, for instance, consists of those born in a given year or period” (e.g., people born in the year 2002, or baby boomers)
“an “event” cohort is those who shared a common experience, such as their first entry into the labor force at a particular time.”
(other examples?)
“It is often hypothesized that individuals in one cohort will, because of their shared background, behave differently than individuals in a different cohort.” For example, “people born in the years immediately after World War II (the baby boomers) may have different political attitudes and affiliations than those who were born in the 1980s.”
Challenges of observational studies
When we step out of the laboratory, our data are observational rather than experimental. This introduces 2 challenges
No control over the values of the independent variable (we do not administer the stimulus)
No control over the allocation of groups (we cannot ensure randomization)
The implications of this design are important
Inferring causality becomes more difficult (in particular, more difficult to establish time order and non-spuriousness)
Case selection becomes more important
When there are many cases, we can address these challenges with statistics
Probability (random) samples cancel out values on variables in which we are not interested
Multivariate analyses allow us to ‘control’ for (or hold constant) third variables (see POLS3650)
Summary: Observational studies
With observational data, the researcher has no control over group allocation and the values on the IV
When the number of cases is large, we can address this by randomization and multivariate analysis
Cross-sectional design involves the study of multiple subjects at a single(or one) point in time
Longitudinal design involves the study of one subject at multiple points in time
Case Study: Definition and purposes
“the detailed examination of a single example of a class of phenomena” (Flyvberg, 2006, p. 220)
Purposes of case studies (Johnson et al., 2020, p. 137)
Idiographic
Hypothesis generating
Hypothesis testing
Idiographic case studies
“aim to describe, explain, or interpret a singular historical episode with no intention of generalizing beyond the case.”
What kinds of singular historical episodes could you chose to study?
Inductive: “lack an explicit theoretical perspective and simply have the purpose of describing all aspects of the case” (descriptive)
Theory-guided: “are explicitly structured by a well-developed conceptual framework”
Example: application of “Kingdon’s “three streams” model of policy making to structure a description of the politics of a particular policy” (e.g., problem stream, policy stream, political stream, policy window)
Hypothesis-generating case studies
“examine or more cases for the purpose of developing more general more general theoretical propositions” that can be tested in future research.”
Example: “researchers might study several cases of conflicts between nations to identify the key factors that seem to have led either to the outbreak of war or to peaceful resolutions of the conflict.”
What other topics could you examine using hypothesis-generating case studies?
Hypothesis-testing case studies
“entail testing hypothesized empirical relationships.”
“These types of case studies include investigations of causal mechanisms...”
What topics could you examine using a hypothesis-testing case study?
Non-explanatory case studies: exploratory and description.
Exploratory – can look at one case to generate new hypotheses
Descriptive – can look at one case to get a “thicker” (or in-depth) description than what is possible in studies of multiple cases
Critiques of non-explanatory case studies
Many scholars doubt the value or even possibility of detecting law-like patterns of reality from case studies
Response to these critiques:
Flyvberg (2006, p. 226): “generalization...is considerably overrated as the main source of scientific progress
In this perspective, narratives (or in-depth description, or account, of a single phenomenon) are a strength, not a weakness
What are your thoughts on the value of case studies? Do findings from case studies have to be generalizable in order to be useful?
Comparative designs
Often the term ‘case’ is used for investigations of a single country that is, in fact, more than one case
Longitudinal studies
studies of the same subject at multiple points in time
every point in time is considered a case
see previous lecture
Comparative case studies - method of difference
“the researcher selects cases in which the outcomes differ, compares the cases looking for the single factor that the cases do not have in common, and concludes that this factor is “the effect, or cause, or a necessary part of the cause, of the phenomenon.”
“applies to situations where the researcher is investigating outcomes that vary in degree (e.g., high, medium, and low levels of an outcome and identifies a factor that also varies in degree”
Process tracing: definition
“refers to case studies that “explicitly unpack mechanisms and engage in detailed empirical tracing of them”
“use deductive reasoning and ask, “If an explanation is true, what would be the specific process leading to the outcome?”
“often involve only one case because of the copious amount of information and detail that is required to trace a causal mechanism and to show that rival explanations do not account for an outcome.”
“depends on logic and has been compared to a detective sifting through evidence in order to solve a mystery.”
Advantages and disadvantages of case studies
Advantages
High internal validity (very accurate measurement of the case itself)
Context-dependent knowledge
Disadvantages
Low external validity (difficult to extrapolate findings to other contexts)
Replication is difficult
Danger of personal investment (blinders)
Comparisons are necessary for explanatory cases
Summary: case studies
Case studies have high internal validity and low external validity
Case studies are difficult to replicate
There is a danger of personal investment when conducting case studies
Oct. 30th
It’s good to use surveys when dealing with a large population. However it is harder to do more in-depth research because you can’t ask why someone wrote the answer they did. It is also difficult to ensure that the answers are honest and truthful.
Steps in a research project
Formulation of research question
Theorization and formulation of hypotheses
Conceptualization and operationalization
choice of research design
Data collection
Data analysis
Formulation of conclusions and dissemination of results
Quantitative Data Collection: Surveys
Quantitative data collection
Involves gathering numerical information on a large number of cases
In the next three lectures, we will review the three most common quantitative methods of data collection in political science
Surveys (quantitative alternative to interviews)
Secondary analysis
Quantitative content analysis
Terminology
Survey: Method of data collection that consists of asking the same questions to many individuals in the exact same order
Respondents: participants in a survey
Questionnaire: list of questions in a survey
Response rate
Response rate: percentage of contacted individual participate in a survey.
Example: you invite 300 university students to participate in a survey asking about their political affiliations. 80 students participate in your survey.
What is your response rate?
divided then multiplied by 100
“As the response rate decreases, the likelihood that the sample will not resemble the population increases–this will lead to poor statistical inference” - bias
You can perform an analysis to see if there is a significant difference in the characteristics of responders and non-responders. Why would you want to do this?
Nov. 4th
What steps could you take to improve your response rate - or the number of partici[ants that complete your survey
Keep the survey as short as possible to avoid incomplete surveys
Send multiple reminders to participants
Promote the survey as widely as possible (eg. social media, posters, other forms of advertisements that are acceptable to the Research Ethics Board)
Communicate the importance of the research to individuals invited to participate in the survey
Offer an incentive to participants that is acceptable to the Research Ethics Board (eg. gift card)
Surveys: Advantages vs. disadvantages
Advantages:
Best method to gauge attitudes and perceptions of a large population
Disadvantages:
Social desirability bias
non-attitudes (when participants don’t care about the issue under study in the survey)
Improper reading of questions or careless completion of the survey questions
Subject to survey design issues (poor design = poor data)
Don’t provide any insights as to the reason so for the responses (superficial)
Ordering of survey questions
How we order questions has major consequences for how participants respond to them
Best practices:
Alternate direction of questions. Why?
Move from general to specific questions
Avoid priming certain conditions over others
priming: persuading the answer
Best practices for the wording of survey questions and answers
To avoid random measurement error:
Keep the questions and answers short and simple
Be precise (as opposed to vague) in terms of your wording
How would you rate the government’s current performance?
How would you rate the Ontario government’s current performance on the issue of climate change
Avoid double-barreled questions (what are these)
To avoid systematic measurement error (bias):
Avoid the inclusion of authorities or experts in question
Avoid argumentative questions
Pilot testing of surveys
Involves administering the survey to a small group of individuals that fit your target population (2 to 3)
These individuals complete the survey and provide important feedback to the researcher on the design and comprehension of the survey
Pilot testing occurs before the widespread deployment of the actual survey
Allows the researcher to modify they survey before its wide-spread deployment to the sample
Survey modes: four common types (ON TESTS AND EXAM)
Mixed-mode surveys
Use a combination of modes to encourage participation and increase response rates
Mail and web
Mail and phone
Mail, phone, and web
etc.
Summary
The quantitative approach to interviewing is survey research, which is ideal for gauging the views and beliefs of large groups of respondents
Response patterns are heavily influenced by the nature of the survey
When evaluating survey results, we should always investigate
Sampling technique
Response rate
Question ordering
Question formulation
Survey mode(s)
Identify sampling method: random or not, random is ideal
Did they show the response rate, they might not always report it (could be considered for critique)
If they don't report, is it a representative (the implications)
Response rate (analysis)
The order of the questions
Questions formation, language
Survey mode(S)
What was the mode and what are the critiques with that (in person?)
did the same person answer twice
Was everyone surveyed (if online), electronically literate?
Nov. 6th
Quantitative Data Collection: Secondary Analysis
Unobtrusive research: Secondary Analysis
Unobtrusive research is the analysis of already existing data
Can you thing of some examples
Major advantages: few concerns about reactivity, few ethical concerns (if collected ethically to start with), verity time efficient
Major limitation: researcher has no control over the nature and availability of data
Types of unobtrusive research
This class discusses two main types of unobtrusive research
Analysis of existing data
Secondary analysis
document analysis
Content analysis
Quantitative
Qualitative
Secondary analysis: What is it
Secondary analysis is the ‘recycling’ of data compiled by others
Researchers perform new analyses on this data
Researchers need to understand the quality of the data compiled by others BEFORE performing their new analyses. Why?
Many data are available that you might expect - little pint in collecting data that are already out there
Ethical implications to collecting already existing data unless there is a very good reason to do so
Useful sources for secondary analysis (NOT TESTING ON)
National
Statistics Canada
Provincial, municipal statistics
Canadian Election Studies
Cross-national
OECD statistics
UN statistics
Freedom House
World Values Survey
Academic institutions have data resource centers
Secondary Analysis: Important considerations
What is the quality of the data?
What do we mean by this? Why determine this ahead of time?
Who compiled the data?
Does the person or organization collecting the data have a stake in the outcomes of the research?
People can create bias, measure incorrectly, word things specifically
How have the data been collected? How have the concepts been measured?
Why do you need to determine this ahead of time?
The way people measure their variables and define them can change the outcome
Are the data applicable to your research question?
Have all data been measured the same way? Why do you need to determine this ahead of time?
Research article analysis
Try to find the research question word for word stated in the article
“What is the effect…”
Explanatory or exploratory
they are looking for a relationship between the independent and dependent variables (explain) = explanatory (its this one)
experiment = MUST manipulate a variable
Use the exact independent and dependent variables stated in the article, also look at how they’re measured
Inductive or deductive
If there is a hypothesis tested it is deductive
If there is a hypothesis formed it is inductive
Key concepts in the study and how they are measured
what they are measuring and operationalizing
Operationalization is how they move to measurement (how did they choose to measure the variable)
how they measure it shows the dependent variable (??)
Research design: experimental, cross-sectional, longitudinal, case study
How did you come to that conclusion
Analysis
Advantages and disadvantages of the research design
Method of data collection: survey, secondary analysis, content analysis qualitative interviews, field research, or document analysis
Survey is quantitative (sometimes secondary analysis?)
Ethical considerations
this ethical consideration is the same as mine
ethical principles slide (justice, , _)
methods is a good place to check (everything hinges on the methodology)
Data collection and analysis slides
Go to the limitations sections of the article
They state the limitations
She wants to see that we follow the slides, especially for methods and data collections (she really wants us to use the slides)
Advantages and disadvantages of methodology
Nov. 11th
Quantitative Data Collection III: Quantitative Content Analysis
Content Analysis
Is the study of recorded communication
Do not confuse it with a literature review
Content analysis investigates primary material (I.e ., First hand material). for other researchers or studies
Any record communication can be subjected to content analysis (Written, Verbal, Non-verbal…)
Content Analysis in Political Science
For political scientists of most interest is political communication, usually between politicians, the public and the media
ex.
Parliamentary min
campaign posts
facebook posts
comments on new stores
press releases
newspaper coverage
what can we learn from record communication? Historical facts
considerations for assessing the validity of historical facts
is the author a credible witness/expert
does the author have a reason to lie/withhold information/exaggerate/ embellish
Can you think of some politicians whose record communications with the public and media have undergone extensive content analysis in terms of historical facts
ex. Trump, Steven Harper made a comment how there ones no colonization
Political attitudes and beliefs
Considerations for assessing the validity of political attitudes and beliefs
who is the author and who is the intended audience
does the author have a reason to lie/withhold information/ exaggerate/ embellish
Discourse
discourse focuses on the structure of political or public communication
there is a link between language and the way we view the world, and that politicians manipulate this for their own ends
it is argued that control and domination of representations allows politicians to generate worldviews consistent with their goals and to downgrade negate or eliminate alternative representations
Discourse: Orwellian Ex
“If a village full of innocent is bombed or thousands of people are relocated as a consequence of aggression and war we can choose to manipulate the representations of such a negative acts as types of positive or neutral events. we could call the first pacification for ex, and the second could be referred to as a rectification of frontiers”
“presented in this way issues such as pain, suffering and homelessness are hidden within neutral, placid or positive representations”
ex. politicians word there sentences that have manipulate them and change the world around us
Quantitative Content Analysis
the quantitative approach to content analysis is to quantify words, phrases and or other elements can see how often words are being repeated
words like illegal or phrases like democracy is at stake
especially appropriate in a deductive study (i.e one that tests theory or hypotheses) that aims to maximize reliability of measurement
Quantitative content Analysis: Manifest vs Latent Content
Manifest (or surface level) content is easier to quantify than latent (hidden) content
“In manifest content analysis, context is derived from the visible and literal meaning of the words—taken at face value.” (Delve, n.d., para 13)
Question: Is the literal meaning of words always easy to derive or consistently derived across multiple persons?
“In latent content analysis, you apply a deeper, interpretive analysis that seeks to infer underlying meaning from the words or phrases you choose to analyze.” (Delve, n.d., para 13)
Strengths and Weaknesses of Quantitative Content Analysis
Strengths
No reactivity
Not very costly in terms of time and money
Easy to replicate, especially, when analyzing public communication
Few ethical concerns; especially, when analyzing public communication
Well-suited for longitudinal research designs
Weaknesses
No control over nature and availability of data
Can be difficult to distinguish truthful from untruthful statements
Measurements and analyses can be difficult
**also have to be concerned with AI- generated communications and edited or manipulated communications
Summary
Content analysis is a type of unobtrusive research, which consists of the investigation of recorded communication
Political scientists are particularly interested in political communication between citizens, politicians, and the media
Quantitative content analysis transforms aspects of communication into numbers
This is most appropriate in deductive studies of manifest content that are most concerned about reliability of measurement
most studies are deductive
Nov. 13th
Qualitative data collection: qualitative interviews
Key features of qualitative interviews
Qualitative interviews are one-on-one discussions between a researcher and research participant
Much more detailed than surveys, and therefore much fewer respondents
Especially useful to investigate:
Internal explanations (examples?)
Historical accounts (examples)
Motivations (examples?)
Qualitative interviews: five aspects that have major implications
Type of interviewees (why?)
Structure of interview
structured interviews: have set question ordering and wording, allow for no improvising, and the researcher takes the lead (pros and cons?), interviewer takes the lead
pro: good for looking for patterns
con: don't know their thoughts on the topic
Unstructured interviews: vary from each other in questions, allow for much improvisation, and the respondent takes the lead (pros and cons?), interviewee takes the lead
An interview guide that contains all of the questions that will be asked is typically required by the REB regardless of the structure of the interview
The choice to conduct structured vs. unstructured interviews balances concerns about reliability, flexibility, and artificiality.
Method of communicating - interviews can take place in-person, over the phone, or in online chat room
choice about method balances concerns about reactivity, expected length, response rate, costs (what is your preference?)
Length of interview
Long interviews will result in more data (more time needed to transcribe and analyze)
Short interviews likely increase response rate, completion rate, and quality of answers (less time neede to transcribe and analyze)
What is your preference in terms of length?
Interview questions
question ordering and formulation have large consequences
Start with warm-up questions, move from abstract to specific
avoid social desirability bias
adjust language to participants
it's important to pilot test your interview guide and interview questions before data collection. Why?
Ways to decrease the potential of reactivity during qualitative interviews
reactivity is a major concern
Techniques to reduce reactivity/increase validity
Ensure you are in a quiet and private location
Consider your appearance: gender, race, attire
Consider cultural conventions
Emphasize how valuable respondent’s views are to you
Use probing questions on short answers
Employ the awkward silence
Minimize interruptions (Don’t interrupt the interviewee while they are talking!)
Stay neutral in terms of your verbal and non-verbal communication
Documenting qualitative interviews
Strategies of documentation
Taking notes (pros and cons?)
Audio recording (pros and cons?)
Video recording (pros and cons?)
No obvious best technique: each has distinct implications for reactivity, the observation of non-verbal cues, and accuracy/comprehensiveness
Which method of documentation do you prefer? Why?
After each qualitative interview is complete
Once the interview is over, write/generate transcripts: detailed (if possible complete) minutes of the interview. This can be very time consuming and costly.
Send transcript to respondent
Reduces ethical risks (guarantees informed participation)
Opportunity for additional validation
You may also want to document important ideas, concerns, or thoughts that come to you about the interview or data while generating transcripts.
Qualitative interviews: strengths and weaknesses
Strengths:
Avoids superficiality of surveys
great corroboration technique
Perfect for studying internal explanations
widely applicable
Weaknesses:
Limited reliability
more artificial than observation
reactivity
imperfect and selective memory of interviewees
Summary: Qualitative Data Collection - Interviews
Qualitative interviews are one on one discussions between researcher and research participant
Much more detailed than surveys, and therefore much fewer respondents
Especially useful to investigate:
Internal explanations
Historical accounts
Motivations
Nov. 18th
Questions:
What is the main research question?
Research goal?
Theoretical approach? Deductive or inductive?
What are the key concepts? How was the researcher decide to measure these concepts?
State whether the study uses an experimental, cross-sectional, longitudinal, and/or case study research design (explain how you came to that conclusion)
State which method(s) of data collection the study has employed (survey research, secondary analysis, content analysis, qualitative interviews, field research, and/or document analysis), and explain how you came to that conclusion.
Evaluate the methods of data collection by discussing whether the researcher has chosen appropriate methods, whether this project could have been conducted with different methods, and how relying on alternative methods of data collection would have affected the conclusions of the study.
Qualitative data collection: field research
Key features of field research…
In field research, researchers are physically present in the social setting they aim to understand
house of commons, cabinet meetings
Very common in descriptive, exploratory and explanatory research
Especially common in case studies or comparative case studies
Advantages and disadvantages if doing field research
Advantages
Compared to interviewing and unobtrusive techniques, field research has several advantages:
Measures behavior more directly
Avoids reported accounts, of which the validity might be difficult to assess
‘Being there’ enhances understanding and exposes more information
Disadvantages
Limited to observable behavior (Why?)
Raises its own problems regarding reliability, validity, and ethics (How?)
Comparatively time-intensive (Why?)
Field research: subtypes
Any project that involves a researcher traveling to the context of study can be considered field research
Some specific subtypes:
Structured observation: quantitative approach aimed at reliable and systematic measurement (e.g. sports statistics)
Ethnography: qualitative approach, aimed at understanding ‘culture’ from insiders’ point of view (common in anthropology)
Field research: setting and identity of researcher
Setting: Is it publicly accessible (open) or not (closed)?
Researcher Identity: Does everyone know they are being researched (overt observation) or not (covert observation)?
What might be some issues in terms of overt and covert observation?
Field research: overt and covert observations
These characteristics are central to a common trade-off between validity and ethics in field research
Overt observation is likely to produce measurement error
Reactivity
Direction and scope of measurement error can differ from one researcher to another
Covert observation overcomes these problems but raises ethical concerns
Unable to ensure voluntary and informed consent
Particularly problematic in closed settings
Field research: role of researcher
Most ethnographers are participant observers: they take part in the social processes they are trying to understand
Other field researchers maintain the role of complete observer: they strictly observe from the sidelines
The choice is partially philosophical–standpoint theorists tend to prefer participant observation (Why?)
The choice has consequences for objectivity and reactivity
Field research: data collection
Most ethnographers attempt to write as comprehensive notes as possible
Other field researchers make more targeted and limited notes
Comprehensive notes might produce more unexpected findings
Targeted notes more feasible in deductive research
What is your preference – comprehensive or targeted notes?
Field research: Trade-Offs between reliability and validity
Decisions about the way we make observations (participant vs. complete observer, ‘comprehensive’ vs. targeted notes) raise another common trade-off: between reliability and validity
In-depth observation of complex social settings is very difficult to conduct reliably
Selective and incomplete observation
More partial and distant observation addresses these problems but raises concerns about measurement validity
Summary: Field Research
In field research, researchers are physically present in the setting they are trying to understand
This immersion enhances understanding and unlocks access to more information, but raises its own methodological challenges
The most qualitative version of field research is ethnography, in which the researcher immerses themselves for a long time in a culture and try to understand it from the insider’s point of view
Overt research likely leads to reactivity, but covert research violates principles of research ethics
Participant observation and extensive note-taking can lead to novel insight, but might be difficult to do reliably and objectively
Nov. 20th
Choosing a method and approach
Question
You want to conduct research on a topic that is of interest to you.
What method and approach (quantitative or qualitative) would you choose?
What would guide your decision making?
Which choices are best
Some researchers feel strongly committed to a method of data collection (interviews, field research, unobtrusive) or approach (quantitative, qualitative)
It is difficult to maintain, however, that one method or approach is inherently superior to another
We can only decide based on the research question which method and approach are most appropriate
We cannot say that some methods are better than others, but we can say that some methods are better to answer certain research questions than others
Data collection: relative strengths
Data collection don’ts
Don’t use interviews
to find external explanations
when more direct measurement is possible
when you are interested in distant history
Don’t use field research
when you are not interested in behavior
Don’t use unobtrusive research
if no useful data are available
Data collection: combinations or mixed methods
Methods of data collection can be fruitfully combined in three ways:
1. Triangulation: corroborating findings based on one method of data collection by findings based on another (e.g., one overarching research question)
2. Facilitation: using one method of data collection to help research using a different method of data collection forward (e.g. one research question drives another research question)
3.Complementation: using different methods of data collection for different components of the same project (e.g., multiple research questions in one study)
Approaches: Don’ts (Why not)
Don’t use qualitative techniques when you want to…
provide a systematic overview of many cases
demonstrate the existence of law-like patterns of behavior
Don’t use quantitative techniques when you want to…
offer detailed narratives of subjective experiences
study a small number of cases
study concepts that are highly complex or onerous to measure
Approaches: Combinations
Qualitative research can facilitate quantitative research
by developing hypotheses that quantitative research can test more rigorously
by suggesting ways to quantify complicated concepts
Similarly, quantitative research can facilitate qualitative research
by identifying interesting patterns that require deeper understanding
by identifying worthwhile cases to study
The combination of quantitative and qualitative research can also have other advantages
the corroboration of findings from quantitative research by qualitative investigation and vice versa strengthens our confidence in their validity (triangulation)
A combination of qualitative and quantitative investigation enable researchers to study different components of a research puzzle (complementation)
Nov. 25th
50 mc questions Lecture 2 - Sept. 11th
Why political scientists conduct research
To learn about important political phenomena
Some examples of important political phenomena provided in text:
“why women make up a larger percentage of legislators in some countries than in others” (Rwanda has the highest percentage of women legislators)
“how some nations exercise power over others” (US, China)
“how political institutions function”
eg. “does the use of non-profit service organizations to deliver public services change government control of and accountability of those services”
eg. home related care, some instances the UN
Empirical research is “a methodology that requires scholars to clearly state hypotheses or propositions that can be evaluated with actual “objective” observation of political phenomena”
Hypothesis: an educated guess or a proposed explanation for a phenomenon. It is a statement that can be tested through experimentation or observation
Proposition: a statement that expresses a judgment or an option. It is a claim that can be either true or false, but it is not necessarily testable in the same way that a hypothesis is
Scientific inquiry
involves the application of the scientific method
the scientific method “involves asking a question, research the question, making and testing a hypothesis, analyzing data, and documenting results
scientific method is systematic
central attention to causation
Scientific inquiry vs. non-scientific inquiry
Scientific inquiry:
scientific method
systematic in its approach
testable
rigourous (doing something to a high standard)
Non-scientific inquiry:
Knowledge may be based on agreement, tradition or authority, personal experience even though it may not be true (partially, fully, or universally true)
typically not systematic or rigourous
Limitations of non-scientific inquiry:
misinformation (Trump)
selective attention to information or bias
historically, some claims based in traditions, or made by authorities, have been false
illogical reasoning
Does science always lead to the truth
science does not always lead to the right answer or the truth, but it does rely on conscious strategies designed to reduce the dangers of relying on agreement and experience
Science is a method for acquiring knowledge
it involved the use of rigorous standards of observation and illogical reasoning to investigate a phenomenon in a systematic and objective manner - helps to reduce bias but may not eliminate bias completely
the difference between scientific and non-scientific inquiry is the use of research methods
Limitations of scientific inquiry
conducting research is not a simple process
the information a researcher chooses to use (or collect), the method that they follow to investigate a research question, and the statistics (or analyses) used to report research findings may affect the conclusions drawn
data quality issues depending on the source
experimental issues (eg. some phenomena are not conducive to experimentation)
This is why research must be critically evaluated
Lecture 3 - Sept. 16th
Empirical research: research based on actual, “objective” observation of a phenomenon
Empiricism
”relying on observation to verify (or refute) propositions (or scientific hypotheses)”
Propositions (differ from hypotheses): can be true or false, a statement that expresses a judgment or an opinion
Example:
“Positive (political) campaigns lead to higher voter turnout than do those that are characterized by mudslinging and name-calling”
Predictor variable: Whether the campaign is positive or negative
Outcome variable: Voter turnout
US “states with easier voter registration system have higher election turnout rates than do states with more difficult systems”
Predictor variable: Difficulty of voter registration systems
outcome variable: election turnout rates
On the EXAM we will have to identify the the independent and dependent variables
Elements of empiricism
”modern political science relies heavily on one kind of knowledge: knowledge obtained through objective observation, experimentation and logical reasoning”
”The ultimate goal of scientific research, which is to always attained, is to use its results to construct theories that explain a political phenomena”
Verification and falsification
“scientific knowledge depends on verification”
“our acceptance or rejections of a statement regarding something “known” must be influenced by observation”
A causes B
Falsifiability is another “key characteristic of scientific claims, meaning that statements or hypotheses can in principle be rejected in the face of contravening empirical evidence”
A does not (always) cause B
Normative vs. Non-normative - ON EXAM
Normative: (values, opinion based)
Subjective (describe personal engagement with reality)
Metaphysical (speculative, supernatural, imperceptible)
”This table is ugly”
“Canada should not have a senate”
Non-normative: (facts)
objective (describes reality)
real
“this table has four legs”
“Canada has a senate”
Most scientists would agree that science is, or should try to be, a non-normative entreprise
Scientific knowledge is transmissible
“Scientific knowledge must be transmissible - that is, the methods used in making scientific discoveries must be made transparent so that other can analyze and replicate findings” (Replicability)
There is a debate among political scientists as to whether they should be compelled to share their data, analytic techniques, and how they drew their conclusions
costs time, money, intellectual property, bias/unethical
Scientific knowledge is cumulative
“substantive (or considerable) findings and research techniques are built upon those of prior studies”
“The process of constantly testing and refining prior research produces an accumulated body of knowledge”
Scientific knowledge is explanatory
Scientific knowledge “provides a systematic, empirically verified understanding of why a phenomenon occurs”
A conclusion can be derived (logically) from a set of general propositions and specific initial conditions
The general propositions asser that when things of type X occur, they will be followed by things of type Y
eg. US “states with easier voter registration systems have higher election turnout rates than do states with more difficult systems.”
X = easier voter registration systems, Y = higher election turnout rates
Important: explanations do not need to “explain or predict a phenomenon with 100 percent accuracy.”
Scientific Knowledge is parsimonious
parsimony = simplicity
Scientific explanations that rely on a few explanatory factors are generally preferred or those with a lot of explanatory factors
Ex. Why do some people trust and follow authoritarian leaders
Immediate personal, social and economic conditions of the individual
All of the factors in explanation 1 plus deep-seated psychological states stemming from traumatic childhood experiences
Theory
Plays an important role in research
The accumulation of observed relationships sometimes leads to the creation of a theory that is, a body of statements that systemize knowledge of and explain relationships between phenomena. The accumulation of observed relationships sometimes leads to the creation of a theory - that is, a body of statements that systemize knowledge of and explain relationships between phenomena.
Overview of empirical research process
Identification of an idea or problem to solve
“Is [voter] turnout among evangelicals [Christians] higher in elections where there are distance differences between candidates on moral issues than in elections where the differences are small?” (e.g., moral issues – same sex marriage, abortion, etc.)
Hypothesis formation
“Evangelical christian’s are more likely than others to base their votes on candidate’s positions on moral issues”
Data collection
“We need to clearly define the concepts of moral values and evangelical Christians”
Create and administer a survey or poll that contains relevant questions
Interpretation and decision
“determine whether or not the observed results are consistent with the hypotheses”
“judging how well data support scientific hypotheses is usually not an easy matter”
Modification and extension
“depending on the outcome of the test, one can tentatively accept, and and on, or modify the hypothesis”
Rejection of a hypothesis can be “both interesting and beneficial”
Lecture 4 - Sept. 18
Assignment 1: personal pronouns are ok
Recall: Key Assumptions of Empiricism
There is a reality out there, governed by law-like patterns
Explanation is the identification of general and parsimonious causal relationships
Observation is the key source of knowledge through verification and falsification
Science should focus on facts and ignore ‘metaphysical’ values
Findings must be transmissible and science will be cumulative
Critiques of empiricism
“Is there really such a thing as the truth?”
Empiricists say yes
“Most political scientists take reality pretty much as a given. That is, they posit that the objects they study – elections, wars, constitutions, government agencies – have an existence independent of observers and can be studied more or less objectively.”
Constructionism - an alternate perspective to empiricism
“Humans do not simply discover knowledge of the real world through neutral processes, such as experimentation or unbiased observation; rather they create the reality they analyze” (Do we agree or disagree)
Example by Johnson et al.
“Consider the term Democratic Party”
citizens behave as if it exists
Instead of having an independent, material existence like an election or a strand of DNA, a political party exists only because citizens behave as if it exists”
“This means that two individuals that come from different social, historical, and cultural backgrounds may not comprehend and respond to the term in the same way”
In other words, reality is subjectively created
The problem of definition and measurement
In lecture 3, we sought to define the terms upper class, lower class, and political power. What happened?
We found a lack of general agreement as to how to best define these complex terms.
Measurement of these 3 terms can also be tricky
we may not be able to measure these complex terms in an objective and complete manner
we may, instead, opt to use different proxy measures, or measures that are a substitute for what we really want to measure but do not capture a complex concept completely, adds subjectivity to what we are studying
Critique of law-like patterns: self-reflection and individuality
“Like any other organisms, humans are aware of their surroundings. They have the additional ability to empathize with others and frequently attempt to reach others’ minds”
“human beings - individually, but especially jointly - are self-interpreting and reflective, capable of assigning meanings to their actions and revising these meanings recursively” (eg. through a process of repetition and extension or building)
examples of when and where meanings of our actions may morph?
The study of voting behavior
“Consider a political scientist who wants to investigate the effects of negative campaign advertising on attitudes”
“Suppose that Jane and Mary are subjects in a study”
“We cannot assume that they will react to experiment stimuli (eg. negative campaign advertising) exactly the sample way, even though they are the same age, gender, political persuasion, and so forth”
“Social scientists have to get around this problem by using groups or samples of individuals and then examining the average effect of the stimulus”
what do we know about averages
Outliers can greatly skew the average, which creates a problem on relying on averages
“The objects political scientists study are multifaceted and conscious beings with volition of their own who often change opinions and behaviors; thus, statements about them must necessarily be tentative, general, and time bound.”
Given the reliance on averages by political scientists, and the complexity and changing nature of human behavior, it begs the question: do law-like patterns really apply to human behavior?
Alternative to identifying patterns
Since constructionists reject the idea of an absolute truth, they insist all we can learn is how different people give meaning to the social world
From the constructionist perspective, explanation involves the identification of:
Intersubjective meanings: understandings of social reality that are shared by multiple people and that, in turn, structure (or shape) social reality
Internal explanations: explanations of human behavior that are offered by actors themselves
Can you think of an example?
Other scientists do not reject the existence of an absolute truth, but question where identifying patterns is the most worthwhile exercise
What if we are interested in things we cannot observe?
What if we are interested in things of which there are only a few instances?
In this perspective, explanation is much more demanding
Use inference to arrive at the best explanation: go back and forth between theory and observation to establish which explanation is best among available alternatives
Discover causal mechanisms: the process by which one things leads to another
Alternative to empiricism - Critical Theory
Critical theory - “the belief that a proper goal of social science is to critique to improve society (by making it more just and humane) rather than merely understand or explain what is going on
Focuses on questions about “what out to be” rather than “what is”
Approaches
Empiricism: truth is objective
Constructionism: the truth is subjective because social reality is subjective
Critical theory: the focus should be on improving society rather than just trying to understand (or explain) what is going on
Sept. 23rd
Recall, key assumptions of empiricism:
There is a reality out there, governed by law-like patterns
Explanation is the identification of general and parsimonious causal relationships
Observation is the key source of knowledge through verification and falsification
Science should focus on facts and ignore ‘metaphysical’ values
Findings must be transmissible and science will be cumulative
Assumption: Observation is the key source of knowledge through verification and falsification
Empiricists insist that observation is the source of all knowledge
Scientific realists argue that reality consists of observable and non-observable parts
observation cannot lead to knowledge about unobservable parts of reality
What are some non-observable parts of reality
Observable vs. Non-observable parts of reality
“the distinction here (in scientific realism) between the observable and the unobservable reflects human sensory capabilities: the observable is that which can, under favorable conditions, be perceived using the unaided senses (eg. planets and platypuses)
the unobservable is that which cannot be detected this way (eg. proteins and protons)
scientific beliefs of observability “generally extend to things that are detectable using instruments”
Assumption: Value-free Observation
Empiricists assume there is a fundamental distinction between facts and values, and insist that science can only make claims about facts
What are facts: something that can be proven
something that is known to have happened or to exist, especially something for which proof exists, or about which there is information
What are values: what we believe is right or wrong
the beliefs people have, especially about what is right and wrong and what is most important in life, that control their behavior
Many doubt that science can really be value-free:
Researchers' values influence their research projects? How?
many researchers believe theories cannot be conducted without some impact from personal values
Researchers’ standpoints (position in society) influence their research projects. What is meant by position in society
Pro insider: You can only truly understand something that you have experienced. Do you agree or disagree with this statement
Pro outsider: it is easier to be objective about something in which you are not personally involved. Do you agree or disagree with this statement
Social standing create bias in observations
Many doubt that political science should be value free
Assumption: Cumulative Science
Empiricists assume that every study builds on existing research and, because of this, science progresses
However, what researchers already (think they) know influences their research!
every observation we make we relate to previous knowledge and observations
Rosenthal effect: research expectations shape research findings
Do you agree or disagree with this statement?
Scientific revolutions
Kuhn (1962) argued that scientific research takes place within context-specific paradigms, which are abandoned during times of scientific revolutions
Paradigms are “a set of assumptions governing how we interact and interpret the world.... Paradigms need constant reinforcement to function. If events occur that cannot be explained by the current paradigm, a new one may be generated.” (Kestrana, 2022, para 3)
Can you think of an example when an existing paradigm was abandoned or a new paradigm created?
Collapse and creation of paradigms
“Scientific paradigms often endure a long time before they are replaced.”
“ For example, one of the earliest descriptions of the universe, Aristotle and Plato's "Two-sphere Universe," lasted about 550 years. This was followed by the Ptolemaic paradigm which lasted even longer before it was replaced by Copernicus, Kepler, and Newton's theories.”
“Since then, as modern technology facilitates greater and faster communication between scientists, paradigms appear and collapse at a faster rate.”
Research ethics
have you participated in a ethics board
Definition of ethics
“the discipline dealing with what is good and bad; a set of moral issues or aspects”
“Ethics are moral obligations that guide us in determining whether a certain behavior is right or wrong”
Why research ethics?
research subjects (or participants) have rights
in the absence of ethics regulations, scientists have violated the rights of their subjects
there can be obvious ethical violations with evil intent
there can also be unintentional (?) ethical violations
September 25
Unethical experiments in the US
Monster study: “Orphaned children with normal speech patterns were told they had poor speech, including a stutter” (…) “developed stutters and suffered negative psychological and behavioral effects”
Milgram obedience experiments: “Study participants were told to shock a “learner” for incorrect answers; however, the study participants did not know that the learner was not a real person, but rather a recording. After each shock, the participant was instructed to increase the voltage of the next shock, despite the learner’s call for them to stop”
Level and types of research-related risk
Different research subjects (or participants) are faced with different ethical risks
no risks when there are no human subjects involved (eg. examining party manifestos)
Large risks when human subjects are vulnerable or sensitive (eg. interviews with war victims) - other examples of subjects that are vulnerable or sensitive?
Large risks when researchers use manipulation (eg. experimental research)
examining party manifestos
large risks when human subjects are vulnerable or sensitive
interviews with war victims
Indigenous communities
children
LGBTQ+
People with disabilities
elderly
Large risks when researchers use manipulation
experimental research
Ethics legislation
Increasing Ethics legislation
today, research on human subjects is impossible without approval from ethics review boards
technology advancement is the reason the legislation has to be revamped consistently
Institutional research boards
In Canada, researchers that perform studies on humans have to complete the Tr-council policy statement: ethical conduct for research involving humans (TCPS2)
Main ethical principles
respect for persons
beneficence
justice
Respect for persons involves consent
Informed participation
Informed consent — participant is provided with information about the study, including the risks and benefits
Implied consent – by completing an anonymous survey, a participant gives their consent to participate in the study
Debrief participants – inform participants of the real goal of the study, if deception was involved at the outset of the study when participants voluntarily agreed to take part in the study
tell participants after the fact, and they debrief you
the debriefing would then make you fully informed; participants are partially informed before going into it
Respect for persons involves voluntary participation
Voluntary participation (not coerced)
Active consent — typically a formal verbal or written agreement to participate
in the study after being informed about the risks and benefits of participation
Passive consent (e.g., a parent signing a form to allow their child to
participate in research)
Exit option – ability to leave the study at any time without providing a reason
Why might a participant chose to exit a study?
your consent can be withdrawn
Respect for persons
Individuals should be treated as autonomous agents
autonomous means independently or without coercion
Persons with diminished autonomy are entitled to protection
who are persons with diminished autonomy?
people with cognitive disabilities
children
Beneficence ( Do no Harm)
Persons are treated in an ethical manner not only be respecting their decisions and protecting them from harm, but also making efforts to secure their well-being
do no harm (i.e. no physical or mental risks, respect privacy when sharing findings, ensure confidentiality of responses and anonymity of participants
what is the difference between confidentiality and anonymity?
Maximize benefits and minimize harms
publish results to the news, media
keeping this anonymous
dont give too much demographic information
Justice
instructs researchers to treat individuals fairly and equally” Ensure that the costs and benefits do not disproportionately affect one group of people over others
typical participants = white men
researchers should not be undertaken on vulnerable populations because they are convenient or because the researcher has easy access
persons of colour, children,etc.
“… research is about strive to involve participant groups who are likely to benefit from the findings, rather than focusing on groups unlikely to benefit .”
Henrietta, 1950s found the cure for polio and covid
Ethics before everything
some researchers argue that observing ethical obligations may have methodological disadvantages
Observing the principle of beneficence (do no harm) might make it difficult to study hurtful things
observing the principle of voluntary participation results in studying only people that want to be studied
Observing the principle of informed participation likely leads to reactivity, the phenomenon that people respond to being studied
Usually research ethics boards (REBs) allow small deviations for methodological purposes if the research has the promise to yield large social benefits
Academic Integrity
A different set of ethical principles that focus on maintaining professional integrity in the conduct of research
proper referencing, citation, and acknowledgement
disclosure or potential conflicts of interest.
Data accessibility and research transparency
Honest presentation of findings
Sept. 30th
Quiz: covers all material from chapters 1 and 2
15 mc, 30 mins to complete
Beginning the research process
Steps in a research question:
Formulation of question
Theorizing and hypothesis formulation
Operationalization
Choice of research design and case selection
Data collection
Data analysis
DIssemination of findings
Finding a subject
starts with personal interest or curiosity
Ideas may come from scholarly or non-scholarly sources (eg. your life, media, course-related materials, social and family networks)
Develop a research question about political phenomena (some examples from the textbook)
“Why is voter turnout for local elections higher in some cities than in others?”
“Do small nations sign more multilateral treaties than large nations?”
“Why do leaders in parliamentary systems call for snap-elections?”
Purposes of literature review
Example research question: “Why is voter turnout for local elections higher in some cities than in others?”
Conduct a literature to:
“see what has and has not been investigated” (about what?)
“to develop general explanations for observed variations [or differences] in a behavior or phenomenon” (what differences?)
“to identify potential relationships between concepts and to identify researchable hypotheses”
Possible hypotheses for the research question?
Conduct a literature to:
“learn how others have defined and measured key concepts” (what concepts?)
“identify data sources that other researchers have used” (what data sources?)
“develop alternative research designs” (more information on this later in the course)
“discover how a research project is related to the work of others”
Finding a subject: Literature review
Consult literature
Peer-reviewed sources. What does this mean?
Journal articles in social science citation index
Books of academic publishers
Editor-reviewed sources
Academic books and journals with an editorial committee
Anything else should be treated with great caution
Identify appropriate electronic scientific databases to search for high quality sources
Google Scholar–widely available
Who has used Google Scholar in the past?
University of Guelph Library
Scientific articles: Web of Science (multi-disciplinary research) – journal database or JSTOR
Books: Use Omni search engine
You do not need ethics approval to do a literature review (EXAM)
Will ask us to identify the phenomena
Select appropriate search terms and use operators in search engines
For example, it I wanted to search for articles on why is voter turnout for local elections higher in some cities than in others, I would use the following terms and operators:
Voter turnout AND “local elections” AND “cities”
Please try this in Google Scholar
What did you get?
Managing references (or citations)
Various types of reference management systems exist to manage and store your sources
BibTeX
EndNote
RefMan
RefWorks
Zotero (free at the University of Guelph)
Reviewing the literature
“Once you have identified references for possible inclusion in a literature review, the next step is to figure out how the references fit together in a way that…
(1) explains the base of knowledge, or what we know about a topic from previous work, with respect to the research question, and
(2) establishes how the current project is going to build on that knowledge.
Researching goal: Description
Description: systematic and exhaustive record of one group of cases (or phenomenon) or one category of things
Examples (McCombes, 2023):
How has the Guelph “housing market changed over the past 20 years?”
“What are the most popular online news sources among under-18s?”
(Other description examples in political science?)
Research goal: Explanation
Explanation: advancement of theory-based account or why a situation or event has occurred (cause and effect)
Aims to test existing ideas systematically
Examples (George & Merkus, 2023):
“Why do undergraduate students obtain higher average grades in the first semester than in the second semester?”
“How does marital status affect labor market participation?”
(Other exploration examples in political science?)
Exploration, Description, or Explanation (WILL BE ASKED TO IDENTIFY WHICH ON EXAM)
Why is voter turnout for local elections higher in some cities than in others?”
“Do small nations sign more multilateral treaties than large nations?”
“Why do leaders in parliamentary systems call for snap-elections?”
How are local elections held in the city of Manila?
Summary
A research project begins with identifying a subject of study, consulting the literature to find out what has already been done, and establishing a research goal
Most research projects pursue one or more of 3 goals:
Exploration: tentative investigation aimed at generating new ideas about a new and understudied subject
Description: exhaustive and systematic record of aspect of reality
Explanation: theory-based account of one thing leading to another (cause and effect)
Oct. 2nd
Research questions and theory
Steps in a research project:
Formulation of question
Theorizing and hypothesis formulation
Operationalization
Choice of research design and case selection
Data collection
Data analysis
Dissemination of findings
Last class we talked about: identifying a research question
What do you need to keep in mind when identifying a research question
Research questions
Once your literature review is complete, you can formulate a research question
This question is usually reformulated (or modified) over the course of the research project, BUT we should start with a question that is feasible
Why?
This question will inform all subsequent methodological choices and steps in a research project (e.g., qualitative, quantitative, mixed-methods)
We can only evaluate the quality of methodological decisions once we know the research question
Formulating a good research question is not easy
Good question have four characteristics:
Specificity
Boundness
Agnosticism
Empirical verifiability
Specificity
Try to be as unambiguous and precise as possible (Why?)
Avoid terms such as “important” and “effective” (normative connotations)
Try to establish what you are not interested in (Why?)
Ask others how they understand your question (Why?)
Specificity is most important for explanatory and descriptive research
Boundedness
Establish empirical domain: time and place covered by your investigation
A smaller domain is not necessarily better
Group activity - Is version A or B better
Version a: Why do some people decide not to vote?
Version b: Why do some citizens of Western democracies decide not to vote in parliamentary elections?
Version a: Do female candidates for parliament receive more negative news coverage than male candidates?
Version b: Do female candidates for parliament in Canada receive more negative news coverage than their male counterparts?
Version a: Does the electoral system have an effect on strategic voting in Canada?
Version b: Does the electoral system have an effect on strategic voting in Western democracies?
Agnosticism
Do not include your expectations in the question (Why?)
Version a: Why do Canadian newspapers have such a left-wing bias?
Version b: Do Canadian newspapers exhibit a specific ideological bias, and if so, why?
Version a: What are the main explanations for Trudeau’s failure to reduce the spread of COVID-19?
Version b: What explains cross-national differences in governments’ level of success in reducing the spread of COVID-19?
Empirical verifiability
Avoid normative questions and concepts
Answering the question should be theoretically possible
Answering the question should be practically feasible
Which version of the research question is better and why
Version a: Which political party in Canada has done the best job governing in Canada?
Version b: Which political party in Canada has brought about the largest reduction in the unemployment rate during its time in government?
Version a: What percentage of the labour force suffers from false consciousness?
Version b: What percentage of the population under the poverty line considers the economic system to be just?
Which version of the research question is better and why
Version a: How does the average West Wing staff member who served under Trump rate his competence as president?
Version b: How does the level of public criticism President Trump has received from his own staff compare to what was leveled against other presidents
Research Variables
Variables are a ‘category’ of characteristics we study
Variables can take on different forms or attributes
Variables can vary from one case to the next
Variables and theories
Theory:
identifies patterns and regularities in the world
makes sense of these patterns (either in whole, or in part)
Often posits the existence of a causal relationship between two or more variables
Independent variables (or predictors or causal variables): “a phenomenon that we think will help us explain political characteristics or behavior”
Dependent variable (or outcome variables): “thought to be caused, to depend upon, or to be a function of an independent variable” (eg. political characteristics or behavior)
Sometimes theory makes more complex predictions between variables
Antecedent variable: “A variable that occurs prior to all other variables and that may affect other independent variables”
Intervening variable: “A variable that occurs closer in time to the dependent variable and is itself affected by other independent variables”
Example 1:
Hypothesis 1: People “who favoured national health insurance [in the United States]” were “more likely to have voted for Barak Obama in 2008 than a person who did not favor such extensive coverage.”
Hypothesis 2: “...people who have inadequate medical insurance are more likely to favor national health insurance”
Independent (or predictor or causal) variable: ?
Dependent (or outcome) variable: ?
Antecedent variable (what might affect the independent variable)?
Example 2:
Hypothesis 1: “voters’ years of formal education affect their propensity to vote”
Hypothesis 2: “formal education creates or causes a sense of civic duty, which in turn encourages voter turnout”
Independent variable: ?
Dependent variable: ?
Intervening variable: ?
Theory and observation: induction and deduction
Induction: using empirical observations to develop a new theory
deduction: testing an existing theory with empirical observations
Purpose of theory:
Simplify reality
Explain a phenomenon
Predict a phenomenon
Summary
The research question guides every aspect of the research process
Good research questions are specific, bounded, agnostic, and empirically verifiable
Researchers make sense of reality by identifying variables (categories of characteristics) and values (the specific characteristics of each individual case)
Theories help us to understand the world around us by simplification and explanation
There is a constant back and forth between theory and observation: researchers develop new theories based on observations (induction), but also test existing theories with systematic observation (deduction)
Oct. 7th
Hypotheses
are a specific research expectation
derives from a theory
can be descriptive
most often explanatory - formulates an expectation regarding the realtionship between two or more variables
Hypotheses as a causal model
Hypotheses as complex causal models
Good hypotheses have 4 characteristics
Falsifiability
Generality
Parsimony
Coherence
Properties of a good hypothesis: Falsifiability
A falsifiable theory can state beforehand which observations or research results it would not expect
If there is no imaginable evidence that would prove the theory wrong, it cannot be subjected to empirical testing
Properties of a good hypothesis: Generality
A general theory has a large domain; that is, it applies to different cases and contexts
If the theory only explains a single event or instance, it offers little more than a description
Recall: Cases
What is under study
Also known as the unit of analysis
What are the cases and contexts in the following research questions (activity)
Do Canadian newspapers exhibit a specific ideological bias, and if so, why?
Why do some citizens of Western democracies decide not to vote in parliamentary elections?
Do female candidates for parliament in Canada receive more negative news coverage than their male counterparts?
Properties of a good hypothesis: Parsimony
A parsimonious theory is simple; that is, it does not include more variables or details than necessary
If a theory is as complex as the process or event it explains, it does not simplify reality
Properties of a good hypothesis: Coherence
A coherent theory makes logical sense, is in line with existing knowledge, and provides a credible and probable account of reality
Example Hypothesis: (Activity)
“In two-party systems, parties will move more and more to the political centre.” (Downs, 1957)
Please evaluate this hypothesis in terms of the following:
Falsifiable?
General?
Parsimonious?
Coherent
Conceptualization, operationalization, and measurement
Conceptualization
Involves defining all concepts in your hypotheses/research question - developing a conceptual definition
Operationalization
Involves specifying how you will measure these concepts - developing an operational definition
Measurement
Involves employing operational definition to record aspects of the real world
Example: moving from hypothesis to operationalization
Example 2: Moving from hypothesis to operationalization
“In two-party systems, parties will move more and more to the political centre” (Downs, 1957)
Conceptualization (How will you define main concepts?)
Operationalization (How will you measure main concepts?)
Conceptual definitions
Are essential to avoid confusion and achieve precision in communicating research findings
Dimensions of the concept are specified
Ideas around conceptualization
Conceptualization (and operationalization) is often contentious, political, or even arbitrary – Why?
When assessing the truth claims of others, always check the conceptual definitions
Conceptualization continues
Many of the concepts political scientists are interested in are not easily observable
Nevertheless, concepts need
empirical referents
Direct observables
Indirect observables (or proxies)
Constructs (i.e., working hypothesis or concept)
Remember: Case = Unit of Analysis
Refers to the events of reality that ‘possesses’ the concepts we aim to operationalize - also referred to as the case
Directly related to our hypothesis and concepts
Questions
Summary
Theoriues help us to understand the world around us by simplification and explanation
There is a constant back and forth between theory and observation: researchers develop new theories based on observations (induction), but also test existing theories with systematic observation (deduction)
Theories generate hypotheses
Many theories claim a causal relationship between two or more variables, which we can visualize in a causal model
Good theories are falsifiable, general, parsimonious and coherent
Oct. 9th
Operationalization is the last step from a hypothesis
Operationalization involves at least 3 sets of choices
Which indicators (variables), and how many?
Which values (attributes of variables), and how many?
Will the approach be quantitative or qualitative?
Recall the possible attributes (from picture of a table above)
Indicators (variables)
Single or multiple indicators?
The more abstract and complex the concept, the better the case for multiple indicators
Would you choose a single indicator or multiple indicators (or variables) for democracy?
Values - B
The set of values for each variable need to be mutually exclusive (eg. do not overlap conceptually) and collectively exhaustive (eg. all possible options are included in the set)
Each case is assigned one value and only one value
Precision of measurement
values can be measured at different levels of precision
Level of measurement has major consequences for data analysis (of limited relevance for this course)
Concept, unit of analysis, indicator, values
Quantitative and Qualitative methods: a narrow definition
In a narrow definition, the distinction between quantitative and qualitative methods is about operationalization
Quantitative research methods use numbers to measure concepts
Qualitative research methods use words to measure concepts
Quantitative and qualitative methods: a broad definitions
some scholars use the terms much more broadly to denote a general approach to the conduct of social science
Quantitative and Qualitative Methods: Strengths and Weaknesses
Quantitative and qualitative research methods are best seen as having distinct strengths and weaknesses
The choice of methods depends crucially on the research goal and question
Quantitative and qualitative methods can be combined fruitfully (see lecture 23)
Comparison
Measurement quality: Error
Measurement error: assigning an incorrect value
Two types of measurement error
Systematic: bias (i.e., the error in measurement is consistent across all cases)
A scale that is out by 1 kg is used to measure the weight of every individual (or case) in a research study (i.e., the incorrect measure is used consistently)
Random (i.e., the error in measurement randomly affects cases)
A scale that is out by 1 kg is randomly used to measure the weight of some individuals (or cases) in a research study (i.e., the incorrect measure is used randomly). A scale that measures weight correctly is used to measure the weight of the remaining individuals (or cases) in that same research study.
Why is this a problem?
Measurement: Validity vs. reliability
Validity: corresponds to ‘true’ values
Face validity
Content validity
Predictive validity
Reliability: refers to the stability of the measurement
Test-re-test
Increase number of observers
Measurement quality
Oct. 16th
Key term: external validity
“The ability to generalize from one set of research findings to other situations.” (p. 353)
“In short, the results of a study have “high” external validity if they hold for the world outside of the experimental situation; they have low validity if they only apply to the laboratory.” (p. 129)
Example: participants in a clinical (or laboratory) study are given a blood pressure medication to reduce their blood pressure. The results that occur in the laboratory also occur in non-laboratory conditions (high external validity)
Key term: Internal validity
“The ability to show that manipulation or variation of the independent [or causal variable] actually causes the dependent [or outcome variable] to change.” (p. 354)
Example: participants in a clinical (or laboratory) study are given a blood pressure medication to reduce their blood pressure. The blood pressure medication given to participants (as opposed to any intervention) is shown to cause a reduction in the participant’s blood pressure. (high internal validity)
Key term: effects-of-causes approach and causes-of-effects approach
Effects of causes appraoch: “starts with a potential cause and works forward to measure its impact on the outcome” (p. 130)
Causes-of-effects approach: “starts with an outcome (e.g., war, election results, passage of a major piece of legislation) and works backward toward the causes.”
Measurement quality: error
Measurement error: assigning an incorrect value
Two types of measurement error
Systematic: bias (i.e., the error in measurement is consistent across all cases)
A scale that is out by 1 kg is used to measure the weight of every individual (or case) in a research study (i.e., the incorrect measure is used consistently)
Random (i.e., the error in measurement randomly affects cases)
A scale that is out by 1 kg is randomly used to measure the weight of some individuals (or cases) in a research study (i.e., the incorrect measure is used randomly). A scale that measures weight correctly is used to measure the weight of the remaining individuals (or cases) in that same research study.
Why is this a problem?
Measurement: Validity vs. reliability
Validity: corresponds to ‘true’ values
Face validity
Content validity
Predictive validity
Reliability: refers to the stability of the measurement
Test-re-test
Increase number of observers
Key Terns: Different types of validity
Example: Diversity measured using ethnic group and religion that people adhere to.
Face validity: “When a measure appears to accurately measure the concept it is supposed to measure....a matter of judgement” (Johnson et al., 2020, p. 353)
Content validity: “Involves determining the full domain or meaning of a particular concept and then making sure that all components of the meaning are included in the measure.” (Johnson et al., 2020, p. 351)
Predictive validity: “ability of a test or other measurement to predict a future outcome.” (Scribbr, 2022, para 1)
Research design: Comparisons
Most research relies on comparisons:
more informative
reduce measurement error (identify anomalies or outliers in the data)
Necessary to advance causal claims
The research design of a project refers to its strategy of comparison
Case selection (this lecture)
Type of comparison (lectures 12-15)
Case selection and sampling
Part of developing a research design is determining which cases will be investigated or studied (eg. countries, individuals, parliament from slide 8)
This decision is straightforward if it is feasible to study all cases that are relevant to your research design -) simply select and study all cases
Often this is not possible and you must select a sample from the larger population that you are interested in
Case selection and sampling diagrammatically
How many university students voted in the last Ontario provincial election?
Population: all university students in the province of Ontario (set of cases we want to make claims about) –1 million university students (hypothetical number)
Sampling frame: create a list of the names and contacts of all university students in Ontario
Sample: select 500 university students from the list of names and contacts (set of cases we end up analyzing)
Goal of sampling
Our goal is to generalize findings from our sample to the population
e.g., what we find in our study sample will be found in the study population)
Sampling error will introduce bias (or an over-representation of a characteristic) into your study (e.g., an over-representation of fourth year university students in your sample)
Why is having an over-representation of fourth year university students in a study about voting behavior a problem?
If your sample is biased, your research conclusions will be biased!
Important: Need to take steps to reduce sampling error because you want to reduce the likelihood of bias in your research conclusions
Probability (random) and non-probability (non-random) sampling
Non-probability (non-random) samples
Convenience sample: Study cases that are easily accessible
Example: I stand at the entrance of the library and ask students as they enter if they voted in the last Ontario provincial election
Purposive (or purposeful) sample: study cases that allow for meaningful comparisons (see next lectures)
Common in qualitative research because you purposefully want to study a set of cases that posses specific characteristics relevant to your research question
Snowball sample: each case suggests new cases to investigate
Example: I ask every student that participates in my study to provide me with the name and contact information of other students they know
Quota sample:
Using known characteristics of the study population (e.g., 1 million university students in Ontario), the research establishes what the study population should like (e.g., the total number, and percentage of 1st, 2nd, 3rd, and 4th year university students in Ontario)
Cases are then selected by a non-probability (non-random) technique to resemble population characteristics (e.g., smaller number, but similar percentages, of 1st, 2nd, 3rd, and 4th year university students in Ontario)
Probability (random) samples
Probability (random) samples are preferred in descriptive and explanatory research of large populations
Avoids selection bias (e.g., of over-representation of 4th year university students in a study about the voting behaviour of all university students)
Allows for inferential statistics (see POLS3650)
Statistics that use data from a study population to make conclusions about the data of a study population
Probability (random) samples
Three well-known types:
Simple random sample: each case has an equal chance of being selected (create a numbered list of all cases and use a random number generator to select cases from that list)
Stratified sample: “elements [or cases] sharing one or more characteristics [(e.g., gender, level of education, income, year in university, etc.)] are grouped and elements [or cases] are selected from each group in proportion to the group’s representation in the total population.” (p. 357)
Example: study population of 1 million university students has 35% of students in year 1, 25% students in year 2, 20 % in year 3, and 20% in year 4. My study sample of 500 university students will have 35% of students in year 1, 25% of students in year 2, 20% of students in year 3, and 20% of students in year 4.
Cluster sample: “used when no list of elements exists.” (p. 351)
Select cluster at random
Then select cluster within selected cluster at random
Finally, select case within smallest cluster at random
Repeat (see diagram to right)
Sampling decision tree: population, probability, or non-probability
Oct. 21st
Causation and research design
Research design: strategy of comparison
The research design of a study refers to its strategy of comparison
Case selection
Types of comparisons
Experimental design (manipulated/controlled comparisons)
Cross-sectional design (comparisons across place at a single point in time)
Longitudinal design (comparisons across time)
Case study design (may or may not be comparisons across cases)
Causation: what is it
Two types of causal relationships:
Deterministic causation
Necessary conditions: B never occurs without A
Sufficient conditions: whenever A occurs, B will follow
Probabilistic causation:
Whenever A occurs, the chance of B increases
Causation does not mean:
Complete explanation about the causal relationship of interest
Absence of outliers (i.e., cases that do not exhibit the same pattern as other cases–they may fall at the extremes)
The causal effect is observable in the majority of cases
What is needed to establish causation
Causation: how do we observe it
To make a convincing case for a causal relationship, we need to demonstrate 4 things
Covariation
Time order
Non-spuriousness
Theoretical support
Causation: Covariation needed
Two variables (i.e., independent and dependent variables) covary when certain values on the independent variable as associated with certain values on the dependent variable
Example: Amount of time spent studying and grade on final exam
Causation: time order needed
The cause is preceded by the effect and not the other way around
Example: Amount of time spent studying and grade on final exam
Causation: non-spuriousness needed
The covariation observed is not produced by a third variable (e.g., causal relationship between urbanization and birth rate is NOT produced by the number of storks)
Causation: theoretical support needed
There is a plausible and logical explanation that connects cause and effect
Example: Amount of time spent studying and grade on final exam
The classic experiment
The classic experiment has been developed for testing causal claims
It inspired most other research designs
It is not the most common, but functions as a model for explanatory research
What does it look like?
The classic experiment: comparisons
The experiment only makes sense if we can make reasonable comparisons between the 2 groups (i.e., experimental and control groups)
The 2 groups need to be similar in their:
initial value of the DV
expected reaction to the stimulus
To achieve this, we assign participants to groups by randomization
What is randomization?
The classic experiment: Causality
The classic experiment allows for establishing causality
Covariation: compare experimental vs control group, pre-test vs post-test
Time order: pretest, stimulus, posttest
Non-spuriousness: manipulated comparisons ensure only difference is the stimulus
It is not important whether the total group of experimental subjects is representative of the total population!
Oct. 23rd
Causation and research design
Research design: decision tree
Recall: The two types of causal relationships
Deterministic causation
Necessary conditions: B never occurs without A
Sufficient conditions: whenever A occurs, B will follow
Probabilistic causation
Whenever A occurs, the chance of B increases
Alternatives to the classic experiment
Variations on experimental design
Field experiments in a natural design
Natural experiments
Quasi experiments
Observational studies (next class)
Field experiments
Mimics classic experiment outside the laboratory
It randomly administers an experimental stimulus to similar groups in the real world
Example field experiment research question: “Which method of informing voters about an upcoming election has the largest effect on their likelihood to come out and vote?”
Quasi experiments
Like field experiments, quasi experiments rely on the logic of administering a stimulus to different groups outside of the laboratory
In sharp contrast, however, in quasi experiments the researchers cannot be sure that the groups are randomized
Example: “Do television debates affect voting decisions?” Aalberg and Jenssen exposed some graduate students to a panel debate for the 2001 Norwegian elections and others to non-political entertainment.
Classic experiment and alternatives compared
Oct. 28th
Observational studies
Why base rational designs
Experiments do not only raise ethical and methodological problems, but they are also rarely suited for our questions
Ethical problems
Methodological problems
Sometimes field experiments, natural experiments, or quasi-experiments offer a solution
Most often the best thing at our disposal is observational design
Observational study
“used to describe designs in which the researcher neither manipulates experimental variables nor randomly assigns subjects to treatment”
The researcher “merely observes causal sequences and covariations.”
Examples?
“Cross-sectional designs and longitudinal designs are two frequently used observation research designs.”
Cross-sectional designs
“Perhaps the most common observation research design is cross-sectional analysis”
“measurements of the independent variable are taken all at the same time or approximately the same time.” (snapshot)
“the researcher does not control or manipulate
the independent variable,
the assignment of subjects to treatment or control groups, or
the conditions under which the independent variable is experienced.”
“If the units of analysis are individuals, the study is often called a survey or poll.”
“if the subjects are geographical entities, such as states or nations or other groupings of units, the term aggregate analysis is frequently applied”
“attributes of the units are measured or observed, and the data recorded.”
What are attributes?
Surveys (Are observational): Advantages and disadvantages
Advantages:
Controls for things that are similar across cases
Control in this instance means holding constant
Data collection is relatively straightforward
Disadvantage:
Not sensitive to time order
Longitudinal designs
These designs “are characterized by the availability of measures of variables at different points in time.”
Example: effect of Pierre Poilievre’s comments to the media on voter’s perception of his [Poilievre’s] competence as a leader over time
Other examples that you can think of?
Longitudinal design: advantages and disadvantages
Advantages:
“change in the level of variables or conditions can be measured and modeled.”
“it is sometimes easier to decide time order or which comes first, X or Y”
“they can in principle estimate three kinds of effects: age, period (history), and cohort.”
Disadvantages:
“the researcher does not control the introduction of the independent variable(s)”
It is difficult, if not impossible, to collect new data about the past
No control group (difficult to isolate the effect of one IV)
Reliability is likely a challenge, meaning of indicators may change over time.
Age and period effects
Age effects: “can be considered a direct measure of (chronological) time and be assessed like other variables”
Example: “an investigator may be interested in the effect of age on political predispositions or ideology. (It is commonly asserted that as people age, they become more politically conservative.)
Period (history) effects: “a period (interval of time) may be thought of as an indicator of history during a period, and the consequences on individuals are period effects. It is the “history” that occurs during the period, not chronological age that matters.”
Example: 1960’s and 1970’s – “events such as Watergate and the Vietnam War adversely affected many citizens’ trust in government, whether they were young or old.”
Cohort effects
“A cohort is defined as a group of people who all experience a significant event in roughly the same time.”
“A birth cohort, for instance, consists of those born in a given year or period” (e.g., people born in the year 2002, or baby boomers)
“an “event” cohort is those who shared a common experience, such as their first entry into the labor force at a particular time.”
(other examples?)
“It is often hypothesized that individuals in one cohort will, because of their shared background, behave differently than individuals in a different cohort.” For example, “people born in the years immediately after World War II (the baby boomers) may have different political attitudes and affiliations than those who were born in the 1980s.”
Challenges of observational studies
When we step out of the laboratory, our data are observational rather than experimental. This introduces 2 challenges
No control over the values of the independent variable (we do not administer the stimulus)
No control over the allocation of groups (we cannot ensure randomization)
The implications of this design are important
Inferring causality becomes more difficult (in particular, more difficult to establish time order and non-spuriousness)
Case selection becomes more important
When there are many cases, we can address these challenges with statistics
Probability (random) samples cancel out values on variables in which we are not interested
Multivariate analyses allow us to ‘control’ for (or hold constant) third variables (see POLS3650)
Summary: Observational studies
With observational data, the researcher has no control over group allocation and the values on the IV
When the number of cases is large, we can address this by randomization and multivariate analysis
Cross-sectional design involves the study of multiple subjects at a single(or one) point in time
Longitudinal design involves the study of one subject at multiple points in time
Case Study: Definition and purposes
“the detailed examination of a single example of a class of phenomena” (Flyvberg, 2006, p. 220)
Purposes of case studies (Johnson et al., 2020, p. 137)
Idiographic
Hypothesis generating
Hypothesis testing
Idiographic case studies
“aim to describe, explain, or interpret a singular historical episode with no intention of generalizing beyond the case.”
What kinds of singular historical episodes could you chose to study?
Inductive: “lack an explicit theoretical perspective and simply have the purpose of describing all aspects of the case” (descriptive)
Theory-guided: “are explicitly structured by a well-developed conceptual framework”
Example: application of “Kingdon’s “three streams” model of policy making to structure a description of the politics of a particular policy” (e.g., problem stream, policy stream, political stream, policy window)
Hypothesis-generating case studies
“examine or more cases for the purpose of developing more general more general theoretical propositions” that can be tested in future research.”
Example: “researchers might study several cases of conflicts between nations to identify the key factors that seem to have led either to the outbreak of war or to peaceful resolutions of the conflict.”
What other topics could you examine using hypothesis-generating case studies?
Hypothesis-testing case studies
“entail testing hypothesized empirical relationships.”
“These types of case studies include investigations of causal mechanisms...”
What topics could you examine using a hypothesis-testing case study?
Non-explanatory case studies: exploratory and description.
Exploratory – can look at one case to generate new hypotheses
Descriptive – can look at one case to get a “thicker” (or in-depth) description than what is possible in studies of multiple cases
Critiques of non-explanatory case studies
Many scholars doubt the value or even possibility of detecting law-like patterns of reality from case studies
Response to these critiques:
Flyvberg (2006, p. 226): “generalization...is considerably overrated as the main source of scientific progress
In this perspective, narratives (or in-depth description, or account, of a single phenomenon) are a strength, not a weakness
What are your thoughts on the value of case studies? Do findings from case studies have to be generalizable in order to be useful?
Comparative designs
Often the term ‘case’ is used for investigations of a single country that is, in fact, more than one case
Longitudinal studies
studies of the same subject at multiple points in time
every point in time is considered a case
see previous lecture
Comparative case studies - method of difference
“the researcher selects cases in which the outcomes differ, compares the cases looking for the single factor that the cases do not have in common, and concludes that this factor is “the effect, or cause, or a necessary part of the cause, of the phenomenon.”
“applies to situations where the researcher is investigating outcomes that vary in degree (e.g., high, medium, and low levels of an outcome and identifies a factor that also varies in degree”
Process tracing: definition
“refers to case studies that “explicitly unpack mechanisms and engage in detailed empirical tracing of them”
“use deductive reasoning and ask, “If an explanation is true, what would be the specific process leading to the outcome?”
“often involve only one case because of the copious amount of information and detail that is required to trace a causal mechanism and to show that rival explanations do not account for an outcome.”
“depends on logic and has been compared to a detective sifting through evidence in order to solve a mystery.”
Advantages and disadvantages of case studies
Advantages
High internal validity (very accurate measurement of the case itself)
Context-dependent knowledge
Disadvantages
Low external validity (difficult to extrapolate findings to other contexts)
Replication is difficult
Danger of personal investment (blinders)
Comparisons are necessary for explanatory cases
Summary: case studies
Case studies have high internal validity and low external validity
Case studies are difficult to replicate
There is a danger of personal investment when conducting case studies
Oct. 30th
It’s good to use surveys when dealing with a large population. However it is harder to do more in-depth research because you can’t ask why someone wrote the answer they did. It is also difficult to ensure that the answers are honest and truthful.
Steps in a research project
Formulation of research question
Theorization and formulation of hypotheses
Conceptualization and operationalization
choice of research design
Data collection
Data analysis
Formulation of conclusions and dissemination of results
Quantitative Data Collection: Surveys
Quantitative data collection
Involves gathering numerical information on a large number of cases
In the next three lectures, we will review the three most common quantitative methods of data collection in political science
Surveys (quantitative alternative to interviews)
Secondary analysis
Quantitative content analysis
Terminology
Survey: Method of data collection that consists of asking the same questions to many individuals in the exact same order
Respondents: participants in a survey
Questionnaire: list of questions in a survey
Response rate
Response rate: percentage of contacted individual participate in a survey.
Example: you invite 300 university students to participate in a survey asking about their political affiliations. 80 students participate in your survey.
What is your response rate?
divided then multiplied by 100
“As the response rate decreases, the likelihood that the sample will not resemble the population increases–this will lead to poor statistical inference” - bias
You can perform an analysis to see if there is a significant difference in the characteristics of responders and non-responders. Why would you want to do this?
Nov. 4th
What steps could you take to improve your response rate - or the number of partici[ants that complete your survey
Keep the survey as short as possible to avoid incomplete surveys
Send multiple reminders to participants
Promote the survey as widely as possible (eg. social media, posters, other forms of advertisements that are acceptable to the Research Ethics Board)
Communicate the importance of the research to individuals invited to participate in the survey
Offer an incentive to participants that is acceptable to the Research Ethics Board (eg. gift card)
Surveys: Advantages vs. disadvantages
Advantages:
Best method to gauge attitudes and perceptions of a large population
Disadvantages:
Social desirability bias
non-attitudes (when participants don’t care about the issue under study in the survey)
Improper reading of questions or careless completion of the survey questions
Subject to survey design issues (poor design = poor data)
Don’t provide any insights as to the reason so for the responses (superficial)
Ordering of survey questions
How we order questions has major consequences for how participants respond to them
Best practices:
Alternate direction of questions. Why?
Move from general to specific questions
Avoid priming certain conditions over others
priming: persuading the answer
Best practices for the wording of survey questions and answers
To avoid random measurement error:
Keep the questions and answers short and simple
Be precise (as opposed to vague) in terms of your wording
How would you rate the government’s current performance?
How would you rate the Ontario government’s current performance on the issue of climate change
Avoid double-barreled questions (what are these)
To avoid systematic measurement error (bias):
Avoid the inclusion of authorities or experts in question
Avoid argumentative questions
Pilot testing of surveys
Involves administering the survey to a small group of individuals that fit your target population (2 to 3)
These individuals complete the survey and provide important feedback to the researcher on the design and comprehension of the survey
Pilot testing occurs before the widespread deployment of the actual survey
Allows the researcher to modify they survey before its wide-spread deployment to the sample
Survey modes: four common types (ON TESTS AND EXAM)
Mixed-mode surveys
Use a combination of modes to encourage participation and increase response rates
Mail and web
Mail and phone
Mail, phone, and web
etc.
Summary
The quantitative approach to interviewing is survey research, which is ideal for gauging the views and beliefs of large groups of respondents
Response patterns are heavily influenced by the nature of the survey
When evaluating survey results, we should always investigate
Sampling technique
Response rate
Question ordering
Question formulation
Survey mode(s)
Identify sampling method: random or not, random is ideal
Did they show the response rate, they might not always report it (could be considered for critique)
If they don't report, is it a representative (the implications)
Response rate (analysis)
The order of the questions
Questions formation, language
Survey mode(S)
What was the mode and what are the critiques with that (in person?)
did the same person answer twice
Was everyone surveyed (if online), electronically literate?
Nov. 6th
Quantitative Data Collection: Secondary Analysis
Unobtrusive research: Secondary Analysis
Unobtrusive research is the analysis of already existing data
Can you thing of some examples
Major advantages: few concerns about reactivity, few ethical concerns (if collected ethically to start with), verity time efficient
Major limitation: researcher has no control over the nature and availability of data
Types of unobtrusive research
This class discusses two main types of unobtrusive research
Analysis of existing data
Secondary analysis
document analysis
Content analysis
Quantitative
Qualitative
Secondary analysis: What is it
Secondary analysis is the ‘recycling’ of data compiled by others
Researchers perform new analyses on this data
Researchers need to understand the quality of the data compiled by others BEFORE performing their new analyses. Why?
Many data are available that you might expect - little pint in collecting data that are already out there
Ethical implications to collecting already existing data unless there is a very good reason to do so
Useful sources for secondary analysis (NOT TESTING ON)
National
Statistics Canada
Provincial, municipal statistics
Canadian Election Studies
Cross-national
OECD statistics
UN statistics
Freedom House
World Values Survey
Academic institutions have data resource centers
Secondary Analysis: Important considerations
What is the quality of the data?
What do we mean by this? Why determine this ahead of time?
Who compiled the data?
Does the person or organization collecting the data have a stake in the outcomes of the research?
People can create bias, measure incorrectly, word things specifically
How have the data been collected? How have the concepts been measured?
Why do you need to determine this ahead of time?
The way people measure their variables and define them can change the outcome
Are the data applicable to your research question?
Have all data been measured the same way? Why do you need to determine this ahead of time?
Research article analysis
Try to find the research question word for word stated in the article
“What is the effect…”
Explanatory or exploratory
they are looking for a relationship between the independent and dependent variables (explain) = explanatory (its this one)
experiment = MUST manipulate a variable
Use the exact independent and dependent variables stated in the article, also look at how they’re measured
Inductive or deductive
If there is a hypothesis tested it is deductive
If there is a hypothesis formed it is inductive
Key concepts in the study and how they are measured
what they are measuring and operationalizing
Operationalization is how they move to measurement (how did they choose to measure the variable)
how they measure it shows the dependent variable (??)
Research design: experimental, cross-sectional, longitudinal, case study
How did you come to that conclusion
Analysis
Advantages and disadvantages of the research design
Method of data collection: survey, secondary analysis, content analysis qualitative interviews, field research, or document analysis
Survey is quantitative (sometimes secondary analysis?)
Ethical considerations
this ethical consideration is the same as mine
ethical principles slide (justice, , _)
methods is a good place to check (everything hinges on the methodology)
Data collection and analysis slides
Go to the limitations sections of the article
They state the limitations
She wants to see that we follow the slides, especially for methods and data collections (she really wants us to use the slides)
Advantages and disadvantages of methodology
Nov. 11th
Quantitative Data Collection III: Quantitative Content Analysis
Content Analysis
Is the study of recorded communication
Do not confuse it with a literature review
Content analysis investigates primary material (I.e ., First hand material). for other researchers or studies
Any record communication can be subjected to content analysis (Written, Verbal, Non-verbal…)
Content Analysis in Political Science
For political scientists of most interest is political communication, usually between politicians, the public and the media
ex.
Parliamentary min
campaign posts
facebook posts
comments on new stores
press releases
newspaper coverage
what can we learn from record communication? Historical facts
considerations for assessing the validity of historical facts
is the author a credible witness/expert
does the author have a reason to lie/withhold information/exaggerate/ embellish
Can you think of some politicians whose record communications with the public and media have undergone extensive content analysis in terms of historical facts
ex. Trump, Steven Harper made a comment how there ones no colonization
Political attitudes and beliefs
Considerations for assessing the validity of political attitudes and beliefs
who is the author and who is the intended audience
does the author have a reason to lie/withhold information/ exaggerate/ embellish
Discourse
discourse focuses on the structure of political or public communication
there is a link between language and the way we view the world, and that politicians manipulate this for their own ends
it is argued that control and domination of representations allows politicians to generate worldviews consistent with their goals and to downgrade negate or eliminate alternative representations
Discourse: Orwellian Ex
“If a village full of innocent is bombed or thousands of people are relocated as a consequence of aggression and war we can choose to manipulate the representations of such a negative acts as types of positive or neutral events. we could call the first pacification for ex, and the second could be referred to as a rectification of frontiers”
“presented in this way issues such as pain, suffering and homelessness are hidden within neutral, placid or positive representations”
ex. politicians word there sentences that have manipulate them and change the world around us
Quantitative Content Analysis
the quantitative approach to content analysis is to quantify words, phrases and or other elements can see how often words are being repeated
words like illegal or phrases like democracy is at stake
especially appropriate in a deductive study (i.e one that tests theory or hypotheses) that aims to maximize reliability of measurement
Quantitative content Analysis: Manifest vs Latent Content
Manifest (or surface level) content is easier to quantify than latent (hidden) content
“In manifest content analysis, context is derived from the visible and literal meaning of the words—taken at face value.” (Delve, n.d., para 13)
Question: Is the literal meaning of words always easy to derive or consistently derived across multiple persons?
“In latent content analysis, you apply a deeper, interpretive analysis that seeks to infer underlying meaning from the words or phrases you choose to analyze.” (Delve, n.d., para 13)
Strengths and Weaknesses of Quantitative Content Analysis
Strengths
No reactivity
Not very costly in terms of time and money
Easy to replicate, especially, when analyzing public communication
Few ethical concerns; especially, when analyzing public communication
Well-suited for longitudinal research designs
Weaknesses
No control over nature and availability of data
Can be difficult to distinguish truthful from untruthful statements
Measurements and analyses can be difficult
**also have to be concerned with AI- generated communications and edited or manipulated communications
Summary
Content analysis is a type of unobtrusive research, which consists of the investigation of recorded communication
Political scientists are particularly interested in political communication between citizens, politicians, and the media
Quantitative content analysis transforms aspects of communication into numbers
This is most appropriate in deductive studies of manifest content that are most concerned about reliability of measurement
most studies are deductive
Nov. 13th
Qualitative data collection: qualitative interviews
Key features of qualitative interviews
Qualitative interviews are one-on-one discussions between a researcher and research participant
Much more detailed than surveys, and therefore much fewer respondents
Especially useful to investigate:
Internal explanations (examples?)
Historical accounts (examples)
Motivations (examples?)
Qualitative interviews: five aspects that have major implications
Type of interviewees (why?)
Structure of interview
structured interviews: have set question ordering and wording, allow for no improvising, and the researcher takes the lead (pros and cons?), interviewer takes the lead
pro: good for looking for patterns
con: don't know their thoughts on the topic
Unstructured interviews: vary from each other in questions, allow for much improvisation, and the respondent takes the lead (pros and cons?), interviewee takes the lead
An interview guide that contains all of the questions that will be asked is typically required by the REB regardless of the structure of the interview
The choice to conduct structured vs. unstructured interviews balances concerns about reliability, flexibility, and artificiality.
Method of communicating - interviews can take place in-person, over the phone, or in online chat room
choice about method balances concerns about reactivity, expected length, response rate, costs (what is your preference?)
Length of interview
Long interviews will result in more data (more time needed to transcribe and analyze)
Short interviews likely increase response rate, completion rate, and quality of answers (less time neede to transcribe and analyze)
What is your preference in terms of length?
Interview questions
question ordering and formulation have large consequences
Start with warm-up questions, move from abstract to specific
avoid social desirability bias
adjust language to participants
it's important to pilot test your interview guide and interview questions before data collection. Why?
Ways to decrease the potential of reactivity during qualitative interviews
reactivity is a major concern
Techniques to reduce reactivity/increase validity
Ensure you are in a quiet and private location
Consider your appearance: gender, race, attire
Consider cultural conventions
Emphasize how valuable respondent’s views are to you
Use probing questions on short answers
Employ the awkward silence
Minimize interruptions (Don’t interrupt the interviewee while they are talking!)
Stay neutral in terms of your verbal and non-verbal communication
Documenting qualitative interviews
Strategies of documentation
Taking notes (pros and cons?)
Audio recording (pros and cons?)
Video recording (pros and cons?)
No obvious best technique: each has distinct implications for reactivity, the observation of non-verbal cues, and accuracy/comprehensiveness
Which method of documentation do you prefer? Why?
After each qualitative interview is complete
Once the interview is over, write/generate transcripts: detailed (if possible complete) minutes of the interview. This can be very time consuming and costly.
Send transcript to respondent
Reduces ethical risks (guarantees informed participation)
Opportunity for additional validation
You may also want to document important ideas, concerns, or thoughts that come to you about the interview or data while generating transcripts.
Qualitative interviews: strengths and weaknesses
Strengths:
Avoids superficiality of surveys
great corroboration technique
Perfect for studying internal explanations
widely applicable
Weaknesses:
Limited reliability
more artificial than observation
reactivity
imperfect and selective memory of interviewees
Summary: Qualitative Data Collection - Interviews
Qualitative interviews are one on one discussions between researcher and research participant
Much more detailed than surveys, and therefore much fewer respondents
Especially useful to investigate:
Internal explanations
Historical accounts
Motivations
Nov. 18th
Questions:
What is the main research question?
Research goal?
Theoretical approach? Deductive or inductive?
What are the key concepts? How was the researcher decide to measure these concepts?
State whether the study uses an experimental, cross-sectional, longitudinal, and/or case study research design (explain how you came to that conclusion)
State which method(s) of data collection the study has employed (survey research, secondary analysis, content analysis, qualitative interviews, field research, and/or document analysis), and explain how you came to that conclusion.
Evaluate the methods of data collection by discussing whether the researcher has chosen appropriate methods, whether this project could have been conducted with different methods, and how relying on alternative methods of data collection would have affected the conclusions of the study.
Qualitative data collection: field research
Key features of field research…
In field research, researchers are physically present in the social setting they aim to understand
house of commons, cabinet meetings
Very common in descriptive, exploratory and explanatory research
Especially common in case studies or comparative case studies
Advantages and disadvantages if doing field research
Advantages
Compared to interviewing and unobtrusive techniques, field research has several advantages:
Measures behavior more directly
Avoids reported accounts, of which the validity might be difficult to assess
‘Being there’ enhances understanding and exposes more information
Disadvantages
Limited to observable behavior (Why?)
Raises its own problems regarding reliability, validity, and ethics (How?)
Comparatively time-intensive (Why?)
Field research: subtypes
Any project that involves a researcher traveling to the context of study can be considered field research
Some specific subtypes:
Structured observation: quantitative approach aimed at reliable and systematic measurement (e.g. sports statistics)
Ethnography: qualitative approach, aimed at understanding ‘culture’ from insiders’ point of view (common in anthropology)
Field research: setting and identity of researcher
Setting: Is it publicly accessible (open) or not (closed)?
Researcher Identity: Does everyone know they are being researched (overt observation) or not (covert observation)?
What might be some issues in terms of overt and covert observation?
Field research: overt and covert observations
These characteristics are central to a common trade-off between validity and ethics in field research
Overt observation is likely to produce measurement error
Reactivity
Direction and scope of measurement error can differ from one researcher to another
Covert observation overcomes these problems but raises ethical concerns
Unable to ensure voluntary and informed consent
Particularly problematic in closed settings
Field research: role of researcher
Most ethnographers are participant observers: they take part in the social processes they are trying to understand
Other field researchers maintain the role of complete observer: they strictly observe from the sidelines
The choice is partially philosophical–standpoint theorists tend to prefer participant observation (Why?)
The choice has consequences for objectivity and reactivity
Field research: data collection
Most ethnographers attempt to write as comprehensive notes as possible
Other field researchers make more targeted and limited notes
Comprehensive notes might produce more unexpected findings
Targeted notes more feasible in deductive research
What is your preference – comprehensive or targeted notes?
Field research: Trade-Offs between reliability and validity
Decisions about the way we make observations (participant vs. complete observer, ‘comprehensive’ vs. targeted notes) raise another common trade-off: between reliability and validity
In-depth observation of complex social settings is very difficult to conduct reliably
Selective and incomplete observation
More partial and distant observation addresses these problems but raises concerns about measurement validity
Summary: Field Research
In field research, researchers are physically present in the setting they are trying to understand
This immersion enhances understanding and unlocks access to more information, but raises its own methodological challenges
The most qualitative version of field research is ethnography, in which the researcher immerses themselves for a long time in a culture and try to understand it from the insider’s point of view
Overt research likely leads to reactivity, but covert research violates principles of research ethics
Participant observation and extensive note-taking can lead to novel insight, but might be difficult to do reliably and objectively
Nov. 20th
Choosing a method and approach
Question
You want to conduct research on a topic that is of interest to you.
What method and approach (quantitative or qualitative) would you choose?
What would guide your decision making?
Which choices are best
Some researchers feel strongly committed to a method of data collection (interviews, field research, unobtrusive) or approach (quantitative, qualitative)
It is difficult to maintain, however, that one method or approach is inherently superior to another
We can only decide based on the research question which method and approach are most appropriate
We cannot say that some methods are better than others, but we can say that some methods are better to answer certain research questions than others
Data collection: relative strengths
Data collection don’ts
Don’t use interviews
to find external explanations
when more direct measurement is possible
when you are interested in distant history
Don’t use field research
when you are not interested in behavior
Don’t use unobtrusive research
if no useful data are available
Data collection: combinations or mixed methods
Methods of data collection can be fruitfully combined in three ways:
1. Triangulation: corroborating findings based on one method of data collection by findings based on another (e.g., one overarching research question)
2. Facilitation: using one method of data collection to help research using a different method of data collection forward (e.g. one research question drives another research question)
3.Complementation: using different methods of data collection for different components of the same project (e.g., multiple research questions in one study)
Approaches: Don’ts (Why not)
Don’t use qualitative techniques when you want to…
provide a systematic overview of many cases
demonstrate the existence of law-like patterns of behavior
Don’t use quantitative techniques when you want to…
offer detailed narratives of subjective experiences
study a small number of cases
study concepts that are highly complex or onerous to measure
Approaches: Combinations
Qualitative research can facilitate quantitative research
by developing hypotheses that quantitative research can test more rigorously
by suggesting ways to quantify complicated concepts
Similarly, quantitative research can facilitate qualitative research
by identifying interesting patterns that require deeper understanding
by identifying worthwhile cases to study
The combination of quantitative and qualitative research can also have other advantages
the corroboration of findings from quantitative research by qualitative investigation and vice versa strengthens our confidence in their validity (triangulation)
A combination of qualitative and quantitative investigation enable researchers to study different components of a research puzzle (complementation)