2: Grand Approaches to Research Design
A framework for the collection and analysis of the data.
What do I want to learn?
What is the nature of the research question?
What kind of explanation will I want?
Considered to be large, team-based approaches to research. Funding for big teams have money set aside for graduate research students.
Explanations
Nomothetic explanation:
Quantitative researchers often explain a phenomenon in terms of the proposed causes and effects that are expressed in laws and principles. General, meant to apply to people who were not part of the study. This is a nomothetic explanation.
Must satisfy three criteria of causation before the explanation is acceptable.
Correlation: proposed cause and effect have to vary together.
Time order: Proposed cause must precede the effect in time. Must show that the cause came before the effect.
Non-spuriousness: alternative explanations for the correlation observed have to be ruled out. Is there an undefined third variable?
Spurious: false or illegitimate.
Quantitative researchers choose research designs that satisfy these criteria.
e.g. what gender may affect which way you vote. Closed ended questions with options, first gender, then voting intention.
The causes are changing. The effect, dependent variable, does not change. We derive a broad model of causes that determines the one dependent variable, vote intention.
How does the urban/rural divide affect vote intention? Survey questions must be used.
Whole sections of surveys focus on demographics, socioeconomic focuses, public policy issues. These are all independent variables that are being studied.
In Quebec, the urban/suburb/rural divide is the greatest influencer of vote intention.
We’ll want a very large sample for statistical validity.
Survey results can be presented numerically using a table, by percentage, or visually using charts.
Idiographic explanation:
Qualitative researchers seek to produce a rich description of a person or group based on perceptions and feelings of the people studied. The aim is not to derive general laws and principles.
May also investigate how phenomena develop over time or how they result from interactions or hierarchy.
Usually involves a detailed story of the people that is based on empathetic understanding.
An open ended question so that people can give you essay-style responses. “I am supporting Quebec Solidarity because I want to vote the bums out and they’ve done a terrible job on issue X and hopefully they’ll take issue Y seriously and I trust them….” so on and so forth.
Let them speak! Not worried about constraints of scientific methods or stats. We do not care about generalizability.
We want to garner people’s deep explanations and opinions. Lots of data will be obtained! But you will have explanations about people’s decision making.
You may obtain opinions and ideas that have otherwise never existed: grounded theory.
Talking to people who hold the knowledge you seek, desiring deep explanations.
Once a design is selected, you must choose a questionnaire, structured interview, participant observation, ethnography, or experiments.
What’s your grand design? Surveys, polls?
Qualitative: interviews & focus groups which comes from the people you’ve interviewed. Focus groups drill down on something specific, e.g. voter apathy.
Mixed designs: questionnaire poll, then pull the interesting responses away into an interview or focus group.
Want to know about how the Senate runs? Observe the Senate. Apparently the Senate is more methodical.
Experimental Design
Experiments are best used to establish causation, especially when isolating the effect of few factors.
In politics, it is generally used for political campaign ads.
Manipulation
Experiments manipulate an independent variable to determine its influence on a dependent variable.
Usually, people are separated into treatment (manipulated independent var.) and control groups, where the dependent variable is then measured.
To social researchers, many independent variables of interest cannot be manipulated. Ethical concerns also preclude experiments. Many variables of interest also have long, complex causes that cannot just be easily manipulated.
Lab experiments: formal, institutional settings.
Field experiments: real-life, every day surroundings.
Classic experimental design
Random assignment to an experimental and control groups. Dependent variables are observed before and after manipulation of the independent variable. We expect the before measurements to be equal.
Notation:
Obs: observations made on a particular independent variable.
Exp: the experimental treatment
T: the timing of the observations.
The goal is to also eliminate rival explanations for the change in the dependent variable.
Other rival explanations are explained but refuted:
history: events other than the experiment
testing: participants may become more experienced at taking a test as a result of the pre-test
instrumentation: changes in the way a test is administered can account for a change in scores pre-test and post-test
mortality: people will leave the experiment before it is over.
maturation: people change over time and may have implications for the dependent variable.
selection: variations between pre-test and post-test may be the result of non-random selection.
Common concerns
Measurement validity: are the variables used in the study adequately measured? Standard scales?
Did the manipulation work? staying off of Facebook?
Generalization of the study’s findings: do the conclusions apply to other people? Can they be used as nomothetic explanations?
Representativeness of the study participants — do they reflect modern society?
Effects of the setting — regionality and applications
Historical effects — generalized into the past and the future?
Effects of pre-testing — participants may become sensitized to the experimental treatment, which affects responses. May be un-applicable to those who aren’t pre-tested.
Reactive effects — participants know that they are in an experiment.
Replicability?
Lab Experiments
Greater control over the research environment, enabling easier establishment of nomothetic causation.
Lab experiments suffer from low external validity — they may not mirror real-world experiences and contexts, inclusive of how the participants are recruited.
Quasi-experiments
Some characteristics of the experimental method but lack some of the features that help researchers establish causation.
Natural experiments: experiment-like conditions are produced by naturally occuring phenomena or changes brought about by people not doing research. Researchers can gather data.
Can be caused by policy changes, examine attitudes/stats before and after the change.
Evaluation research: examines the effects of organizational innovations such as a longer school day or greater worker autonomy.
Cross-sectional design
Observations are taken at one point in time; no before and after comparisons. No manipulation of independent variables, snapshots taken. Questionnaires and structured interviewing, structured observation & analysis of statistics or diaries.
Used by sociologists.
Entails the collection of usually quantitative data on more than one case. Interested in variation between people, families, nation states, and different groups. Many cases used to encompass all variations and produce large sample sizes.
2+ variable data collection, examined to detect patterns of association. Independent & dependent variables are examined simultaneously, making time order hard to prove.
Replicability characterizes cross-sectional research, as long as the procedures for selection, administration, and analysis are published.
Other designs may need to be employed to substantiate casual inference.
External validity is strong when there is a random sample. Most quantitative social research employs a cross-sectional design.
Given how some variables are pre-determined, it is easier to establish a casual relationship. Many variables of interest can be assumed to be temporally prior to other variables.
Longitudinal Design
Neil Nevitte longitudinal designs — attitudinal change towards institutions, politicians, voting behaviour. Research on 4 different timepoints to see the exaggeration of an effect. Same questionnaire. Requires different timestamps.
People no longer voted with their parents. Liberal ideas began to be accepted. Help inquiries with social research groups. Reflected in the ‘88 and ‘93 elections.
“Leduc Pammett elections” — embrace of Reform and the Bloc in 1993. Case studies around electoral reform.
Cases are examined at a particular time and again at later times, without the manipulation of an independent variable.
Insight into time-order of variables.
Panel study: the same people, households or other groups are studied on at least 2 different occasions. Needs rules for handling new entrants and exits.
Cohort study: people sharing the same experience are studied overtime, but the same people may not be studied each time. e.g. different graduating classes are cohorts.
Both usually take the form of repeated survey research using a self-completion questionnaire or structured interview.
Issues
Sample attrition: people die, move away, withdraw.
Few guidelines for determining the best timing for data collection. Continued participation can affect behaviour.
Case Study Design
A detailed and intensive analysis of a single case. Communities, families, organizations, persons, events, provinces, economic sectors.
Tendency to associate with qualitative research. Users usually favour participant observation or unstructured interviews. Can also use qualitative methods or a combination.
The case is an object of interest in its own right and provide an elucidation of the case.
Goal: collect in-depth, qualitative data that may be unique to time and place is characteristic of case study research. Often idiographic in nature. Tends to take an inductive approach.
Research design criteria are based on the appropriateness of these criteria upon their work. External validity is a topic that is being debated.
Types of case:
Critical case: clearly specified hypothesis, and case is chosen on the ground that it will allow a better understanding of the circumstances under which the hypothesis will/will not hold.
Extreme case: refute common-held hypotheses
Revelatory case: observe and analyze a phenomenon previously inaccessible to scientific investigation.
Problems can arise when research involves comparison of 2+ cases. Multiple-case study approaches result in ignorance of specific case details.
Case studies in omnibus bills and being linked back to one central theme.
Research orientation & design
Research design | Quantitative | Qualitative |
Experimental | Comparisons between experimental and control groups on the dependent variable. | Not used in qualitative research, may inspire or be inspired by qualitative findings. |
Cross-sectional | Survey question and structured observation on a sample at a single point in time. | Qualitative interviews or focus groups at a single point in time. |
Longitudinal | Survey research on a sample on more than one occasion in panel or cohort studies. Content analysis of documents relating to different time periods. | Ethnographic research over a long period, qualitative interviewing on more than 1 occasion, analysis of documents across time periods. |
Case study | Survey research is conducted on a single case with a view to reveal important features about its nature. | Intensive study done by interviewing of a single case, organization, person, family, or community. |
The divisions are not always perfect.
Case studies are fair for exams.