HDO
What are the key features that define experiments as a research method? Provide an example of each feature.
manipulation of the independent variable, random assignment of participants to experimental and control conditions, and experimental control of other factors that could influence the outcome of the experiment
What is one example of âmanipulatingâ an independent variable?
if conducting an experiment on how writing effects health, the research might instruct some participants to write about traumatic experiences and others to write about neutral experiences.
How is random assignment different from random sampling?
random assignment is how the participants are placed into each group while random sampling is how those participants are selected from the population.
Why are experiments the best research method for establishing causality? (related: be sure that you really know the three criteria necessary for establishing causality â we covered this Week 2)
experiments provide the strongest evidence which supports the requirements to establish causality.
Why are experiments particularly good at solving the problem of spuriousness? (related: what is the âproblem of spuriousnessâ? How does random assignment address the problem of spuriousness? How does experimental control address the problem of spuriousness?)
well conducted experiments tend to be high in internal validity by ruling out extraneous variables that may produce a spurious association. spuriousness occurs when two factors appear related but are not. random assignment helps eliminate the threat to internal validity which influences spuriousness and controlling each variable from the start helps avoid spuriousness.
What is the experimental group? What is the control group?
experimental group: group which receives the variable being tested, control group: group which is not receiving any treatment
Can you still end up with experimental and control groups that differ on important dimensions even with random assignment? Why? And what should you (as the researcher) do to address this possibility?
Variation in experimental design: (1) different timing/measurement of the dependent variable â what is the difference between a post-test only control group design vs. a pre-test/post-test control group design?
post- test only control group is when the dependent variable is tested after te manipulation of the independent variable. The pretest-posttest control group is when the dependent variable is measured both before and after the manipulation of the independent variable.
Variation in experimental design: (2) number of independent variables manipulated. What is a factorial design? What are the benefits of using a factorial designâwhat does this experimental design get us/show us that a non-factorial experimental design does not? (Hint: the Rivera and Tilcsik 2023 study provides an example to illustrate this point)
factorial design is when two or more independent variables are studied in a single experiment. This is beneficial because it provides evidence of the impact of each factor and the joint effect of the factors.
Factorial design notation: what does a â2 x 2â factorial design tell us about this experiment?
when a design has 2 independent variables, each with 2 levels
indv1: negative gender stereotype, indv2: positive college student stereotype, each have 2 categories/levels: present or absent
Experimental context: what is a field experiment? What is one advantage of each type of experimental context (lab vs. field experiment)?
field experiments: a study carried out in a natural, real world experiment, one advantage over a lab experiment is that behavior is observed in a real-life setting in which individuals are unaware they are participants in a study. Field experiments allow for less control in design and implementation.
What are the major strengths of experiments as a research method? Provide an example of each strength.
experiments provide sound evidence of a causal relationship,
What is internal validity? What is external validity?
internal validity: studies with evidences that rules out the possibility that facts other than manipulated independent variable are responsible for the measured outcome. External validity: experimental findings that are generalized to other settings, measurements, populations, and time periods.
What are the major limitations of experiments as a research method? Provide an example of each limitation. Is there anything researchers can do to try to address (or minimize) these limitations?
experiments may not reflect real-world situations due to different behaviors within participants and settings, participants may cause biases, and humans can be difficult to measure.
What is the threat of âselectionâ (or âselection biasâ) and why are experiments good at eliminating this threat? Â
present in any experiment that has all variables except for random assignment, which could affect the outcome of the study, due to the possibility that those groups could be bias.
What are âreactive measurement effectsâ and why are they a threat to external validity?
when participants are aware they being observed, it is possible this affect their behaviors which is called reactive measurement effect. This would affect the validity because participants would try to figure out the purpose of the study which could lead to bias, they may alter their behavior to be favorable. It would be difficult to generalize the findings beyond the specific context of which the study was conducted.
What are âmanipulation checksâ intended to accomplish?
procedures used in experimental research to assess whether the manipulation of the independent variable had the intended effect on the participants. They are intended to confirm the validity of the results, the interpretation of the study, and allow for feedback so adjustments can be made.
What are the key features that define surveys as a research method? Provide an example of each feature.
a large number of respondents are chosen to represent the population of interest (bigger than interviews), close-ended questions, quantitative research method (gain estimates of characteristics, behaviors, and attitudes, and test hypotheses)
What does it mean to use survey data for the goal of âdescriptionâ? Can you provide an example? What does it mean to use survey data for the goal of âexplanationâ? Can you provide an example?
using data for the goal of description is presenting the data in a clear and concise manner to provide an overview of the observation. Using data for the goal of explanation involves exploring casualty or associations between different factors to understand the trends and patterns.
What limitations do surveys have (compared to experiments) when the goal is âexplanationâ? What advantage do surveys have (compared to experiments) when the goal is âdescriptionâ?
compared to experiments surveys have many limitations when it comes to explanation such as causality, control over variables, biases and cofounding factors. As far as advantages cost and time, ethical considerations, and real world context are advantages.
What is the difference between a cross-sectional and longitudinal survey design? What is one advantage that a longitudinal survey design has that a cross-sectional design does not share?
a cross sectional survey design is when data is collected at a single point in time providing a snapshot of the population while a longitudinal survey design is when data is collected from the same individuals or groups over multiple points in time.
an advantage of longitudinal is the ability to examine changes in variables and relationships over time.
What is face-to-face data collection? What is web-based data collection? Name one advantage and one disadvantage of each type of data collection (the general goal here to be able to justify why you might pick one mode of data collection over another for a particular survey â what are the principles a researcher should keep in mind?).
face to face date collection is interviews that directly interact with respondents to gather information. One advantage of this is high response rates while a disadvantage is the cost and time.
web based data collection is the administration of surveys or online questionnaires where responses are surveyed. An advantage of this is that it is cost effective while a disadvantage would be the possibility of biases.
If you survey the entire population that you care about (rather than a sample of that population), do you have to use statistics when reporting your survey findings? What about if you only surveyed a sample of the population you care about? Do you have to use statistics when you report these results? (hint: the answer to this question also draws from what you learned about sampling during Week 5 of our course)
statistics are still essential for summarizing and interpreting the data. Even if surveying the entire population, statistics give insights into the characteristics of the population. When surveying just a sample of the population the statistics are helping for drawing inferences about the rest of the population that was not surveyed.
What the primary benefit of drawing a probability sample for your survey?
it allows researchers to make statistical inferences about the broader population without having to ask every member of the population.
What are three ways that surveys are often bad (we discussed these during lecture)?
sampling bias, poor question design, and response bias
Why is a low response rate on a survey a problem? What kind of bias does it introduce? [check your understanding: do you really understand what selection bias is?].
a low response rate on a survey is a problem because it may result in a non representative sample of the population which can lead to sampling bias. This can also introduce non response bias which can distort survey results and lead to misleading conclusions. This also diminishes the precision of the survey.
What are the six principles to keep in mind for writing good survey questions? Can you write an example question that violates one of these principles?
clarity, relevance, unbiased, specificity, consistency, and sensitivity
an example of this would be âwould you agree that our companyâs product is the best on the market?â This violates the principle of biases because you uses suggestive language.
Why should we pre-test survey questionnaires?
pre testing is important so we can identify ambiguities and confusion, asses response variability, test survey logistics, minimize errors, and enhance validity and reliability
What should you (the researcher) do if your survey may be biased (say, because you werenât able to collect a random sample)?
researchers should acknowledge the limitations, consider alternative sampling methods, interpret the results cautiously, and validate the findings.
What are the three major advantages of surveys as a research method? Can you provide an example of each of these advantages (or describe it in your own words)?
efficiency and cost-effectiveness: a company sends out an online survey to their customers in hope to receive feedback on their business
versatility and flexibility: a sociologist is studying behavior towards climate change so he designs a survey with open and close ended questions to capture both quantitative and qualitative data
standardization and comparability: a government agency is conducting a survey using standardized questions and sampling methods to collect data from diverse demographic groups which allows for comparison
What are two disadvantages of surveys as a research method? For each disadvantage, identify another research method that does not have this disadvantage.
limited depth of information due to close ended questions, qualitative interviews or focus groups allow for deeper discussion about participants
potential for bias response which can compromise the validity and reliability, experimental research methods allow researchers to have better control over responses.
Big picture: when should you use qualitative over quantitative research methods for your study? Can you provide one example of a scenario where it makes more sense to use qualitative research methods? Can you provide one example of a scenario where it makes more sense to use quantitative research methods?
Qualitative: great for mysteries or developing new theory, Quantitative: great for testing theories
if a researcher is looking to understand the factors influencing job satisfaction qualitative methods would provide the best responses from employees.
if a researcher is looking at the relationship between exercise and blood pressure, qualitative research methods would be appropriate for testing hypotheses and providing numerical data.
Similarly: what do you lose the ability to say with qualitative research methods? What do you (as the researcher) gain?
qualitative research may limit the ability to generalize findings or make statistical inferences about relationships between variables. As the researcher you gain richness and depth of data, contextual understanding, flexibility and adaptability, and participant perspectives.
What is ethnography (definition, example)? What are the features that make ethnographic studies distinctive from in-depth interview based studies?
ethnography is the qualitative research method that involves cultures, communities, and social groups through field work and observation. An ethnographer interested in understanding the social dynamics of rural farming might spend serval months living among the farm studying the interactions, networks, rituals, etc.
immersive feedback, naturalistic observation, holistic understanding, and contextual embeddedness
Can you define (and provide an example of) each of the following types of ethnographic observations: covert vs. overt observation; participant observation vs. non-participant observation
covert: researchers observing participants without their knowledge or consent by blending into the research setting; example of this would be a researcher studying customer service interactions in a retail store but posing as a shopper
overt: researchers openly observing participants with their knowledge and consent; an example of this would be a researcher studying students in a classroom and openly tells the teacher and students about their presence.
participant observation: researchers actively participating in activities, rituals, and interactions of the group/community being studied; an example of this is an anthropologist studying tribal communities and fully immersing themselves in the community to gain deeper insights.
non participant observation: researchers observing behaviors and interactions from an external and non participatory position; an example of this would be studying public gatherings, such as protests or demonstrations, without actively participating in the event,
Be ready to explain the following concepts, as they relate to ethnography: choosing a research site or setting, gaining access to a research site, establishing roles and relationships with those in the field, choosing who or what to observe (see if you can find an example of each concept in the Calarco reading)
choosing a research site or setting is based on the research questions, objectives, and cultural/social setting they wish to study, researchers should consider the richness of data when selecting a site.
gaining access to a site involves establishing rapport, trust, and cooperation with the community who is in control of the setting. This involves networking, building relationships, and seeking permission to gain entry.
establishing roles and relationships with those in the field requires researchers to navigate relationships and adopt perspectives of the culture, along with providing clear communication , cultural sensitivity, and ethical conduct. They must consider factors such as access, relevance, and diversity.
Why is establishing strong relationships with those in the field so important for obtaining high-quality ethnographic data?
facilitating trust, access, credibility, insider perspectives, and ethical conduct. Strong relationships enhance the validity, reliability, and richness of the data, contributing to the overall rigor and integrity of the research.
What are the systematic/scientific steps ethnographers should follow in gathering and analyzing data? (e.g. this is the slide on field jottings, analytic memos etc. â can you walk someone through how ethnographic data is systematically collected and analyzed?)
starts with field jottings, field notes, writing analytic memos, and analyzed qualitatively
What is reflexivity? Why is this practice considered so important in qualitative research in particular? Can you provide an actual example of reflexivity from either the Calarco or Rivera article?
a common practice in qualitative research, researchers reflect on how their characteristic and presences shape the research process. Relationships are also shaped by the researchers experiences and personal characteristics
What is the main finding from Calarcoâs ethnography of homework rule enforcement at public schools? (again, can you give me a precise 1-2 sentence description of what this study discovered in your own words?)
What are the major advantages of ethnography as a research method (hint: we touched on this point indirectly multiple times in class, but it might also help to review the end of Chapter 9 in the textbook and the short reading from Pascoe 2018)? What is one way that the Calarco study illustrates these specific advantages?
rich and detailed data, holistic understanding, contextual sensitivity, flexibility and adaptivity, insider perspective
the calarco study provided very rich and intensive observations and data
What are the major disadvantages of ethnography as a research method (hint: we touched on this point indirectly multiple times in class, but it might also help to review the end of Chapter 9 in the textbook)? What is one way that the Calarco study illustrates these disadvantages? Â Â
very time intensive, resource intensive, subjectivity and bias, limited generalizability, ethical challenges, complexity, researcher reflexivity
the calarco study was very time intensive as she observed the school for 3 hours, twice a week for around 2 years
10
Interviews (this may be updated/refined after class on 3/28)
What are the key features that define interviews as a research method? Provide an example of each feature.
intended to yield rich and detailed information on participants experiences
primarily made up of open ended questions
generally flexible layout
What is the goal of the qualitative analysis of interview data? To ask the same question in a different way: what are interview studies trying to accomplish?
the goal is to yield rich and detailed information on participants experiences and interpretations
Big picture: how are interviews different from a survey? How are interviews different from ethnography? What are the relative advantages and disadvantages of each method?
What is a âsemi-structuredâ interview? Can you provide an example?
a semi structured interview is an interview with specific objectives but the interviewer has some freedom in meeting those objectives; an example of this would be arriving to an interview with specific questions and changing them up during the interview
What is a focus group? What are the disadvantages and advantages of this research method (can you draw from the New Coke case to provide an example of the advantages or disadvantages of this method)? Can you name one condition under which it would be very appropriate to use focus groups? One condition under which it would be less appropriate to use focus groups?
a focus group is an interview method in which a researcher collects data from a group by moderating group discussion
an advantage to this is that it allows the researcher to observe social interaction among participants while a disadvantage is it adds an extra layer of reactivity
it is important when individuals cannot or do not accurately anticipate the views of others
What principles should researchers (in a high-quality interview study) follow in selecting and recruiting interviewees?
Best practices for interviewing effectively: what are they? Can you provide an example of each practice?
obtain in depth responses, you do not need to cover every questions, listen t what the person is saying and ask for more information based off responses, let the interviewee talk
How do you analyze interview data? What is âcodingâ? Can you provide an example of coding? Why is this method an example of inductive logical reasoning?
look for emergent patterns and themes in the transcripts
coding is the sorting of responses into categories represented by words, more than one person should code the same transcript and compare
The relative advantages and disadvantages of interviews:
What kinds of insight does ethnographic observation provide that interviews alone cannot?
the researcher is able to speak directly with the evaluators about important topics when selecting a candiate, it also allows us to get detailed information about how the interviewee views the world
Can you list one example of when you would probably want to choose to use ethnography over interviews? One example of when you would probably want to choose interviews over ethnography?
if you are trying to evaluate a specific population in an area, pace, etc, ethnography will allow the researcher to immerse themselves in the findings
if you are looking to get information from employers, interviews would be the most helpful because you can have in depth conversations
What is one benefit of combining (some) interviews and (some) observations in the same study?
this would allow for a very in-depth understanding of the data and allow for maximum findings
[Note: the questions above are big questions â and to fully answer them, you should combine what you have learned from (1) lecture; (2) textbook Chapter 9; (3) Lamont and Swidler 2014; (4) the short Pascoe 2018 reading on ethnography].
What is the main finding from Riveraâs interview-based study of hiring practices at professional service firms? (again, in your own words, in 1-2 sentences)
Why were interviews in particular a good fit for answering Riveraâs research question? In other words, what specific insights did INTERVIEWS provide in this specific study that other research methods could not have provided?
the interviews allowed her to study the subjective interpretations and social processes of candidates
What are three advantages of qualitative research (both ethnography and interviews)? Can you provide an example of each advantage or explain it in your own words? Draw from lecture and from Chapter 9.
ability to observe people in their natural settings, in depth understanding of meanings, processes, and contexts, and allows for a flexible research design
What are three disadvantages of qualitative research (both ethnography and interviews)? Can you provide an example of each advantage or explain it in your own words? Draw from lecture and from Chapter 9.
does not allow for generalizing to a specific population, fewer controls on the research process, not a very efficient research strategy