Sampling and Integration in Mixed Methods Research

Introduction to Sampling and Integration in Mixed Methods

• This lecture marks the final session before the holiday break, focusing on the critical issues of sampling and integration within mixed methods research.

• Integration is emphasized as a foundational element of mixed methods, distinguishing it from simply conducting two separate studies.

• Learning Objectives:

  • Identify the definition and importance of sampling in quantitative ($quant$), qualitative ($qual$), and mixed methods research.

  • Understand key sampling issues across three primary mixed methods designs: convergent, explanatory, and exploratory.

  • Recognize the different levels at which integration can occur (design, methods, and interpretation).

  • Evaluate the "fit" between integrated findings, including confirmation, expansion, and discordance.

The Nature and Importance of Sampling

• Definition: Sampling is the process of selecting a subset of individuals, cases, or data from a larger population to participate in research.

• Impact on Research Claims: The sampling method directly determines the scope and validity of the conclusions that can be drawn.

• Quantitative Considerations: Sampling methods influence the generalizability of findings to the broader population.

• Qualitative Considerations: Sampling influences the depth, credibility, and quality of the data, determining which stories are told and which patterns are observed.

• Sampling and Integration: The chosen sampling approach can either facilitate the easy linking of datasets or limit the potential for meaningful integration.

Quantitative Sampling Procedures and Issues

• Population Clarity: Researchers must define the parameters of the population of interest (who they are and where they are located).

  • Example: A study claiming to represent "Pacific people" that only samples Samoan participants is limited in the conclusions it can draw about the broader Pacific population.

• Sample Size and Statistical Power:

  • Power: This refers to the probability of finding a statistical effect when an actual effect exists in the population.

  • Power Analysis: A technical procedure used to determine the minimum sample size required to detect a specific effect.

  • Issues with Size:

    • Samples that are too small result in low power and unreliable findings.

    • Samples that are too large may waste resources or detect statistically significant effects that have no practical or real-world meaning.

• Probability Sampling Approaches: These involve random selection where everyone in the population has an equal chance of being chosen.

  • Random Sampling: The ideal approach where selection is purely by chance.

  • Stratified Sampling: Subgroups (strata) are created (e.g., age brackets $15$ to $25$ and $26$ to $35$), and random sampling is conducted within those specific brackets to ensure representation.

  • Cluster Sampling: Groups (clusters) are randomly selected (e.g., five specific laboratory streams), and everyone within those clusters is surveyed.

• Non-Probability Sampling Approaches: These do not give everyone an equal chance of selection, often leading to bias.

  • Convenience Sampling: Selecting easily accessible participants (e.g., psychology undergraduate students).

  • Snowball Sampling: Initial participants recruit others they know, causing the sample to grow like a snowball rolling down a hill.

• Sampling Bias:

  • Systemic Exclusion: For instance, online surveys may exclude those with limited access to or interest in technology, such as older generations.

  • Non-Response Bias: Occurs when certain types of people choose not to respond.

  • Self-Selection Bias: Occurs because people generally volunteer to participate based on specific personal characteristics.

Qualitative Sampling: Purpose over Numbers

• Core Logic: Unlike quantitative sampling, the goal is not broad representation but depth, meaning, and context. The focus is on "information-rich cases."

• Sample Size and Quality: Large numbers do not necessarily indicate high-quality qualitative research; very small samples can yield profound insights.

• The Concept of Saturation:

  • A common guide for sample size where researchers continue data collection (e.g., interviews) until no new insights or themes are emerging.

  • While often cited as a goal, saturation is not always necessary for a high-quality depth of analysis.

  • Example: Typically, a researcher might conduct approximately $15$ individual interviews to reach this point.

• Purposive (or Percussive) Sampling:

  • Intentional recruitment of specific people or groups who meet very narrow criteria to provide rich detail.

  • Example: Targeting individuals who have specifically undergone Cognitive Behavioral Therapy ($CBT$) for anxiety-related issues.

• Snowball Sampling in Qualitative Contexts: Often used to reach marginalized or difficult-to-contact communities (e.g., searching for the Tuvaluan community, a small Pacific subgroup in New Zealand, through word-of-mouth connections).

Sampling Issues in Specific Mixed Methods Designs

• Convergent Design:

  • The goal is to collect $quant$ and $qual$ data at the same time.

  • Recommendation: Use the same individuals for both strands to facilitate easier comparison. Using different samples can introduce extraneous information that complicates integration.

  • Size Disparity: Normally, the $quant$ sample is much larger than the $qual$ sample.

  • Parallel Questioning: Regardless of the strand, researchers should address the same concepts using parallel questions (e.g., self-esteem scales in the survey and open-ended self-esteem questions in the interview).

• Explanatory Sequential Design:

  • Starts with $quant$ data to identify trends, followed by $qual$ data to explain them.

  • Recommendation: Follow up with the same individuals from the first phase.

  • Sampling Logic: The $quant$ results guide the $qual$ sampling. For example, a researcher might select individuals who scored in the highest $10\%$ on a depression measure for a follow-up interview to explain those high scores.

• Exploratory Sequential Design:

  • Starts with $qual$ insights to build a subsequent $quant$ study.

  • Sampling Logic: The $qual$ phase uses specific criteria; the $quant$ phase uses standard power analysis to determine size.

  • Integration through Building: Themes or wordings from the $qual$ phase are used to generate hypotheses or develop items for a new measurement scale.

Levels of Integration

1. Design Level Integration

• Integration is built into the sequence and relationship of study phases.

• In Convergent designs, integration occurs because both strands address a single research question from different angles.

• In Exploratory and Explanatory designs, integration occurs through the sequencing where one phase directly informs or explains the other.

2. Methods Level Integration

• Connecting: The datasets are linked through the participants (e.g., selection for phase $2$ is based on participation in phase $1$).

• Building: Using results from one phase (like themes or specific participant quotes) to inform the data collection of the next (e.g., developing survey items).

• Merging: Bringing separate datasets together for analysis to look for alignment or divergence. This requires conceptual matching (e.g., ensuring a survey scale for self-esteem matches the conceptual interview questions).

• Embedding: Integration occurs at multiple points throughout the study, common in intervention trials or experiments (e.g., collecting data pre-, during, and post-therapy).

3. Interpretation and Reporting Level Integration

• Narrative Integration: Describing findings in a single report.

  • Weaving: Presenting $quant$ and $qual$ findings together, theme by theme.

  • Contiguous: Presenting sections separately (all $quant$ then all $qual$) within the same report.

  • Staged: Publishing different phases of a study in separate papers over several years (common in longitudinal research).

• Data Transformation:

  • Content Analysis: A process of turning $qual$ data into $quant$ data by coding and counting the frequency of words or concepts.

  • Example: Categorizing $2000$ open-ended survey responses about COVID-19 vignettes (e.g., descriptions of characters "Sione" or "Louise") to count how often "anxiety" was correctly identified.

• Joint Displays: The use of visual representations, such as tables or matrices, to present $quant$ and $qual$ data side-by-side.

  • Example Table Content: A theme like "Academic Workload" paired with a statistic (e.g. $74\%$ reported high stress) and a supporting participant quote about assignments being due at the same time.

Evaluating the Fit of Integration

• Confirmation: Both the quantitative and qualitative data support the same conclusions (e.g., surveys and interviews both show high stress during exams).

• Expansion: The strands address the same phenomenon but in ways that deepen understanding. One dataset expands on the other (e.g., numbers show workload predicts stress; interviews explain it is due to "uncertainty").

• Discordance: Findings are inconsistent or contradictory. This is not a failure but an opportunity for further research.

  • Responses to Discordance: Examine bias, re-analyze data, collect more data, or revisit the validity of the constructs (e.g., checking if the "well-being" measure aligns with the participants' cultural worldview).

Questions & Discussion

Participant Question: If you are doing something like content analysis where you are quantifying qualitative data, doesn't the small sample size of qualitative research make that difficult?

Speaker Response: That is a typical issue with small qualitative samples. However, content analysis is ideal for very large qualitative datasets. For example, in a study I conducted on mental health literacy during COVID-19, I collected $2000$ survey responses. I provided vignettes—small scenarios—of characters like "Sione" (Tongan) or "Louise" (European) experiencing anxiety. Participants gave open-ended responses regarding what the character was experiencing, ranging from a single word to up to $50$ words. Because the responses were hundreds or thousands of "shallow" answers, I could use content analysis to transform that data into frequencies. This can also be used in media analysis, where you might count the frequency of specific topics across news articles over a set time period.