Mixed Methods Research on Belief Bias

Integration in Mixed Methods Research

Integration in mixed methods research involves intentionally bringing together quantitative and qualitative approaches to achieve a greater understanding of a topic.
Researchers should articulate how and to what extent they integrate quantitative and qualitative approaches.
Integration is considered both the greatest advantage and the greatest challenge in mixed methods research.
Approaches to achieving integration vary, with some focusing on specific procedures, others on general stages, and still others on the rationale for using mixed methods.
Fetters et al.’s (2013) framework is used because of its combination of generality, specificity, and pragmatism.

Levels of Integration (Fetters et al., 2013)

Study Design Level: Conceptualization of the study and the design implemented.
- Basic designs: explanatory sequential (QUAN -> qual), exploratory sequential (QUAL -> quan), and convergent (QUAN + QUAL).
- Sequential designs: data collected and analyzed in phases.
- Convergent designs: data collected and analyzed independently, then brought together.
- Integration at the design level influences decisions about integration at other levels.
Methods Level: Linking methods of data collection and analysis.
- Integrating databases through sampling (connecting).
- Using one data collection procedure to inform another (building).
- Bringing databases together for analysis and comparison (merging).
- Linking data collection and analysis at multiple points (embedding).
- The study design informs integration at the methods level.
- Explanatory sequential design may use nested sampling.
Interpretation and Reporting Level: Mixing data sets to be more informative than alone.
- Describing data in a report (narrative).
- Converting one data type into another (quantizing qualitative data).
- Using a joint display (visual representation of data analyses).
- Integration at the study design level is crucial for rigorous mixed methods studies.
- Integration at the methods level is less common than at the interpretation and reporting level due to a lack of examples.

Purpose of the Article

To illustrate how integration can be achieved at the methods level and at the interpretation and reporting level in an explanatory sequential design.
The context is a study on belief bias in high school students’ evaluations of scientific arguments.

Belief Bias in Reasoning

Sound scientific reasoning involves evaluating the plausibility of a claim, evidence, methods, and source.
Belief bias undermines scientific reasoning, causing individuals to evaluate information based on consistency with their beliefs rather than evidence quality.
Belief bias is a failure to reason independently of one’s beliefs, affecting the evaluation of information.
Adolescents are susceptible to belief bias despite their capacity for abstract and critical thinking.
Klaczynski and Gordon (1996) found that adolescents evaluated belief-consistent studies as stronger and more valid than belief-inconsistent studies, despite identical flaws.
Klaczynski and colleagues replicated these findings across various belief-relevant topics (e.g., religion, gender, occupational goals).
Age is not strongly related to the development of scientific reasoning beyond childhood.
Belief bias is demonstrated in adults across various topics including capital punishment, nuclear power safety, HIV/AIDS, gun control, affirmative action, child care, climate change, and vaccinations.
Understanding how adolescents reason about belief-related scientific evidence is important for promoting scientific reasoning and minimizing belief bias.
Adolescents are an understudied population in belief bias research, with limited knowledge about the reasoning behind biased judgments.
Previous research predominantly used quantitative designs, providing effects at the group level.
Qualitative inquiry can be useful for understanding individual differences.
A mixed methods design enables investigation at both the group and individual levels.
This study investigates adolescents’ scientific reasoning about belief-relevant arguments.

Method: Integration at the Study Design Level

Explanatory sequential design was used to investigate how adolescents evaluated belief-relevant arguments about climate change.
The study consisted of a quantitative argument rating task and qualitative interviews.
The design began with collection and analysis of argument ratings (quantitative data), followed by collection and analysis of interviews (qualitative data).
Overarching question: How do adolescents evaluate belief-consistent and belief-inconsistent arguments with equally compelling justifications?
Quantitative phase: Participants rated the strength of arguments about climate change supported by plausible, fictional data.
Arguments had the same weaknesses to assess belief bias.
Qualitative phase: A subset of participants were interviewed to gain insights into their reasoning.

Method: Integration at the Methods Level

Integration implemented through connecting and building.
Connecting: Linking data through the sampling frame.
- Quantitative findings used to develop sampling criteria for the qualitative phase.
- Extreme-case sampling: identifying individuals with higher and lower levels of belief bias.
- Bias score: the difference in summed scores for belief-consistent and belief-inconsistent arguments.
- Lower bias score: objective reasoning.
- Higher bias score: less-objective reasoning.
- Participants were divided into two qualitatively distinct groups.
- Four students from each group were randomly selected for interviews.
Building: Using results from one data collection procedure to inform the other.
- Quantitative findings at the individual level differed from the group level.
- Interview protocol developed to investigate these differences in reasoning.

Method: Integration at the Interpretation and Reporting Level

Integration implemented through narrative and joint display.
Integration through Narrative: Describing quantitative and qualitative findings in a single report.
- Contiguous approach: reporting quantitative and qualitative findings in different sections, then organizing in an integrated results matrix.
Joint Display: Integrated results matrix to juxtapose quantitative results and qualitative findings.

Quantitative Phase

Setting: Suburban, all-male, public secondary school in New Zealand.
Participants: 62 male secondary students (mean age = $13.95$ years, $SD = 0.6$ ).
Instruments:
- Topic beliefs scale: measured beliefs about human impact on climate (9-point Likert-type scale, $\alpha = .86$ ).
- Argument strength rating task: Adapted from Taber and Lodge (2006); participants rated strength of arguments on a 9-point scale.
- Participants evaluated six evidence-based arguments on climate change (three against, three for). Each argument used one of three types of evidence: temperature, sea level, or glacier.
Data Collection: IRB approval, parental consent, and participant assent were obtained. Conducted in regular classrooms.
Data Analysis: Repeated-measures ANOVA to analyze argument strength ratings. $2$ (belief-consistency) x $3$ (evidence type).
- Eta squared ( $\eta^2$ ) for effect size ( $\eta^2$ = $0.01$ small, $0.06$ medium, $0.14$ large).

Qualitative Phase

Data Collection: Interview protocol designed to prompt explanations of argument ratings.
- Questions started with follow-up questions, moved to probe questions, and specification questions.
- Individual interviews were conducted 10 days after the experiment.
Retrospective reporting is susceptible to post hoc rationalizations minimized by time gap and context cues (Ericsson & Simon, 1993; Nisbett & Wilson, 1977).
Data Analysis: Thematic analysis using a five-step process (Braun & Clarke, 2006).
- Broad holistic scoring, extraction of descriptive phrases, initial coding, category development, and theme identification.
- In vivo coding was used to categorize relevant phrases.

Results

Quantitative Phase: Main effect for argument type was significant, $F(1, 61) = 8.00$ , $MSE = 5.13$ , p < .01, $\eta^2 = .116$ . Participants rated belief-consistent arguments higher (M = 5.03) than belief-inconsistent arguments (M = 4.37).
Qualitative Phase:
- More-objective (M = 6.5, SD = 1.5) and less-objective (M = 6.2, SD = 1.7) students did not differ in reading comprehension, $t(1, 17) = 0.38$ , $p = .712$ .
- More-objective (M = 1.23, SD = 0.87) and less-objective (M = 1.87, SD = 1.23) students did not differ in strength of beliefs, $t(1, 18) = 1.35$ , $p = .194$ .

More-Objective Group

Explicitly focused on the quantity of evidence.
Applied the same evaluation criteria independently of belief consistency.
Example quote from P31: "Because this [evidence] was based on one glacier and one area between two years, and 'coz it was one glacier, it doesn’t mean that all the glaciers around the world are the same."

Less-Objective Group

Focused on the quantity of evidence but only for belief-inconsistent arguments.
Applied evaluation criteria differently based on belief consistency.
Focused on plausibility or believability of the evidence when arguments were belief-consistent.
Example quote from P27: "Coz, it’s only giving one data set, like just one country . . . but in another country it could be different."

Discussion

Quantitative data suggest that adolescents can reason independently from their beliefs, though belief bias is common.
Scores at the individual level differed, with some students being more-objective and others less-objective.
Students in the more-objective group rated belief-consistent and belief-inconsistent arguments similarly.
Students in the less-objective group rated belief-consistent arguments higher.
The interview data indicated that students in both groups applied the same evaluation criteria to belief-inconsistent arguments.
Only students in the more-objective group applied the same standards to belief-consistent arguments.
Students in the less-objective group evaluated arguments differently based on belief consistency.
Holding a belief did not necessarily lead to biased reasoning; rather, biased reasoning occurred when individuals applied a more critical standard of evaluation to belief-inconsistent arguments.
Conceptual metacognition about the influence of beliefs on reasoning, and procedural metacognition to control against this influence is discussed as a possible explanation.

Contribution to Mixed Methods Research

The study illustrates how integration can be achieved at the methods and interpretation levels in an explanatory sequential design.
Integration at the study design level occurred through the intentional use of an explanatory sequential mixed methods design.
Integration at the methods level occurred through connecting and building.
Integration at the interpretation level occurred through narrative and joint display.
Multilevel mixed design.

Overview

Integration in mixed methods research is a vital approach that combines quantitative and qualitative methodologies to enhance the understanding of research topics. The process involves careful consideration of how these different methods can complement each other, enabling researchers to capture a more holistic view of their subjects. This integration is both a significant advantage and a notable challenge in mixed methods research, requiring researchers to articulate their approaches clearly and systematically. This study focuses on the integration mechanisms at various levels within an explanatory sequential design, specifically exploring how adolescents evaluate belief-relevant arguments related to climate change.

Integration in Mixed Methods Research

Integration in mixed methods research involves intentionally bringing together quantitative and qualitative approaches to achieve a greater understanding of a topic.
Researchers should articulate how and to what extent they integrate quantitative and qualitative approaches.
Integration is considered both the greatest advantage and the greatest challenge in mixed methods research.
Approaches to achieving integration vary, with some focusing on specific procedures, others on general stages, and still others on the rationale for using mixed methods.
Fetters et al.’s (2013) framework is used because of its combination of generality, specificity, and pragmatism.

Levels of Integration (Fetters et al., 2013)

Study Design Level: Conceptualization of the study and the design implemented.- Basic designs: explanatory sequential (QUAN -> qual), exploratory sequential (QUAL -> quan), and convergent (QUAN + QUAL).
- Sequential designs: data collected and analyzed in phases.
- Convergent designs: data collected and analyzed independently, then brought together.
- Integration at the design level influences decisions about integration at other levels.
Methods Level: Linking methods of data collection and analysis.- Integrating databases through sampling (connecting).
- Using one data collection procedure to inform another (building).
- Bringing databases together for analysis and comparison (merging).
- Linking data collection and analysis at multiple points (embedding).
- The study design informs integration at the methods level.
- Explanatory sequential design may use nested sampling.
Interpretation and Reporting Level: Mixing data sets to be more informative than alone.- Describing data in a report (narrative).
- Converting one data type into another (quantizing qualitative data).
- Using a joint display (visual representation of data analyses).
- Integration at the study design level is crucial for rigorous mixed methods studies.
- Integration at the methods level is less common than at the interpretation and reporting level due to a lack of examples.

Purpose of the Article

To illustrate how integration can be achieved at the methods level and at the interpretation and reporting level in an explanatory sequential design.
The context is a study on belief bias in high school students’ evaluations of scientific arguments.

Belief Bias in Reasoning

Sound scientific reasoning involves evaluating the plausibility of a claim, evidence, methods, and source.
Belief bias undermines scientific reasoning, causing individuals to evaluate information based on consistency with their beliefs rather than evidence quality.
Belief bias is a failure to reason independently of one’s beliefs, affecting the evaluation of information.
Adolescents are susceptible to belief bias despite their capacity for abstract and critical thinking.
Klaczynski and Gordon (1996) found that adolescents evaluated belief-consistent studies as stronger and more valid than belief-inconsistent studies, despite identical flaws.
Klaczynski and colleagues replicated these findings across various belief-relevant topics (e.g., religion, gender, occupational goals).
Age is not strongly related to the development of scientific reasoning beyond childhood.
Belief bias is demonstrated in adults across various topics including capital punishment, nuclear power safety, HIV/AIDS, gun control, affirmative action, child care, climate change, and vaccinations.
Understanding how adolescents reason about belief-related scientific evidence is important for promoting scientific reasoning and minimizing belief bias.
Adolescents are an understudied population in belief bias research, with limited knowledge about the reasoning behind biased judgments.
Previous research predominantly used quantitative designs, providing effects at the group level.
Qualitative inquiry can be useful for understanding individual differences.
A mixed methods design enables investigation at both the group and individual levels.
This study investigates adolescents’ scientific reasoning about belief-relevant arguments.

Method: Integration at the Study Design Level

Explanatory sequential design was used to investigate how adolescents evaluated belief-relevant arguments about climate change.
The study consisted of a quantitative argument rating task and qualitative interviews.
The design began with collection and analysis of argument ratings (quantitative data), followed by collection and analysis of interviews (qualitative data).
Overarching question: How do adolescents evaluate belief-consistent and belief-inconsistent arguments with equally compelling justifications?
Quantitative phase: Participants rated the strength of arguments about climate change supported by plausible, fictional data.
Arguments had the same weaknesses to assess belief bias.
Qualitative phase: A subset of participants were interviewed to gain insights into their reasoning.

Method: Integration at the Methods Level

Integration implemented through connecting and building.
Connecting: Linking data through the sampling frame.- Quantitative findings used to develop sampling criteria for the qualitative phase.
- Extreme-case sampling: identifying individuals with higher and lower levels of belief bias.
- Bias score: the difference in summed scores for belief-consistent and belief-inconsistent arguments.
- Lower bias score: objective reasoning.
- Higher bias score: less-objective reasoning.
- Participants were divided into two qualitatively distinct groups.
- Four students from each group were randomly selected for interviews.
Building: Using results from one data collection procedure to inform the other.- Quantitative findings at the individual level differed from the group level.
- Interview protocol developed to investigate these differences in reasoning.

Method: Integration at the Interpretation and Reporting Level

Integration implemented through narrative and joint display.
Integration through Narrative: Describing quantitative and qualitative findings in a single report.- Contiguous approach: reporting quantitative and qualitative findings in different sections, then organizing in an integrated results matrix.
Joint Display: Integrated results matrix to juxtapose quantitative results and qualitative findings.

Quantitative Phase

Setting: Suburban, all-male, public secondary school in New Zealand.
Participants: 62 male secondary students (mean age = $13.95$ years, $SD = 0.6$ ).
Instruments:- Topic beliefs scale: measured beliefs about human impact on climate (9-point Likert-type scale, = .86).
- Argument strength rating task: Adapted from Taber and Lodge (2006); participants rated strength of arguments on a 9-point scale.
- Participants evaluated six evidence-based arguments on climate change (three against, three for). Each argument used one of three types of evidence: temperature, sea level, or glacier.
Data Collection: IRB approval, parental consent, and participant assent were obtained. Conducted in regular classrooms.
Data Analysis: Repeated-measures ANOVA to analyze argument strength ratings. $2$ (belief-consistency) x $3$ (evidence type).- Eta squared (^2) for effect size (^2 = $0.01$ small, $0.06$ medium, $0.14$ large).

Qualitative Phase

Data Collection: Interview protocol designed to prompt explanations of argument ratings.- Questions started with follow-up questions, moved to probe questions, and specification questions.
- Individual interviews were conducted 10 days after the experiment.
Retrospective reporting is susceptible to post hoc rationalizations minimized by time gap and context cues (Ericsson & Simon, 1993; Nisbett & Wilson, 1977).
Data Analysis: Thematic analysis using a five-step process (Braun & Clarke, 2006).- Broad holistic scoring, extraction of descriptive phrases, initial coding, category development, and theme identification.
- In vivo coding was used to categorize relevant phrases.

Results

Quantitative Phase: Main effect for argument type was significant, $F(1, 61) = 8.00$ , $MSE = 5.13$ , p < .01, ^2 = .116. Participants rated belief-consistent arguments higher (M = 5.03) than belief-inconsistent arguments (M = 4.37).
Qualitative Phase:- More-objective (M = 6.5, SD = 1.5) and less-objective (M = 6.2, SD = 1.7) students did not differ in reading comprehension, $t(1, 17) = 0.38$ , $p = .712$ .
- More-objective (M = 1.23, SD = 0.87) and less-objective (M = 1.87, SD = 1.23) students did not differ in strength of beliefs, $t(1, 18) = 1.35$ , $p = .194$ .

More-Objective Group

Explicitly focused on the quantity of evidence.
Applied the same evaluation criteria independently of belief consistency.
Example quote from P31: "Because this [evidence] was based on one glacier and one area between two years, and 'coz it was one glacier, it doesn’t mean that all the glaciers around the world are the same."

Less-Objective Group

Focused on the quantity of evidence but only for belief-inconsistent arguments.
Applied evaluation criteria differently based on belief consistency.
Focused on plausibility or believability of the evidence when arguments were belief-consistent.
Example quote from P27: "Coz, it’s only giving one data set, like just one country . . . but in another country it could be different."

Discussion

Quantitative data suggest that adolescents can reason independently from their beliefs, though belief bias is common.
Scores at the individual level differed, with some students being more-objective and others less-objective.
Students in the more-objective group rated belief-consistent and belief-inconsistent arguments similarly.
Students in the less-objective group rated belief-consistent arguments higher.
The interview data indicated that students in both groups applied the same evaluation criteria to belief-inconsistent arguments.
Only students in the more-objective group applied the same standards to belief-consistent arguments.
Students in the less-objective group evaluated arguments differently based on belief consistency.
Holding a belief did not necessarily lead to biased reasoning; rather, biased reasoning occurred when individuals applied a more critical standard of evaluation to belief-inconsistent arguments.
Conceptual metacognition about the influence of beliefs on reasoning, and procedural metacognition to control against this influence is discussed as a possible explanation.

Contribution to Mixed Methods Research

The study illustrates how integration can be achieved at the methods and interpretation levels in an explanatory sequential design.
Integration at the study design level occurred through the intentional use of an explanatory sequential mixed methods design.
Integration at the methods level occurred through connecting and building.
Integration at the interpretation level occurred through narrative and joint display.
Multilevel mixed design.

Overview

Integration in Mixed Methods Research

Integration in mixed methods research involves intentionally bringing together quantitative and qualitative approaches to achieve a greater understanding of a topic.
Researchers should articulate how and to what extent they integrate quantitative and qualitative approaches.
Integration is considered both the greatest advantage and the greatest challenge in mixed methods research.
Approaches to achieving integration vary, with some focusing on specific procedures, others on general stages, and still others on the rationale for using mixed methods.
Fetters et al.’s (2013) framework is used because of its combination of generality, specificity, and pragmatism.

Levels of Integration (Fetters et al., 2013)

Study Design Level: Conceptualization of the study and the design implemented.- Basic designs: explanatory sequential (QUAN -> qual), exploratory sequential (QUAL -> quan), and convergent (QUAN + QUAL).
- Sequential designs: data collected and analyzed in phases.
- Convergent designs: data collected and analyzed independently, then brought together.
- Integration at the design level influences decisions about integration at other levels.
Methods Level: Linking methods of data collection and analysis.- Integrating databases through sampling (connecting).
- Using one data collection procedure to inform another (building).
- Bringing databases together for analysis and comparison (merging).
- Linking data collection and analysis at multiple points (embedding).
- The study design informs integration at the methods level.
- Explanatory sequential design may use nested sampling.
Interpretation and Reporting Level: Mixing data sets to be more informative than alone.- Describing data in a report (narrative).
- Converting one data type into another (quantizing qualitative data).
- Using a joint display (visual representation of data analyses).
- Integration at the study design level is crucial for rigorous mixed methods studies.
- Integration at the methods level is less common than at the interpretation and reporting level due to a lack of examples.

Purpose of the Article

To illustrate how integration can be achieved at the methods level and at the interpretation and reporting level in an explanatory sequential design.
The context is a study on belief bias in high school students’ evaluations of scientific arguments.

Belief Bias in Reasoning

Sound scientific reasoning involves evaluating the plausibility of a claim, evidence, methods, and source.
Belief bias undermines scientific reasoning, causing individuals to evaluate information based on consistency with their beliefs rather than evidence quality.
Belief bias is a failure to reason independently of one’s beliefs, affecting the evaluation of information.
Adolescents are susceptible to belief bias despite their capacity for abstract and critical thinking.
Klaczynski and Gordon (1996) found that adolescents evaluated belief-consistent studies as stronger and more valid than belief-inconsistent studies, despite identical flaws.
Klaczynski and colleagues replicated these findings across various belief-relevant topics (e.g., religion, gender, occupational goals).
Age is not strongly related to the development of scientific reasoning beyond childhood.
Belief bias is demonstrated in adults across various topics including capital punishment, nuclear power safety, HIV/AIDS, gun control, affirmative action, child care, climate change, and vaccinations.
Understanding how adolescents reason about belief-related scientific evidence is important for promoting scientific reasoning and minimizing belief bias.
Adolescents are an understudied population in belief bias research, with limited knowledge about the reasoning behind biased judgments.
Previous research predominantly used quantitative designs, providing effects at the group level.
Qualitative inquiry can be useful for understanding individual differences.
A mixed methods design enables investigation at both the group and individual levels.
This study investigates adolescents’ scientific reasoning about belief-relevant arguments.

Method: Integration at the Study Design Level

Explanatory sequential design was used to investigate how adolescents evaluated belief-relevant arguments about climate change.
The study consisted of a quantitative argument rating task and qualitative interviews.
The design began with collection and analysis of argument ratings (quantitative data), followed by collection and analysis of interviews (qualitative data).
Overarching question: How do adolescents evaluate belief-consistent and belief-inconsistent arguments with equally compelling justifications?
Quantitative phase: Participants rated the strength of arguments about climate change supported by plausible, fictional data.
Arguments had the same weaknesses to assess belief bias.
Qualitative phase: A subset of participants were interviewed to gain insights into their reasoning.

Method: Integration at the Methods Level

Integration implemented through connecting and building.
Connecting: Linking data through the sampling frame.- Quantitative findings used to develop sampling criteria for the qualitative phase.
- Extreme-case sampling: identifying individuals with higher and lower levels of belief bias.
- Bias score: the difference in summed scores for belief-consistent and belief-inconsistent arguments.
- Lower bias score: objective reasoning.
- Higher bias score: less-objective reasoning.
- Participants were divided into two qualitatively distinct groups.
- Four students from each group were randomly selected for interviews.
Building: Using results from one data collection procedure to inform the other.- Quantitative findings at the individual level differed from the group level.
- Interview protocol developed to investigate these differences in reasoning.

Method: Integration at the Interpretation and Reporting Level

Integration implemented through narrative and joint display.
Integration through Narrative: Describing quantitative and qualitative findings in a single report.- Contiguous approach: reporting quantitative and qualitative findings in different sections, then organizing in an integrated results matrix.
Joint Display: Integrated results matrix to juxtapose quantitative results and qualitative findings.

Quantitative Phase

Setting: Suburban, all-male, public secondary school in New Zealand.
Participants: 62 male secondary students (mean age = $13.95$ years, $SD = 0.6$ ).
Instruments:- Topic beliefs scale: measured beliefs about human impact on climate (9-point Likert-type scale, = .86).
- Argument strength rating task: Adapted from Taber and Lodge (2006); participants rated strength of arguments on a 9-point scale.
- Participants evaluated six evidence-based arguments on climate change (three against, three for). Each argument used one of three types of evidence: temperature, sea level, or glacier.
Data Collection: IRB approval, parental consent, and participant assent were obtained. Conducted in regular classrooms.
Data Analysis: Repeated-measures ANOVA to analyze argument strength ratings. $2$ (belief-consistency) x $3$ (evidence type).- Eta squared (^2) for effect size (^2 = $0.01$ small, $0.06$ medium, $0.14$ large).

Qualitative Phase

Data Collection: Interview protocol designed to prompt explanations of argument ratings.- Questions started with follow-up questions, moved to probe questions, and specification questions.
- Individual interviews were conducted 10 days after the experiment.
Retrospective reporting is susceptible to post hoc rationalizations minimized by time gap and context cues (Ericsson & Simon, 1993; Nisbett & Wilson, 1977).
Data Analysis: Thematic analysis using a five-step process (Braun & Clarke, 2006).- Broad holistic scoring, extraction of descriptive phrases, initial coding, category development, and theme identification.
- In vivo coding was used to categorize relevant phrases.

Results

Quantitative Phase: Main effect for argument type was significant, $F(1, 61) = 8.00$ , $MSE = 5.13$ , p < .01, ^2 = .116. Participants rated belief-consistent arguments higher (M = 5.03) than belief-inconsistent arguments (M = 4.37).
Qualitative Phase:- More-objective (M = 6.5, SD = 1.5) and less-objective (M = 6.2, SD = 1.7) students did not differ in reading comprehension, $t(1, 17) = 0.38$ , $p = .712$ .
- More-objective (M = 1.23, SD = 0.87) and less-objective (M = 1.87, SD = 1.23) students did not differ in strength of beliefs, $t(1, 18) = 1.35$ , $p = .194$ .

More-Objective Group

Explicitly focused on the quantity of evidence.
Applied the same evaluation criteria independently of belief consistency.
Example quote from P31: "Because this [evidence] was based on one glacier and one area between two years, and 'coz it was one glacier, it doesn’t mean that all the glaciers around the world are the same."

Less-Objective Group

Focused on the quantity of evidence but only for belief-inconsistent arguments.
Applied evaluation criteria differently based on belief consistency.
Focused on plausibility or believability of the evidence when arguments were belief-consistent.
Example quote from P27: "Coz, it’s only giving one data set, like just one country . . . but in another country it could be different."

Discussion

Quantitative data suggest that adolescents can reason independently from their beliefs, though belief bias is common.
Scores at the individual level differed, with some students being more-objective and others less-objective.
Students in the more-objective group rated belief-consistent and belief-inconsistent arguments similarly.
Students in the less-objective group rated belief-consistent arguments higher.
The interview data indicated that students in both groups applied the same evaluation criteria to belief-inconsistent arguments.
Only students in the more-objective group applied the same standards to belief-consistent arguments.
Students in the less-objective group evaluated arguments differently based on belief consistency.
Holding a belief did not necessarily lead to biased reasoning; rather, biased reasoning occurred when individuals applied a more critical standard of evaluation to belief-inconsistent arguments.
Conceptual metacognition about the influence of beliefs on reasoning, and procedural metacognition to control against this influence is discussed as a possible explanation.

Contribution to Mixed Methods Research

The study illustrates how integration can be achieved at the methods and interpretation levels in an explanatory sequential design.
Integration at the study design level occurred through the intentional use of an explanatory sequential mixed methods design.
Integration at the methods level occurred through connecting and building.
Integration at the interpretation level occurred through narrative and joint display.
Multilevel mixed design.