Study Notes: How Cross-Examination on Subjectivity and Bias Affects Jurors' Evaluations of Forensic Science Evidence

Core Research Objectives and Context

• Research Overview: This trial simulation experiment, conducted by William C. Thompson and Nicholas Scurich, investigated how county courthouse jurors evaluate forensic science evidence when challenged on the grounds of subjectivity and contextual bias.

• Primary Focus: The study examined reactions to a prosecution witness (a forensic odontologist) testifying about bite mark evidence under various cross-examination conditions.

• Key Finding Summary: Jurors found experts less credible and were less likely to convict when exposure to potentially biasing task-irrelevant information or the subjectivity of the analysis was admitted. Conversely, adopting context management (blinding) procedures appeared to "immunize" experts against these challenges.

• General Perception of Forensic Science: Previous research (Lieberman et al., 2008) suggests people generally view forensic science as highly trustworthy, believing the chances of false findings are extremely low.

• Scientific Criticism: Distinguished scientific bodies have criticized forensic practitioners for relying on methods that are inadequately validated, overstating findings, and failing to avoid bias (NAS 2009; PCAST 2016).

• Legal Standing: Despite scientific criticism, judges rarely rule such evidence inadmissible under $Daubert$ or $Frye$ standards. Consequently, the burden falls on lawyers to challenge "shaky but admissible" evidence through vigorous cross-examination, as suggested in $Daubert\,v.\,Merrell\,Dow\,Pharmaceuticals$ ($509\,U.S.\,579,\,1993$).

Theoretical Background: Subjectivity and Contextual Bias

• NAS 2009 Conclusion: The National Academy of Sciences reported that forensic experts are vulnerable to cognitive and contextual bias, which "renders experts vulnerable to making erroneous identifications."

• Empirical Support for Bias: Studies have demonstrated bias across multiple disciplines:

  • Latent Prints: Dror et al. (2006) showed fingerprint examiners were less likely to identify a print if told a suspect had a solid alibi.

  • Document Examination: (Miller, 1984; Stoel et al., 2014).

  • Bite Mark Analysis: (Osborne et al., 2014; Pretty & Sweet, 2010).

  • Other Fields: Bloodstain pattern analysis, forensic anthropology, and DNA analysis (Dror & Hampikian, 2011; Thompson, 2009).

• Relationship to Ambiguity: Research suggests context has a stronger influence when the data being interpreted are ambiguous and a weaker effect when the interpretation is obvious (17).

• Public Misconception: Television programs often portray pattern-matching as automated computer processes, whereas in reality, these are subjective human decisions reached after painstaking analysis.

Dror's Taxonomy and the Nature of Task-Irrelevant Information

• Five-Level Taxonomy (Dror et al., 2015): Distinguishes levels of task-irrelevant information:

  • Level 1: Information from the trace evidence itself.

  • Level 2: Information from reference samples.

  • Level 3: Case information (e.g., detectives' theories, witness statements).

  • Level 4: Examiners’ base-rate expectations.

  • Level 5: Organizational and cultural factors.

• Task-Relevance Definitions (NCFS):

  • Task-Relevant Information: Helps the examiner draw conclusions from the physical evidence designated for examination through the correct application of accepted analytic methods. For example, knowing the surface curvature of a fingerprint lift to assess distortion.

  • Task-Irrelevant Information: Factors that should not be considered, including a suspect's criminal history, statements to police, alibi information, or other implicating evidence (NCFS, 2015).

Context Management and Blinding Procedures

• Procedural Solutions: Forensic laboratories can reduce Level 3 bias by using "case managers" who mediate between investigators and examiners (36, 39, 40).

• Blinding Logic: This division ensures examiners only receive information necessary for the scientific test. They may learn the case background only after recording their final conclusions.

• Juror Sensitization: The study sought to determine if jurors appreciate the importance of blinding, especially when the forensic examination requires high levels of subjective judgment.

The Debate Over Blinding: Arguments and Professionalism

• Opposition Arguments: Some practitioners argue that professionalism and a commitment to objectivity are sufficient to overcome bias without procedural blinding.

• Thornton (2010) Quote: "I reject the insinuation that we do not have the wit or the intellectual capacity to deal with bias, of whatever sort. If we are unable to acknowledge and compensate for bias, we have no business in our profession…"

• Scientific Rebuttal: Critics argue this view is psychologically naive. Research on the "bias blind spot" (Pronin et al., 2002) shows people often recognize bias in others but not in themselves. Blinding is standard in medicine because conclusions often rest on researcher's subjective evaluations.

The Criminalist's Paradox and Evidence Independence

• Double-Counting: This occurs when a forensic scientist incorporates non-scientific evidence (e.g., a suspect's confession) into their scientific judgment. To the jury, it appears there are two independent pieces of evidence (confession + forensic match), when in reality they are non-independent.

• The Paradox: Thompson (36, 55) identifies "the criminalist's paradox" where an expert's effort to be "right" (by looking at all facts) actually undermines the independence and value of the scientific conclusion, potentially misleading the trier of fact.

Experimental Methodology: Participants and Procedure

• Recruitment: Participants were recruited from a jury assembly room at a county courthouse in the suburban Southwest United States.

• Sample Size: $N = 225$.

• Demographics:

  • Gender: $54\%$ female.

  • Age: Median is $42$ (Interquartile Range = $28$).

  • Education: $56\%$ graduated from at least a 4-year college.

  • Income: Median annual family income between $\$60,000$ and $\$80,000$.

  • Political Beliefs: $27\%$ liberal, $36\%$ middle-of-the-road, $37\%$ conservative.

• Procedure: Participants read a criminal case synopsis involving a bar owner extortion case where an assailant bit the owner. Police located a suspect (a gang affiliate). An expert, Dr. Krauss, concluded "to a scientific certainty" the bite mark matched the suspect.

• Compensation: Participants were paid $\$10$.

Experimental Design and Manipulations

• Design: $2 \times 4$ between-participants factorial design.

• Factor 1: Subjectivity-Cross Manipulation:

  • Condition A (Subjectivity-Cross): Included 10 specific questions highlighting interpretation, inconsistencies (e.g., a $4\,mm$ measurement difference), and the lack of formal standards.

  • Condition B (No Subjectivity-Cross): These questions were omitted.

• Factor 2: Task-Irrelevant Information Exposure:

  • Blind Condition: Expert did not see the biasing information due to lab blinding procedures.

  • Ignored Condition: Expert saw the information (knowing the suspect was a "known Crips gang member" who "keeps skating on charges") but claimed he had ignored it.

  • Used Condition: Expert saw the information and admitted considering it to see the "big picture."

  • Control Condition: No mention of exposure to task-irrelevant information.

Detailed Transcript Content: Subjectivity Cross-Examination

• Verbatim Interchange Highlights:

  • Q: "But that required some interpretation, didn’t it?" / A: "Well, yes, there is always an element of interpretation."

  • Q: "But when you measured the inter-canine distance on Mr. Wilson’s teeth it was four millimeters shorter?" / A: "Yes, but in my judgment that could have been caused by stretching of Mr. Johnson’s skin…"

  • Q: "There aren’t any formal rules in your field about what constitutes a match?" / A: "There is no formula… I rely on my knowledge, training and experience…"

  • Q: "So it’s a match because you say it’s a match?" / A: "Yes. I’m a qualified expert in this field."

Detailed Transcript Content: Exposure to Task-Irrelevant Information

• Biasing Notation: The phone log noted: "Suspect known Crips gang member. Keeps skating on charges. Never serves time. This time his gun jammed… Murphy [the lead detective] wants to connect him to the bite mark."

• Used Condition Rationale: The expert stated: "It is important for a forensic scientist to take into account all the evidence… You have to consider the big picture. If you don’t have all the facts you might miss something important."

• Prosecution Rebuttal (Blind): "…he didn’t even know whose teeth he was comparing… so how could Dr. Krauss have been biased?"

• Prosecution Rebuttal (Used): "Dr. Krauss did what every good forensic scientist would do… considered all the facts."

Measures and Statistical Results: Scientific Credibility

• Measure: Composite of four Likert items (rated 1–7) including trustworthiness, credibility, following proper procedures, and subjectivity vs. objectivity. (Cronbach’s $\alpha = 0.88$).

• Scientific Credibility Findings (ANOVA):

  • Main Effect: Exposure: F(3,196) = 5.29,\,p < 0.001,\,\eta_p^2 = 0.084.

  • Main Effect: Subjectivity: F(1,196) = 10.72,\,p < 0.001,\,d = 0.496.

  • Interaction: F(3,196) = 2.80,\,p < 0.001,\,\eta_p^2 = 0.046.

• Contrasts (Bonferroni):

  • Blind vs. Ignored: Mean difference = $0.57$, $95\%\,CI\,[0.204,\,0.945]$, $p = 0.02$.

  • Blind vs. Used: Mean difference = $0.744$, $95\%\,CI\,[0.335,\,1.12]$, p < 0.001.

  • Blind vs. Control: Mean difference = $0.58$, $95\%\,CI\,[0.190,\,0.944]$, $p = 0.02$.

• Interpretation: The expert was most credible when blind. Subjectivity cross-examination significantly reduced credibility specifically in the Control condition ($95\%\,CI$ difference between No/Yes Cross: $[0.35,\,0.794]$ and $[-1.12,\,-0.358]$, respectively).

Measures and Statistical Results: Verdict Analysis

• Model Significance (Binary Logistic Regression): \chi^2 = 23.34,\,df = 7,\,p < 0.001.

• Conviction Likelihood:

  • Participants not hearing the subjectivity cross were over nine times more likely to convict ($Exp(B) = 9.05,\,95\%\,CI\,=\,2.57,\,31.84$).

  • Participants in the Blind condition were over six times more likely to convict than those in the Control condition ($Exp(B) = 6.01,\,95\%\,CI\,=\,1.84,\,20.32$).

• Credibility as a Driver: When credibility ratings were added to the regression (\chi^2 = 137.19,\,df = 8,\,p < 0.001), the main effects of experimental variables disappeared. For each unit increase in credibility, the odds of a guilty verdict increased by $14.8$ ($95\%\,CI\,[6.88,\,31.91]$).

8Implications and Discussion

• Sensitivity to Quality: Results indicate that jurors are sensitized to scientific quality via cross-examination regarding subjectivity and lack of standards.

• Juror Scientific Sophistication: Unlike previous studies on social science blinding (McAuliff et al., 2009), jurors in this study appreciated the danger of contextual bias involving criminal history.

• Immunization via Blinding: Blinding not only makes the science more rigorous but protects the expert from cross-examination that would otherwise lower their credibility and the resulting conviction rate.

• Strategic Risks for Lawyers: Cross-examination about task-irrelevant information (like gang affiliation) carries a risk of reminding jurors of prejudicial facts about the defendant, even if intended to show expert bias.

Limitations and Future Research Directions

• Problematic Terminology: The transcript used "reasonable scientific certainty" and "match," phrases criticized by the NCFS (2015) but chosen for realism in current court environments.

• Stimulus Material: The study used written transcripts; future work should use video testimony and include jury deliberations for higher ecological validity.

• Scope of Disciplines: The findings based on bite mark evidence should be replicated in other pattern-matching fields like latent prints, tool marks, and footwear analysis.

• Level 2 Bias: Future research is needed to see if jurors appreciate "sequential unmasking" (Krane et al., 2008) or "linear sequential unmasking" (Dror et al., 2015) designed to mitigate Level 2 bias (exposure to reference samples before evidentiary interpretation).