Quality Concerns In Forensic Science
By Way of Review:
On March 11, 2004, ten bombs in backpacks and other small bags exploded on four commuter trains. One bomb was defused. The police did controlled explosions of the the others
Spanish authorities collected fingerprints from the detonator and sent them to worldwide agencies
Based on fingerprint, the FBI identified Brendon Mayfield
The FBI’s match was based on 20 points of similarities. It was one of may possible matches
The initial match was inappropriately made by an experienced tech, but was affirmed by the agency
Mayfield was in the database as a suspected domestic terrorist
He converted to Islam after marrying an Egyptian national, the wife of one of his college professors. He was never overt about his religion. In fact, until his arrest most people didn’t know he was practicing Islamic
After college he went to law school and in 1999 began practicing family law
In 2003 he represented Jeffrey Leon Battle in a child custody case
Battle was a member of the Portland 7- a group of people convicted of trying to go to Afghanistan to help the Taliban
Most people described Mayfield as patriotic
He served in the US army
Mayfield was innocent. He was only a suspect because of his legal philosophy and religion
The fingerprint match was made by an under qualified technician. Proper protocol would have been to have a certified examiner confirm the match
Why:
This case is famous but not isolated
Crime labs are like any other testing lab. Protocols are often based on “this is how we do it” rather than best practices
This led to the NAS report “A Path Forward” and has major impact on forensic science
A Path Forward:
The committee that was formed sponsored eight meeting. Four of these meetings which were open to the public
The four open meeting included testimonials and recommendations from representatives in the various fields of forensic science, laboratory administrators, academia, statisticians, and lawyers
Main Findings:
The forensic science system, encompassing both research and practice, has serious problems that can only be addressed by a national commitment to overhaul the current structure that supports the forensic science community in this country. This can only be done with effective leadership and highest levels of both federal and state government, pursuant to national standards, and with a significant infusion of federal funds
Recommendations:
During closed meetings, the committee deliberated, reviewed, and drafted a report with 13 recommendation
Create a National Institute of Forensic Sciences
Standardizes terminology and reporting practices
Expand research on the accuracy, reliability, and validity of the forensic sciences
Remove forensic science services from the administrative control of law enforcement agencies and prosecutors’ offices
Support forensic science research on human observer bias and sources of error
Develop tools for advancing measurement, validation, reliability, information sharing, and proficiency testing and to establish protocols for examinations, methods, and practice
Requires the mandatory accreditation of all forensic laboratories and certification for all forensic science practitioners
Laboratories should establish routine quality assurance procedures
Establish a national code of ethics with a mechanism for enforcement
Support higher education in the form of forensic science graduate programs to include scholarships and fellowship
Improve the medico legal death investigation system
Support AFIS interoperability through the development of standards
Support the use of forensic science in homeland security
NIJ Response to the NAS Report:
NIJ recognizes the need for fundamental research and currently supports projects that address the accuracy and reliability of forensic methods
Target areas: Handwriting, Firearms/Bullet, Tool Marks, Footwear impressions, Tire impressions, Bitemarks and fingerprints
A Decade Later:
Most disciplines have consistent terminology
National certifications exists for most fields
Not all labs are certified/ accredited and there is nor requirement to do so in most states
Major labs are adopting quality system
History of Quality:
The need for quality control is not new. It is a natural outgrowth of mass production
Evidence of established protocols and procedures can be found in Ancient Rome
Quality standards were one of the primary selling points for goods for Medievals Guilds
This is sometimes referred to as the Craftsmanship model
The focus was on training of individuals and relying on their skill and judgement
This model dominated manufacturing until the post WWII industrialization explosion
The craftmanship model has dominated forensic science until very recently
This is partially because large forensic labs are relatively new
Large high volume labs really start happening with the war on drugs when large volume of samples were being received, Need drives the Bus
Back to manufacturing: The strength of the United States in WWII was based on industrial might
Legislation was enacted to move the civilian economy into wartime manufacturing
High volume production of critical elements highlighted the need for regulated production standards. Bombs needed to explode, and at the right times
Product safety and consistency were important. In order to easy problems, the army began a standardized inspection regimen
This is the Mil-Std-105. It dictated the number of random samples needed in any batch by containers needed to confirm meeting specification
The sampling guidelines are still used in some industries
Next came statistical control charts- the work of Walter Shewhart who had lain the foundation of control charts in the 1920s
Which brings to W. Edwards Deming. A statistician with the USDA and Census bureau. He was an advocate of the Shewhart system and set the foundation of modern QC in Japan after the war
DMAIC:
This first phase is to Define the system. In this phase the customer requirements and project goals are established
Ultimately the customers is the court. In criminal cases, the police and investigators use the data. But it always has to end in the court. In the courtroom setting it is important to recognize that the customers
The second phases to Measure the relevant aspects of the current processes. This has two factors
1. the objective accuracy of the results
2. Jury outcome. As an industry the need for both has been clearly identified
After the relevant data is collected the third phase is to Analyze the data and determine causation. As mentioned above, causes like errors, deliberate misstatement and lack of relevant education of jurors are all contributing to the situation
Based on these data the improve phase can begin. In this phase a plans for correcting the defects need to be made. Once a “new” process is decided then tested and validated it can be implemented and then monitored in the Control phase
As an industry, forensic science is in between the analyze and improve phases. Sources of error relating to laboratory procedures are being addressed through review and implementation of laboratory Quality Control Systems (“North Carolina State Crime Laboratory Quality Policy Statement”
However this approach is still ignoring the need of the most important customer to understand the evidence in court. To address this remaining factor to the overall quality of forensic evidence one needs to study the jurors real ability to understand the scientific evidence and then if a deficiency is found, identify and implement the needed training
Up Next:
Now we should discuss 6-sigma and ISO
If you study QC 6-Sigma is most often quoted on the engineering side of things. However in forensic science we follow ISO
Look at both system
Six Sigma:
Six Sigma is a statists based approach to approving business process by decreasing defects and problems
It is a team based approach
There are two basic guiding philosophies DMAIC and DMADV
DMAIC: Define, Measure, Analyze, Improve, and Control
DMADV: Define, Measure, Analyze, Design, Verify
Define purpose of the project, process or service, identify and set realistic and measurable goals, as seen from the perspectives of the organization and the stakeholders, to create the schedule and guidelines for the review and to identify and assess potential risks. A clear definition of the project is established during this step, and every strategy and goal must be aligned with the expectations of the company and the customers
Define the purpose of the project, process or service, identify and set realistic and measurable goals
In context of laboratory services, the purpose is fairly concrete: Provide accurate information that assists the trier or fact in reaching an accurate verdict
Additional goals may include assisting investigators, time or cost efficiencies or any other needed outcome
Measurement: Obtaining and recording the factors that are critical to quality. This includes defining requirements and markets segments, evaluating which components most important to the quality, risk assessment, determining process capability and product capability, Once the values for these factors are known, then an effective approach can be taken to start the production process
Measurement: Obtaining and recording the factors that are critical to quality. It is important here to determine which metrics are critical to the stakeholder and to translate the customer requirements into clear project goals
Analysis: Includes developing design alternative identifying the optimal combination of requirements to achieve value within constraints, developing conceptual designs, evaluating then selecting the best components, then developing the best possible design. It is during this stage that an estimate of the total life cycle cost of the design is determined. After thoroughly exploring the different design alternatives, what is the best design option available for meeting the goals?
Here the needs of laboratory sciences differ from production. The trial and error approach often used in business in not sufficient for the needs of labs
Best practice models come from reviewed literature professional peers and first person research
Define: Here the plan for what will be done is finalized
Verify: Make sure the new system meets the goals, Essentially starting the circle again
ISO:
Established in 1947 The International Organization for Standardization is a non-governmental worldwide federation of national standards bodies for approx 100 countries, one from each country
ISO promotes the development of standardization and related activities in the world with a view to facilitating the international exchange of goods and services and to developing cooperation in the spheres of intellectual, scientific, technological and economic activity
Trivia Point: ISO is not an acronym. If it were it would be ISO
It is derived from the Greek isos meaning equal which is the root of the prefix iso- that occurs in a host of terms such as isometric and isonomy equality of laws, or of people before the law
ISO standards are general they say what sort of procedures a certified entity must have in place
The procedures themselves are developed by the entity implementing the system
Standard operating procedures would another step down the chain
At regular intervals the certifying agency would audit the entity to see that the Quality Procedures satisfy the international standard
The audit would also verify that the SOP support the QP
And of course the audit would verify that the company is following their established system
Unlike Six Sigma, there is no established team structure for the development of the procedures
The system is more flexible
Better address laboratory needs
Why ISO?
Six Sigma, establishes a regression to the mean. Implicity it seeks to minimize error
However, in forensic there is a 0 error allowed this is a threshold model
International standardization:
Striving for industry wide standardization is that results from consensus agreements reached between all players in that industrial sector
What does that mean for forensic science?
All participating bodies would have the same level of expectations of performance and accuracy