GFALE II Domain VII Root Cause Analysis in Forensic Science

Corrective Actions

potential solutions that eliminate or minimize the risk of repeating the nonconforming or departure from policies and procedures.

*It is a requirement when any error or nonconformity is identified

How are corrective actions initiated?

As required by ISO 17025

-an investigation is initiated to determine root cause of situation or condition

_________ is a critical step of determining corrective actions and may be the most important part of establishing proper corrective actions.

Root cause analysis (RCA)

"just culture"

"a culture that recognizes that competent professionals make mistakes and acknowledges that even competent professionals will develop unhealthy norms (shortcuts, "routine rule violations"), but has zero tolerance for reckless behavior."

Preventative Action

establish procedures to identify needed improvements and potential sources of nonconformities

-proactive process

-follows similar process of RCA to identify the best solutions to prevent or minimize the chance of nonconformity from occurring

Goal of RCA

-to learn from adverse events & "near misses" (nonconformities)

-to implement PROACTIVE change in order to reduce further similar events that might compromise lab report or opinion integrity.

An important feature of the RCA is that it is a __________ analysis:

blame-free

Labs under ISO 17025 are required to

establish policy and procedure and shall designate appropriate authorities for implementing corrective action when nonconforming work or departures from the policies and procedures in the management system or technical operations have been identified

Department of Justice should encourage FSSPs & FSMPs to

strive to be "high reliability organizations" and ensure a culture of constant self-monitoring and self-improvement by incorporating established practices of "just culture" and learning from errors

Root Cause Analysis policies should include:

- objective GUIDANCE as to when a RCA should be conducted

- the regular provision of appropriate TRAINING to key personnel on how a RCA should be conducted

- TRAINING to all employees within the FSSP on RCA principles and processes, to enhance the quality of RCA and its acceptance within the lab

- proper CONSTRUCTION of investigative team conducting RCA

- definition & procedures for INVESTIGATION that identifies extent of adverse events and causal factors in blame-free environment, prioritizing continuous improvement of lab quality, safety, reliability by learning from adverse events

-RECOMMENDATIONS that identify proactive action to minimize chance of future recurrence of adverse events identified in RCA

- GUIDELINES that define when and how to identify other cases that may have also been affected by identical or similar adverse event, and obligation to conduct a retrospective re-analysis of and address such cases

- COMMUNICATION of existence of adverse event to individuals impacted by adverse event

-PROVISION OF SAFE HARBOR to employees who report adverse events or near misses, including immunity for participation in RCA and limitations on disclosure of materials generated in course of RCA

- IMPLEMENTATION of intervention designed to minimize the chance of future similar adverse events & to appropriately redress injury caused by adverse event

- DOCUMENTATION of both adverse event & proposed interventions in manner that does not reveal specific individuals or case information, & makes learnings from RCA publicly available for review and benefit of other FSSPs and FSMPs

What 3 types of error does Dr. James Reasons give?

1. Decision error:

one made because information, knowledge, or experience is lacking

2. Skill-based error:

one made while engaged in a familiar task

3. Perceptual error:

one made because input to one of five senses is degraded or incomplete

The 3 types of error typically fall into 1 of 4 categories:

1. unsafe acts:

those performed by operator

2. preconditions for unsafe acts:

environmental factors

3. supervision:

management actions affecting operator

4. organizational influences:

culture, policies, or procedures of the organization that affect the operator

What are violations?

1. Routine violation:

habitual, repeat departures, enabled by "bending of rules"

2. Exceptional violation:

a willful departure outside norms, not condoned by management

What are adverse events?

"intentional departures from accepted practice";

*any occurrence or worsening of an undesirable or unintentional consequence, including an abnormal laboratory finding, or outcome caused by or associated with particular action by FSSP or FMSP

What is a near miss?

unplanned event that did not result in injury, illness, or damage but had potential to do so

-only a fortunate break in the chain of events prevented an injury, fatality or damage

Types of unsafe acts

*unsafe act:

violations--> routine, exceptional

*unsafe act:

errors--> skill-based error, decision error, perceptual error

What are the preconditions for unsafe acts?

*environmental factors:

physical environment, technological environment

*personnel factors:

communication/coordination/planning, fitness for duty

*conditions of operator:

adverse mental state, adverse physiological state, chronic performance limitation

What can cause adverse effects in supervision?

- inadequate supervision

- inappropriate planned operations

- failure to address a known problem

- supervisory violation

What are the causes of undue organizations influences?

- resource management

- organizational climate

- organizational process

RCAs are ___________, not ___________, and their purpose is _________, not __________.

event reviews; performance evaluations

learning; punishment

The purpose of the RCA is to identify any and all ______ factors.

contributing

Personnel & discipline issues should be handled through a _______ process from RCA

separate

Are activities and output of RCA admissible as evidence?

no; are excluded from discovery in litigation to ensure this purity of purpose

The "just culture" focus of RCA creates shared accountability:

the system is responsible for providing an environment that is optimally designed for safe care & staff is responsible for their choices of behavior and for reporting system vulnerabilities

Construction of RCA

*should be performed by a multidisciplinary team:

-4-10 members from all levels of staff, with fundamental knowledge of the specific area involved

- people who were not involved with specific incident to ensure objectivity in the review

* facilitator appointed who was not directly implicated in incident

- no placing blame or speculating

- schedule that balances ambitions and time, allowing for sufficient debate

Investigation in RCA

*event should be analyzed for causal factors

*detailed review of event by team

* identify problems- WHAT went wrong

- is this one time adverse event or recurring error?

*identify root causes/ contributing factors- WHY it went wrong?

- focus on objective causes & minimize "bad apple" causation conclusions where possible

* prioritize factors that contributed to harm, evaluating both their severity & the probability that these factors will cause harm in future

* develop interventions that conform with prioritization and likelihood of repetition of various factors

Recommendation in RCA

* team should make specific, prioritized recommendations for interventions that are intended to prevent occurrence of similar events.

-These recommendations should be made in writing and stored for future review as needed

Implementation of RCA

implement those interventions, considering quality of analysis, cost of suggested interventions, and their likely real-world impact on safety and reliability

Evaluation in RCA

evaluate the interventions and take subsequent additional action as needed

Professional Standards and a "Just Culture"

balances "blame free" event reviews with need for professionals, including FSSPs, to be personally accountable for adherence to reasonable standards of professional conduct

What should be done to preserve integrity of RCA as a blame-free event review?

*any disciplinary process be additional to, and separate from, RCA

*that individual in charge of making determinations about disciplinary action be informed by, but not reporting to or involved in, RCA itself

3 Examples of root cause analysis deployed lab environments

1. Veterans Health Administration

2. FBI Lab

3. Washington DC Department of Forensic Science

Steps in Washington DC Department of Forensic Science - STEP 1

* RCA team agrees on PROBLEM STATEMENT:

-a concise, complete, and accurate single sentence describing what the problem (nonconformity) is

-should be understood & agreed upon by all members of RCA team

* Examples:

-proficiency failed

-case jacket missing documentation

-wrong individual was identified

-sample was contaminated

-wrong results were reported

* solutions should NOT be offered at this stage of the process

The _________ is central to the RCA: it acts as the reference point and compass for the analysis

problem statement

Steps in Washington DC Department of Forensic Science - STEP 2

* RCA team engages in "5 Whys" Approach

-starting with problem statement, team asks: "why (did this process fail)?" repeatedly until root cause is identified

-these questions generate a recommendation for a new process that will help avoid an undesired situation in the future, rather than simply blaming a forgetful employee.

Steps in Washington DC Department of Forensic Science - STEP 3

* sorts root cause into one of four contexts defined by the nature of cause and effect:

*Simple:

-stability, clear cause & effect relationships;

-the "right" answer is often obvious

*Complicated:

-clear cause & effect relationship but not immediately obvious;

-at least one "right" answer exists;

-requires expertise;

-testing may be required to obtain necessary information;

-better practices, rather than best, may be appropriate

*Complex:

-unpredictable;

-no one "right" answer may exist;

-"adequate" may be the best answer;

-answers reveal themselves upon exploration

*Chaotic:

-searching for the right answer is pointless;

-turbulence predominates;

-cause and effect relationships are impossible to determine due to constant change and flux

Steps in Washington DC Department of Forensic Science - STEP 4

*REMEDIATION STAGE

- corrective actions are RESPONSIVE to an event

- preventative actions are PREDICTIVE and are meant to stop a problem BEFORE IT OCCURS

remediation

sets in place actions and processes intended to stop and prevent the root cause that lead to the problem.

-AKA: a corrective or preventative action

Steps in Washington DC Department of Forensic Science - STEP 5

* SOLICITS FEEDBACK on the effects of the remediation

* feedback is necessary to inform system about the results of the solution attempted , the remediation

-results of remediation attempt will inform whether it was successful or not and, if so, to what degree

- if satisfied with results, step 6 of the process can continue

Steps in Washington DC Department of Forensic Science - STEP 6

*original process can RESUME WITH REMEDIATED CHANGES IN PLACE

-clear criteria for resuming the process must be established and agreed upon by the team prior to the remediation

Steps in Washington DC Department of Forensic Science - STEP 7

* PERIODIC CHECKS SHOULD BE MADE TO PROVIDE FEEDBACK ON THE CONTINUAL PREVENTION OF THE NONCONFORMITY

- to ensure that the remediation continues to prevent the nonconformity.

-The periodicity should be relevant and meaningful to the remediation and the timing of new process

What does ISO/IEC 17025 require?

*that all selected changes resulting from corrective action investigations be documented and implemented.

*labs are required to monitor results of corrective actions to ensure the effectiveness of the solutions;

-this monitoring should similarly be documented

What should RCA panel communicate?

panel conducting RCA has obligation to communicate adverse event to individuals or agencies involved in casework that my have been affected by the adverse event

Because when adverse events affect casework, the life & liberty of a human may be (or may not be) affected, forensic science service providers have a duty to:

correct the impact of the adverse event, which should include a new, amended, or supplemental report with the correct results and an explanation of the initiating adverse event distributed to the various parties in a case.

What are the top 3 barriers to RCAs?

-lack of time (55%)

-unwilling colleagues ( 34%)

- inter-professional differences (31%)

*each of these can be addressed by:

-experienced facilitation & support from senior management within the organization

What are characteristics of successful RCA facilitators?

* must be interested in facilitating and documenting problems

* must have excellent listening skills

* must be naturally inquisitive

* must be comfortable speaking in front of a group

* must be detail oriented

* must have a relatively calm disposition

* must have a good rapport with front-line personnel and management

Properly done, RCAs can be time-consuming team activities that include:

*investigation

*development

*implementation

*evaluation

-they should be deployed with an eye towards the SEVERITY & RISK of a problem

What is a level 1 nonconformity (FBI)?

situations or conditions that directly affect and have a fundamental impact on the quality of work product or integrity of evidence

What is a level 2 nonconformity (FBI)?

situations or conditions that may affect the quality of the work, but does not, to any significant degree, affect the fundamental reliability of the work product or the integrity of the evidence

Safety Assessment Code (SAC) Matrix (VHA)

*balances the severity of an error (measured by its impact on the patient and levels of patient care) with the frequency of the error occurring

*VHA evaluates whether a RCA is needed every time a "Serious Adverse Event" has occurred

-MUST conduct for a 3, MAY conduct for a 2, UNLIKELY to conduct for a 1

*RCAs should be conducted both on actual adverse events and on adverse events that could have occurred but for a fortuitous intervention or timely discovery

-should score "near misses" in the SAC Matrix as if they were an event that actually happened

It is also important that the RCA process include steps designed to understand whether or not ___________________, and if so, the extent of the adverse events

the error has been repeated

How can providing a "Safe Harbor" encourage transparency and reporting of error?

providing a "safe" environment-- an environment that encourages and prevents negative use of important quality and/or reliability information--

*enhances participation in RCAs,

*and thus improves both their frequency and their substance

What are the key characteristics of a Safe Harbor?

1. qualified immunity for participants

a. an individual should not be disciplined in any way for participating in RCA or offering a candid and good faith assessment of the role of other in an incident under review

b. an individual who reports an error should receive positive consideration from any disciplinary body if the individual self-reports an error WITHIN A REASONABLE TIME AFTER THE ACCIDENT (this does not protect the individual from any liability)

2. Protection from discovery for notes, minutes, correspondence, and/or reports generated as part of RCA.

-Materials generated in RCA should not be discoverable in civil or criminal litigation related to incident under Peer Review Protection Acts

3. nothing in safe harbor should be viewed as limiting discovery rights of individuals to information about the underlying facts related to an adverse event.

Purpose of Safe Harbor is

to ensure that no one is penalized as a result of his or her participation in a valuable event review designed to improve fair administration of justice

Why do recurrent defects occur?

*human error is often incorrectly identified as root cause of defect.

-the corrective action implemented is retraining, but actual root cause is not identified and remediated

Why is human error an insufficient root cause?

doesn't identify real problem causing defect

How can root case be identified?

asking "why did the human err?" will help you identify the true root cause of a problem

What is Title 21 of the US Code of Federal Regulations?

*corrective and preventative action.

*it states that identifying product defects and the cause of defects, and implementing actions to correct the problem are required as part of a robust quality management system (QMS).

-regulation does not say how to do this

-regulation does not identify good/bad corrective & preventative action programs, systems or steps

What can be inferred from having 3 or more root causes identified after a robust RCA investigation has been performed?

the root cause has not yet been found and additional investigation is required

Visual inspection example:

What is the acceptable level of true rejection according to Judi See?

90% or higher, so the rate of reliability of human error visual inspections is not acceptable because it is lower than this (85%)

In RCA investigation, "why did the human err?" will

result in better root cause for which a solution may reduce recurrence of the event if the same human error should repeat

Why might a human err?

1) confusing procedure:

poorly written; vague, or superfluous instructions; hard to follow

2) internal (personal) distractions:

personal life, illness; injury; disability

3) external distractions:

noisy environment; frequent interruptions; poor layout or workflow

4) unaware the procedure existed:

inaccessible to employee; inadequate communication of procedural changes

5) procedural updates:

updates occur too often; too many revisions; employee can't keep up with procedural changes; procedures released inconsistently

6) intentional misuse or willful misconduct:

employee is disgruntled or dissatisfied

7) inadequate electronic clearance on automated systems:

another employee's ID or login was used; unauthorized workarounds; unable to access automated systems

8) missed a step in procedure or work instruction:

instructions are unclear or illegible

9) inadequate paper-based systems:

printer errors; misnumbered or missing pages

10) inadequate software-based systems:

software glitch

11) employee apathy:

lack of motivation; bored with the job; unaware of the effect on quality

12) employee unable to perform certain steps:

inaccessible or inappropriate fixtures, tools, or equipment

Truly robust manufacturing system is:

one that will produce quality products despite the constant potential for human error

Corrective Action: Retraining

What can the investigator ask about corrective actions?

1) is the defect of such an impact that it must be corrected?

2) is this an isolated incident?

what is a robust corrective action?

ensures consistency across processes when multiple individuals are performing the same tasks, which reduces or even eliminates human variation

What is a core responsibility of forensic organizations?

*risk management.

-managing risk should be comprehensive and systematic, scaled appropriately to potential outcomes based on likelihood, consequence, and impact

Investigations into unexpected outcomes, even positive ones, provides feedback into a system of continuous quality and improvement: the "________" is more important than the "_________" for an organization to learn and grow

why; what

What are hazard risks?

risks with negative outcomes

What are opportunity risks?

risks with positive outcomes

What is a high reliability organization (HRO)?

one that avoids catastrophes in a risky setting or complex environment where accidents are expected

Ex: hospitals, airlines, military, forensic science

What should a risk management plan consist of?

*should be proportionate, aligned, comprehensive, embedded, and dynamic;

*needs to be appropriate and cost effective;

*appropriate to the level of risk (or proportionate & scaled to the potential adverse or unwanted outcomes);

*comprehensive to the whole organization's commitment to its purpose, mission, values;

*an ongoing process, dynamic in response to changing circumstances and supportive of an organization's resilience

What are the 4 responses to risk?

* tolerate:

risk may be tolerable

* treat:

while the organization continues the activity giving rise to the risk, action is taken to constrain the risk to an acceptable level

*transfer:

shifting responsibility for the risk may be the best option for the organization

*terminate:

some risks can be remediated only by terminating the activity giving rise to the risk

Quality

an acceptable range of variation in an outcome or product; that is, it is an explicit statement of the level of acceptable risk manifest in a product

What is RCA?

method of problem solving (ultimately, risk mitigation) that tries to identify event, typically a fault, problem, or a nonconformity in a quality system that exceeds the acceptable range

What is necessary and sufficient to have initiated a nonconformity?

root cause

-no prior causes can be used to explain the nonconformity as the root cause is the basis for the chain of actions that occurred

What is the causal chain?

an action in one area can create other actions or results in others

Why should periodic checks be made?

provide feedback on continual prevention of the nonconformity

-are the changes still preventing the nonconformity?

Who should perform an RCA?

*should be a team effort

recommended that, at minimum, a quality specialist, subject matter expert not involved, and manager be members of the team, with others joining as necessary

Why should the problem statement be kept general?

being too specific can suggest avenues of investigation based on misleading assumptions;

-it also shouldn't cost more to look for problems that aren't there than it does to make the mistake in the first place

What is the key to getting to the root cause of a nonconformity?

avoid assumptions and mental traps (both a type of bias)

A standard phrase in RCA thought is:

"People do not fail; processes do."

5 Whys Approach

identifies root causes by first stating the problem and then repeatedly asking the questions "Why?" until the root cause is identified

Example of problem statement and 5 Whys

The analysis of a sample/specimen was not completed by deadline

1. Why? The instrument failed to complete the run

2. Why? The instrument ran out of carrier gas

3. Why? The tank of carrier gas emptied mid-run

4. Why? More gas was not ordered

5. Why? An employee forgot to order more gas

*It might take more than 5 whys to get to the root cause

What is the purpose of the 'Whys'?

they provide a problem analysis, a breakdown of effects and causes leading from the problem to the root cause

What should be avoided in a 5 Why approach?

*mental blocks or biases on the part of the team.

-team should not stop at superficial symptoms rather than continuing on to deeper causal events.

-team should have resources needed to ask questions and identify root cause;

-under-resourcing a root cause can create further non-conformities

Thinking that only one cause exists is dangerous because

*a cause never stands alone;

-a root cause is not one cause but a system of causes working together, understanding the interaction between multiple factors

A Remediation is essentially a __________ to be tested, a proposed solution that can be checked against how well it solved the problem, if it did at all

hypothesis

Types of Causes

- symptoms

- first-level causes

- higher-level causes

Symptoms

these are not regarded as actual causes, but rather as signs of existing problems

First-level causes

causes that directly lead to a problem

Higher-level causes (root cause)

causes that lead to first-level causes.

-although they do not directly cause the problem, higher-level causes form links in the chain of cause-and-effect relationships that ultimately create the problem

-"the evil at the bottom" that sets in motion cause-and-effect chains

Root cause analysis steps

1. define the event

-succinctly describing the event or deviation that triggered the RCA

2. Find causes

-coming up with as broad a range of potential causes of the problem as possible

3. Find the root cause

-zooming in on the main culprit

4. Find solutions to solve the problem and prevent the event from reoccurring

5. Take action,

-implementing solutions to ensure that things stay that way

6. Measure and assess to determine whether the solution(s) work and solved the problem

What can trigger and RCA process?

* internal triggers:

employees observing poor practices, or someone having witnessed an event

* external triggers:

customers, suppliers, media or other stakeholders

* system triggers:

reviews, surveys, or audits

* specific incidents:

employee/customer injuries or fatalities, damage to equipment, or other events that exceed a predetermined limit

How is the RCA project plan visualized?

using the Gantt chart

What is the main purpose of the Gantt chart?

depict the project tasks and the schedule and to provide the basis for monitoring progress

What are the steps in the Gantt chart?

1. list all tasks or activities

2. for each activity, define latest finish date and earliest start date, the duration, and any dependence on other activities

3. place the tasks in empty Gantt chart, with the timeline reflecting overall duration of project

4. schedule activities by drawing bars that correspond to duration of activities

5. use diamonds to depict milestones in the plan

What are two approaches to collect information and data?

- interview:

-gather information from those involved in the event, either directly or indirectly

-when you want to gather verbal info

- survey:

-collect data from a large number of respondents

-when you want to collect data about people's attitudes, feelings, opinions

What are the main steps in using interviews?

1. prepare interview guide

2. test the questions in order to eliminate ambiguity

3. make appointment with people you want to interview

4. make sure you have privacy and are not disturbed during the interview

5. ask the questions and make sure they are understood

6. record the answers, digitally or in writing

7. if relevant, obtain the interviewee's confirmation that the answers were understood correctly

What are the steps in using surveys?

1. define the objective of the survey and what information is required to achieve this objective

2. decide how the survey will be undertaken: written (via mail, e-mail, or online) or verbal (by telephone or in person)

3. develop the questionnaire and test it with employees uninvolved in the survey

4. identify sample of respondents and send the survey to them

5. collect data according to the chosen approach and analyze it

What are the steps to using a flowchart to map an event?

1. gather those employees working in the process in a meeting room with whiteboard facilities and plenty of adhesive notes in different colors

2. define the start and end points of the process as well as boundaries between parallel processes

3. identify main activities or tasks undertaken during process (sometimes it is useful to start with final outcome and work backward)

4. create adhesive notes in diff colors to represent activities, products, documents, other elements of process in question

5. map process by moving notes around on board until they reflect most realistic picture of process in question

6. butcher paper is helpful in case flowchart of process becomes too large to be easily readable on standard page