Lecture 10: Aviation System Safety Analysis

Aircraft Systems Overview

• System: A collection of components intended to function together to perform a task.

• An aircraft is a system comprising structure, propulsion, controls, etc. to carry people or cargo from A to B.

• Systems are usually composed of sub-systems (and eventually, components)

• Flight controls (actuators), hydraulics (pumps), electrical power generation (circuit breakers, connectors), landing gear (brakes, doors), Pitot-static (heated pitot tube, air data computer)

• All of these are subsystems of the aircraft system

Systems interact with each other; system definitions must include the interfaces.

Essential/Mission Systems

• For airworthiness - consider only systems essential for flight.

  • Flight controls, propulsion, navigation, etc.

• Mission systems - required for the intended function of the aircraft but not essential for safe flight

  • In flight entertainment (IFE), weapons, research sensors, NRC helicopter research fly-by-wire flight control systems

Mission systems must at least be assessed to ensure they “do no harm” to essential systems nor introduce new hazards!

Systems Context - Simple v. Complex

• This presentation mainly intended to discuss sophisticated, complex aircraft

• Many cert rules are based on basic lessons learned long ago on small, simple aircraft, and they still apply because there has been no change to the technology

• The need for predictable handling qualities such as prescribed “stick force gradient”

Case: Turkish Airlines Flight 1951 B737 Amsterdam - 25 Feb 2009

• Incorrect autopilot mode selection and one faulty radio altimeter (undetected) caused the auto-throttles to decrease thrust to idle early approach (flare for landing)

• The crew noticed low airspeed too late to increase thrust and prevent the aircraft from aerodynamic stall

• Crashed 1.5 km north of runway 18R killing 9 passengers and crew including all 3 pilots

Crew error due to lack of “automation mode awareness” is a system safety issue.

Boeing 737 Max 8

Lion Air 610, Indonesia Oct 29, 2018

• Dove into sea 13 minutes after T-O, 189 fatalities

• Newly delivered airplane (2 mths, 800 hrs in service)

• Experienced crew

Ethiopian Airlines 302, Ethiopia March 10, 2019

• Dove into ground 6 minutes after T-O, 157 fatalities

• Also a new airplane (4 months in service)

• Experienced captain, inexperienced first officer

Single failure in a system caused the loss of 346 lives and 2 aircraft.

System Safety Assessment - General

• Much of system safety analysis is imagining how systems can fail

• Need to analyze potential failures in a rigorous and systematic way

• Must include potential human factor errors (due to maintenance and flight crew actions)

• Think about possible cascading failures

System Safety Assessment - “What If?”

The airworthiness mindset is to always ask: What can possibly go wrong?

• Failure Mode (or Failure Condition) - In what ways can this system fail to function as intended; or function when it should not!

• Failure Effect - What are the consequences of each failure condition?

• Probability - How often will each failure condition occur?

• Mitigation - What (if anything) should be done about them?

Safety/Risk

• The level of safety is the inverse of the level of risk

• For any given hazard there is:

  • Some probability that the hazard will result in an unsafe event

    • For components & systems “failure probability” is the inverse of “reliability”

  • Some level of severity associated with the unsafe event

  • Risk is the product of these two

RISK = severity x probability

Regulations = “Thirteen-O-Nine” pt a

CS 25.1309 Equipment, systems, and installations

(a) The aeroplane equipment and systems must be designed and installed so that:

(1) Those required for type certification or by operating rules, or whose improper functioning would reduce safety, perform as intended under the aeroplane operating and environmental conditions.

(2) Other equipment and systems are not a source of danger in themselves and do not adversely affect the proper functioning of those covered by subpara a1

TL;DR:

• Required equipment must function as intended under all conditions - not necessarily the same as “function as designed”

• Non-required systems must “do not harm”

Regulations = “Thirteen-O-Nine” pt b

CS 25.1309 Equipment, systems, and installations

(b) The aeroplane systems and associated components, considered separately and in relation to other systems, must be designed so that -

(1) Any catastrophic failure condition

(i) is extremely improbable; and

(ii) does not result from a single failure; and

(2) Any hazardous failure condition is extremely remote; and

(3) Any major failure condition is remote

TL;DR:

• No single failure shall lead to a catastrophic condition

• Failure conditions of increasing severity must be increasingly unlikely

Regulations = “Thirteen-O-Nine” pt c

CS 25.1309 Equipment, systems, and installations

(c) Information concerning unsafe system operating conditions must be provided to the crew to enable them to take appropriate corrective action. A warning indication must be provided if immediate corrective action is required.

Systems and controls, including indications and annunciations must be designed to minimize crew errors, which would create additional hazards.

TL;DR:

• Crew must be informed/annunciations for warnings and cautions

• Design to ensure crew takes the correct action in response

Examples: consistent use of colours (red = warning, amber = caution)

Regulations = “Thirteen-O-Nine” pt d-f

CS 25.1309 Equipment, systems, and installations

(d) Compliance with the requirements of para b must be shown by analysis, and where necessary, by appropriate ground, flight, or simulator tests. The analysis must consider:

(1) Possible modes of failure, including malfunctions and damage from external sources.

(2) The probability of multiple failures and undetected failures.

(3) The resulting effects on the aeroplane and occupants, considering the stage of flight and operating conditions, and

(4) The crew warning cues, corrective action required, and the capability of detecting faults.

(e) In showing compliance with paragraphs (a) and (b) of this section with regard to the electrical system and equipment design and installation, critical environmental conditions must be considered. For electrical generation, distribution, and utilization equipment required by or used in complying with this manual, except […] the ability to provide continuous, safe service under foreseeable environmental conditions may be shown by environmental tests, design analysis, or reference to previous comparable service experience on other aircraft.

(f) Electrical Wiring Interconnection Systems (EWIS) must be assessed in accordance with the requirements of 525.1709

TL;DR:

• Analysis and/or testing is required - methods are not prescriptive but must be acceptable to the cert authority

• Must consider external sources (birds, lightning, HIRF)

• Consider environmental conditions

Probability and Severity

For increasing severity decreasing probability allowed

Qualitative Probability (for a fleet of airplanes of one type)

Probable failure conditions… anticipated to occur one or more times during the entire operational life of each aeroplane.

Remote… unlikely to occur to each aeroplane during its total life… may occur several times when considering the total operational life of a number aeroplanes of the type.

Extremely remote… not anticipated to occur to each aeroplane during its total life… may occur a few times in whole fleet.

Extremely improbable… not anticipated to occur during the entire operational life of all aeroplanes of one type.

Where do the quantitative values come from?

• Historical serious accident rate 1 per million hours of flight (1 accident per 1×10^6 hours)

• About 10% due to systems Failure Conditions (FC)

• Safety target for new a/c: probability of a serious accident from all system FC < 10 ^-7 per flight hour (at least as good as histroical record)

• Arbitrary assumption of 100 catastrophic FCs per airplane

Safety target for Catastrophic FC for any system is therefore < 1 × 10^-9 per flight hour.

Fail Safe Design Concept

• Failure of any single system element should be assumed regardless of probability and must not be catastrophic

• Subsequent failures during same flight, should also be assumed, unless their joint probability with the first failure is shown to be extremely improbable (< 10^-9 per hr)

• Apply design principles to ensure acceptable LOS w probabilities no greater than:

  • REMOTE; <10^-5 per hour for MAJOR FCs

  • EXTREMELY REMOTE; <10^-7 per hour for HAZARDOUS FCs

  • EXTREMELY IMPROBABLE; < 10 ^-9 per hour for CATASTROPHIC FCs

Failsafe Design Principles pt I

Designed Integrity and Quality, including life limits

• Ensure intended function and prevent failures (simple, robust, but may need…)

Redundancy or backup systems

• to enable continued function after any single failure; eg two or more engines, hydraulic systems, flight control systems, etc. (adds complexity)

Isolation and/or segregation of systems, components, and elements

• So failure of one does not cause the failure of another (ensure independence but requires space/weight)

Proven reliability

• Well understood designs with known low failure rates/high reliability (slows innovation!)

Failsafe Design Principles pt II

Failure warning or indication

• to provide detection (risks crew overload)

Flight crew procedures

• Corrective action after failure detection (difficult to quantify human performance)

Checkability

• ability to check a component’s condition (flight or maintenance crew workload)

Design failure effect limits

• ability to sustain damage; limit safety impact (analysis and testing effort/cost)

Failsafe Design Principles pt III

Design failure path

• to control and direct the effects of failure in a way that limits its safety impact (analysis and testing effort/cost)

Margins or factors of safety

• To allow for undefined/unforeseeable adverse conditions (usually adds weight)

Error-tolerance

• Consider effects of errors during design, test, manufacture, operation, and maintenance (requires strong human factors knowledge)

Simplified Process Overview

FHA’s ——Validation of Safety Targets———>

PSSA ——-Verification of Safety Targets——> SSA

SSA: System Safety Analysis

PSSA: Preliminary SSA
FHA: Functional Hazard Assessments

• Aircraft and system levels

Definition - Validation

The determination that the requirements for a product are sufficiently correct and complete.

Top down from aircraft to system to sub-system to component.

Definition - Verification

The evaluation of an implementation to determine that applicable requirements are met.

Bottom up from component to sub-system to system.

General Heat Systems Configuration

How the (EPS) functions:

• Pilot commands motor torque via “throttle” thrust lever

• Pilot commands motor RPM via propeller control lever

• Battery provides HV DC electricity

• Motor controller takes HV DC and torque command and outputs HV AC to motor

• Battery Management System (BMS) monitors battery and makes HV connection to motor controller

• Displays and annunciators show status and warnings to pilot

Functional Hazard Assessment (FHA) At Aircraft and System Levels

• High level, qualitative assessment of basic aircraft functions

• Identify each Failure Condition (FC) and the rationale for its severity classification

• System FHA iterative as systems evolve; consider interactions of systems

• Generate top level events for subsequent system analysis using Fault Tree Analysis (FTA) during Preliminary System Safety Analysis (PSSA)

• Derive Development Assurance Levels (DALs) for h/w and s/w based on system criticality

Clarification regarding back-up and redundant systems

A back-up system only operates when the primary systems cannot; it replaces the capability of the primary system

A redundant system consists of multiple components operating simultaneously; the failure of any one does not cause an overall system failure due to the capacity of the other components

Preliminary System Safety Analysis (PSSA)

• Iterative analysis embedded in overall development process down to “item level”

• Completes the FC list including those due to latent faults and common causes

• Derives system & item safety requirements

• Identify protective strategies such as built-in test, dissimilarity, monitoring, flight, or maintenance crew tasks/intervals

• Include h/w and s/w errors to assist in establishing DALs

• Define verification strategies to be used during SSA - ie qualitative vs quantitative and specific test/analysis methods to show compliance

P/SSA tools and methods from ARP4761

Fault Tree Analysis (FTA)

• Boolean logic (AND, OR, NOT, etc.) as introduced in prior examples

Dependence Diagrams (DD)

• Uses paths to depict the relationship of failure effects;

• Parallel paths for AND gates; series paths for OR gates

Markov Analysis (MA)

• Statistical method

Failure Modes and Effects Analysis (FMEA)

Top Down Methods: Fault Tree Analysis (FTA)/Dependence Diagrams (DD)

• Quantify probability of occurrence for the top event vs. safety target

• Establish reliability budgets and Development Assurance Levels (DALs) for hardware and software

• Assess the impact of a design modification

• Identify the need for a design modification and/or identify unique situations that require special attention

• Show compliance with qualitative and/or quantitative safety objectives as part of the SSA

• Establish crew/maintenance tasks and intervals necessary to meet the requirements of the safety assessment

Failure Modes and Effects Analysis (FMEA) - Bottom Up Method

• FMEA is a systematic bottom up method of identifying failure modes and determining effects on the next higher level

• Generally used to address effects resulting from single failures

• Assess for each phase of flight and include “failure detection/indication” provisions

• Generates lengthy document - collect failure modes of similar effect in summary document (FMES)

Common Cause Analysis - CCA

• CCAs consider possible violations of failure condition independence

  • Three types of analysis - Common Mode Analysis (CMA); Particular Risk Analysis (PRA); Zonal Safety Analysis (ZSA)

• Adding apparent redundancy introduces complexity; difficult to make such systems truly independent

• Even truly independent systems may be affected by a common event due to their physical proximity or response to a common external condition

  • Egs: fuel starvation or contamination affecting all engines; icing affecting all Pitot tubes, bird strikes, etc.

Common Cause Analysis - CMA

Common Mode Analysis (CMA) - assess independence of faults due to:

• s/w errors (not possible to test for all possibilities; use multiple code implementations)

• Complex h/w failure (use multiple dissimilar computers, etc.)

• Production/repair flaw (factor of safety; high production/process QA)

• Installation/maintenance errors (dissimilar connections; eg hydraulic hoses and electric cables)

• Cascading faults (auto load shedding)

Eg: dual electrical generating system analyzed as though the two systems are completely independent. Either system can provide more than enough electrical power to systems essential for flight but neither system can provide enough power to sustain all loads under all conditions. One system fails for a reason that has been anticipated and designed for; second system continues to function provided crew sheds some electrical load. If load is not shed, the second system may fail due to overheating, etc.

Therefore, the two systems were not truly independent. (highlights the importance of human factors - required crew actions in this case - may drive design change to automatic load shedding).

Common Cause Analysis - PRA

Particular Risk Analysis (PRA) - due to factors external to the system:

• Fire

• High energy devices

• Hail, ice, snow

• Bird strike

• Lightning

• HIRF

• Leaking fluids

• Flailing shafts

• Tire tread separation or wheel rim release

• New particular risks for electric propulsion systems due to external events: battery thermal runaway, high voltage hazards to occupants

Assign a probability of “1” to each of these for analysis

Common Cause Analysis - ZSA

Zonal Safety Analysis (ZSA)

• Observance of basic installation rules such as segregation of:

  • Power and signal wiring;

  • electrical and fluid lines

• Interference between systems

• Potential for maintenance errors

Requires knowledge of physical installation (not just schematic)

Common Cause Analysis - ZSA cont’d

• Requires experienced engineers and maintenance staff to do this properly! Knowledge of where problems have occurred in the past is essential

• Maintenance error potential is particularly important:

  • Can a valve be installed in reverse?

  • Can a pipe or connector be physically connected to more than one other pipe/connector in the same bay?

  • Does removal of an item for servicing require removal of another item?

• Use of 3D CAD models for this purpose; waiting until the actual aircraft is available can be very expensive (ie need to reroute wiring or hydraulics, etc.)

Common Cause Mitigations and Examples

Heat EPS Common Cause Analysis

• Common Mode (CMA) - independence of DC-DC converter failure conditions?

• Particular Risk (PRA) - due to external factors (high ambient temperatures - risk of TR?)

• Zonal Safety (ZSA) - electric power and coolant lines in close proximity?

*Requires knowledge of phsyical installation (not just schematic)

Software and Complex Hardware - Development Assurance Levels (DALS)

DAL Assigned based on s/w criticality

Top level failure condition severity classification → DAL for s/w

• Catastrophic → A

• Hazardous → B

• Major → C

• Minor → D

• No safety effect → E

Not possible to test every aspect of s/w and complex electronic h/w

Level A, B very expensive - limited to safety critical applications

Human Factors

Actions of Flight Crews & Maintenance Personnel must be accounted for in system design:

Potential for errors due to:

• Misleading or conflicting information

• Insufficient or excessive information

• High workload (eg some phases of flight)

• Poor ergonomics (eg control function)

Some crew procedures are required to meet safety targets (ie pre-flight checks for latent/dormant faults)

Other Topics - MEL & CMR

Minimum Equipment List (MEL)

• A list of equipment/functions which need not be operative for flight

• based on stated compensating precautions (eg operational or time limitations, flight crew procedures, or ground crew checks)

Certification Maintenance Requirements (CMR)

• Periodic maintenance or flight crew check required to achieve compliance with CS 25.1309(b) for Hazardous and Catastrophic Failure Conditions

• Where such checks cannot be accepted as basic servicing or airmanship they become Certification Maintenance Requirements

• A CMR is identified as an operating limitation on the Type Certificate for the aeroplane