Cards: System failures and errors

What is the aim of the lecture regarding system failure?

To identify causes of system failure through case studies.

What is the aim of the lecture regarding system errors and failures?

To understand different theories as to why system errors and failures occur.

What is the aim of the lecture regarding making systems dependable?

To consider how systems can be made more dependable.

What type of failure was the Titanic?

A catastrophic failure of a large system.

What were the costs of the Titanic failure?

Very costly failure in terms of: Money, Human life, Organisational reputation.

When were many mistakes made during the Titanic project?

Many mistakes made during all phases of design and development.

What kind of system was the Titanic?

Very complex socio-technical system.

What type of control systems did the Titanic have?

Safety critical control systems.

What cutting-edge technology was involved in the Titanic?

Data communications, Engineering technologies.

What kind of management structures did the Titanic have?

Complex management structures.

What was the context of the Titanic system?

Complex political and organisational context.

What perspectives are needed to understand what went wrong with the Titanic?

Entire system perspective, considering Technical Components, People, knowledge, processes, Organizational context, Environment.

What was the Post Office case study described as by the BBC?

‘the most widespread miscarriage of justice in UK history’ (BBC).

When was a new accounting software system produced by Fujitsu (Horizon) installed at the Post Office?

1999.

Between which years did over 700 post office branch managers receive criminal convictions?

Between 2004 and 2014.

What were branch managers accused of?

Faulty accounting and theft.

What was the reality about the Horizon system?

Horizon was faulty and had falsely suggested cash shortfalls.

What were the implications of the Horizon fault?

Severe implications, with many people wrongly imprisoned.

What did Lord Justice Holroyde say about the Horizon system?

‘there were serious issues about the reliability of Horizon’.

What happened when PO staff complained about bugs in the system?

PO staff members complained of bugs in the system, but were not taken seriously.

What conclusion was drawn about the Horizon software and PO staff?

Conclusion drawn that Horizon software must be correct; and that PO staff had stolen money.

What organizational context factors contributed to the Post Office scandal?

Over-trust in technology? Lack of respect for workers? Embarrassment that an expensive tech contract was failing?.

What legal system failings contributed to the Post Office scandal?

Failings in the legal system – legal presumption of proper functioning of computers?.

When did all passengers, pilots and cabin crew die in Boeing 737 Max crashes?

October 2018 and March 2019.

What did Boeing designers use that had to be repositioned?

Boeing designers used larger engines which had to be repositioned forward and higher.

What did the repositioned engines cause?

This caused unwanted extra lift and pitch-up at high angle of attack.

What software was used to reduce pitch-up and the risk of stall?

Maneuvering Characteristics Augmentation System (MCAS).

How did MCAS achieve its function?

Software was used to automatically push the nose down.

What sensors did MCAS use?

This uses the AoA sensors.

What did the MCAS system adjust?

The MCAS system adjusts the angle of the stabiliser.

What did MCAS force the nose to do?

This lifts the tail, hence forces the nose down.

What were the characteristics of the MCAS system?

The system is covert, forceful, and persistent.

What was the nature of the solution chosen for the Boeing 737 MAX problem?

Software solution chosen for what was a hardware problem (size of engine, and design of plane).

Was there open communication about the risks of the Boeing 737 MAX system?

Seems to have been little open communication around the risks of the system.

Were pilots' concerns listened to regarding the Boeing 737 MAX?

Pilots raised concerns which were not listened to.

Were some pilots aware of the new Boeing 737 MAX system?

Some pilots were not even aware of the new system and how it worked.

What market forces influenced the Boeing 737 MAX case?

Market forces pushing airline companies to make larger, faster planes – and for cheaper.

What are Regulatory failures?

Lack of information; under-trained personnel; lack of regulation.

What are Managerial Failures?

Safety climate, lines of command and responsibility, quality control.

What are Hardware Failures?

Design failure; requirements failure; implementation failure.

What are Software Failures?

Requirements failures; specification failures.

What are Human Failures?

Slips, lapses & mistakes; team factors, human error.

What can happen when failure in one part of a complex system coincides with failure of a different part?

This combination can cause cascading failures of other parts.

Are there many possible combinations of failures in complex systems?

Yes, in complex systems these are many possible combinations.

What characterizes complex interactions in a complex system?

Unfamiliar, unplanned, or unexpected sequences which are not visible or immediately comprehensible.

What characterizes tightly coupled systems?

Time-dependent processes, Rigidly ordered processes (sequence B must follow sequence A), Very little slack.

When is a system particularly prone to failure?

If a system has interactive complexity and is tightly coupled it is particularly prone to failure.

What is Reason’s Swiss Cheese Model?

Successive layers of defences, barriers, & safeguards, with some holes due to active failures and other holes due to latent conditions, which can align to cause losses.

What is a limitation of the Swiss Cheese Model according to Leveson (2004)?

Independence of the barriers is assumed and some randomness in whether the “holes” line up.

What is a limitation of the Swiss Cheese Model according to Dekker (2002) regarding layers of defence?

Layers of defence are not static or constant, and not independent of each other either. They can interact, support or erode one another.

What does Dekker say the Swiss Cheese Model doesn’t explain?

The Swiss Cheese Model doesn’t explain what the holes are, how and why they got there, how the holes line up, etc..

What is the most important property for most complex socio-technical systems?

Dependability.

What is dependability a judgment about?

Judgement about the user’s trust in a system.

What does dependability reflect?

Reflects the extent of the user’s confidence that it will operate as expected and will not ‘fail’ in normal use.

How is dependability defined by Mellor?

“Dependability is defined as that property of a computer system such that reliance can justifiably be placed on the service it delivers.”.

What is System failure?

When the system does not deliver the service its users expect.

What is System error?

Where the behaviour of the system does not confirm to its specification.

What is System fault?

Incorrect system state not expected by the designers of the system.

What is Human error or mistake?

Human behaviour that results in faults being introduced into a system.

What is Fault avoidance?

Preventing the occurrence or introduction of faults.

What is Fault tolerance?

Delivering correct service, though faults are present.

What is Fault removal?

Reducing number or severity of faults.

What is Fault forecasting?

Estimating number of faults, future occurrence, consequences.

What is Availability (as a primary attribute of dependability)?

Ability of system to deliver services when requested.

What is Reliability (as a primary attribute of dependability)?

Ability of the system to deliver services as specified.

What is Safety (as a primary attribute of dependability)?

Ability of the system to operate without catastrophic failure.

What is Security (as a primary attribute of dependability)?

Ability of the system to protect itself against accidental or deliberate intrusion.

What is Timeliness (as a secondary attribute of dependability)?

The ability of the system to respond in a timely way to user requests.

What is Survivability (as a secondary attribute of dependability)?

The ability of a system to continue to deliver its services to users in the face of deliberate or accidental attack.

What is Recoverability (as a secondary attribute of dependability)?

The ability of the system to recover from user or system errors.

What is Maintainability (as a secondary attribute of dependability)?

The ease of repairing the system after a failure has been discovered or changing the system to include new features.

What are the key points about System errors and failures?

System failures are the result of many compounding factors.

Are failures more likely in simple or complex systems?

Failures are more likely in complex systems.

What is crucial for complex systems?

Ensuring dependability is crucial for complex systems.