Risk and Responsibility

Key Concepts

Liability

Liability is legal responsibility for harm caused by a product, system, action, or failure to act.

In computing, liability may arise when:

A system malfunctions.
A product is defective.
A developer fails to exercise reasonable care.
A company makes false promises about a product.
A system causes foreseeable harm.
Users rely on incorrect or misleading information.

Safety-critical system

A safety-critical system is a system whose failure could cause:

Injury.
Death.
Severe environmental damage.
Major property loss.
Serious disruption of essential services.

Examples include:

Aircraft-control systems.
Medical devices.
Radiation therapy machines.
Nuclear plant systems.
Industrial-control systems.
Autonomous vehicles.
Emergency-response systems.

Negligence

Negligence is the failure to exercise the level of care that a reasonable professional would use under similar circumstances.

A negligence case generally considers whether:

A duty existed.
The duty was breached.
The breach caused harm.
Actual damage occurred.

Accountability

Accountability is the obligation to explain, justify, and accept responsibility for decisions and outcomes.

Accountability is broader than legal liability. A person may be professionally accountable even when a court does not find legal liability.

Strict liability

Strict liability assigns responsibility for harm caused by a defective product or activity, even if the responsible party did not intend to cause harm and may have exercised reasonable care.

Values in design

Values in design means that design decisions often reflect assumptions about:

Whose needs matter.
What risks are acceptable.
Which users are prioritized.
How much privacy is provided.
How accessible a system should be.
Who receives control.
Who bears the consequences of failure.

A system is not completely neutral because its design choices affect people differently.

Computer Liability

When a computer system fails, several questions may arise:

Who designed the system?
Who developed the software?
Who approved the product?
Who operated the system?
Was the failure foreseeable?
Were reasonable testing procedures followed?
Were warnings provided?
Was the system used as intended?
Did organizational pressure contribute to the failure?

Responsibility may be shared among:

Designers.
Developers.
Manufacturers.
Vendors.
Organizations.
Operators.
Users.
Regulators.

Three Legal Frames for Liability

Negligence

The main question is:

Did the developer, manufacturer, or operator fail to exercise reasonable professional care?

Examples of possible negligence include:

Failure to test a system adequately.
Ignoring known defects.
Failing to provide warnings.
Deploying software despite serious unresolved risks.
Failing to train users.
Neglecting maintenance.
Failing to monitor system behavior.

Strict liability

The main question is:

Was the product defective, regardless of how much care was taken?

Strict liability focuses on the condition of the product rather than only on the behavior of the developer or manufacturer.

Possible product defects include:

Design defects.
Manufacturing defects.
Inadequate safety warnings.
Missing safety features.
Dangerous defaults.

Breach of warranty

The main question is:

Did the system fail to perform as explicitly or implicitly promised?

A warranty may be:

Express: A specific claim or promise made by the seller.
Implied: A reasonable expectation that the product is fit for its ordinary purpose.

Example:

A company promises that its backup system will automatically recover data after a failure. If the system does not perform that function, the customer may claim a breach of warranty.

Hardware Malfunction

Hardware components may fail because of:

Age.
Heat.
Electrical stress.
Moisture.
Vibration.
Manufacturing defects.
Poor maintenance.
Power surges.
Physical damage.

Component wear

Mechanical and electronic parts degrade with use and age. Examples include:

Hard-drive failure.
Battery degradation.
Overheating processors.
Damaged sensors.
Worn mechanical components.
Failing power supplies.

Environmental stress

Environmental conditions may cause unpredictable faults.

Examples include:

Power fluctuations.
Lightning.
Excessive heat.
Water exposure.
Dust.
Vibration.
Electromagnetic interference.

Redundancy

Redundancy uses backup components or systems to reduce the effect of a failure.

Examples include:

Backup servers.
RAID storage.
Dual power supplies.
Multiple network links.
Backup sensors.
Emergency generators.

Redundancy lowers the probability of complete failure, but it does not eliminate risk. It also adds:

Cost.
Complexity.
Maintenance requirements.
Possible configuration errors.
Additional failure points.

Software Malfunction

Software may fail because of:

Programming errors.
Ambiguous requirements.
Hidden edge cases.
Incorrect assumptions.
Poor testing.
Update regressions.
Integration problems.
Legacy code.
Insecure configurations.
Unexpected user behavior.

Hidden edge cases

An edge case is an unusual situation or input that may not have been considered during development.

Example:

A system works for dates within a normal range but fails when a leap year, time-zone conversion, or invalid date is entered.

Update regressions

A regression occurs when a software update fixes one problem but unintentionally breaks another feature.

Example:

A security update corrects authentication but prevents legitimate users from accessing the system.

Integration failures

Individually reliable components may behave incorrectly when combined.

Example:

The database works correctly.
The application works correctly.
The network works correctly.
However, an unexpected data-format difference causes the complete system to fail.

Reliability is continuous

Testing before release is not enough. Reliability requires:

Continuous monitoring.
Maintenance.
Updates.
Regression testing.
Incident review.
User feedback.
Periodic risk assessment.

Safety in Computing

Safety means preventing or reducing physical harm, environmental damage, and severe operational consequences.

Safety is different from ordinary convenience or performance. A system may be inconvenient when it fails, but a safety-critical system may injure or kill people.

Safety principles

Anticipate failure before deployment.
Consider misuse and abnormal conditions.
Test beyond normal operating conditions.
Use independent safety mechanisms.
Provide clear warnings.
Train operators.
Monitor system performance.
Maintain manual override options.
Document assumptions.
Review changes carefully.

Safety versus security

Safety	Security
Protects people and the environment from system failure	Protects systems and information from intentional or unauthorized harm
Focuses on accidents and unsafe conditions	Focuses on attacks, misuse, and unauthorized access
Example: Preventing a radiation overdose	Example: Preventing an attacker from changing treatment settings

Safety and security often overlap. A cyberattack on a medical device can become a safety incident.

Misinterpretation of Information

Computer systems may produce incorrect, incomplete, or uncertain information. Users may misread or over-trust that information.

Common causes

Ambiguous labels.
Poor interface design.
Missing units.
Incomplete context.
Confusing alerts.
Complex data.
Over-reliance on automation.
Insufficient user training.

Automation bias

Automation bias occurs when users trust computer-generated recommendations too much and fail to question them.

Example:

A medical decision-support system incorrectly flags a patient as low risk. A healthcare worker accepts the recommendation without reviewing the original data.

Preventing misinterpretation

Systems should:

Clearly label outputs.
Display units and time frames.
Show uncertainty where relevant.
Explain important limitations.
Provide warnings for unusual conditions.
Make critical information visible.
Require confirmation for high-risk actions.
Train users to verify important results.

Users should:

Cross-check high-stakes information.
Avoid assuming that automated output is always correct.
Report unusual results.
Understand the system’s limitations.

Liability for Defective Information

When software provides incorrect or misleading information that causes harm, the legal treatment may vary.

Product liability

Software may be treated like a manufactured product. A defect may create liability regardless of fault, depending on applicable law and context.

Professional liability

Software may be treated more like professional advice. Liability may depend on whether the developer or organization failed to exercise reasonable care.

Shared responsibility

Courts and investigators may consider:

Developer diligence.
Testing procedures.
User training.
System warnings.
Disclaimers.
Foreseeability of misuse.
Operator behavior.
Organizational decisions.
Whether the information was presented clearly.

Safety-Critical System Evaluation

Safety-critical systems require more rigorous evaluation than ordinary software.

Formal verification

Formal verification uses mathematical methods to prove that code satisfies defined specifications.

It is useful for:

Flight-control systems.
Medical devices.
Cryptographic systems.
Industrial controllers.
Operating-system components.

Formal verification cannot prove that the specification itself is correct. A system can satisfy a flawed requirement perfectly.

Redundancy testing

Redundancy testing examines whether backup systems work during simulated failures.

Testing may include:

Power loss.
Network failure.
Sensor failure.
Hardware failure.
Incorrect input.
Component disconnection.
Communication delay.
Software crash.

Independent audit

An independent audit is performed by people or organizations separate from the development team.

Independent reviewers may identify:

Conflicts of interest.
Hidden assumptions.
Unreported defects.
Insufficient testing.
Weak documentation.
Unsafe design decisions.

Other evaluation practices

Hazard analysis.
Failure-mode analysis.
Fault-tree analysis.
Stress testing.
Penetration testing.
Code review.
Simulation.
User training.
Safety certification.
Incident-response planning.
Post-deployment monitoring.

Therac-25 Case

What happened

The Therac-25 was a radiation therapy machine used in the 1980s. A software race condition allowed the machine to deliver massive radiation overdoses to several patients, some fatally.

Why it mattered

Earlier versions of the system used hardware safety interlocks. In the newer system, some protections were removed or replaced with software controls.

This created excessive dependence on software correctness.

Software can cause physical harm.
Safety mechanisms should be independent.
Hardware fail-safes may be necessary.
Software errors can interact with user-interface and design problems.
Reported failures should be taken seriously.
Safety-critical systems require rigorous testing.
A system should not rely entirely on one protection layer.
Responsibility includes anticipating unusual operating conditions.

Values in Design

Design choices embed values and priorities.

Whose values?

Designers may unintentionally prioritize:

The majority over minorities.
Efficiency over privacy.
Profit over safety.
Speed over testing.
Experts over ordinary users.
Technical users over people with disabilities.
Organizational convenience over public welfare.

Trade-offs

Common design trade-offs include:

Priority	Possible benefit	Possible risk
Cost reduction	Lower product price	Fewer safety features or less testing
Speed	Faster release	Unresolved defects
Convenience	Easier user experience	Reduced security or privacy
Automation	Improved efficiency	Automation bias and loss of human judgment
Data collection	Better personalization	Privacy violations
Centralization	Easier management	Larger impact if compromised
Simplicity	Easier operation	Less flexibility or control

Every design trade-off raises an important question:

Who receives the benefit, and who bears the risk?

Accessibility

Accessibility means designing systems that can be used by people with different:

Physical abilities.
Sensory abilities.
Languages.
Technical knowledge.
Educational backgrounds.
Devices.
Internet conditions.

Examples of accessible design

Screen-reader compatibility.
Keyboard navigation.
Sufficient color contrast.
Captions and transcripts.
Clear language.
Adjustable text size.
Support for multiple languages.
Alternative input methods.
Error messages that explain how to recover.
Compatibility with low-bandwidth environments.

Ignoring accessibility can build exclusion directly into a system, even when exclusion was not intended.

Transparency

Transparency means making important system decisions, limitations, and risks understandable and visible.

Transparency may involve:

Documenting design decisions.
Explaining data collection.
Recording changes.
Disclosing known limitations.
Reporting defects.
Explaining automated decisions.
Identifying responsible personnel.
Preserving audit trails.

Transparency supports accountability. If decisions are hidden, it becomes difficult to determine who made them and why.

Hardware Safety Switches

Physical override

A physical switch allows an operator to stop a process independently of software.

Examples include:

Emergency-stop buttons.
Manual power cutoffs.
Physical shutdown switches.
Mechanical interlocks.

Software-only control

Removing a hardware switch means that the system depends more heavily on software logic.

Design question

Does the convenience or cost reduction justify removing a fail-safe that humans can use when software cannot be trusted?

Removing independent controls may:

Reduce cost.
Simplify hardware.
Improve automation.
Increase dependence on software.
Remove the operator’s last line of defense.
Increase the consequences of software failure.

Accountability in a Computerized Society

As decisions move from people to computerized systems, responsibility may become difficult to trace. However, responsibility does not disappear simply because a computer made the immediate decision.

Accountability chain

Designer → Developer → Organization → Operator → Regulator

Each part of the chain may have different responsibilities.

Designer

Responsible for:

Identifying requirements.
Considering user needs.
Evaluating risks.
Including safety and accessibility.
Avoiding harmful assumptions.

Developer

Responsible for:

Writing reliable code.
Testing software.
Reporting defects.
Following professional standards.
Documenting decisions.
Avoiding intentional concealment of risks.

Organization

Responsible for:

Providing resources.
Setting realistic schedules.
Establishing safety policies.
Supporting ethical escalation.
Conducting audits.
Accepting responsibility for business decisions.

Operator

Responsible for:

Following procedures.
Receiving proper training.
Monitoring system output.
Reporting errors.
Avoiding unauthorized use.
Knowing when to stop or question the system.

Regulator

Responsible for:

Establishing standards.
Monitoring compliance.
Investigating serious failures.
Protecting public welfare.
Enforcing applicable requirements.

Responsibilities of Computer Scientists

Computer scientists and IT professionals should:

Anticipate misuse.
Consider failure modes.
Communicate risks honestly.
Follow professional codes.
Refuse to build systems that knowingly endanger users.
Maintain current technical skills.
Document design and testing decisions.
Protect privacy and security.
Report serious defects.
Consider social and environmental effects.
Design for accessibility.
Support independent review.
Accept responsibility for professional decisions.

Being an expert means having responsibilities beyond simply writing code that compiles or works under normal conditions.

Responsibilities of Computer Users

Responsibility also belongs to people who operate and rely on systems.

Users should:

Verify instead of assuming

Cross-check computer-generated information before making high-stakes decisions.

Report errors

Report:

Malfunctions.
Unexpected behavior.
Incorrect outputs.
Security warnings.
Unusual results.
Data inconsistencies.

Stay trained

Users should understand:

What the system can do.
What the system cannot do.
When output may be unreliable.
How to respond to failures.
When human judgment is necessary.

Follow procedures

Users should:

Use systems as instructed.
Avoid bypassing safety controls.
Protect credentials.
Complete required training.
Follow emergency procedures.
Avoid unauthorized changes.

Sample Case Analysis

Case

A hospital deploys a decision-support system that recommends patient treatment. The system occasionally produces incorrect recommendations, but the hospital continues using it without warning doctors or reviewing the results.

Main issues

Patient safety.
Misinterpretation of automated information.
Professional accountability.
Inadequate testing.
Lack of transparency.
Possible negligence.
Over-reliance on automation.

Stakeholders

Patients.
Doctors.
Nurses.
Hospital administrators.
Developers.
Vendor.
Regulators.
The public.

Ethical analysis

Utilitarianism: Continued use may improve care for many patients, but serious errors may cause severe harm.

Deontology: The hospital and professionals have duties to protect patients, disclose limitations, and avoid unsafe practices.

Virtue ethics: Responsible professionals should be honest about system limitations and should not hide known errors.

Rights-based ethics: Patients have rights to safety, informed care, privacy, and appropriate treatment.

Recommended action

Suspend or limit unsafe use.
Investigate the cause of incorrect recommendations.
Validate the system independently.
Notify healthcare professionals of limitations.
Require human review.
Document errors.
Improve testing and monitoring.
Report serious risks to appropriate authorities.