Fault tolerance fdcc

Fault Tolerance

Introduction

  • Fault tolerance refers to the ability of a distributed system to continue operating correctly in the event of a failure.

    • Essential for ensuring system reliability and availability.

Types of Faults in Distributed Systems

  • Various faults that distributed systems may face:

    • Failure of a link: Connection between nodes may break.

    • Failure of a site: A complete node or server may become unresponsive.

    • Loss of message: Messages sent between nodes may be lost due to network issues.

    • Failure of power: Loss of power can disrupt system operations.

Recovery Steps After Failure

  • Steps required for recovery from these failures:

    • Post-repair integration with the main system should transition smoothly and gracefully.

    • Both parties involved in a link must be notified in case of a link failure.

    • Implement mechanisms for recovery to handle faults systematically.

    • Maintain systematic logging of failures for analysis and response.

System Failure Modes

  • Distributed systems face unique failures:

    • Site failure: Entire site or node becomes unresponsive.

    • Loss of messages: Handled by protocols like TCP-IP to ensure messages are sent reliably.

    • Failure of a communication link: Network protocols can reroute messages to alternative links.

    • Network partition: A scenario where a network is split into disconnected subsystems.

      • Distinction is often unclear between network partitioning and site failures.

Approaches to Achieve Reliable Systems

  • Approaches to ensure reliability in systems:

    • Fault prevention: Strategies put in place to prevent failures from occurring.

    • Fault removal: Fixes and updates to eliminate existing flaws.

    • Fault tolerance: Enable the system to keep operating in the presence of faults.

    • All of the mentioned: These strategies combined enhance system reliability.

Dependability in Distributed Systems

  • Dependability consists of several aspects:

    • Availability: Measurement of how ready a system is for immediate use.

    • Reliability: Measurement of continuous functionality without failure over time.

    • Safety: Assurance that failures will not lead to catastrophic outcomes.

    • Maintainability: Ease with which a system can be repaired or updated.

Availability vs. Reliability

  • Availability is a measure of readiness, while reliability reflects long-term performance without failure.

    • Example: A system with high reliability could still have low availability if it shuts down periodically for maintenance.

Safety and Maintainability

  • Safety: Indicates the degree to which a system can fail without causing serious harm. Essential in critical systems like nuclear power plants.

  • Maintainability: Refers to the ease with which a system can be repaired. High maintainability can correlate with high availability.

Definitions of Faults

  • Failure: Occurs when a system cannot meet its operational specifications.

  • Error: A manifestation of a fault which can result in a system failure.

  • Fault: The underlying cause of an error; it may be transient (temporary), intermittent (recurs unpredictably), or permanent (continuous).

Failure Models

  • Types of failures encountered:

    • Crash failure: A server stops working correctly.

    • Omission failure: A server fails to process requests or receive messages.

    • Message delivery failings: Lost or missed messages significantly impact system operations.

Reliable Group Communication

  • Reliable multicast strategies ensure messages sent to a group are delivered to all group members.

    • Handling scenarios such as membership changes during communication and sender crashes is crucial.

    • Delivery can be organized to ensure messages are received in order, enhancing reliability.

Process Resilience

  • Process resilience can be achieved through replication:

    • Organized identical processes into groups where messages to a group are delivered to all members.

    • If a member fails, other members can continue operations.

Recovery Mechanisms

Types of Recovery

  1. Backward Recovery: Returns the system to a previous, correct state.

  • Example: Using checkpoints to restore the system state.

  1. Forward Recovery: Moves the system to a new, correct state after an error.

  • Example: Erasure correction techniques to reconstruct lost data from available information.

Checkpointing for Recovery

  • Checkpointing is crucial for backward recovery:

    • Regularly save system states (checkpoints) for potential rollbacks during failures.

    • Checkpoints need to be consistent to ensure valid recovery.

Message Logging

  • Combine message logging with checkpointing to improve recovery:

    • Message logging can either be pessimistic (log before processing) or optimistic (log after processing).

    • Ensures that the system can recover effectively by replaying messages between checkpoints.