SA: Chapter 4.4

Fault Tolerance: Motivation and Availability Requirements

• Service Level Agreements (SLAs): Frequently specify strict availability requirements.

• Amazon Web Services (AWS) Service Commitment:

  • AWS aims for a "Monthly Uptime Percentage" of at least $99.95\%$ for Amazon EC2 and Amazon EBS.

  • Failure to meet this commitment makes customers eligible for a Service Credit.

• High Availability ("5 nines"): This term refers to a system targeted at $99.999\%$ availability.

• Definition of Fault Tolerance: The ability of a system to mask or repair faults in such a way that the cumulative service outage period does not exceed a required value over a specific time interval.

Circuit Breaker Pattern

• Definition: "Prevent cascading failures by preventing requests to external services that begin to fail" [Bonér 2017].

• Applicability:

  • Deliberately breaks the flow of requests from one component to another when the recipient is either failing or overloaded.

  • Provides the recipient "breathing room" to recover from load-induced failures.

  • Empowers the sender to decide that requests should fail immediately rather than wasting time and resources waiting for negative replies.

• Mechanism and States:

  • Open State: Requests fail immediately; a timeout is set to try again later.

  • Closed State: Normal operation where requests are processed.

  • Half-Open/Success Transition: If the next request succeeds after a timeout, the system moves back to the Closed state. If it fails, it returns to the Open state and resets the timeout.

• Standard Implementations: Included in frameworks like Akka, Lagom, and Hystrix.

Fault Tolerance Infrastructure Desiderata

• Support Component Compartmentalisation:

  • The infrastructure must provide means to isolate faulty components so they do not cause a total system crash.

• Support Suspension of Component Interactions:

  • When a component fails, calls to it should be suspended until it is fixed or replaced.

  • The new component should continue work without dropping data.

  • The call handled at the time of failure must not disappear, as it may be critical for recovery or for diagnosing the fault.

• Provide Component Lifecycle Management:

  • Faulty components must be isolated and, if impossible to recover, terminated or re-initialized to a correct starting state.

  • Must have defined lifecycles (start, restart, terminate).

• Enable Separation of Concerns:

  • Fault-recovery code should be separated from normal processing code (recovery is a cross-cutting concern).

Bulkheading: Principles from Shipbuilding

• Origin: The term comes from segmenting a vessel into fully isolated compartments.

• Core Principle: If the hull is breached, only affected compartments fill with water, keeping the ship afloat.

• The Titanic Example:

  • Featured $15$ bulkheads and was considered unsinkable.

  • Failure Cause: To save money and for passenger convenience, bulkheads extended only a few feet above the waterline and were not sealed at the top.

  • When $5$ compartments near the bow were breached, the bow dipped, allowing water to flow over the top of the bulkheads into adjacent compartments until the ship sank.

• Distributed Computing Parallel: Managing fault tolerance at the level of entire application servers can result in one failure stalling or overloading others if they are not truly isolated.

Failures of Object-Oriented (OO) Exception Handling

• Log Processing Example Components:

  • FileWatcher: Watches file systems for new files; notifies processors.

  • LogFile: The target file to be read.

  • LogProcessor: Reads files line by line and creates row objects.

  • Row / Writer: Writes rows to storage (database).

  • Connection / Database: The physical DB connection.

• The Setup Problem:

  • A DbWriter refers directly to a connection val con = DbFactory.createConnection(url).

  • A logProcessor refers directly to the writer.

• Critical Weaknesses:

  • Replacing Dependencies: If a DBConnectionException occurs, replacing the broken connection in the object graph is not a first-class feature.

  • Direct Communication: Objects communicate directly, making isolation difficult.

  • Entanglement: Fault-recovery and functional code are tangled together.

  • Concurrency Issues: It is difficult to ensure other threads do not use the faulty connection while it is being replaced.

Actor Supervision in Akka

• Implementation:

  • Supervisors wrap message processing behavior and catch exceptions.

  • The mailbox of a crashed actor is suspended until the supervisor decides the outcome.

  • Decisions are based solely on the magnitude or cause of the crash (exception type).

  • Domain logic remains in the message processing behavior, separate from recovery logic.

• Recovery Options:

  1. Restart: The crashed actor is re-created. It continues processing messages via its ActorRef.

  2. Resume: The same actor instance continues; the crash is ignored.

  3. Stop: The actor is terminated and no longer participates in messaging.

• Addressing Desiderata:

  • Compartmentalisation: Actors can be terminated/removed; the hierarchy allows run-time instance replacement.

  • Suspension: Mailbox suspension prevents work loss during failure.

  • Lifecycle: Actors are active components that can be started, stopped, and restarted.

  • Separation: Message-processing and supervision flows are orthogonal.

Custom Supervision Strategy Implementation

• Exception Model:

  • Error instances: Denote unrecoverable errors (e.g., DiskError).

  • Exception instances: Denote recoverable errors (e.g., ParseException, DbBrokenConnectionException, UnexpectedColumnsException).

• Supervision Logic Examples:

  • LogProcessor: Uses .onFailure[ParseException](SupervisorStrategy.resume) to continue processing other files if one cannot be parsed.

  • FileWatcher: Uses .onFailure[ClosedWatchServiceException](SupervisorStrategy.restart) to attempt to restart the directory change service.

  • DbWriter:

    • Uses .onFailure[UnexpectedColumnsException](SupervisorStrategy.resume) to bypass data format errors.

    • Uses SupervisorStrategy.restartWithBackoff for DbBrokenConnectionException.

• Backoff Algorithm Specifics:

  • minBackoff: $3$ seconds.

  • maxBackoff: $30$ seconds.

  • randomFactor: $0.1$.

  • withResetBackoffAfter: $15$ seconds.

  • This results in successive retry delays at approximately $3, 6, 12, 24, 30$ seconds.

Error Kernel Pattern

• Definition: "In a supervision hierarchy, keep important application state or functionality near the root while delegating risky operations towards the leaves" [Kuhn 2017].

• Applicability:

  • Systems where components have varying reliability requirements and failure severities.

  • When state at the top is expensive to re-create, whereas leaf-level failures are frequent and expected.

• Hierarchy Strategy:

  • Push activities with high failure risks downward.

  • Failure domains should coincide with responsibility boundaries.

  • This allows the most "dangerous" actors to be at the bottom, where their faults can be handled by multiple layers of supervisors.

Review Questions

• When is a Circuit Breaker pattern appropriate in micro-service architectures?

• Sketch the interaction flow between actors implementing the Ask pattern using an ephemeral child actor.

• What is the relationship between the Builder and Aggregator design patterns?

• What logic is required for reliable message delivery on the sender and recipient sides? What Akka features facilitate this?

• Illustrate the Domain Object pattern in Akka, highlighting differences between domain-level and actor-level messages.

• What is event sourcing? Provide three advantages and three disadvantages.

• Compare three options for preventing service overload from incoming requests.

• How do flow control patterns compare to the circuit breaker pattern?

• How does Akka satisfy the infrastructure desiderata for fault tolerance?

• Implement an example actor system separating normal message processing from fault recovery logic.