Chapter 4 - Managing transactions with sagas

0.0(0)
Studied by 0 people
call kaiCall Kai
learnLearn
examPractice Test
spaced repetitionSpaced Repetition
heart puzzleMatch
flashcardsFlashcards
GameKnowt Play
Card Sorting

1/26

encourage image

There's no tags or description

Looks like no tags are added yet.

27 Terms

1
New cards

Using the Saga pattern to maintain data consistency

Sagas are mechanisms to maintain data consistency in a microservice architecture without having to use distributed transactions.

You define a saga for each system command that needs to update data in multiple services.

A saga is a sequence of local transactions. Each local transaction updates data within a single service using the familiar ACID transaction frameworks and libraries.

The system operation initiates the first step of the saga. The completion of a local transaction triggers the execution of the next local transaction.

2
New cards

Sagas vs ACIS

Sagas differ from ACID transactions in a couple of important ways. As I describe in detail in section 4.3, they lack the isolation property of ACID transactions.

Also, because each local transaction commits its changes, a saga must be rolled back using compensating transactions.

3
New cards

An example SAGA: the createOrder() saga

services that participate in a saga communicate using asynchronous messaging. A service publishes a message when a local transaction completes. This message then triggers the next step in the saga. Not only does using messaging ensure the saga participants are loosely coupled, it also guarantees that a saga completes. That’s because if the recipient of a message is temporarily unavailable, the message broker buffers the message until it can be delivered.

<p>services that participate in a saga communicate using asynchronous messaging. A service publishes a message when a local transaction completes. This message then triggers the next step in the saga. Not only does using messaging ensure the saga participants are loosely coupled, it also guarantees that a saga completes. That’s because if the recipient of a message is temporarily unavailable, the message broker buffers the message until it can be delivered.</p>
4
New cards

Sagas use compensating transactions to roll back changes

Unlike ACIS transaction, sagas can’t be automatically rolled back, because each step commits its changes to the local database. This means, for example, that if the authorization of the credit card fails in the fourth step of the Create Order Saga, the FTGO application must explicitly undo the changes made by the first three steps. You must write what are known as compensating transactions.

5
New cards

What happens if a step in a Saga fails?

  • If step Tn+1 fails, all previous steps T1…Tn must be undone.

  • Each step Ti has a compensating transaction Ci that reverses it.

  • Undo is done in reverse order: T1… Tn, Cn…C1.

  • Example: If credit card authorization fails in step 4, the first 3 steps must be rolled back with compensating transactions.


It’s important to note that not all steps need compensating transactions. Read-only steps, such as verify ConsumerDetails(), don’t need compensating transactions. Nor do steps such as authorizeCreditCard() that are followed by steps that always succeed.

(145-147)

<ul><li><p>If step T<sub>n+1</sub> fails, all previous steps T<sub>1</sub>…T<sub>n</sub> must be undone.</p></li><li><p>Each step T<sub>i</sub> has a compensating transaction C<sub>i</sub> that reverses it.</p></li><li><p>Undo is done in reverse order: T<sub>1</sub>… T<sub>n</sub>, C<sub>n</sub>…C<sub>1</sub>.</p></li><li><p>Example: If credit card authorization fails in step 4, the first 3 steps must be rolled back with compensating transactions.</p></li></ul><p></p><p>It’s important to note that not all steps need compensating transactions. Read-only steps, such as verify ConsumerDetails(), don’t need compensating transactions. Nor do steps such as authorizeCreditCard() that are followed by steps that always succeed.</p><p>(145-147)</p>
6
New cards

Coordinating Sagas

A saga’s implementation consists of logic that coordinates the steps of the saga. When a saga is initiated by system command, the coordination logic must select and tell the first saga participant to execute a local transaction. Once that transaction completes, the saga’s sequencing coordination selects and invokes the next saga participant. This process continues until the saga has executed all the steps. If any local transaction fails, the saga must execute the compensating transactions in reverse order.

There are a couple of different ways to structure a saga’s coordination logic: 

  • Choreography—Distribute the decision making and sequencing among the saga participants. They primarily communicate by exchanging events.

  • Orchestration—Centralize a saga’s coordination logic in a saga orchestrator class. A saga orchestrator sends command messages to saga participants telling them which operations to perform.


7
New cards

Coordinating Sagas - Choreography-based sagas

One way you can implement a saga is by using choreography. When using choreography, there’s no central coordinator telling the saga participants what to do. Instead, the saga participants subscribe to each other’s events and respond accordingly.

8
New cards
<p>Implementation of Create order SAGA using Choreography</p>

Implementation of Create order SAGA using Choreography

Figure 4.4 shows the design of the choreography-based version of the Create Order Saga. The participants communicate by exchanging events. Each participant, starting with the Order Service, updates its database and publishes an event that triggers the next participant.


The Create Order Saga must also handle the scenario where a saga participant rejects the Order and publishes some kind of failure event. For example, the authorization of the consumer’s credit card might fail. The saga must execute the compensating transactions to undo what’s already been done. Figure 4.5 shows the flow of events when the AccountingService can’t authorize the consumer’s credit card.

<p>Figure 4.4 shows the design of the choreography-based version of the Create Order Saga. The participants communicate by exchanging events. Each participant, starting with the Order Service, updates its database and publishes an event that triggers the next participant.</p><div data-type="horizontalRule"><hr></div><p>The Create Order Saga must also handle the scenario where a saga participant rejects the Order and publishes some kind of failure event. For example, the authorization of the consumer’s credit card might fail. The saga must execute the compensating transactions to undo what’s already been done. Figure 4.5 shows the flow of events when the AccountingService can’t authorize the consumer’s credit card.</p>
9
New cards

communication-related issues that you must consider when implementing choreography-based sagas.

The first issue is ensuring that a saga participant updates its database and publishes an event as part of a database transaction. Each step of a choreography-based saga updates the database and publishes an event. For example, in the Create Order Saga, Kitchen Service receives a Consumer Verified event, creates a Ticket, and publishes a Ticket Created event. It’s essential that the database update and the publishing of the event happen atomically. Consequently, to communicate reliably, the saga participants must use transactional messaging, described in chapter 3.

The second issue you need to consider is ensuring that a saga participant must be able to map each event that it receives to its own data. For example, when OrderService receives a Credit Card Authorized event, it must be able to look up the corresponding Order. The solution is for a saga participant to publish events containing a correlation id, which is data that enables other participants to perform the mapping.

10
New cards

Benefits & Drawbacks of Choreography-based sagas


<p></p>
11
New cards

Orchestration-based sagas

Orchestration is another way to implement sagas. When using orchestration, you define an orchestrator class whose sole responsibility is to tell the saga participants what to do.

The saga orchestrator communicates with the participants using command/async reply-style interaction.

To execute a saga step, it sends a command message to a participant telling it what operation to perform. After the saga participant has performed the operation, it sends a reply message to the orchestrator. The orchestrator then processes the message and determines which saga step to perform next.

<p>Orchestration is another way to implement sagas. When using orchestration, you define an orchestrator class whose sole responsibility is to tell the saga participants what to do.</p><p>The saga orchestrator communicates with the participants using command/async reply-style interaction.</p><p>To execute a saga step, it sends a command message to a participant telling it what operation to perform. After the saga participant has performed the operation, it sends a reply message to the orchestrator. The orchestrator then processes the message and determines which saga step to perform next.</p>
12
New cards

Implementing the the createOrder (as an example) Saga using Orchestration

Note that in final step, the saga orchestrator sends a command message to OrderService, even though it’s a component of Order Service. In principle, the Create Order Saga could approve the Order by updating it directly. But in order to be consistent, the saga treats Order Service as just another participant.

Diagrams such as figure 4.6 each depict one scenario for a saga, but a saga is likely to have numerous scenarios. For example, the Create Order Saga has four scenarios. In addition to the happy path, the saga can fail due to a failure in either Consumer Service, Kitchen Service, or Accounting Service. It’s useful, therefore, to model a saga as a state machine, because it describes all possible scenarios.

<p>Note that in final step, the saga orchestrator sends a command message to <em>OrderService,</em> even though it’s a component of Order Service. In principle, the Create Order Saga could approve the Order by updating it directly. But in order to be consistent, the saga treats Order Service as just another participant.</p><p>Diagrams such as figure 4.6 each depict one scenario for a saga, but a saga is likely to have numerous scenarios. For example, the Create Order Saga has four scenarios. In addition to the happy path, the saga can fail due to a failure in either Consumer Service, Kitchen Service, or Accounting Service. It’s useful, therefore, to model a saga as a state machine, because it describes all possible scenarios.</p>
13
New cards

Modeling saga orchestration as state machines

A state machine consists of a set of states and a set of transitions between states that are triggered by events. Each transition can have an action, which for a saga is the invocation of a saga participant.

The transitions between states are triggered by the completion of a local transaction performed by a saga participant.

The current state and the specific outcome of the local transaction determine the state transition and what action, if any, to perform.

(check img) → The state machine’s initial action is to send the VerifyConsumer command to Consumer Service. The response from Consumer Service triggers the next state transition. If the consumer was successfully verified, the saga creates the Ticket and transitions to the Creating Ticket state. But if the consumer verification failed, the saga rejects the Order and transitions to the Rejecting Order state. The state machine undergoes numerous other state transitions, driven by the responses from saga participants, until it reaches a final state of either Order Approved or Order Rejected.

(153-154)

<p>A state machine consists of a set of states and a set of transitions between states that are triggered by events. Each transition can have an action, which for a saga is the invocation of a saga participant. </p><p>The transitions between states are triggered by the completion of a local transaction performed by a saga participant. </p><p>The current state and the specific outcome of the local transaction determine the state transition and what action, if any, to perform.</p><p>(check img) → The state machine’s initial action is to send the <em>VerifyConsumer </em>command to Consumer Service. The response from Consumer Service triggers the next state transition. If the consumer was successfully verified, the saga creates the Ticket and transitions to the Creating Ticket state. But if the consumer verification failed, the saga rejects the Order and transitions to the Rejecting Order state. The state machine undergoes numerous other state transitions, driven by the responses from saga participants, until it reaches a final state of either Order Approved or Order Rejected.</p><p>(153-154)</p>
14
New cards

Saga orchestration and transactional messaging

Each step of an orchestration-based saga consists of a service updating a database and publishing a message.

For example, Order Service persists an Order and a Create Order Saga orchestrator and sends a message to the first saga participant. A saga participant, such as Kitchen Service, handles a command message by updating its database and sending a reply message. Order Service processes the participant’s reply message by updating the state of the saga orchestrator and sending a command message to the next saga participant. As described in chapter 3, a service must use transactional messaging in order to atomically update the database and publish messages. Later on in section 4.4, I’ll describe the implementation of the Create Order Saga orchestrator in more detail, including how it uses transaction messaging.

I recommend using orchestration for all but the simplest sagas.

<p>Each step of an orchestration-based saga consists of a service updating a database and publishing a message.</p><p>For example, Order Service persists an Order and a Create Order Saga orchestrator and sends a message to the first saga participant. A saga participant, such as Kitchen Service, handles a command message by updating its database and sending a reply message. Order Service processes the participant’s reply message by updating the state of the saga orchestrator and sending a command message to the next saga participant. As described in chapter 3, a service must use transactional messaging in order to atomically update the database and publish messages. Later on in section 4.4, I’ll describe the implementation of the Create Order Saga orchestrator in more detail, including how it uses transaction messaging.</p><p>I recommend using orchestration for all but the simplest sagas.</p>
15
New cards

Handling the lack of isolation when using sagas

The challenge with using sagas is that they lack the isolation property of ACID transactions. That’s because the updates made by each of a saga’s local transactions are immediately visible to other sagas once that transaction commits.

This behavior can cause two problems. First, other sagas can change the data accessed by the saga while it’s executing. And other sagas can read its data before the saga has completed its updates, and consequently can be exposed to inconsistent

data. You can, in fact, consider a saga to be ACD without I (Isolation):

  • Atomicity—The saga implementation ensures that all transactions are executed

or all changes are undone.

  • Consistency—Referential integrity within a service is handled by local databases.

Referential integrity across services is handled by the services.

  • Durability—Handled by local databases.


16
New cards

Overview of anomalies

This lack of isolation potentially causes what the database literature calls anomalies. An anomaly is when a transaction reads or writes data in a way that it wouldn’t if transactions were executed one at time.

  • Lost updates—One saga overwrites without reading changes made by another saga.

  • Dirty reads—A transaction or a saga reads the updates made by a saga that has

not yet completed those updates.

  • Fuzzy/nonrepeatable reads—Two different steps of a saga read the same data and get different results because another saga has made updates.

All three anomalies can occur, but the first two are the most common and the most challenging. Let’s take a look at those two types of anomaly, starting with lost updates.

17
New cards

Anomalies - Lost updates

knowt flashcard image
18
New cards

Anomalies - Dirt updates

knowt flashcard image
19
New cards

Countermeasures for handling the lack of isolation

20
New cards

The structure of a saga

The countermeasures paper mentioned in the last section defines a useful model for the structure of a saga.

shown in figure 4.8, a saga consists of three types of transactions:

  • Compensatable transactions—Transactions that can potentially be rolled back using a compensating transaction.

  • Pivot transaction—The go/no-go point in a saga. If the pivot transaction commits, the saga will run until completion. A pivot transaction can be a transaction that’s neither compensatable nor retriable. Alternatively, it can be the last compensatable transaction or the first retriable transaction.

  • Retriable transactions—Transactions that follow the pivot transaction and are guaranteed to succeed.


<p>The countermeasures paper mentioned in the last section defines a useful model for the structure of a saga.</p><p>shown in figure 4.8, a saga consists of three types of transactions:</p><ul><li><p>Compensatable transactions—Transactions that can potentially be rolled back using a compensating transaction.</p></li><li><p>Pivot transaction—The go/no-go point in a saga. If the pivot transaction commits, the saga will run until completion. A pivot transaction can be a transaction that’s neither compensatable nor retriable. Alternatively, it can be the last compensatable transaction or the first retriable transaction.</p></li><li><p>Retriable transactions—Transactions that follow the pivot transaction and are guaranteed to succeed.</p></li></ul><p></p>
21
New cards

Countermeasure: Semantic Lock

When using the semantic lock countermeasure, a saga’s compensatable transaction sets a flag in any record that it creates or updates.

The flag indicates that the record isn’t committed and could potentially change. The flag can either be a lock that prevents other transactions from accessing the record or a warning that indicates that other transactions should treat that record with suspicion.

It’s cleared by either a retriable transaction—saga is completing successfully—or by a compensating transaction: the saga is rolling back.

<p>When using the semantic lock countermeasure, a saga’s compensatable transaction sets a flag in any record that it creates or updates. </p><p>The flag indicates that the record isn’t committed and could potentially change. The flag can either be a lock that prevents other transactions from accessing the record or a warning that indicates that other transactions should treat that record with suspicion. </p><p>It’s cleared by either a retriable transaction—saga is completing successfully—or by a compensating transaction: the saga is rolling back.</p>
22
New cards

Countermeasure: Commutative updates

One straightforward countermeasure is to design the update operations to be commutative. Operations are commutative if they can be executed in any order. An Account’s debit() and credit() operations are commutative (if you ignore overdraft checks). This countermeasure is useful because it eliminates lost updates.

Consider, for example, a scenario where a saga needs to be rolled back after a compensatable transaction has debited (or credited) an account. The compensating transaction can simply credit (or debit) the account to undo the update. There’s no possibility of overwriting updates made by other sagas.

23
New cards

Countermeasure: pessimistic view

Another way to deal with the lack of isolation is the pessimistic view countermeasure. It reorders the steps of a saga to minimize business risk due to a dirty read.


<p>Another way to deal with the lack of isolation is the pessimistic view countermeasure. It reorders the steps of a saga to minimize business risk due to a dirty read.</p><p></p>
24
New cards

Countermeasure: Read value

The reread value countermeasure prevents lost updates. A saga that uses this countermeasure rereads a record before updating it, verifies that it’s unchanged, and then updates the record. If the record has changed, the saga aborts and possibly restarts.

<p>The r<strong>eread value </strong>countermeasure prevents lost updates. A saga that uses this countermeasure rereads a record before updating it, verifies that it’s unchanged, and then updates the record. If the record has changed, the saga aborts and possibly restarts.</p>
25
New cards

Countermeasure: Version file

The version file countermeasure is so named because it records the operations that are performed on a record so that it can reorder them. It’s a way to turn noncommutative operations into commutative operations.

<p>The version file countermeasure is so named because it records the operations that are performed on a record so that it can reorder them. It’s a way to turn noncommutative operations into commutative operations.</p>
26
New cards

Countermeasure: By value

The final countermeasure is the by value countermeasure. It’s a strategy for selecting concurrency mechanisms based on business risk.

An application that uses this countermeasure uses the properties of each request to decide between using sagas and distributed transactions. It executes low-risk requests using sagas, perhaps applying the countermeasures described in the preceding section. But it executes high-risk requests involving,

<p>The final countermeasure is the by value countermeasure. It’s a strategy for selecting concurrency mechanisms based on business risk. </p><p>An application that uses this countermeasure uses the properties of each request to decide between using sagas and distributed transactions. It executes low-risk requests using sagas, perhaps applying the countermeasures described in the preceding section. But it executes high-risk requests involving,</p>
27
New cards

Creating sagas - OrderService example

This would need the following steps:

  1. creating a new order

  2. publish the createOrder event

  3. save the new created order in the database

  4. create the saga state

  5. publish the saga state to consumers

(complete example: 163-175)

<p>This would need the following steps:</p><ol><li><p>creating a new order</p></li><li><p>publish the createOrder event</p></li><li><p>save the new created order in the database</p></li><li><p>create the saga state</p></li><li><p>publish the saga state to consumers</p></li></ol><p>(complete example: 163-175)</p>