--- title: Compensating Actions url: https://www.grahambrooks.com/event-driven-architecture/compensating-actions/ date: 2017-05-25T23:15:26-06:00 author: Graham Brooks description: Consistency - the absence of contradiction is a challenge in any application. Event driven asynchronous distributed systems turn up the dial of complexity. What does consistency mean? tags: - architecture series: - event-driven-architecture --- # Compensating Actions ```asciidoc = Measuring Consistency :permalink: "/event-driven-architecture/consistency-measurement/" :paro: _code-samples/paro :sample: _code-samples/event-driven-architecture/samples.json This is a followup to link:/event-driven-architecture/consistency/[consistency]. Specifically this is about measuring consistency in a distributed system within different consistency scopes. Event or message driven systems using CQRS present the most consistency challenges so I am going to assume that we are trying to measure consistency for a asynchronous event driven system. image:/images/cqrs-event-service-consistency.svg[align="center"] In CQRS requests that change the state of a system are called commands. Commands are received by the command processor [1]. The processed command generates a state change that is stored in a local store and an event published [2]. Once this processing is complete control returns to the caller. The event bus transports the event to the query processor [3]. The query processor receives the event and updates its own store. At this point a Query [4] is consistent with the command [1]. == Transaction Consistency .Transaction Consistency image:/images/transaction-consistency-scope.svg[align="center"] The command defines the transaction boundary. Once the command has been processed the command processor achieves transaction consistency. == Service/Cluster Consistency image:/images/service-consistency-scope.svg[align="center"] For service level consistency the event has to be transported from the command processor to the query processor and processed successfully. Quite often the Command and Query processors are in different processes. High availability means multiple instances so even if the command and query processors are in the same process it is likely that the event will be picked up but another member of the cluster. For Transaction and service consistency measurement the service can be responsible for monitoring its consistency. Service activity from the command and query processors can be reconciled. The service can also be responsible for alerting when service SLAs are not met. == Capability Consistency image:/images/capability-consistency-scope.svg[align="center"] Once beyond a single service it is not possible for each service to monitor its own consistency. Service +B+ does not know that Service +A+ has published an event if it has not received one. At this point in consistency measurement we need to introduce another party to take care of this measurement. image:/images/consistency-observer.svg[align="center"] The 3rd party observer is a system implementation of the OO Observer pattern. The Observer relies on activity data published by all the components. In the example above Service A tells the observer that it processed a command [1] and published an event [2]. This activity 'report' sets up an expectation in the observer that Service B will be taking action. Capability consistency is reached when an activity report is received from Service B that it processed the event [3] and updated the external service [4]. Activity reports include details of the data being processed and the data being produced. TIP: I processed [command/event] and produced [event(s)] 'I' Refers to the system that processed the data. This would typically be a unique service name and version. The version is important during updates that introduce a new event dependency. 'command/event' refers to the type and identity of the data being processed. 'events' lists the event types and identities of the events produced. Activities signal to the observer that there is a consistency expectation that needs to be met. == Enterprise Consistency Enterprise consistency is a catch-all level and includes all systems implemented and used by the enterprise. It is also the hardest to measure because it includes both real-time user facing systems and warehousing systems. Consistency measurements at this level are typically computed offline and infrequently. == The Observer The observer outlined above is a form of distributed tracing. http://zipkin.io[Zipkin] and https://cloud.spring.io/spring-cloud-sleuth/[Spring Cloud Sleuth] are examples of libraries and tools that support distributed tracing. There are some key differences though. Both Zipkin and Spring Cloud Sleuth are measurement tools that provide a way to investigate latency and execution path issues. For Consistency measurement we need to add expectations so that each execution path can be compared against it. === Expectations Each component in a distributed messaging system has a set of messages/events that it is interested in. e.g. As an order management system I am interested in new customer registrations. Given that we have an interest list from each service or component we can infer that if a new customer registration event happens then we should see an activity report from the order management system. When processing an event it is perfectly reasonable for a service to generate its own event. In this case the activity report would include the events produced from handling the received event. This leads to expectation chaining. An activity report can extend the 'flow' to include new expectations for the produced event. .Single service expectation [source,clojure] ---- include::{paro}/test/paro/consistency_test.clj[tag="single-service-expectation"] ---- The expectation declaration above indicates that the `order-management` system expects to process `new-customer-registration` events. If a component publishes a `new-customer-registration` then the observer expects to receive an activity report from the `order-management` system. image:/images/consistency-measurement-expectation-setting.svg[align="center"] When we receive the activity data from the customer management system this sets up an expectation that to be consistent the order management system must report that is has processed the event. .Customer Management activity report [source,clojure] ---- include::{paro}/test/paro/consistency_test.clj[tag="producing-activity-report"] ---- Receiving this activity report sets up an expectation that the `order-management` system will report that it has processed the `new-customer-registration` event as part of the flow identified by the `flow-id`. .Order Management activity report [source,clojure] ---- include::{paro}/test/paro/consistency_test.clj[tag="consuming-activity-report"] ---- Once the activity report has been received we can conclude that consistency was achieved in 50 seconds. The first measurements we need to make Measurement measure how well we achieve a particular consistency level against its SLA or expectations. == Consistency scopes Let's compare this with the same interaction that communicates through events. The client interaction only involves the sales system directly. As part of the client call an event is produced and later consumed by the other components. If the client calls the shipping system immediately after placing the order it is likely that the shipping system will know about the order because the event is still in transit. Until all the components process the event the system is inconsistent. Components contradict each other about whether an order exists or not. [plantuml, event-driven-change,svg, align="center",width="100%"] ---- include::_pages/event-driven-architecture/patterns/plantuml.skin[] participant Client participant Sales participant EventBus participant Orders participant Shipping Client -> Sales: Place Order Sales -> EventBus: New Order Sales -> Client: Order Confirmation ... EventBus -> Orders: New Order ... EventBus -> Shipping: New Order ---- We could rely on smart clients that understand inconsistency and poll until the order becomes available - but what happens if the event never arrives or can't be processed for some reason. We now have clients polling indefinitely waiting for the system to be consistent. Clients might accept notifications or events. Each system could publish client events back to the UI. This avoids the polling scenario. This could be a websocket or any server->client notification system. [plantuml, async-clien-tevent-driven-change,svg, align="center",width="100%"] ---- include::_pages/event-driven-architecture/patterns/plantuml.skin[] participant Client participant Sales participant EventBus participant Orders participant Shipping Client -> Sales: Place Order Sales -> EventBus: New Order Sales -> Client: Order Confirmation ... EventBus -> Orders: New Order Orders --> Client: Order accepted ... EventBus -> Shipping: New Order Shipping --> Client: Shipping scheduled ---- If all goes well this interaction allows provides a responsive highly available user experience. If an event is not received within an acceptable period the user can be informed perhaps with a "Your order has been received. Confirmation coming soon". Note: These scenarios come up in typically consistent systems when a required service becomes unavailable. A key difference is that when designing event driven systems these concerns are there at the start and not later when something goes wrong. Being an immediate concern means that we can discuss these scenarios earlier in the development. == Consistency Failure Scenarios Failure modelling The problems arise when something goes wrong. Events can't be processed. Maybe the are lost in transmission, contain unexpected data etc. While likely to be rare the impact of failure is significant for many systems.. This example only covers a single client call. In reality there will be many client calls and lots of events traversing the system. If this activity is continuous then the overall system is always in an inconsistent state. If you have read about event driven systems you have probably heard about eventual consistency, that eventually the system will become consistent. In the case of a single client transaction all the data stores will reflect the same state. based on replicating state contradictions are common. These systems are by definition eventually consistent. Events take time to be transported between components. While the event is in transit different components have a different view of that state. ## Command processing Processing the same command repeatedly should result in the same behavior as processing the command once *unless* the state of the entity changes during repeated command execution. Contiguous commands of the same action is equivalent to a single command for the same action. c1(1) c1(1) … === c1(1) REST and command (mutating request) handling POST /api/resource/new In rest the /api/resource path might be used since the new and POST are equivalent. In CQRS the path might contain the action such as new for an account or moved for an address. When creating a new resource POST `/api/resource/new` would typically return a 201 (Created) status code. A modifying request `/api/resource/close` that does not result in a new resource or sub-resource would return 200 (OK) or 204 (No Content). Assuming that a repeated command can be identified as a repeat by the receiving service the result of the call or function is not Idempotent because the status code should not be the same. Returning 201 (created) multiple times would not make sense the client. Status code 409 (Conflict) would be more appropriate for subsequent requests. ## Side Effects Events do not exist in a vacuum. To be effective events need affect not only the state of the system but they need to affect the state of the integration systems as well. An event may result in one or more events because of such a change. These side effects ## Event processing Events are published in response to state changes to an aggregate root that are significant to the domain. Event subscribers update or process the event. Updates can be idempotent iff the latest event is received multiple times. However in a replay situation all past events are received. Consider the sequence E1..E2..E3. When processed a value is first set to 1 then 2 and finally 3. Replay would result in E1..E2..E3 being received in order again but the subscriber state would ideally stay in state 3. ## Event processing to actions Events can cause subsequent events to be published to down-stream subscribers. In a replay scenario it would be inappropriate to re-issue the downstream event or take processing action (e.g. sending an email). ### 201 Status code (http://restpatterns.org) The request has been fulfilled and resulted in a new resource being created. The newly created resource can be referenced by the URI(s) returned in the entity of the response, with the most specific URI for the resource given by a Location header field. The response SHOULD include an entity containing a list of resource characteristics and location(s) from which the user or user agent can choose the one most appropriate. The entity format is specified by the media type given in the Content-Type header field. The origin server MUST create the resource before returning the 201 status code. If the action cannot be carried out immediately, the server SHOULD respond with 202 (Accepted) response instead. ## Subscriber replay detection Replay and Compensating actions. `Replay is the` === Compensating transaction pattern https://msdn.microsoft.com/en-us/library/dn589804.aspx Type: ''[Design | Organizational | ...] Problem:: Business transaction errors and processing errors in event sourced systems cannot easily be corrected by changing the state of data because the data state is computed on demand. Address errors in systems that do not directly maintain state but rather model state as a sequence of events that when applied using a function result in the effective system state. State = \(f(f(null, Event0), Event_1_)\) Context:: In event sourced systems the current state is derived by applying a function that applies an event to the previous data state. This could be represented as follows. The initial state of some data elements S~0~ is defined as s~0~ = f(null, E~0~) subsequent data sates would then be derived by applying the function to the previous data state. S~n~ = f(S~n-1~, E~n~) Forces:: Compensating transactions can be used as part of expected application behavior to reverse previous actions. It can also be used to correct anomalous state caused by programming or more generally processing errors. Proposed Solution:: A compensating transaction is introduced into the event stream modeled as an event such that the result of applying the appropriate function to the desired data results in the expected state. Target State S~t~ = f(S~n~, E~compensation~) When modeled in this way a compensating transaction event is indistinguishable from a non compensating event to the function that applies the event to the application data state. In its basic form the compensating transaction needs to be introduced into the event stream at the top most position. In the CQRS pattern events are emitted in response to a command. To apply a compensating transaction to an entity a compensating command would be required. Compensating Command [plantuml, diagram-class, png] ---- @startuml class Event { EntityId } class Entity { id Event[] events static apply(event) } Entity -- Event: accepts @enduml ---- Resulting Context:: [what happens when you apply it, good and bad] Design Rationale:: [rationale] Related Patterns:: AntiPatterns you should beware of:: Pattern Category:: [classify it] Also Known As:: [other names] Examples in the Literature:: Examples in Practice:: Context In event sourced systems state chages are modeled as events. State corrections cannot be applied In business terms a compensating transaction Compensating actions are taken to address a processing error that results in the system being in an inappropriate state. These actions are best described by example. Lets assume that a subscriber maintain === Recall Pattern Context:: An event is generated in error, perhaps due to a bug or inconsistent state. Forces:: Recalls are much simpler to process if they are sent immediately after the event to be recalled. As historical context builds it can be more difficult to process the recall. Consider a financial transaction event that is received and processed for an account. [source,json] ---- include::{sample}[tag="transaction-event"] ---- A recall for the event might look something like [source,json] ---- include::{sample}[tag="transaction-event-recall"] ---- Proposed Solution:: A recall is the complete reversal of a previous event. It is not the deletion although in many cases the effect will be the same. Reversing the effects of a previous event and in particular any processing that may have happened because of the event is typically non-trivial. Transaction fees may have accumulated on the account because of the transaction. It is not uncommon for transactions to be disputed months after the transaction has been processed. However, adjusting the account balance with a complete event history is simpler than special processing in an non-event driven system. It's also relatively common that a recall might be issued on a previous recall. Resulting Context:: Design Rationale:: Patter Category:: Also Known as:: Examples in the Literature:: Examples in Practice:: A recall is a compenstaing ation ## When things go wrong So you have your services up and running maybe running in production and working with client requests. All's good with your system and the world. If only things could be that simple. In the world of real systems things go wrong and systems misbehave quite often at the worst possible time. We have a pretty good track set of tools and techniques for fixing transnational systems. Most revolve around updating the system state directly to the data store. For relational store that means writing some SQL and running it against the affected database. While relatively cheap to implement in terms of time and effort these approaches have problems. Quite often they are run by bypassing the usual release procedures. They are often difficult to test in an automated way and require peer review. Data changes while a solution is being developed so the fix may not even work when it is finally applied to an environment. These problems often lead to even riskier situations where changes are being defined and applied during the update window for the fix. Interestingly, event based systems and particularly event sourced systems are typically very difficult to fix this way. Events being the immutable store of changes against an entity or aggregate root means that any changes to that entities data has to be processed and result in an event that corrects the problem. This is one example of a compensating action. Types of compensating actions: Loss Compensation:: Loss compensation is used to address the problem of data lost in transit. Error Compensation:: Using existing APIs to make a change to resolve a problem with a client requests. Bug Compensation:: These are the most complex to resolve and require custom code to be developed that addresses changes caused by a bug in the system. == Loss Compensation In an event driven eventually consistent system when do you know when an event has been lost? Is the loss of a message to a single subscribr the same as the loss to all substribers. Is one a special case of the more general situation. == Error Compensation == Bug Compenstation ```