The Ops Community ⚙️: Avraham Neeman

Event Streams Are Nothing Without Action

Avraham Neeman — Tue, 22 Nov 2022 08:54:00 +0000

Each data point in a system that produces data on an ongoing basis corresponds to an Event. Event Streams are described as a continuous flow of events or data points. Event Streams are sometimes referred to as Data Streams within the developer community since they consist of continuous data points. Event Stream Processing refers to the action taken on generated Events.

This article discusses Event Streams and Event Stream Processing in great depth, covering topics such as how Event Stream Processing works, the contrast between Event Stream Processing and Batch Processing, its benefits and use cases, and concluding with an illustrative example of Event Stream Processing.

Event Streams: An Overview

Coupling between services is one of the most significant difficulties associated with microservices. Conventional architecture is a “don’t ask, don’t tell” architecture in which data is collected only when requested. Suppose there are three services in issue, A, B, and C. Service A asks the other services, “What is your present state?” and assumes they are always ready to respond. This puts the user in a position if the other services are unavailable.

Retries are utilized by microservices as a workaround to compensate for network failures or any negative impacts brought on by changes in the network topology. However, this ultimately adds another layer of complexity and increases the expense.

In order to address the problems with the conventional design, event-driven architecture adopts a “tell, don’t ask” philosophy. In the example above, Services B and C publish Continuous Streams of data, such as Events, and Service A subscribes to these Event Streams. Then, Service A may evaluate the facts, aggregate the outcomes, and locally cache them.

Utilizing Event Streams in this manner has various advantages, including:

Systems are capable of closely imitating actual processes.
Increased usage of scale-to-zero functions (serverless computing) as more services are able to stay idle until required.
Enhanced adaptability

The Concept of Event Stream Processing

Event Stream Processing (ESP) is a collection of technologies that facilitate the development of an Event-driven architecture. As previously stated, Event Stream Processing is the process of reacting to Events created by an Event-driven architecture.

One may behave in a variety of ways, including:

Conducting Calculations
Transforming Data
Analyzing Data
Enriching Data

You may design a pipeline of actions to convert Event data, which will be detailed in the following part, which is the heart of Event Stream Processing.

The Basics of Event Stream Processing

Event Stream Processing consists of two separate technologies. The first form of technology is a system that logically stores Events, and the second type is software used to process Events.

The first component is responsible for data storage and saves information based on a timestamp. As an illustration of Streaming Data, recording the outside temperature every minute for a whole day is an excellent example. In this scenario, each Event consists of the temperature measurement and the precise time of the measurement. Stream Processors or Stream Processing Engines constitute the second component.

Most often, developers use Apache Kafka to store and process Events temporarily. It also enables the creation of Event Streams-based pipelines in which processed Events are transferred to further Event Streams for additional processing. Other Kafka use cases include activity tracking, log aggregation, and real-time data processing. Kafka enables software architects design scalable architectures for large systems due to the decoupling of different components in the system.

Event Stream Processing vs. Batch Processing

With the development of technology, businesses deal with a much bigger number of data than they did ten years ago. Therefore, more sophisticated data processing technologies are necessary to keep up with this rate of change. A conventional application is responsible for the collection, storage, and processing of data, as well as the storage of the processed outputs.

Typically, these procedures occur in batches, so your application must wait until it has sufficient data to begin processing. The amount of time your application may have to wait for data is unacceptable for time-sensitive or real-time applications that need quick data processing.

In order to solve this difficulty, Event Streams enter the fray. In Event Stream Processing, every single data point or Event is handled instantaneously, meaning there is no backlog of data points, making it perfect for real-time applications.

In addition, Stream Processing enables the detection of patterns, the examination of different degrees of attention, and the simultaneous examination of data from numerous Streams. Spreading the operations out across time, Event Stream Processing requires much less hardware than Batch Processing.

The benefits of using Event Stream Processing

Event Stream Processing is used when quick action must be taken on Event Streams. As a result, Event Stream Processing will emerge as the solution of choice for managing massive amounts of data. This will have the greatest impact on the prevalent high-speed technologies of today, establishing Event Stream Processing as the solution of choice for managing massive amounts of data. Several advantages of incorporating Event Stream Processing into your workflow are as follows:

Event Stream Pipelines can be developed to fulfill advanced Streaming use cases. For instance, using an Event Stream Pipeline, one may enhance Event data with metadata and modify such objects for storage.
Utilizing Event Stream Processing in your workflow enables you to make choices in real-time.
You can simply expand your infrastructure as the data volume grows.
Event Stream Processing offers continuous Event Monitoring, enabling the creation of alerts to discover trends and abnormalities.
You can examine and handle massive volumes of data in real-time, allowing you to filter, aggregate, or filter the data prior to storage.

Event Streams use cases

As the Internet of Things (IoT) evolves, so does the demand for real-time analysis. As data processing architecture becomes more Event-driven, ESP continues to grow in importance.

Event Streaming is used in a variety of application cases that span several sectors and organizations. Let’s examine a few industries that have profited from incorporating Event Stream Processing into their data processing methodologies.

Besides helping big sectors, it also addresses specific problems we face on a daily basis. Here are some examples of how this can be used.

Use case 1: Pushing Github notifications using Event Streams

Event streams are a great way to stay up-to-date on changes to your codebase in real-time. By configuring an event stream and subscribing to the events you’re interested in, you can receive push notifications whenever there is an activity in your repository. We hope this use case has will help you understand how to use event streams in GitHub push notifications.

Here we are taking an example of creating a chrome extension that makes use of event dreams to provide real-time GitHub push notifications.

The GitHub Notifier extension for Google Chrome allows you to see notifications in real time whenever someone interacts with one of your GitHub repositories. This is a great way to stay on top of your project’s activity and be able to respond quickly to issues or pull requests. The extension is available for free from the Google Chrome store. Simply install it and then sign in with your GitHub account.

Once you’ve done that, you’ll start receiving notifications whenever someone mentions you, comments on one of your repositories, or even when someone stars one of your repositories. You can also choose to receive notifications for specific events such as new releases or new Pull Requests. Stay up-to-date on all the latest activity on your GitHub repositories with GitHub Notifier!

Use case 2: Internet of Things in Industry (IIoT)

In the context of automating industrial processes, businesses may incorporate an IIoT solution by including a number of sensors that communicate data streams in real-time. These sensors may be installed in the hundreds, and their data streams are often pooled by IoT gateways, which can deliver a continuous stream of data further into the technological stack. Enterprises would need to apply an event stream processing approach in order to make use of the data, analyze it to detect trends, and swiftly take action on them. This stream of events would be consumed by the event streaming platform, which would then execute real-time analytics.

For instance, we may be interested in tracking the average temperature over the course of 30 seconds. After that, we want the temperature only to be shown if it surpasses 45 °C. When this condition is satisfied, the warning may be utilized by other programs to alter their processes in real-time to prevent overheating.

There are many technologies that can help automate the processes. Camunda’s Workflow Engine is one of them which implements this process automation and executes processes that are defined in Business Process Model and Notation (BPMN), the global standard for process modeling. BPMN provides an easy-to-use visual modeling language for automating your most complex business processes. If you want to get started with Camunda workflow, the Camunda connectors is a good starting point.

Use case 3: Payment Processing

Rapid payment processing is an excellent use of event stream processing for mitigating user experience concerns and undesirable behaviours. For instance, if a person wishes to make a payment but encounters significant delays, they may refresh the page, causing the transaction to fail and leaving them uncertain as to whether their account has been debited. Similarly, when dealing with machine-driven payments, delay may have a large ripple impact, particularly when hundreds of payments are backed up. This might result in repeated attempts or timeouts.

To support the smooth processing of tens of thousands of concurrent requests, we may leverage event streaming processing to guarantee a consistent user experience throughout.

A payment request event may be sent from a topic to an initial payments processor, which then changes the overall amount of payments being processed at the moment. A subsequent event is then created and forwarded to a different processor, which verifies that the payment can be completed and changes the user’s balance. A final event is then generated, and the user’s balance is updated by another processor.

Use case 4: Cybersecurity

Cybersecurity systems collect millions of events in order to identify new risks and comprehend relationships between occurrences. For the purpose of reducing false positives, cybersecurity technologies use event streaming processing to augment threats and give context-rich data. They do this by following a sequence of processes, including:

Collect events from diverse data sources, such as consumer settings in real-time.
Filter event streams so that only relevant data enters the subjects to eliminate false positives or benign assaults.
Leverage streaming apps in real-time to correlate events across several source interfaces.
Forward priority events to other systems, such as security information and event management (SIEM) systems or security automation, orchestration, and response (SAO&R) systems (SOAR).

Use Case 5: Airline Optimization

We can create real-time apps to enhance the experience of passengers before, during, and after flights, as well as the overall efficiency of the process. We can effectively coordinate and react if we make crucial events, such as customers scanning their boarding passes at the gate, accessible across all the back-end platforms used by airlines and airports.

For example, based on this one sort of event, we can enable three distinct use cases, including:

Accurately predicting take-off times and predicting delays
Reduce the amount of assistance necessary for connecting passengers by giving real-time data
Reduce the impact of a single flight’s influence on the on-time performance of the other flights.

Use case 6: E-Commerce

Event stream processing can be used in an e-commerce application to facilitate “viewing through to purchasing.” To do this, we may build an initial event stream to capture the events made by shoppers, with 3 separate event kinds feeding the stream.

Customer sees item.
A customer adds an item to their shopping cart.
A customer puts an order.

We may assist our use cases by applying discrete processes or algorithms, such as:

An hourly sales calculator that parses the stream for ‘Customer puts order’ events and keeps a running tally of total revenues for each hour.
A product look-to-book tracker that reads “Customer sees item” from the stream and keeps track of the overall number of views for each product. Additionally, it parses ‘Customer puts order’ events from the stream and keeps track of the total number of units sold for each product.
A new ‘Customer abandons’ cart event is created and posted to a new topic when an abandoned cart detector – which reads all three kinds of events and uses the algorithm described previously to identify customers who have abandoned their shopping cart – detects abandoned carts.

Conclusion

In a world that is increasingly driven by events, Event Stream Processing (ESP) has emerged as a vital practice for enterprises. Event streams are becoming an increasingly important data source as more and more companies move to a streaming architecture. The benefits of using event streams include real-time analytics, faster response times, and improved customer experience. They offer many benefits over traditional batch processing.

In addition, there are a number of use cases for event streams that can help you solve specific business problems. If you’re looking for a way to improve your business performance, consider using event stream processing.

Join 4500+ others and sign up for our data engineering newsletter

Originally published at memphis.dev by By Avraham Neeman Co-Founder & CPO @Memphis.dev.

Event Sourcing Outgrows the Database

Avraham Neeman — Wed, 12 Oct 2022 12:24:02 +0000

Introduction
Event sourcing is not a new word, if you are working in tech you must have come across event sourcing. Event sourcing is a powerful tool and is adapted by many large organizations as their database architectural design. It has the capability to scale up and serve the needs of the modern data industry.

In this article, we will understand more about event sourcing and why it’s gaining popularity. We will also discuss the most asked question: Is event sourcing going to outgrow databases?

What is Event Sourcing?
Event sourcing is an architectural pattern to store data as a sequence of events. An event is nothing but a context to the business operation. For instance a customer has requested a refund but you don’t know why? Event sourcing gives the context why refund was made.

Let’s understand some key terms associated with event sourcing:

Events:
Events represent changes in the state of data. These are immutable facts that provide context to the business. Let’s take an example of an e-commerce store. All the data changes will be stored as events like ProductAdded, ProductOrdered, ProductShipped, PaymentReceived etc.

Events are recorded in the past tense and provide the source of truth to the current business state. In addition to the context events also store metadata to provide more information.

Event Source database:
Event source database also known as event source db records all the events in an append-only database. In event source db history of changes are maintained in chronological order.

Event source db can also be referred to as event logs and they can not be changed as events are immutable. There’s another counter argument that says that event source db can be changed by adding another event the state of the event source db or its outcome is changing.

The event source db for the e-commerce store will record each event along with associated metadata. Suppose there are two events in the Product’s event source db and a third product is added to the event source db. This added product is not new but is returned by the customer so the context of the event is returned product and the number of events is updated. The event source db holds all these events in chronological order.

The context and chronological order of events provide useful information for in-depth analysis.

Streams:
Event streaming is a practice that involves capturing and processing data in real time within a distributed system using a streaming database. Popular examples of streaming databases include Memphis.dev, Apache Kafka, Apache Flink, and AWS Kinesis.

A key advantage of event streaming is that it allows organizations to capture events as they occur, providing a more complete history of modifications related to a particular event. Event streams represent the order in which events occur and can be short-lived or long-lived depending on the scenario. Each event within the stream is identified by a unique numeric value that increments as events are updated, allowing for the retrieval of the original state of events through these identifiers.

Event streams are often used in modern architectures to create a robust data processing and analysis platform. By tracking events in real-time, organizations can gain valuable insights into their operations and make data-driven decisions based on up-to-date information.

For example, in an e-commerce store scenario, the payment object has its own unique identifier and its event stream flows as follows: Payment confirmed -> Payment received -> Refund requested -> Refund amount deducted. By monitoring these events in real time, organizations can quickly identify potential issues and make adjustments as needed to improve their operations.

Query View:
In event sourcing query models represents logical transitioning of source write model to read model. They are also referred to as Projections or View Models. In the query view there are two types of concepts: read model and write model. Let’s recall the example of an e-commerce store where the write model events that are added to the query view are Order placed, payment received, order dispatched, product deducted and then we use the query view to generate a summary of all the orders made and the payments received in the read model.

Why do we need event sourcing?
Event sourcing is an excellent choice in a variety of applications. Let’s discuss a few scenarios where event sourcing is an acceptable solution.

1.Event sourcing is useful in auditing systems where logs can be stored in chronological order and has on-demand back up option.

2.Traditional methods collect data in specific locations to be used only when needed. By quickly responding to newly available information, an event-driven approach can be more effective. By subscribing to the stream, an organization can receive updates about new occurrences and respond to them immediately. This makes it simpler to model and create intricate business procedures.

3.It is possible to migrate legacy systems to contemporary distributed architectures slowly, eventually replacing particular functionalities with event-sourced services. While writes are directed to the services, the legacy system’s current read pathways can continue to be used.

4.Dependent services can “catch up” when the originating service resumes operation if one goes down. When each service comes back up, synchronization may be accomplished because events are stored in the stream in a specific order.

5.In an event-sourced system, data travels in one direction through separate models to read or update data. Due to the singular responsibility that each part of the data flow has, it makes it easier to reason about data and troubleshoot problems.

How’s event source database (db) different from traditional databases?
Data is stored in databases using CRUD operation that is create, read, update and delete. Whenever a change happens the record is updated in the database and it preserves the current state of the system. In all relational and non-relational databases records can be deleted and the state of the system will be lost.

In event source db events are immutable; they can’t be deleted or altered. Event source db preserves the history of logs in chronological order. By tracking changes discrepancies between audit data and transactional data are avoided. Just like in CRUD system design, event sourcing stores the events in tables but in chronological order. Since the data is in order with the latest data at the top, filtering event sourcing is easier as compared to traditional databases.

Does event sourcing outgrows databases?
In real-world applications multiple concurrent users are updating records in the data store and often data is not updated in all places. This results in inconsistency across data stores. There is no mechanism to store the metadata of the history of changes that can be used for in-depth analysis.

Event sourcing also provides context to the change happening in the database which helps in answering the business questions. Event sourcing works better with microservices and is reliable to share data between other services.

Here’re a few advantages that make event sourcing a better choice than traditional databases:

1.Events can be saved using an append-only operation and are immutable. Tasks that deal with events can run in the background while the user interface, workflow, or process that started them can continue. This, along with the absence of conflict during transaction processing, can greatly enhance the performance and scalability of an application, particularly at the presentation level or user interface.

2.Events are straightforward objects that explain an action that took place together with any additional information needed to fully describe the action that the event represents. A data store is not directly updated by events. They are merely recorded for handling when necessary. This can make administration and implementation simpler.

3.For a domain expert, events often have meaning. However, object-relational impedance mismatch might make it challenging to comprehend complicated database tables. Tables are made up objects that depict the system’s current condition rather than actual events.

4.Because event sourcing does not require updating data store objects directly, it can assist prevent conflicts caused by concurrent modifications. The domain model must still be built to withstand queries that can cause an inconsistent state, though.

5.Tasks respond to events by performing actions as they are raised by the event source db.

6.The jobs and events are separated, which offers flexibility and extensibility. Tasks are aware of the type of event that occurred and its data, but not the operation that caused it.

7.Additionally, each event can be handled by numerous tasks. This makes it possible to integrate additional services and systems that are limited to monitoring new events raised by the event source db with ease. The event sourcing events, however, frequently have a very low level, thus it would be essential to create particular integration events in their place.

Challenges with Event-sourcing
Despite having a lot of advantages, event sourcing has a lot of challenges as well. Let’s discuss a few challenges associated with Event-sourcing.

1.Event source db is immutable and serves as a permanent repository for data, event data should never be modified. Adding a new event to the event source db is the sole option to update an entity to reverse a modification. It can be challenging to mix current events in the store with the new version if the format (rather than the contents) of the persistent events needs to change, possibly during a migration. It could be necessary to introduce new events that utilize the new format or loop through all the existing events making adjustments to bring them into compliance with the new format. To keep both the old and the new event forms, think about using a version stamp on each iteration of the event schema.

2.Querying data or reading data from event source data stores can be difficult as there’s no standard sql mechanism. To read the data event stream is extracted against the event identifier.

3.The event source db may contain events that were stored by multi-threaded programmes and multiple instances of apps. Both the consistency of events in the event source db and the timing of events that have an impact on a particular entity are crucial (the order that changes occur to an entity affects its current state). Every event should have a timestamp to assist avoid problems. Another typical technique is to assign an incremental identification to each event that results from a request. The event source db may reject an event that matches an existing entity identifier and event identifier if two operations attempt to add events for the same entity at the same time.

4.Event sourcing reduces the likelihood of conflicting data changes, the application must still be able to handle inconsistencies brought on by eventual consistency and the absence of transactions. A customer may need to be informed or a back order may need to be created, for instance, if an event in the data store indicating a reduction in stock inventory occurs while an order is being placed for that item.

5.Once event source systems have been capturing events for some time, another difficulty will arise. It becomes vital to find a technique to handle historical events because, although it’s one thing to record all events that a system has handled, failing to understand that history renders the event log completely useless. The whole event log may need to be reprocessed in order to bring the system’s data universe up to date during system failure recovery events or while migrating derived state stores. Periodic system state snapshots may also be necessary for systems handling large amounts of events, where processing the entire event log again would go beyond any recovery time goals, so that recovery can start from a more recent known good state. Organizations must take into account how events are formed, how that structure can vary over time as the collection of fields changes, and how events with earlier structures might be processed using the current business logic given changes in how the business runs over time. It may be possible to future-proof the event recording by using a defined, extensible event schema, but it may also be necessary to add additional processing rules to the most recent business logic to ensure that earlier event structures can still be understood. Periodic snapshots could also be used to distinguish between significant changes to the event structure, where the expense of maintaining prior events ends up being more than their intrinsic value.

Conclusion
We have studied the concepts of event sourcing and its advantages and disadvantages in detail. As a final verdict, event sourcing is a great architectural design pattern to store data. However, it can only bring value when used the right way. There are few scenarios where traditional database technologies are a better option and should be used. Event sourcing is going to be adapted massively during incoming years but it can’t replace traditional databases. The CRUD based databases are here to stay and they also serve a huge number of real-world applications.

Thanks
to Avraham Neeman Neeman Co-Founder & CPO @Memphis.dev for the amazing writing.

Join 4500+ others and sign up for our data engineering newsletter