The occasion processor for the Grug Brained Developer
Right this moment we’re releasing an open-source challenge referred to as Kawa, an occasion processor so easy and straightforward to make use of that even grug brained developers like us can grok it.
Kawa is a labour of affection borne from Go design rules, and strives to attain the elusive trifecta of simplicity, efficiency, and reliability. We consider it is potential to construct a easy, straightforward to make use of framework with out sacrificing reliability or efficiency. Kawa is the occasion processor powering our occasion pipeline at RunReveal and permits us to scale to hundreds of occasions per second, per core.
One thing as conceptually easy as log processing also needs to be easy in apply. We’re not going to make any wild claims to help each conceivable use case, however we do wish to make it straightforward so that you can take logs from wherever they’re and get them to wherever they are often extra helpful. We additionally need it to be straightforward to implement new sources and locations if one does not exist but to your favourite software program. One aim of ours is to allow builders construct sources and locations in simply a few hours, presuming they’re acquainted with the patterns utilized in Kawa.
On this publish we’ll discover the issues and motivation for this software program in addition to dig into the design and implementation of the framework.
Why is it nonetheless so laborious to do distributed log processing? That is a query we stored asking ourselves when it got here time to construct a brand new logging, metrics and occasion pipeline at RunReveal. That is one thing we have accomplished just a few instances in our careers and it hasn’t gotten any simpler! There are dozens if not a whole bunch of choices accessible, however each one we have encountered comes with fairly main tradeoffs.
On one finish of the spectrum, we’ve instruments aimed extra on the particular use circumstances of amassing telemetry. These usually require excessive quantity, low overhead processing and are not fairly as involved about precisely as soon as processing, so long as the info will get to the place it must be. Regardless of coming from a extra pragmatic angle, they nonetheless miss the mark on ease of use. Vector is a well-liked versatile telemetry daemon, however extending it requires data of Rust and the method to construct plugins or lengthen the software program is poorly documented. OpenTelemetry is a challenge that goals to standardize all telemetry assortment, however by making an attempt to please everybody and make one thing idiomatic for all languages the result’s a little bit of a sizzling mess. The interface used for log collection within the opentelemetry collector makes grug’s head spin. Wanting on the contribution statistics we won’t say we’re terribly shocked.
On the opposite finish of the spectrum, we have got versatile, extra generalized stream processing frameworks impressed by Google’s Millwheel. Frameworks like Dataflow, Apache Beam, Apache Flink, Spark Streaming purpose to be extremely constant and versatile, however the complexity skyrockets because of this. These frameworks will let you construct execution flows that are deep directed acyclic graphs and are appropriate for extremely scalable workflows that may do very superior calculations throughout huge datasets. We like the sort flexibility, however the frameworks are unsuitable for targeted workloads like working a lean log collector daemon.
At RunReveal we wished one thing easy sufficient that might be deployed in a single binary that we may iterate on rapidly, course of arbitrary knowledge varieties, and nonetheless keep a excessive degree of effectivity and efficiency. We wished a framework which might match simply as properly in a tiny daemon working on IoT gadgets as it will working in a big cluster doing analytics workloads. The present instruments at our disposal that we mentioned above are nice for those who’re sticking to the paved street for which they have been constructed, they usually all replicate an amazing lesson in engineering: outline the issue you are seeking to resolve and resolve it properly. We’re taking classes from these frameworks and utilizing them to tell our future growth for our opinionated use case.
The ideas underpinning Kawa will be understood in one-sitting.
Sources are the place the occasions comes from. Locations are the place the occasions are headed. And Handlers course of the occasions.
We’d like to have the ability to deal with errors or failures gracefully, course of occasions with at-least-once supply semantics, and scale to hundreds of occasions per second per core whereas remaining environment friendly.
With these necessities, we got here up with the Supply, Vacation spot and Handler abstractions. They’re maybe finest understood by wanting on the core handler loop in processor.go, discovered on the root of the repository, together with their interface definitions.
// processor.go
func (p *Processor[T1, T2]) deal with(ctx context.Context) error {
for {
msg, ack, err := p.src.Recv(ctx)
if err != nil {
return fmt.Errorf("supply: %w", err)
}
msgs, err := p.handler.Deal with(ctx, msg)
if err != nil {
return fmt.Errorf("deal with: %w", err)
}
err = p.dst.Ship(ctx, ack, msgs...)
if err != nil {
return fmt.Errorf("vacation spot: %w", err)
}
}
}
// varieties.go
kind Supply[T any] interface {
Recv(context.Context) (Message[T], func(), error)
}
kind Vacation spot[T any] interface {
Ship(context.Context, func(), ...Message[T]) error
}
kind Handler[T1, T2 any] interface {
Deal with(context.Context, Message[T1]) ([]Message[T2], error)
}
This loop encapsulates the essence of the framework. Some issues of observe that we’ll dig deeper into within the following sections:
- Every stage is a blocking name, which should be cancellable by the passed-in context.
- Every iteration of the loop handles one occasion at a time from a supply.
- An error returned from any stage at this degree is deadly.
- Handlers aren’t allowed to acknowledge occasion processing success or failure.
Blocking APIs
Sustaining and debugging concurrent code turns into sophisticated fast. That is why we made the aware choice to make each name crossing an API boundary seem synchronous. What might be a large number of channels, asynchronous features or callbacks, now turns into readable in a really crucial fashion.
Word that this does not imply that every name is obstructing the processor or stop any of the phases from doing their very own concurrent work behind the interface. Sources and locations will usually have to take action, to deal with fetching and studying in new occasions whereas ready for different occasions to be processed by the handler, and despatched to their locations.
One Factor at a Time
An important choice in growing this framework was to deal with one occasion at a time. This method simplifies interfaces for sources and locations, assures higher supply, fine-tunes flexibility in parameters, and cleans up error dealing with.
Sounds limiting? Think about dealing with a number of occasions in a batch. Now, contemplate the intricacies of acknowledging the success or failure of processing these messages. That is the place the precept of “one piece movement,” a top quality management technique adopted by lean manufacturing, comes into play.
Within the context of our occasion processing, a similar precept helps deal with points that come up from errors or exceptions processing any arbitrary set of occasions. We deal with these points individually, not in batches. This method ensures better-informed selections, particularly when non permanent community or system errors make ahead progress momentarily unavailable.
Don’t simply examine errors, deal with them gracefully
That is taken straight from a Go proverb. Each error within the core loop is at present thought of a deadly error and returned to the caller. We do that as a result of we consider errors must be dealt with as near the supply of that error as potential, because the nearer you might be to the place it happens, the extra context you are more likely to have to have the ability to efficiently deal with it.
Moreover, each supply, vacation spot and handler goes to have completely different concepts about what errors will be retried or ignored. Subsequently, we go away the accountability of error dealing with to the Sources, Locations and Handlers. There could also be developments on this space because it pertains to some stream processing methods skill to deal with unfavourable acks, however we’re not taking over that work at this time.
Handlers are simply features
Maybe most notably in all of this, a handler is only a operate which accepts an occasion, and returns zero or extra occasions, however doesn’t have the flexibility to acknowledge the occasion was processed.
A number of occasions returned with out an error, then the supply occasion was efficiently processed. An empty slice and no error, the supply occasion was efficiently processed and no occasions should be despatched to locations. Any error returned is deadly, whatever the state of the returned slice.
We do not move the ack operate into the handler as a result of it will create two potential locations for it to be referred to as: from handlers and from locations. If we wish to assure supply to a vacation spot, then any code calling ack contained in the handler would break that contract.
Due to these properties, handlers are by default idempotent. They take the complexity out of writing the handlers which are literally doing work and implementing the enterprise logic. Handlers are the place customized stream-processors will be added to change, alert, multiplex, or suppress logs. A lot of the core logic at RunReveal is carried out in handlers together with the WebAssembly for alerting and transformations.
So if every thing is obstructing, and occasions are processed separately, how on earth is it potential to scale this technique?
First off, we’re not making an attempt to be the quickest occasion processor on the market. That is a fools errand finest left for the excessive frequency merchants and efficiency junkies. We do have a requirement to course of hundreds of occasions per second per core, ideally within the 10k-100k vary.
To realize that aim, we lean into Go’s strengths in concurrency and parallelism.
First, Sources and Locations should be protected for concurrent use, which can be a problem for these first getting began with Go, however these are additionally probably the most reusable elements so ought to solely must be constructed as soon as.
Then, as long as the handlers getting used are idempotent, we will trivially parallelize the processor, which means we will scale to nonetheless many or few go routines as we would like to regulate throughput and useful resource utilization. Fairly neat, huh?
Separating the concurrency of Sources, Handlers and Locations is immediately impressed by Rob Pike’s “Concurrency is just not Parallelism” speak, which is to at the present time our favourite presentation on the subject of concurrent designs.
Message supply ensures for any system is determined by many elements. An important of which is the ensures supplied by the methods any given software program interacts with. A system can solely present as a lot assurance because the methods that it integrates with.
As such, supply ensures might differ relying on the supply or vacation spot being utilized in query. If there is no write ensures for the vacation spot, or in case your supply can not successfully monitor the supply standing of messages, then supply ensures are going to be weak, if current in any respect.
Recognizing that the ensures are depending on the integrations, we have carried out a easy and versatile system for locations to speak again to the sources the supply standing of any given message by way of acknowledgement callbacks.
Each supply carried out in Kawa will get to outline it is personal callback for acknowledging whether or not or not a message was efficiently processed. Vacation spot plugins should name this callback operate if and provided that the message was efficiently dedicated to the vacation spot. This property offers us at-least-once processing.
Though Kawa continues to be in its early phases, it has already confirmed to be instrumental in manufacturing at RunReveal by enabling us to construct sources and locations rapidly. We hope that you will get related worth out of the framework.
You can use it today! If this framework seems compelling to you as a library then now’s the time to get entangled and provides us your suggestions. We wish it to be broadly helpful past simply log processing or telemetry purposes whereas nonetheless holding it easy.
When you’re taken with simply the daemon, you should utilize that at this time too! See the documentation for learn how to configure it to ship nginx logs to RunReveal or one other vacation spot of your selecting like S3.
We have got quite a lot of polish that also must be accomplished (extra docs, devoted docs website, testing, modularization), however it is unlikely that the core rules or interfaces will change a lot earlier than reaching v1.0.