a Quick, Sturdy Job Queue for Go + Postgres

Years in the past I wrote about my trouble with a job queue in Postgres, wherein desk bloat attributable to long-running queries slowed down the employees’ capability to lock jobs as they hunted throughout thousands and thousands of useless tuples looking for a reside one.

A job queue in a database can have sharp edges, however I’d understated in that writeup the advantages that got here with it. When used nicely, transactions and background jobs are a match made in heaven and fully sidestep an entire host of distributed techniques issues that in any other case don’t have straightforward remediations.

Take into account:

• In a transaction, a job is emitted to a Redis-based queue and picked up for work, however the transaction that emitted it isn’t but dedicated, so not one of the information it wants is accessible. The job fails and can must be retried later.

• A job is emitted from a transaction which then rolls again. The job fails and also will fail each subsequent retry, pointlessly consuming assets regardless of by no means having the ability to succeed, finally touchdown the useless letter queue.

• In an try and work across the information visibility downside, a job is emitted to Redis after the transaction commits. However there’s a quick second between the commit and job emit the place if the method crashes or there’s a bug, the job is gone, requiring handbook intervention to resolve (if it’s even seen).

• If each queue and retailer are non-transactional, all the above and extra. As a substitute of knowledge not being seen, it might be that it’s in {a partially} prepared state. If a job runs within the interim, all bets are off.

Work in a transaction has different advantages too. Postgres’ NOTIFY respects transactions, so the second a job is able to work a job queue can wake a employee to work it, bringing the imply delay earlier than work occurs right down to the sub-millisecond degree.

Regardless of our operational bother, we by no means did change our database job queue at Heroku. The worth of switching would’ve been excessive, and regardless of blemishes, the advantages nonetheless outweighed the prices. I then spent the following six years staring right into a maelstrom of pure chaos as I labored on a non-transactional information retailer. No normal for information consistency was too low. Code was a morass of conditional statements to guard towards 1,000,000 doable (and possible) edges the place precise state didn’t line up with anticipated state. Job queues “labored” by brute power, bludgeoning jobs by till they might attain some extent that might be tacitly referred to as “profitable”.

I additionally picked up a Go behavior to the purpose the place it’s now been my language of selection for years now. Working with it professionally throughout that point, there’s been quite a lot of moments the place I needed I had a superb framework for transactional background jobs, however didn’t discover any that I notably cherished to make use of.

So just a few months in the past, Blake and I did what one ought to typically by no means do, and began writing a brand new job queue venture constructed particularly round Postgres, Go, and our favourite Go driver, pgx. And eventually, after lengthy discussions and far consternation round API shapes and implementation approaches, it’s prepared for beta use.

I’d prefer to introduce River (GitHub link), a job queue for constructing quick, hermetic purposes.

Designed for generics

One of many comparatively new options in Go (since 1.18) that we actually wished to take full benefit of was the usage of generics. A river employee takes a river.Job[JobArgs] parameter that gives strongly typed entry to the arguments inside:

kind SortWorker struct {
    river.WorkerDefaults[SortArgs]
}

func (w *SortWorker) Work(ctx context.Context, job *river.Job[SortArgs]) error {
    type.Strings(job.Args.Strings)
    fmt.Printf("Sorted strings: %+vn", job.Args.Strings)
    return nil
}

No uncooked JSON blobs. No json.Unmarshal boilerplate in each job. No kind conversions. 100% reflect-free.

Jobs are uncooked Go structs with no embeds, magic, or shenanigans. Solely a Sort implementation that gives a singular, steady string to determine the job because it spherical journeys to and from the database:

kind SortArgs struct {
    // Strings is a slice of strings to type.
    Strings []string `json:"strings"`
}

func (SortArgs) Sort() string { return "type" }

Past the fundamentals, River helps batch insertion, error and panic handlers, periodic jobs, subscription hooks for telemetry, distinctive jobs, and a bunch of different options.

Job queues are by no means actually completed, however we’re fairly happy with the API design and preliminary function set. Try the project’s README and getting started guide.

With performance in mind

One of many causes we like to jot down issues in Go is that it’s quick. We wished River to be a superb citizen of the ecosystem and designed it to make use of quick methods the place we may:

• It takes benefit of pgx’s implementation of Postgres’ binary protocol, avoiding loads marshaling to and parsing from strings.

• It minimizes spherical journeys to the database, performing batch selects and updates to amalgamate work.

• Operations like bulk job insertions make use of COPY FROM for effectivity.

We haven’t even begun to optimize it so I received’t be exhibiting any benchmarks (which are typically deceptive anyway), however on my commodity MacBook Air it really works ~10k trivial jobs a second. It’s not sluggish.

You could be considering: Brandur, you’ve had bother with job queues in databases earlier than. Now you’re selling one. Why?

Just a few causes. The primary is, as described above, transactions are actually only a actually good concept. Possibly the most effective concept in sturdy service design. For the previous few years I’ve been placing my cash the place my mouth is and constructing a service modeled totally round transactions and powerful information constraints. Information inconsistencies are nonetheless doable, however particularly in a relative sense, they functionally don’t exist. The period of time this protects operators from having to manually fiddle in consoles fixing issues can’t be overstated. It’s the distinction between night time and day.

Single dependency stacks

One more reason is that dependency minimization is nice. I’ve written beforehand about how at work we run a single dependency stack. No ElastiCache, no Redis, no bespoke queueing parts, simply Postgres. If there’s an issue with Postgres, we will repair it. No must develop experience in function hardly ever used, black field techniques.

This concept isn’t distinctive. An attention-grabbing improvement in Ruby on Rails 7.1 is the addition of Solid Cache, which 37 Indicators makes use of to cache in the identical database that they use for the remainder of their information (identical database, however totally different cases of it in fact). Ten years in the past this is able to’ve made little sense since you’d desire a sizzling cache that’d serve content material from reminiscence solely, however developments in disks (SSDs) has been so nice that they measured an actual world distinction within the double digits (25-50%) transferring their cache from Redis to MySQL, however with an enormous enhance in cache hits as a result of a disk-based system permits cache area to widen expansively.

Ruby non-parallelism

An enormous a part of our queue downside at Heroku was the design of the precise job system we had been utilizing, and Ruby deployment. As a result of Ruby doesn’t help actual parallelism, it’s generally deployed with a process forking model to maximise efficiency, and this was the case for us. Each employee was its personal Ruby course of working independently.

See Also

JMIRx Bio – The Loch Ness Monster: If It’s Actual, May It Be an Eel?

This produced lots of competition and pointless work. Operating independently, each employee was individually competing to lock each new job. So for each new job to work, each employee contended with each different employee and iterated thousands and thousands of useless job rows each time. That’s lots of inefficiency.

A River cluster might run with many processes, however there’s orders of magnitude extra parallel capability inside every as particular person jobs are run on goroutines. A producer inside every course of consolidates work and locks jobs for all its inner executors, saving lots of grief. Separate Go processes should still take care of one another, however many fewer of them are wanted because of superior intra-process concurrency.

Improvements in Postgres

Throughout my final queue issues we’d’ve been utilizing Postgres 9.4. We’ve got the advantages of 9 new main variations since then, which have introduced lots of optimizations round efficiency and indexes.

• A very powerful for a queue was the addition of SKIP LOCKED in 9.5, which lets transactions discover rows to lock with much less effort by skipping rows which might be already locked. This function is outdated (though no much less helpful) now, however we didn’t have it on the time.

• Postgres 12 introduced in REINDEX CONCURRENTLY, permitting queue indexes to be rebuilt periodically to take away detritus and bloat.

• Postgres 13 added B-tree deduplication, letting indexes with low cardinality (of which a job queue has a number of of) be saved rather more effectively.

• Postgres 14 introduced in an optimization to skip B-tree splits by eradicating expired entries as new ones are added. Very useful for indexes with lots of churn like a job queue’s.

And I’m positive there’s many I’ve forgotten. Each new Postgres launch brings dozens of small enhancements and optimizations, and so they add up.

Additionally thrilling is the potential addition of a transaction timeout setting. Postgres has timeouts for particular person statements and being idle in a transaction, however not for the whole length of a transaction. Like with many OLTP operations, long-lived transactions are hazardous for job queues, and it’ll be an enormous enchancment to have the ability to put an higher sure them.

Anyway, check out River (see additionally the GitHub repo and docs) and we’d recognize it when you helped kick the tires a bit. We prioritized getting the API as polished as we may (we’re actually making an attempt to keep away from a /v2), however are nonetheless doing lots of energetic improvement as we refactor internals, optimize, and customarily nicen issues up.