Executing Cron Scripts Reliably At Scale

Cron scripts are answerable for vital Slack performance. They guarantee reminders execute on time, electronic mail notifications are despatched, and databases are cleaned up, amongst different issues. Through the years, each the variety of cron scripts and the quantity of knowledge these scripts course of have elevated. Whereas typically these cron scripts executed as anticipated, over time the reliability of their execution has often faltered, and sustaining and scaling their execution atmosphere grew to become more and more burdensome. These points lead us to design and construct a greater option to execute cron scripts reliably at scale.

Working cron scripts at Slack began in the way in which you may count on. There was one node with a replica of all of the scripts to run and one crontab file with the schedules for all of the scripts. The node was answerable for executing the scripts regionally on their specified schedule. Over time, the variety of scripts grew, and the quantity of knowledge every script processed additionally grew. For some time, we may hold transferring to larger nodes with extra CPU and extra RAM; that stored issues operating more often than not. However the setup nonetheless wasn’t that dependable — with one field operating, any points with provisioning, rotation, or configuration would carry the service to a halt, taking some key Slack performance with it. After repeatedly including an increasing number of patches to the system, we determined it was time to construct one thing new: a dependable and scalable cron execution service. This text will element some key parts and issues of this new system.

System Elements

When designing this new, extra dependable service, we determined to leverage many present providers to lower the quantity we needed to construct — and thus the quantity now we have to take care of going ahead. The brand new service consists of three most important parts:

1. A brand new Golang service known as the “Scheduled Job Conductor”, run on Bedrock, Slack’s wrapper round Kubernetes
2. Slack’s Job Queue, an asynchronous compute platform that executes a excessive quantity of labor shortly and effectively
3. A Vitess desk for job deduplication and monitoring, to create visibility round job runs and failures

Scheduled Job Conductor

The Golang service mimicked cron performance by leveraging a Golang cron library. The library we selected allowed us to maintain the identical cron string format that we used on the unique cron field, which made migration less complicated and fewer error inclined. We used Bedrock, Slack’s wrapper round Kubernetes, to permit us to scale up a number of pods simply. We don’t use all of the pods to course of jobs — as a substitute we use Kubernetes Chief Election to designate one pod to the scheduling and have the opposite pods in standby mode so one among them can shortly take over if wanted. To make this transition between pods seamless, we applied logic to stop the node from happening on the prime of a minute when doable since — given the character of cron — that’s when it’s probably that scripts will should be scheduled to run. It’d first seem that having extra nodes processing work as a substitute of only one would higher resolve our issues, since we gained’t have a single level of failure and we wouldn’t have one pod doing the reminiscence and CPU intensive work. Nevertheless, we determined that synchronizing the nodes can be extra of a headache than a assist. We felt this manner for 2 causes. First, the pods can change leaders in a short time, making downtime unlikely in apply. And second, we may offload virtually the entire reminiscence and CPU intensive work of really operating the scripts to Slack’s Job Queue and as a substitute use the pod only for the scheduling part. Thus, now we have one pod scheduling and several other different pods ready within the wings.

Job Queue

That brings us to Slack’s Job Queue. The Job Queue is an asynchronous compute platform that runs about 9 billion “jobs” (or items of labor) per day. It consists of a bunch of theoretical “queues” that jobs stream via. In easy phrases, these “queues’” are literally a logical option to transfer jobs via Kafka (for sturdy storage ought to the system encounter a failure or get backed up) into Redis (for brief time period storage that enables extra metadata of who’s executing the job to be saved alongside the job) after which lastly to a “job employee” — a node able to execute the code — which truly runs the job. See this article for extra element. In our case, a job was a single script. Though it’s an asynchronous compute platform, it may well execute work in a short time if work is remoted by itself “queue”, which is how we had been in a position to make the most of this technique. Leveraging this platform allowed us to dump our compute and reminiscence issues onto an present system that would already deal with the load (and far, way more). Moreover, since this technique already exists and is vital to how Slack works, we decreased our construct time initially and our upkeep effort going ahead, which is a wonderful win!

Vitess Database Desk

Lastly, to spherical our service out, we employed a Vitess desk to deal with deduplication and report job monitoring to inner customers (different Slack engineers). Our earlier cron system used flocks, a Linux utility to handle locking in scripts, to make sure that just one copy of a script is operating at a time. This only-one requirement is happy by most scripts often. Nevertheless, there are just a few scripts that take longer than their recurrence, so two copies may begin operating on the similar time. In our new system, we report every job execution as a brand new row in a desk and replace the job’s state because it strikes via the system (enqueued, in progress, completed). Thus, after we wish to kick off a brand new run of a job, we are able to test that there isn’t one operating already by querying the desk for energetic jobs. We use an index on script names to make this querying quick.

Moreover, since we’re recording the job state within the desk, the desk additionally serves because the backing for a easy internet web page with cron script execution data, in order that customers can simply search for the state of their script runs and any errors they encountered. This web page is very helpful as a result of some scripts can take as much as an hour to run, so customers need to have the ability to confirm that the script remains to be operating and that the work they’re anticipating to occur hasn’t failed.

Conclusion

General, our new service for executing cron scripts has made the method extra dependable, scalable, and consumer pleasant. Whereas having a crontab on a single cron field had gotten us fairly far, it began inflicting us plenty of ache and wasn’t maintaining with Slack’s scale. This new system will give Slack the room wanted to develop, each now and much off into the long run.

Need to assist us work on programs like this? We’re hiring! Apply now