Harry's Engineering

Harry's
Engineering
Blog

09.07.14

Cronut - Sweeten up your day

By: Sunny Ng

Like it or not, we always end up having to schedule tasks that mysteriously run throughout the day. Whether they’re clean-up jobs that run every hour or heavy-duty critical jobs that we want to perform overnight when there’s much lighter web traffic, it’s always important that these jobs do in fact, run. In fact, many of the jobs we run at Harry’s are critical to our business. They allow us to ensure our customers’ orders are shipped in a timely manner. Shipments arriving on time mean happy customers. And happy customers mean better business!

I don't always schedule cron jobs

###Cron jobs aren’t always reliable

The problem is, it’s difficult to guarantee if the cron jobs will actually run. Even if they do run, it’s not always easy to tell if a job actually completed. Some jobs may be optional and may only perform an operation when a threshold is reached, then how do you know if that job should even do anything? And so, how can you tell if a job has run or not? Sure, you can have your jobs send out emails on each of its completion (eww…), but that’s a lot of noise and it’s easy to miss when something didn’t happen. Not to mention this gets really out of hand when you have many jobs that occur at different intervals around the clock.

Not sure if slow day

###Introducing Cronut: a scheduling job dead man’s switch Cronut is an open-source standalone Rails app that allows you to set a schedule of when expected jobs are to happen using intervals or cron expressions, and notify you if an expected job hasn’t run. Cronut expects each of your jobs to send a POST request (using something like curl) to a unique URL and if that has not happened by a certain time, it will notify you via either email or PagerDuty, a service we find really useful. It works out of the box with Heroku along with some simple security features, but it is flexible enough that it can probably be deployed in other ways.

###How it works This app has an admin interface for users to add in jobs. We allow two types of jobs to be scheduled: cron jobs and interval jobs. Cron jobs are pretty straight-forward, they’re jobs scheduled with a cron expression, which a lot of our jobs are based on, because we’re old school like that. These jobs are run based on specific times throughout the day. Interval jobs are jobs that occur once per specific period of time. The next expected time for jobs like these is based on the last occurence of the job. For instance, maybe you would like to make sure that there’s at least one transaction every 5 minutes at any given time.

Say we have a job that runs daily at 4am, and that it’s important for it to finish within the hour before we start to get more traffic. If we have a system that just tells us the job has run once a day regardless of time, then we can’t rely on it to alert us when the job runs at the wrong time. So we also implemented the concept of buffer time as an optional attribute. When buffer time is set, then we expect the job scheduled to finish within a certain amount of time before or after the expected scheduled time to allow some variance.

The app runs a background job via either Clockwork or Heroku Scheduler that checks every minute or 10 respectively, and sends out notifications if any of the jobs has expired.

We support two types of notifications: email and PagerDuty—two easy ways to escalate if a problem arises.

After you get it all set up, you can just sit back and enjoy the rest of your day.

###Download

Get the code from GitHub

###Feedback

Welcomed via email (sunny@h*****.com) or twitter: @_blahblahblah or pull requests on GitHub.

Check it out! Let us know what you think, and hopefully you can help improve it and make it better!

###License

This code is released under the MIT License.