Harry's Engineering



Kontrast: Fast automated visual QA for your website

By: Ilya Rubnich

TL;DR: We're open-sourcing a visual diff tool that alerts us to unexpected changes in our web frontend before we deploy our code.

Here at Harry’s we’re crazy about testing. We write unit tests, feature tests, integration tests, model tests, controller tests, JavaScript tests, oh my. We even wrote a Jasmine helper that let us write more JavaScript tests.

Last fall we introduced a new kind of test into our growing suite - the frontend test. We wanted to visually compare our test website with our production website before deploying to make sure everything on our frontend was in the right place.

We were able to run our frontend tests using Wraith - a great plugin written by the BBC News developers. Wraith did a great job, but at the expense of build times. It was also difficult to break apart and parallelize these tests so they could work on our CI server, and doing so did not significantly reduce build times. We needed a better solution.

Enter Kontrast

Inspired by Wraith, we created our own tool that is fast, easy to configure, easy to use, and supports parallelization out of the box.

In its most basic form, Kontrast lets you specify a test URL, a production URL, and a set of paths. With one command, Kontrast loads up the test and production versions of each path in a browser, takes screenshots of them, and visually compares them. After all this is done, Kontrast generates a gallery with its results so you can see exactly what’s different on each page.


Set up your test suite:

# config.rb
Kontrast.configure do |config|
    # Set your test and production URLs
    config.test_domain = "http://localhost:4000"
    config.production_domain = "http://engineering.harrys.com"

    # Tell Kontrast to load these pages in a 1280px-wide browser window
    config.pages(1280) do |page|
        page.home "/"
        page.cronut "/2014/07/09/cronut.html"

Run Kontrast:

$ kontrast local_run --config ./config.rb

Here’s a sample of the resulting gallery. The blue streaks you see are the differences between the pages:

Kontrast Gallery

P.S: If you’re running Rails, Kontrast will generate its config into your config/initializers folder and you won’t ever need to specify the --config flag as long as you use bundle exec.


Kontrast is very customizable - you can hook into many stages of the test suite. For example, here’s how you can have Kontrast send a message to a HipChat room if there are any diffs in your build:

# Make sure you load the HipChat gem
require "hipchat"

config.after_gallery do |diffs, gallery_path|
    hipchat_room = "Kontrast Results"
    hipchat_user = "KontrastBot"

    if !diffs.empty?
        msg = "Kontrast Diffs: #{diffs.keys.join(', ')}. Don't push to production without reviewing these. You can find the gallery at #{gallery_path}."
        client = HipChat::Client.new(ENV["HIPCHAT_TOKEN"])
        client[hipchat_room].send(hipchat_user, msg, :color => "red")

Check out the GitHub page for more examples. If you find a cool way to extend Kontrast, let us know and we’ll include it in our documentation!

But Wait, There’s More!

Billy Mays

If you use a CI tool such as CircleCI that supports multiple containers, then we have some great news for you. If you call within the next 30 minutes set a few more config options, you can have Kontrast run in parallel across all your containers, dramatically reducing its running time.

How we Parallelize

To set up parallelization, you need to have an Amazon S3 bucket. Why’s that? Well, Kontrast needs to combine test results from all containers to produce a gallery. During each test, Kontrast creates and uploads its results to S3. After a container finishes running all its tests, a small manifest file containing a list of test results in JSON format is created and uploaded to S3. Then after all containers finish running the test suite, one container downloads all the manifest files and uses them to create a single gallery with the combined test results.

The best part is that everything runs quickly. The Harry’s frontend test suite consists of 45 pages and runs in less than 2 minutes. That’s 90 pages (test & production) that have to be loaded in a browser, screenshotted, compared, and uploaded. To make this possible, Kontrast takes advantage of the Workers gem to perform testing and uploading in background threads. It also does as little uploading and downloading as possible - only the minimum required to coordinate between containers.

You can check out how to set up parallelization on Kontrast’s GitHub page.

Download & Documentation

Check out Kontrast’s GitHub page for details on installing Kontrast and for the full documentation.