How to smash visual bugs like a pro: introducing Argus Eyes

Auteur
Maarten Groeneweg
Datum

Last year I felt a bit depressed. I realised I was spending hours and hours opening pages and checking if everything was still looking like it was supposed to be. Damn! Why was I spending my valuable time with mind-numbing work?

Continuous (bug) Delivery

Every time we touch code there is the risk of introducing bugs. Often those bugs are not what we’re working on but in functionality in existing parts of the site. This is called regression, also expressed as ‘damn this used to work before’. Often the term regression is used for functional issues but it’s important to acknowledge that regression can appear in every aspect of your site. It can be a performance or security issue but also a UI bug. In the taxonomy of bugs the UI bugs are the visual regression bugs.

When working with continuous delivery we change and deploy our code daily. But without the correct automated quality gates in place we’ll end up with continuous delivery of bugs. Our goal is to deliver functionality very fast to production, so there is very little time available for testing. Due to its time constrains the traditional manual regression test is not viable anymore. This is why we use automated tests as our quality gates. They enable us to release often without jeopardising quality.

The UI matters

To reduce the number of bugs we write Unit Tests and Automated Functional Tests. Both are helpful for finding functional defects but they won’t identify most UI bugs. That’s a big problem. Users don’t use CSS selectors to find a button, so if the button is not in the right place they won’t click on it.

Besides that, brand identity is important for our clients. We’re hired to create a digital user experience which is an increasingly big part of their overall identity. This is why we put a lot of effort in creating a well designed site. And this is why we don’t want to have any visual regression bugs.

Test A.S.A.P.

Traditionally testing is an activity which happens at the end of the development process. And that is a bad thing. Due to the slow feedback cycle it will cost more time to fix any found issues. In scrum it will also cause tiny waterfalls for each user story. After development each story will be handed over to the tester. This will lower the focus of the team because members are working on several stories at once, aside from hurting the overall velocity.

The fact that early testing is so important also means that testing should be a team effort. This allows developers to test their own changes even before these are committed into source control.

Argus Eyes

The hours I spent on checking pages reduces the risk of UI bugs. But to be honest I’m not perfect and I will not spot every tiny little detail. We reached the situation in which we had an ok-ish quality gate for visual regression bugs in place. But it was slow and it was too late in the development process.

It was time for a change. This is how Argus Eyes was born. A tool for automated visual regression testing. Argus Eyes is based on three core principles.

1. Fail fast, fail cheap

Speed is everything. If the tests are running faster, we can run them more often and earlier in the development process. They should run so fast that you can run them on your development machine before pushing your code out. Also the process of reviewing the test results should be super easy. Argus Eyes is reporting all deviations from the baseline, but not every difference is a bug.

2. Component-based

We build our frontend templates as reusable components. So it makes sense to tests on component level as well. It has the big advantage that it allows us to give better feedback on a failed test. If you have an issue with the footer, it will not show that all pages changed, it will show you that you have an issue with the footer.

3. Configuration driven

We love code but not as much as we love automation. That’s why Argus Eyes is configuration driven. We can generate most of the configuration with our development tooling. But it’s not tightly coupled to the exact tools we’re using. You can integrate it with your favourite tooling.

Getting started

1. Install Argus Eyes

First ensure that you’ve installed Node.js, version 4 or higher. Then install Argus Eyes globally with npm:

$ npm install argus-eyes -g

2. Configuration

Now you can create a JSON configuration file in your project, name it argus-eyes.json.

First define the resolutions you would like to test. You specify those in an array called ‘sizes’. Each resolution is defined in the format [width]x[height]:

{ "sizes": ["320x480", "1280x768"] }

Then create an array of pages. Each page should have a name and url on which the page can be opened. Also it needs to contain an array of components which are references to the component names.

"pages": [
    {
        "name": "homepage",
          "url": "http://localhost:3000/",
          "components": ["navigation", "news-items"]
    }
]

And lastly, your component definitions are also stored in an array. Each component needs to have a name which can be referenced in pages.components. It also needs to have a selector, which is any valid CSS3 selector with which the element can be found.

Your config should look somewhat like this now:

{
    "sizes": ["320x480", "1280x768"],
    "pages": [
        {
            "name": "homepage",
            "url": "http://localhost:3000/",
            "components": ["navigation", "news-items"]
        }, {
            "name": "contact",
            "url": "http://localhost:3000/contact.html",
            "components": ["navigation"]
        }
    ],
    "components": [
        {
            "name": "navigation",
            "selector": ".nav"
        }, {
            "name": "news-items",
            "selector": ".news-items"
        }
    ]
}

3. Create a baseline

To generate a baseline for comparison you run the capture command and provide a name for the first set of screenshots. In this example I just call it baseline but you’re free to give it any name.

$ argus-eyes capture baseline

This will create a folder baseline. In that folder you’ll find screenshots of each component on each page in the defined resolutions.

4. Introduce some bugs

Now is the time to work on that new feature you should be working on, fixing bugs or do some kind of refactor work. For the sake of demonstration you should create a visual bug, I settled for changing the font size for the navigation component.

5. Create new screenshots

Run the capture command again to create screenshots of the site after the changes are made. Make sure that you provide a new name for this set of screenshots. For now I just call it ‘new’ but if you’re using Gitflow it’s a good practice to use the name of your branch.

$ argus-eyes capture new

6. Compare ‘new’ with the baseline

Now you have two sets of screenshots stored on your disk. You can simply run the compare command to find the differences. You’ll need to provide the names of the sets of screenshots you would like to compare. In this case ‘baseline’ and ‘new’.

$ argus-eyes compare baseline new

When differences are found those will be logged as an error. In that case the process will exit with code 1. Visual diff images are generated with differences highlighted in red.

Next steps

So now you can find your visual regression bugs. But what’s next? The answer is easy: more automation !! Because Argus Eyes is configuration driven you can integrate it in your frontend build tooling. This way you won’t have to maintain the config file manually.

Also ensure that Argus Eyes runs on your build server. Just because every quality measure which is not forced will be forgotten once.

More info

If you want more info about Argus Eyes have a look at http://arguseyes.io or on our GitHub. When you have any questions just create an issue or contact me on twitter via @LazyTesting.

Tags

Testing Automation