How to test your website UI using rules-based visual E2E tests

Automated testing has been one of those advancements in software development that has so much potential but still has yet to really make the impact that we all know it is capable of. We've spent the past few months trying to push the boundaries of what's been traditionally done and have come up with some interesting learnings we'd like to share.

Today's UI tests just aren't good enough

Being a design focused end-to-end agency that builds a lot of website and mobile apps, we found that there's a gap in the user interface side of automated end-to-end testing. Things like font styles, sizing, responsive layout, and many other visual aspects just aren't covered by traditional automated tests.

Visual snapshot diffs are often used to close this gap, however they really don't do a great job of validating the concrete expectations of designs that are nowadays clearly defined by tools like Zeplin and Invision.

Often you'll get false positives where the diffs look good, but the baseline has never matched the original designs. Other times you have a subtle change in layout break all of the diffs and have to re-baseline by manually inspecting the designs.

Webdriver.io + Programmatic UI Testing

Webdriver.io is one of my favorite Javascript E2E testing frameworks because of it's simplicity and extensibility - it works with many different test runners, reporters and other extensions making it a tool of choice for those looking to experiment and innovate.

We've created a set of Webdriver.io extensions and packaged it up as @nascentdigital/wdio-extend. The package provides a set of easy to use Webdriver.io and Chai extensions that help you build reliable visual tests.

For example, take the following section from our website:

What if we wanted to validate that the main title had the correct styling on desktop? Using our extensions it's pretty simple.

it("should have correct copy", () => {

    // open page
    browser.open("https://nascentdigital.com/work");
  
    // resize to "md" (i.e. 1199px)
    browser.setBreakpoint("md");

    // wait for element to render
    const title = $(`*[data-component="Hero"] *[data-name="title"]`);
    title.waitForDisplayed();

    // validate text
    expect(title).to.have.text("Our Work");

    // validate style
    expect(title).to.have.style({
        "font-family": "Faktum",
        "font-size": "90px",
        "font-weight": 600,
        "font-style": "normal",
        "line-height": "100px",
        "letter-spacing": "normal",
        "color": "#000000"
    });
});

That looked pretty simple, but what about testing across multiple breakpoints?

// validate the title style for the phone, tablet, and desktop
browser.forBreakpoint(["xs", "sm", "md"], breakpoint => {
  
    expect(title).to.have.style(styles[breakpoint]);
});

The extensions do all of the heavy lifting of resizing the browser to match your configured breakpoints, while still letting you specify your testing logic using the existing methods. It even handles some of the trickier nuances such as dealing with mobile devices tha can't have the window resized - in those cases only the mobile breakpoint will be called.

This paradigm can become pretty powerful as you combine more of the extension methods. Another scenario we run into is having to validate the responsive layout of elements:

it("should layout title + subtitle correctly", () => {

    // open page
    browser.open("https://nascentdigital.com/work");
  
    // resize to "xs" (i.e. 599px)
    browser.setBreakpoint("xs");
  
    // wait for elements to render
    const navLogo = getNavLogo();
    const title = getHeroTitle();
    const subtitle = getHeroSubtitle();

    // validate the title is under the logo and is left-aligned
    expect(title).to.be.positioned(["below", "leftAligned"], navLogo);
  
    // validate the title is also above and left-aligned with the caption
    expect(title).to.be.positioned(["above", "leftAligned"], subtitle);
});

This is just a teaser, but we're actively developing the extensions and more tests to help improve the quality of user experiences we create.