The Good and the Bad of Cypress

After many years working in Selenium, in 2019 the team evaluated Cypress as an alternative for testing our more modern front end code. For those projects we were using webdriver.io to run Selenium which worked well enough, but we were facing issues with flakiness that we attributed to network issues with Browserstack. What we found was a mixed bag; the development experience was far better, but CI was a much tougher road.

Development

Getting started with cypress is straight forward. Install it as a dev dependency (npm install cypress --save-dev) in your project and start it up (npx cypress open). This will set up files which you will mostly delete. Assuming you have a test application you are ready to write tests; if you want to play with a toy application an example can be found here with both cypress and Selenium versions of the tests.

When you get right down to it, despite differing test philosphies your tests can look very similar regardless of which framework you go with. Here is a test from that example that clicks a button to open a modal and closes it written for cypress:

beforeEach(() => {
  cy.visit('/')
});

it('can open a modal and close it', () => {
  cy.getByDataTest('modal')
    .should('not.exist');

  cy.getByDataTest('open-button').click();

  cy.getByDataTest('modal')
    .should('exist');

  cy.getByDataTest('close-button').click();

  cy.getByDataTest('modal')
    .should('not.exist');
});

That same test can easily be written with webdriver. It's a slightly different syntax, but it reads almost identically:

beforeEach(() => {
  browser.url('/');
});

it('can open a modal and close it', () => {
  expect(getByDataTest('modal')).not.toBeExisting();

  getByDataTest('open-button').click();
  
  expect(getByDataTest('modal')).toBeExisting();

  getByDataTest('close-button').click();

  expect(getByDataTest('modal')).not.toBeExisting();
});

So while there are some advanced features in Cypress, at its core porting tests between the two is relatively straight forward.

What do you gain?

It's when you are actually developing tests that you notice that there are two real advantages to the cypress test runner. The first is that as you edit your test files the test runner will seamlessly re-run your tests with low latency. This means the cycle-time of writing tests is minimal:

This is made possibly by the fact that Cypress combines the test runner and the browser whereas with Selenium they are seperate constructs. So even if you are watching the test files when using Selenium it will launch a third window which takes time and space.

The second is the ability to walk backwards in time through your test run to see what went wrong:

Selenium has no similar tooling that I am aware of; you can configure HAR files and videos but they pale in comparsion to being able to click through the history of each step and see exactly what happened.

What do you lose?

By moving to Cypress you do lose some explicit features of Selenium. While Cypress is expanding its browser coverage as of early 2020 there is only beta support for Firefox and there is no support for older versions of Internet Explorer. Depending on your user base that may or may not be a deal breaker. Personally, I have found running tests in multiple browsers to be less useful that I had wished. Modern versions of browsers are much more consistent than the days of IE6 I have yet to find a bug that was caught by running tests against Chrome and IE that wasn't caught by running tests against just Chrome. In practice I have also found IE tests to be much more flakey than tests against other browsers. I did not rigorously examine why, so this could be a function of the tests I was running or a property of the web driver for the version of IE I was testing against. Regardless, given their low utility I ended up stopping automated testing against IE completely.

The other large difference between Selenium and Cypress is that while Selenium run remote commands against an actual browser using drivers often maintained by the browser themselves, Cypress is much more integrated. In their own words:

Most testing tools (like Selenium) operate by running outside of the browser and executing remote commands across the network. Cypress is the exact opposite. Cypress is executed in the same run loop as your application.

This key difference means that Cypress does not interact with the browser in the same way that a user does; it runs tests against a close approximation of a browser inside their test runner. So far this has not been an issue personally, but this in theory could mean tests are passing but functionality is broken for real users.

Continuous Integration

The mechanics of running Cypress in CI are relatively straightforward. You can either run it as a headless electron based browser directly on the machine with the command npx cypress run or you can run it in a docker container:

version: '3'
services:
  web:
    /* your app configuration */

  cypress:cypress:
    image: cypress/included:3.4.0
    environment:
        - CYPRESS_baseUrl=${BASE_URL}
    container_name: cypress
    volumes:
        - ./cypress:/var/www/cypress

When running tests in CI using docker you should keep a couple things in mind:

• You should skip installing cypress on the local machine via npm install by setting the environment variable CYPRESS_INSTALL_BINARY=0
• You should only create videos for each failed test by setting "videoUploadOnPasses": false in the cypress.json file

If you are only running a handful of tests than either of those solutions should be enough. However, once you get to any reasonable number of tests on a non-trivial application you will either have incredibly long testing times or you will need to invest in parallizing your tests and generally scaling the system.

Conclusion

After evaluating Cypress we ultimately decided to use it going forward and have executed a migration plan. Overall, I think there is no disagreement that the development experience is much improved over writing tests in selenium. However, we continue to need to work on scaling it in CI which has proven to much more effort than we initially expected.