Lucid

At the start of this year, we could only write end-to-end tests using Selenium in Scala here at Lucid. This was just fine for the developers here who mostly write in Scala. The problem was that learning Scala and Selenium was a high bar of entry for developers to just write an end-to end-test. We have many devs who almost exclusively write in TypeScript. As newcomers to Scala, making an end-to-end test for new features was so difficult that often the tests just wouldn’t get written. When I found out about Puppeteer, it seemed like the right tool to solve this problem. Developers could write tests in TypeScript, a language they are more familiar with. We already used Jasmine for writing unit tests, so the ability to create Puppeteer tests with Jasmine was an obvious win. Devs can also connect Chrome DevTools when running tests, which allows them to use a debugger they are familiar with. All of these features looked ideal for lowering the bar of entry to writing end-to-end tests. Puppeteer also came with a few advantages over Selenium.

Simpler JavaScript execution

A powerful feature of both Selenium and Puppeteer is the ability to run JavaScript in the browser. The uses of this feature are nearly endless, and using this feature in Puppeteer is nearly effortless. Compare these two snippets of code: Scala + Selenium

val evalResult = Json.parse(driver.executeAsyncScript(“””
    var callback = arguments[arguments.length - 1];
    asyncFunction().then(callback);
“””).asInstanceOf[String])

TypeScript + Puppeteer

const evalResult = await page.evaluate(() => asyncFunction());

Right away the TypeScript version is simpler and comes with some additional advantages. First, the TypeScript version automatically handles exceptions. If asyncFunction fails in the Selenium version, you would not get an error; instead it would time out. You could, and probably should, make a wrapper function that simplifies calling JavaScript and correctly handles errors if you go with Selenium. However, since the base implementation is already simpler, Puppeteer is a better choice here. No need to modify the interface. The Puppeteer version also has the advantage of being type checked by TypeScript. You can declare functions and variables used inside of evaluate, and if you have syntax or type errors, TypeScript will catch those errors. In Selenium, you won’t catch the errors until you attempt to run the test. The core of these advantages comes down to having the test driver use the same language the browser does. This makes connecting the two much more seamless. As a note, you can write Selenium tests in TypeScript and implement a similar seamless implementation of evaluate, but that wasn’t an option for our Scala code— that is why I am listing this as a reason I wanted to switch to Puppeteer.

Network interception

This is the most powerful advantage that Puppeteer has over Selenium. Your test code can log, modify, block, or generate responses of requests made by the browser. This may not seem like a very useful feature at first glance, but it helps solve many problems that would be hard to solve otherwise.

Testing handling failed requests

By having Puppeteer selectively fail some requests, you can verify that your product fails gracefully in these situations. You can use this process to verify correct error messaging when an upload fails. You can verify that your page layout doesn’t fall apart if images don’t load.

Mocking out third-party services

I have firsthand experience using network interception in this category. We wanted to write some automated tests for Salesforce blocks. The problem? Our Salesforce plugin relies on making calls to the Salesforce API. If we wrote the test to log in to an existing Salesforce account to run these tests, we would run into a few problems: we would have to rely on our test machines having a reliable connection to Salesforce, changes in the Salesforce GUI would result in tests suddenly failing with no fault on our end, and Salesforce could detect we are logging in as bot and put up a CAPTCHA or require two-step authentication. Yikes. Puppeteer solved this problem for us. We were able to write a mock Salesforce API that runs locally. Any requests to Salesforce are intercepted by Puppeteer, and fake data is returned in its place. Using this method, were were able to write an automated test that covered much more than unit tests without having to actually interact with the Salesforce API.

Testing offline mode

Puppeteer can also simulate going offline. You can write tests to make sure your product handles losing an internet connection correctly. This ability was essential to writing unit tests for the offline mode functionality we recently added to our product. We were able to create unit tests to verify that changes are saved offline and that when connectivity returns, the changes are saved to the server. Without this functionality, we would be stuck either relying solely on unit tests or trying to fake offline mode in a way that wouldn’t best test our offline functionality.

Debugging

Since Puppeteer can be notified of all requests and responses coming from the browser, you can also simply log that information—which is super useful when trying to diagnose problems for failing tests that run on our build server. As part of our build system, when one of our Puppeteer tests fails, we get a screenshot of all tabs open at the time of failure, as well as a full console log dump complete with all requests made by the browser. This information helps diagnose test failures from build reports without having to run them locally. It is also super valuable when a test only fails on our build servers, where you don’t see the test run and only get the test result.

A single browser, a single language

This one may sound like a disadvantage. If I were writing an article on why Selenium is better than Puppeteer, I would certainly include that Selenium gives you more options about what language you want to use, and, more importantly, lets you run your tests on multiple browsers. So why am I giving Puppeteer points for lacking those features? Those features come with a cost. When searching for code examples on Selenium, you will often find examples in another language. You may search online and find a great tutorial on how to use Selenium or a great code snippet showing what you want to accomplish, but they might be in a language you aren’t using and aren’t familiar with. Also, the promise of “write once, run on any browser” that Selenium gives doesn’t always hold up in the real world. For some reason, some tests will pass in one browser and not in another without a clear reason why. Bugs in the different browser drivers could prevent a test from reliably running on all browsers. If you are willing to put in the extra work, you can run your tests on multiple browsers, but only needing to target a single browser greatly simplifies your development load.

Should you choose selenium over Puppeteer?

For my situation, I think choosing Puppeteer over Selenium was the right choice. I don’t mean to say that everybody should migrate away from Selenium. We still write and maintain Selenium tests for those who prefer them. I also am not recommending that everybody should choose Puppeteer over Selenium. I chose Puppeteer because it provides simpler Javascript execution, network interception, and a simpler, more focused library. I hope these points are useful so you can make an informed choice if you’re looking to make end-to-end tests.

Why Puppeteer is Better than Selenium