Selenium IDE: The Good, the Bad, and the Ugly

Here at Lucid Software, we are always trying to release high quality code to users as quickly as possible. For every release, we spend three to five days doing a full regression to make sure the code is user ready. Three to five days is not as fast as possible. In an effort to release more quickly, we are always on the lookout for new software to help us improve our testing process. We encourage all members of the team to look around for new and interesting tools that will make testing at Lucid faster and more complete. This summer, as part of this effort to find new tools, one of our interns began experimenting with Selenium IDE. Selenium IDE is an add-on for Firefox that allows anyone to create macros through commands and then translate them into reproducible steps with the Selenium Firefox Driver. Commands include clicking on a element, typing into a field, and dragging an element from one location to another. The add-on can be downloaded from the Mozilla app store with tutorials on how to get started.

To understand how we were interested in incorporating Selenium IDE into our development cycle, you first need to understand where our testing infrastructure stands as a company. At Lucid Software we have 13 full-time testers on two teams: a team of nine manual testers and a team of four testers dedicated to maintaining and improving our UI Automation framework. Lower level tests, such as integration and unit tests, are the responsibility of each developer. Additionally, all developers write some UI tests for new features they implement. As a relatively young company only five years old, we currently only have about half of our regression tests automated. Automating more of the regression tests is one of our top priorities company wide. As part of this effort, we began experimenting with Selenium IDE. From our investigation and experimentation, we found out a great deal about Selenium IDE; hence, we present The Good, The Bad, and The Ugly.

The Good

1. Fast setup. As a browser add-on, Selenium IDE can be set up very quickly. We have had four or five different developers and QA members try the tool, and all of them were able to get going in under 10 minutes.
2. Simple interface. Selenium IDE doesn’t require a CS degree or knowledge of programming languages to set up most tests. In fact, it’s mainly just point and click. There is a whole list of possible asserts to look for when it comes to debugging, and it highlights red where the test fails. Overall, the tool is built for any web user with even the simplest understanding of testing.
3. Quick turnaround time. Compared to writing Selenium tests with our framework, Selenium IDE was incredibly fast. We timed writing a simple test to make sure our Lucidchart editor loads. Even with most of the methods written and available in our homegrown Selenium framework, it took about ten minutes from the time the test was started until we were able to run it. With Selenium IDE, it took two.

The Bad

1. Only runs in Firefox. About 25% of traffic to our sites is on Firefox, while almost 70% comes through Chrome. While it's good for us to have tests in both Firefox and Chrome, we want our tests to mimic the workflows of the majority of our users.
2. Working with Canvas elements is hard. I mean really hard. In both our products, the canvas element is about 80% of the editor’s real estate. With our homegrown Selenium solution, we make use of the Actions object to move things around on the canvas, click at locations, and drag and drop blocks. Without access to the Actions object in Selenium IDE, this became just a matter of trying to click at the correct offset from the canvas object to make selections and move them.
3. Lack of conformity to a standard. Selenium IDE is an open source project, and it shows. The list of commands and syntaxes varies widely. Some just use the command field and a target. Others require the target to be listed in the value field. Still others require you to add specific fields or take some away. While the versatility is good, the lack of standard means that any tester needs to spend a lot of time learning the different commands and what they expect in order to do anything complicated. The simplicity and ease of the product is lost through poor adherence to standards. This problem will likely become worse as more commands are added.

The Ugly

1. The waiting game. This is the number one complaint around the web with Selenium in any fashion. With Selenium IDE, it is even worse. Waiting on a site to execute Javascript requires the browser to wait for a particular action. In Selenium IDE, it tries to execute as quickly as possible. For example, as soon as a button gets added to the DOM, it tries to click the button, even if the browser is still calculating position and the button won’t work yet. You need to wait until the button is fully set up, but Selenium IDE moves on as soon as the button is added to the DOM.  In order to work around this problem, the options in Selenium IDE are using a loop and building out some retry code or simply putting in a wait command to tell it to wait a little longer.
2. Access to the API. One of the advantages of our home grown Selenium framework is direct access to backend APIs. We use these to create users, add users to teams, create documents, and clean up at the end of the test. With Selenium IDE, we don’t have this access, so every single test creates a new user in our test environment that never gets cleaned up.
3. Reliability. To be completely frank, simple tests flake out around 5% of the time. Once we ramped up a little and tried to do some complex testing, we found we were getting false positives about 20-25% of the time. On these tests, we spent hours trying to fine-tune waiting periods, change the speed of the test, and click on more clear components. Even with hours of effort, the best we could get was about a 10% fail rate.

The Verdict

Selenium IDE is a user-friendly, easy-to-use tool. You can set it up very quickly, and anyone can use it. At Lucid Software, we have decided to add Selenium IDE to our toolset for our Quality Assurance Specialists, but only in a limited capacity at the discretion of each team member. It was not reliable enough or full-featured enough for us to warrant porting out old UI tests forward or even making it a primary tool in automation. The cost to maintain was just too high.

Our Quality Assurance Specialists have found use for it in repeating common tasks that they do every day. They use it to set up team situations, set up a new account with specific A/B tests, or even just clean up old testing documents by going through all the documents and deleting them. These are all simple tasks that can fail without hurting anything, as the QA Specialist can see what failed and simply finish the process manually. Selenium IDE accomplishes the task faster and allows the specialist to focus on other things while the process is being run.