Tuesday, February 7, 2012

Pros and Cons of Selenium

Over the last few projects I have worked on, there has been a need for testing the web UI. Doing manual testing on the UI is expensive and time consuming (never mind that many bugs are ignored as rerunning the same tests over and over again is boring). Many groups have started using Selenium to do the web UI testing with mixed results.

Selenium (http://seleniumhq.org/) is a great tool and does what it's designed to do very well. Selenium is very good for doing directed tests where there are not a lot of steps. Anything more complicated than that, and Selenium starts to give you troubles. Selenium is also very easy to get set up and going, especially with Selenium IDE and Webdriver. Another great point of Selenium is that you can use almost any popular programming language from Java to Ruby to Python.

However, there are some concerns of using Selenium as well. First, they are slow. Usually, Selenium tests need to start the browser and stop the browser between each test. Another factor which increases the length of the tests is that there are a lot of sleeps or pauses in the the test while waiting for an element to become present. This slowness will cause the tests to be run less and less frequently creating tests which rust and become expensive to maintain.

Second, they are brittle and non-deterministic. Often times the test will break because of a network failure, or a page took too long to load. When the test is run again, it will pass. This causes a lot of problems when you're trying to rely on the tests as there tends to be a lot of extra noise caused by this brittleness. How do you know if a test is actually broken, or the site had a hiccup?

Finally, Selenium tests are easy to create. Wait, What? You just said that was one of the good things of Selenium. Well, yes, ease of creating tests on it's own is a good thing. However, what happens when a developer goes and changes some piece of functionality on your site and suddenly, a whole slew of tests break? Also, the more tests you create, also means the longer the tests take to run (see my previous point) meaning it takes longer to close the feedback loop.

My next post will talk about some of the lessons I learned on how to overcome some of these issues.

1 comment: