Selenium: What to do

In my last post, I went through some of the pros and cons of Selenium.

Now I'd like to go through some of the lessons I learned to keep Selenium tests maintainable and valuable for a team.

Run continuously. How often does a UX developer change the user experience? Isn't that their job? When things change in the UI, Selenium will break, that is the downside using the UI to drive our tests.  With CI servers becoming more and more popular, groups have gotten into the habit of running their tests after every commit, otherwise they rust. Why don't they do it with their Selenium tests? I have found after all of the other tests pass, kicking off your Selenium tests is the logical next step. We avoid test rust, a huge Herculean effort at the end of an iteration to make the tests all pass again when there is always so much time, and reduce the feedback cycle as much as possible.

Don't over reach. Do not make too many tests. These tests take a while to run. The more tests you create, the longer they take and the longer the feedback cycle. I've found that only testing your regression tests, or acceptance tests, is more than enough. By only testing the basics, you allow your QA members to do some exploratory testing to ensure that the application is working.

Practice the DRY principal religiously. Let me give you an example. You have an e-commerce site and the developers change something in step one of your checkout flow, but you have ten test which go through this flow. Do you want to find every place that you're calling the newly changed code? Trust me, hunting these references down becomes a wicked time sink. Instead, if you have a step which is repeated more than once, create a method for that step. You'll be happy that you do. which leads me to

Constantly refactor. These are not just some test scripts which people don't need to pay attention to. You will end up creating a DSL for your website with Selenium as the driver. If a method's name doesn't make sense, change it.. If there is code repetition, combine it. If there is any code which is not clear, refactor. You may not have your most senior people maintaining this code, but to reduce the costs, make sure they have opportunities to review this code.

Find ways to make the tests faster. Probably part of the last point, but it bears repeating. These tests are slow. If you can combine tests, do it. If you don't need to run ten times through a step, do it. If you can create data needed for the tests directly with a database connection, do it. Or if you can clan up after the tests without going through the UI, do it. The faster the tests, the shorter that feedback loop, and the smaller the costs are to your group to create, run and maintain these tests.

Use Web driver. With web driver, you don't need to start up an extra server in order to run your tests. Also, Selenium 2 uses web driver and it really is a big step forward.

If you have any question or comments, please don't hesitate to contact me.

Pros and Cons of Selenium

Over the last few projects I have worked on, there has been a need for testing the web UI. Doing manual testing on the UI is expensive and time consuming (never mind that many bugs are ignored as rerunning the same tests over and over again is boring). Many groups have started using Selenium to do the web UI testing with mixed results.

Selenium (http://seleniumhq.org/) is a great tool and does what it's designed to do very well. Selenium is very good for doing directed tests where there are not a lot of steps. Anything more complicated than that, and Selenium starts to give you troubles. Selenium is also very easy to get set up and going, especially with Selenium IDE and Webdriver. Another great point of Selenium is that you can use almost any popular programming language from Java to Ruby to Python.

However, there are some concerns of using Selenium as well. First, they are slow. Usually, Selenium tests need to start the browser and stop the browser between each test. Another factor which increases the length of the tests is that there are a lot of sleeps or pauses in the the test while waiting for an element to become present. This slowness will cause the tests to be run less and less frequently creating tests which rust and become expensive to maintain.

Second, they are brittle and non-deterministic. Often times the test will break because of a network failure, or a page took too long to load. When the test is run again, it will pass. This causes a lot of problems when you're trying to rely on the tests as there tends to be a lot of extra noise caused by this brittleness. How do you know if a test is actually broken, or the site had a hiccup?

Finally, Selenium tests are easy to create. Wait, What? You just said that was one of the good things of Selenium. Well, yes, ease of creating tests on it's own is a good thing. However, what happens when a developer goes and changes some piece of functionality on your site and suddenly, a whole slew of tests break? Also, the more tests you create, also means the longer the tests take to run (see my previous point) meaning it takes longer to close the feedback loop.

My next post will talk about some of the lessons I learned on how to overcome some of these issues.