Thought I might share with you a few of my experiences with using Selenium. It is a little overdue as I have only recently been able to bring myself to talk about it.
With JIRA 3.10, we introduced the AJAX-based ‘User-picker’ and ‘Issue-picker’. A very handy feature, if I don’t say so myself.
auto-complete.png


Having recently completed some AJAX eye candy for ShipIt V, I found myself tasked to do this, along with one of our new recruits – Brad (a Javascript guru*). Our architecture is based on dwr for the communication and yui for all the Javascript goodness. We went a bit nuts, making the yui autocomplete do things it was never meant to do – subsections, titles, non-selectable items… We also implemented our own dwr datasource for yui that enabled yui and dwr to integrate seemlessly. More on that in my next post.
As a good friend suggested, Selenium seemed like the perfect tool for the job of testing this dynamic feature. We scheduled 16 hours for writing the tests – a couple of hours to write the tests, the rest for figuring out how to use it. This turned out to be a bit of an under-estimation.
We hit a few road blocks:
Key Events
I guess the the first sign of things to come was when we were having trouble testing the autocomplete pop-up i.e. we couldn’t get it to pop. It turns out selenium.type(…) doesn’t actually simulate a user typing into an input box. Go figure. So we typed in the text and triggered everykey stroke seperately – selenium.keyDown(…); selenium.keyPress(…); selenium.keyUp(…);. It worked!! in Firefox… In other browsers, it would now would print every character twice. So a common interaction such as typing some text into an input box turned into:

public void typeWithFullKeyEvents(String locator, String string, boolean reset)
{
if (reset)
{
selenium.type(locator, "");
}
char[] chars = string.toCharArray();
// Some browsers clear the text field when we start typing,
// so we need to pre-populate this string with the existing contents
StringBuffer sb = new StringBuffer(selenium.getValue(locator));
for (int i = 0; i < chars.length; i++)
{
char aChar = chars[i];
String key = Character.toString(aChar);
sb.append(aChar);
selenium.keyDown(locator, key);
// some browser dont actually input any characters on these events
// supposedly to prevent JS spoof attacks. So we type for them
if (!SeleniumProxyUtil.getInstance().getUserAgent().equals("firefox"))
{
selenium.type(locator, sb.toString());
}
selenium.keyPress(locator, key);
selenium.keyUp(locator, key);
}
}

Other gotchyas – the key events will not give the field focus.
Timings
Due to our non-selectable items (section labels etc.), we wanted to test that some elements attributes do not change. We fire a mouse event on the element, and then use an xpath locator to check that the element’s attribute did not change. Worked fine on my local box, though it needed a slight pause between the event fire and test on the Bamboo box. This was a very common fix for a lot of the things that would break – “Add/increase the pause”.
xpath
So we used xpath locators for our tests. Seemed resonable. Right? Wrong! These xpath expressions are evaluated on the browser. Each browser constructs its DOM differently and may evaluate the expressions differently. We ran into trouble locating attributes on DIV items inside list items… The solution – write javascript locators. We ended up using yui to locate our things for us.
WTF?
To make things a little nicer for our users we decided to put in an autoscroll feature. I.e. when the pop-up pops, scroll the browser window to display the whole pop-up if it was hanging below the bottom of the screen.
Our tests were still a little flakey on Bamboo, though we had just had our greatest number of consecutive successful builds (5), so I wasn’t overly surprised when they broke again. Only 3 out of 40 odd tests. After putting in/increasing numerous pauses, I still couldn’t get them to pass. Why would scrolling the screen cause the tests to fail? We fire events on selected elements – not co-ordinates. The build kept on breaking…
BadGraph.png
We reached our lowest successful build percentage…
BadPercentage.png
I decided to have a look at what was happening on the server (admittedly I should have done this earlier). To do this I needed to open an ssh tunnel to connect the vncviewer client to the vncserver. You can open a tunnel with ssh using this: {{ ssh -L 5901:localhost:5901 username@remote.host.com }} and then for vncserver instance 1, localhost will pipe through to remote.host.com. Increment the remote (second) port number by one to get to each successive vncserver instance, e.g. 5902 for instance 2. Leave the local (first) port number as 5901 so when you run vncserver as if to connect to your local instance 1 it will actually pipe through to remote instance 2.
So what did I see? I saw the mouse positioned directly in the midle of the screen, when the browser window scrolled, the mouse was positioned over one of the non-selectable items. Only 3 of tests caused the screen to scroll.
How to fix? We installed xwarppointer, to enable us to move the mouse pointer through bash, and tucked it away in a little corner.
Ahhh, the green!
green.png
Lessons Learnt
1) As much as you would like it to be, the Selenium client is not a user! It won’t trigger all the events a user will and different browsers rely on different events!
2) Cross browser my arse!
3) Pauses are a Selenium Test’s friend!
4) When running tests, keep the mouse the f&#k away!
Or, are we just using it wrong?
Was it worth it?
Tough to say. We are getting more used to all of Selenium’s little quirks, though we are still struggling to get it to pass on IE and Safari. It did help us pick up a ThreadLocal leak that could have been quite bad, but is hasn’t really picked up any UI bugs.
I guess I will put off a positive or negative conclusion until we build up our suite of tests, delve further into AJAXy features and have gotten used to Selenium some more.
Cheers,
Nick
JIRA Developer
* In the JIRA team anyone who knows what a namespace in javascript is considered a Javascript guru.

Fresh ideas, announcements, and inspiration for your team, delivered weekly.

Subscribe now

Fresh ideas, announcements, and inspiration for your team, delivered weekly.

Subscribe now