My research into UI-testing tools continues. A nice resource that has been guiding my efforts, although a bit out of date, does have a nice compilation of candidates for functional testing, load-testing, and etc.
A next candidate, PesterCat, tests at the HTTP request-response level. So it's moreless HTTPUnit with a nice GUI around it. This precludes both the "identify the element to click on" model and the "add pause here" stuff needed in both Selenium and Tellurium; imo, this is a Good Thing. The playback doesn't actually launch a browser at all, it just runs the HTTP requests (so it runs very quickly relative to the launch-a-browser approach, which is also nice) and lets you check the HTTP response for what you're looking for. Assertions can be done with Xpath, regular expressions, etc.; I've not explored that aspect fully, but was able to succeed on a simple assertion without too much trouble. So one (possibly show-stopping) downside of this is one can't test what the user-facing stuff looks like, can't test what the Javascript has done after the response is received, etc. So, perhaps not well-suited to AJAX-centric webapps.
The GUI is simple, clean and intuitive. Just like you'd expect for a $50 tool - not bogged down with too many features, so the GUI is still usable.
In theory it can be automated, since the scripts are exported to XML that can then be run from Ant scripts - but the evaluation license doesn't support saving anything so I couldn't test drive that part. The problem I see is that sending requests at the HTTP level calls for data-specific values that make tests less repeatable - e.g. instead of clicking on "My FooBar" and hitting "Delete", you might have to send a request with the ID of the object, as represented on the back end (since the ID is the unique identifying attribute, a typical setup). And, when the test is next executed, that ID could be different (especially if you're creating test data in your tests; you'd have to e.g. parse the returned HTML to figure out the ID), so you can't rely on that break of encapsulation (as usual).
PesterCat requires SSL to be turned off and must be configured as the browser proxy, but only for recording of tests, not for playback (since there's no browser needed for playback). The proxying noticeably slows things down when recording a test (that's with Firefox 2.0; this might be faster with 3.x). It is considerably easier constructing tests than with Selenium or Tellurium - since it's simply replicating HTTP requests, and you're not goofing around with HTML ID's, Xpath, pauses, etc. - but as noted, it's also more narrow in how it can be applied.
If it weren't for how slow the proxy is, this tool would actually be quite useful for monitoring HTTP requests, since you can filter out the GIF's, JS, PNG, CSS, etc. requests, all of which clutter the FireBug console pretty badly. We're also going to look into how useful PesterCat might be in fleshing out the URLs needed to construct JMeter tests, which last time I checked was one of the harder-than-I-wanted-it-to-be aspects of that load-testing tool.
Friday, August 21, 2009
A Quick Look at Tellurium
Tellurium builds on top of Selenium to present a more object-oriented abstraction, potentially resulting in more maintainable test code, and perhaps even making it easier to construct the tests to begin with; both of these things would require more familiarity on my part to judge fairly.
While Tellurium provides an IDE (TrUMP), it is not used for the same purpose as the Selenium IDE - i.e. rather than click-and-record of sequential events, it facilitates click-and-generate of (Groovy) code snippets for a given UI component or hierarchical collection of components. These become the basis of so-called "UI Module" classes, to which the developer can add methods that access the UI component. Test classes - whether in TestNG, Junit, Groovy, etc. - then leverage that set of methods to write test cases. This approach encapsulates the complex lookup (XPath, etc.) needed around UI components with access methods around those components, and supports separating the concerns of UI component vs tests written against that component. This of course sets the stage for reuse of the UI components in different test classes. Since Tellurium provides a relative location lookup model, it insulates the test cases against changes to the UI (i.e. an absolute XPath expression could break if any of the enclosing path components changes, e.g. when the UI is laid out differently).
Good Things and Random Notes
Most notably, Tellurium provides an OO approach. This is good for quick construction of test code and, more importantly, ease of maintenance on that codebase over time.
As a convenience wrapper around Selenium, Tellurium starts up the Selenium server automatically when one runs a test case.
To use TrUMP, the Tellurium IDE, SSL must be turned off; this is however probably a good idea anyway for testing purposes. The IDE is used by turning Record on, clicking on one or more UI components (e.g. a form, a table, link, etc.), and then clicking on Generate in the IDE. This (ultimately) produces a set of snippets, with any components that are part of a hierarchy in the page represented as such. Each snippet presents all attributes in the given component; you can next proceed to customize things (include/exclude snippets, rename for better abstraction, etc.). Next, you'd export the snippets to a Groovy class that represents the given UI Module, and as mentioned add methods to support access by test classes.
One roadblock that took some time to solve was dealing with iframes; the browser instance became confused, losing track of its connection to Tellurium. To deal with this type of thing, set useMultiWindows = true in the TelluriumConfig.groovy file.
When things go wrong - i.e. unexpected failures in your test case - a screenshot is automatically saved to facilitate revisiting things in a post-mortem.
Criticism
As of this writing, TrUMP is not compatible with Firefox 3.5.2; I don't know yet about other 3.x releases.
The export to Groovy is a heavy-handed "save as file" approach; I would prefer just a quick export-to-window-in-TrUMP model so I could more quickly copy-and-paste into my IDE.
I had no luck using TrUMP to capture webpage elements that were generated by post-processing AJAX, i.e. links constructed dynamically by JavaScript based on the data returned. Nor did I succeed with TrUMP in capturing Dojo menus that appear from dropdown buttons (a typical Dojo approach). In both of these cases I had to do things manually, using various approaches like Selenium, FireBug "Inspect" (in particular leveraging the Search functionality in 3.5), and old-fashioned examination of the page source. The good news is that once a particular UI component's location has been fleshed out, that effort is reusable since the resulting UI module is reusable.
While Tellurium provides an OO approach - which IMO is a Good Thing - there are as such some (not unreasonable) assumptions around just what behaviors certain objects should have. For example, it's not uncommon in Dojo widgets to find onclick event handlers (moreless) attached to TD tags; but from an OO point of view, a TD should not be clickable. This appeared to be show-stopper until I (quite by accident) stumbled on an attribute that could be retrofitted to a UI module component description that instructs it to "respond" to click events (i.e. add respond:["click"] to the generated Groovy UI Module description).
The "quite by accident" serendipity described above should be better; in fairness, Tellurium is a nascent product, but I hope that soon some first-class JavaDoc and other reference documentation is provided (or, it could be that I just need to poke around more; but, again, this shouldn't take any effort at all to find - it "should" just jump out at me).
Conclusion
All in all, I'm giving a qualified two thumbs up. Tellurium out-of-the-gate shows great potential, and I personally prefer it to Selenium. It's moreless the kind of thing that I'd probably have ended up doing myself to get the level of abstraction I'd want; I'm grateful to the folks at Google to have done it for me.
There is still a great deal of facility I've yet to explore with this tool. It appears to have a fair amount of power, and I look forward to writing up more insights and tips around Tellurium.
Tuesday, August 4, 2009
Selenium IDE Tips
This is a quick cheatsheet for various Selenium (IDE-based) commands. There's nothing too jaw-dropping here; just a collection of tips that I've found useful.
General Usage
Here is a suggested development technique with caveats. There are lots of ways to accomplish the same thing; this is just my own approach:
To confirm there's a DIV that contains an onClick handler with a given name and some template text of a given value:
...use this:
The @onclick can be replaced with any attribute of the given DIV tag. The bracketed "contains" conditions can be chained, and are AND'd together.
To check/uncheck a checkbox in a table with a given ID that contains a cell with the given template text:
...use this:
Avoid the annoying 30-second delay after clicking a link that use a JavaScript command as the HREF:
Set the timeout as needed:
This will still get logged in the Selenium IDE as an ERROR, which unfortunately will result in the test being marked as failed. This of course makes it difficult to determine that your tests are in fact all green, since these failures show up as red. But I've just gotten into the habit of clearing the log before I start a test suite, and just ignoring the "[error] Timed out after 1000ms" log messages. If any other log messages are at ERROR level, these tell me there is something to look into; else, I consider my test suite to have passed.
To select an item from a menu:
...use this:
To enter a carriage-return character in an input field with a given ID:
So the number 13 is the HTML code for the carriage return character. Here's a reference of more character codes.
To click on a link with the given template text:
...use this:
To consume an alert:
General Usage
Here is a suggested development technique with caveats. There are lots of ways to accomplish the same thing; this is just my own approach:
- Start by creating a directory that is named by the use case you're testing.
- Create a test suite in that directory, naming it TestSuite.html. Using this convention will help distinguish the test suite from its test cases, all of which should use an HTML suffix.
- Create test cases one by one, immediately give the test case a title via the Properties menu item (right click over the test case object), copy that name onto your clipboard before hitting OK, and then immediately save it to the use case directory by pasting that name into the file-save dialog and appending .html.
- Do not expect to be able to re-use general purpose test cases from other use-case directories (e.g. a login sequence) - I've had to duplicate these in each use case that I use it in. This might be just me doing something wrong; I might need to revisit it to confirm. Obviously it sure would be nice if we could reuse things this way.
- Do not expect to "Add Test Case" using an existing test case which you then alter, without altering the other uses of it within your test suite. Using my techniques here, one test case == one HTML file; modifying it will modify all instances of it within that use case.
- Save often - ideally save test cases as soon you alter them, and save the test suite as soon as you add/delete test cases to/from it. If you change test suites, the IDE will not warn you that you have unsaved changes.
To confirm there's a DIV that contains an onClick handler with a given name and some template text of a given value:
<div onclick="doSomething()">foo</div>
...use this:
assertElementPresent //div[@onclick[contains(., 'doSomething') ]][contains(., 'foo')]
The @onclick can be replaced with any attribute of the given DIV tag. The bracketed "contains" conditions can be chained, and are AND'd together.
To check/uncheck a checkbox in a table with a given ID that contains a cell with the given template text:
<table id="myID">
<tr>
<td nowrap="nowrap">
<input type="checkbox">check me</input>
</td>
</tr>
</table>
...use this:
click //table[@id='myID']//td[contains(.,'check me')]//input
Avoid the annoying 30-second delay after clicking a link that use a JavaScript command as the HREF:
<a href="javascript:doSomething()">Do Something</a>
Set the timeout as needed:
setTimeout 1000
This will still get logged in the Selenium IDE as an ERROR, which unfortunately will result in the test being marked as failed. This of course makes it difficult to determine that your tests are in fact all green, since these failures show up as red. But I've just gotten into the habit of clearing the log before I start a test suite, and just ignoring the "[error] Timed out after 1000ms" log messages. If any other log messages are at ERROR level, these tell me there is something to look into; else, I consider my test suite to have passed.
To select an item from a menu:
select menu label=three
...use this:
select menu label=three
To enter a carriage-return character in an input field with a given ID:
keyPress inputFieldID 13
So the number 13 is the HTML code for the carriage return character. Here's a reference of more character codes.
To click on a link with the given template text:
<a href="foobar">Click Me</a>
...use this:
click //a[contains(text(),'Click Me')]
To consume an alert:
storeAlert
Subscribe to:
Posts (Atom)
Posts
