What the f**k to test?

Was reading my emails and some blog posts today and came across these:

http://lifehacker.com/5911385/seven-helpful-sites-that-tell-you-what-the-fk-to-do

https://github.com/soulwire/WTFEngine

it made me think, as a tester, it would be interesting to have/see a site about “What the f**k to test” or something similar. It could be useful to the entry level and novice testers just getting into testing profession. And it could be useful to all of us testers when we happen to feel lazy or have “tester’s block” not being able to test or decide how/what to test.

Or do we actually have something like that already? If so, let me know.

If not, anyone interested in helping build such a site/service? It would be a site that randomly decides to give you the tester a tip on what to test for your site/application/system (in general terms that are not specific to your site/application/system).

Chrome version upgrade can break Chrome extensions?

This is a short blog post today:

Who would have thought this, but some recent Chrome version upgrade can break the UI of your Chrome extension (by changing extension popup window size and font sizes). I only found out after reading a user review.

For reference to the particular issue: https://code.google.com/p/autosmsclients/issues/detail?id=3

and now I have to find time to go debug and fix this, just because Chrome decided it wanted to change things on how a Chrome toolbar UI renders, and without notifying it’s extension developers about this…sigh…

Integrating AutoIt, Sikuli, and other tools with Selenium when running tests in Selenium Grid

A true integration for tools like AutoIt, Sikuli, etc. with Selenium would be when they are able to run under Selenium Grid configuration just like Selenium. That would probably first require that they can run over WebDriver API, JSONWireProtocol as with AutoItDriverServer. But when Grid automatically selects which node to run your Selenium tests with, would it not do the same for AutoIt, except if you specifically request a certain capability set that let’s you know which exact node the test will go to. But anyhow, it might get confusing how to map/correlate which grid node runs the Selenium part of test vs the AutoIt/Sikuli part, etc. So maybe having AutoIt/Sikuli run under Grid configuration isn’t the whole answer either.

Regardless if AutoIt/Sikuli ran under Grid configuration or not, you could still deploy your Selenium tests that use AutoIt/Sikuli, etc. under Grid. I did blog about this a while back but never gave concrete examples. In StackOverflow tradition, people seem to want specific examples. So I’ve finally worked up a simple illustrative example.

To have AutoIt/Sikuli work with Selenium while it’s running under grid mode, you need to find a way to determine the node host that runs the Selenium tests at runtime, so that you can then call/run AutoIt/Sikuli code on the same target node host, so that everything runs on the same correct machine (rather than the dilemma novices encounter where Selenium executes on the remote node but AutoIt/Sikuli runs on the localhost, causing mismatch and test fails).

The easiest example solution is to extract out the node host information from the Selenium Grid API by providing the WebDriver session ID that you first need to extract. Once you have that information, you can then use various ways to remotely execute AutoIt, Sikuli against the node host that you’ve extracted. All this is presented in this Github gist:

https://gist.github.com/daluu/3b4746f6b672c49f7e8f

of all the various ways in the sample gist, I personally prefer calling AutoIt, Sikuli over WebDriver API, where possible rather than to resort to using PSExec.exe, SSH, etc. I’d go for web service route as alternative to WebDriver API. Sticking with PSExec.exe type option as last resort. But you are free to pick whichever option works best for you.

WebDriver API and JSONWireProtocol is not just for web and mobile applications testing, it can be for desktop too!

And so, here’s a blog post about just that. I’ve found that the WebDriver API in general, with regards to common location strategies of ID, name, class, tag, and kind of XPath, along with element manipulations–such as click, type/sendKeys, get/set text, get attributes, mouse operations, key up/down, taking screenshots, finding elements & validating properties (enabled, visible, selected)–all that being common across web, mobile, and desktop.

Selenium/WebDriver started off for web applications. Then came along Appium, ios-driver, etc. which expanded it to the mobile space, first iOS then Android, and now somewhat even Windows Phone.

But there has been very little in the desktop area. The first for it was Appium for Mac. And we’ve seen nothing else since. Though if you consider unofficial Selenium/WebDriver-like APIs, then maybe there’sa few more: Twin and sikuli-remote-control. But now, I’ve worked out some proof of concept prototypes (based off the old Appium Python implementation) to give you more options to desktop automation using WebDriver API and for Windows, not Mac!

https://github.com/daluu/AutoItDriverServer
https://github.com/daluu/AutoPyDriverServer

Check them out. Pretty interesting. The AutoIt one has a Selenium integration demo showing automation of AutoIt and Selenium together against websites using 2 drivers – a “web” driver and an AutoIt driver. They’re not Selenium Grid compatible yet, something to look into for the future. But at least they work for remote deployment. Though there is a way to make it work unofficially with Selenium Grid deployment of Selenium tests (but where these desktop UI servers are just not officially part of Grid as nodes) – this solution will be presented in a future blog post.

And last, I had wanted to complete with a three’s company big bang, but I couldn’t get Sikuli working just yet, maybe in the future…

https://github.com/daluu/SikuliDriverServer

So try them out, submit feedback, send pull requests with enhancements and bug fixes, etc.! ;-)

P.S. all this made possible by the great work of others. Like the Appium team for the server base I used. And if/when I work on the Java server version – the Selenium team or ios-driver team, and .NET server version – Jim Evans for his Strontium server implementation. As well as the work of those who build great free or open source tools like AutoIt, Sikuli, AutoPy.

Polymorphic security measure could be a pain for Selenium if running tests with that enabled

Came across this site today:

https://www.shapesecurity.com/technology/

Interesting technology. But if the site to be tested uses real time obfuscation of the (form) element IDs, names, classes, etc. that could be a pain to automate tests against since you’d have to get a handle on the element location to perform the automation. Perhaps one would need or use a backdoor option to disable the obfuscation when running automation to test the site normally.

Either that or utilize an API provided by that solution perhaps that lets you real time map original element location identifiers to the obfuscated ones to use with Selenium at run time.

Be interesting to hear of anyone automate testing of a site that uses such technology.

Organizing tests by repetition or granularity?

I’ve seen this often in my job covering test scenarios manually as well as adapting those tests for automation. Have others come across same issue?

In terms of manual, and more correctly, exploratory testing, as a human tester, you can choose how you perform your tests and it is often logical to consolidate areas of testing (or in documentation parlance, “test cases”) around things that would otherwise be repetitive. e.g. re-using same test data to test slightly varying scenarios. Where workflow or sequence of test steps are common across several tests, you may as well run them sort of together like multitasking rather than separately one after another, repeating the common steps all over again.

One example is testing a workflow that happens to be mostly same across desktop, tablet, mobile. Only UX or UI is different. Do the test 3x? Or run test nearly 1x by testing all 3 at same time where/when possible using one test/path to do most of the workflow and verifying across all 3 paths at the critical points.

But that’s easy to do for exploratory (and undocumented manual) testing. But what do you do when it comes time to put this on pen & paper for others like an offshore team to execute said tests, who are one or more connections/layers away from the intimacy of the feature being test than you are? And then converting such documented tests to automation.

Do we strive to follow the human (exploratory test) nature/convention of consolidating the test coverage around the repetitive workflow and shareable test data minimizing amount of test cases and test scripts to create OR strive to keep granular individual specific tests for tracking/auditing purposes and simplify or attempt to idiot proof things, even though it will be in ways redundant in repeating test steps and generating more repeated tests data that could have been shared/reused otherwise?

One argument could be to do it granular since automation and/or hardware is cheap. But it turns out unless your site is 1990s basic HTML, repeating test steps can still be slow against modern websites/applications against modern browsers. And running tests in parallel helps with this but adds complexity and still can take long time once your test codebase grows big. So in some cases, maybe consolidating testing isn’t necessarily bad, you have to weight the pros & cons with short & long term automation coverage/infrastructure in mind.

Just wondering what experiences and approaches folks have taken in the real world.

Selenium page objects beyond pages like a cart object?

Selenium page objects is a design pattern to help you model test code better. But one doesn’t have to follow the full guidelines of the design pattern.

Some people have used it to model parts of pages as well (headers, footers, navigation, templates, widgets, etc.).  But perhaps it can be useful for more than that, and some people might have already done this or not, as I couldn’t really search anything up or don’t know what/how to search specifically for this. If people have already done this, they haven’t widely publicized it.

What I have found so far is this: https://github.com/cheezy/page-object/wiki/Indexed-Properties

It’s an interesting piece to review. I had this similar thought in mind recently and decided to blog about it:

A shopping cart page doesn’t really do much really. It contains cart items, and offers a visual call to action (click button) that then takes you to checkout. That along with standard site (header/footer/navigation) template actions (login, logout, links to other areas of site).

The core functionality in the shopping cart page really belongs to the cart items and what you can do with them. So in my mind, having the shopping cart page object manipulate cart item actions doesn’t seem quite appropriate for object oriented design.

For example, this would be how you might typically implement the cart page in basic page object model:

cartPage.updateQuantity(cartItemIndex, quantity);

cartPage.removeCartItem(cartItemName);

cartPage.getCartItemName(cartItemId);

//obviously access cart item by index, text string name of item, or by some unique ID

However, perhaps you can extrapolate the cart items outside of the page object to manipulate individually as a collection or set of related WebElements (name, quantity field, remove button, etc.) or for more advanced usage as an encapsulated cart item object model itself.

Both modeling options are presented here below (since I find it hard to list code in a “basic” WordPress blog)

https://gist.github.com/daluu/8e4c43d653c07eb0c57a

I would note that modeling cart items in such a way, while being more object oriented, can make implementing the cart page object (particularly the getCartItems method) and associated procedure to locate and group the related cart item elements together functionally more complicated. Because often, the web application will not have an easily implemented UI that has all the related elements easily related and uniquely identifiable to each other, especially on a cart page with N number of cart items.

Usually that may require sophisticated use of CSS and XPath patterns to locate and relate the set of elements for N cart items on the page. So this this whole approach is not something novice page object model and Selenium users can easily tackle. It takes time and skill to do. But worth trying out.

In the long run, I feel this type of approach is more maintainable, scalable, and makes the tests more readable. It just requires more thought in architectural design and more work upfront to implement. However, the complexity to implement could be reduced if you can get the developers to make the element locator values easily defined w/o resorting to custom CSS and XPath, and make it work for N cart items, and X related cart item elements (e.g. item color, item description, item this, item that, for every cart item)

If you ask me personally which cart item object model I prefer, it is the latter one that resembles a page object rather than the one that is simple a container of WebElements for a cart item.

What are your thoughts on modeling things or objects on a page like a page object. Cart item is the one that tends to come to mind, but there are others, for which they can be thought of as objects but not widgets, nor headers, footers, or navigation for page object modeling. Some other possible examples include a search result, a category listing, etc.

Also, please do inform me if you come across other articles about using page objects for things like cart items, search results, etc. where we’re not working with a page but some other object per se.