Shorthand to XPath and CSS in developer console and javascript libraries but not WebDriver API?

This thought just came to mind while I blogged and commented online about finding elements by XPath and CSS using the DOM following DOM conventions (accessed/executed in javascript whether in browser developer console or with WebDriver JavascriptExecutor) like:

document.getElementById(‘someId’)

document.getElementByXPath(‘//someXPath’)

document.getElementByCssSelector(‘someCss’)

where some people pointed out you could just do

$x(‘//someXPath’) and $$(‘someCss’)

in the browser developer console (Firefox, Chrome) to do same thing. And yes, it is much shorter to type out.

And of course, you can do similar if your web application already uses jQuery or other javascript libraries, in which case you can execute their ways of locating elements with WebDriver’s JavascriptExecutor. If the web application under test doesn’t use them, the alternative if you want those location strategies is to inject the jQuery and like libraries into the page under test using JavascriptExecutor first before you can utilize them.

So it seems a bit interesting how some people prefer shorthand use in javascript library and in developer console, but how we don’t have much complaints or work done on the WebDriver side to mimic same changes on the web development/debugging side.

Like how document.getElementByXyz() is considered unnecessary extra work but driver.findElement(By.xpath()).click() is ok instead of targeting for something like driver.$x(‘//someXpath’).click() or driver.$$(‘some css’).click(). Granted in the WebDriver case, due to object oriented programming and instantiating classes, you’d always have a driver object that you can’t skip typing though you could shorten it by naming it “d”. But maybe we can also shorten the find by XPath and find by CSS methods with alias methods. And if there’s an argument of no, then one might also ask, since we can live with this with WebDriver, why complain that you can and should use shorthand in the browser console and jQuery?

I guess it is easier to code and debug when shorthand but standardization and consistency of APIs and naming to me is more important. Less to remember & learn, especially for novices. To me

document.getElementByXyz() clearly maps better to  driver.findElement(By.xyz()) than $x() and $$(), and vice versa. Same applies for the multiple elements version of the methods.

Selenium automation with execution of custom site specific javascript

Expanding on my previous post: Javascript is your ally for Selenium / WebDriver

I am wondering if anyone has used WebDriver’s javascript execution facility to execute custom javascript code (not general code used for browser/Selenium limitation workarounds) but javascript code that’s specific to their website or web application under test.

I asked a colleague within our organization and he said his department recommends avoiding such use. While I understand that it is best not to dig into specific detailed functionality of the app under test for UI automation it does have some uses:

Limitations of the UI and element locators (IDs, class names, name attributes or other DOM attributes, visual text on screen) in providing a good way to define locators in an object oriented way programmatic parameterized way so that your locators are more dynamic and not static. For example have a method that selects a color and we “select/click” the element (finding it) based on color name rather than some less user friendly/understandable strategy. In some cases, that’s not possible, but might be if you execute app-specific javascript code to find out the info you want that is not exposed via the UI. Same goes for checking application state that might not be fully exposed visually in the UI. I’ve found it helpful in my case.

The other useful case may be for more white box javascript API & unit testing integrated within Selenium UI tests for extended test coverage. This might be useful compared to issues that may come up with pure javascript API/unit tests alone per this blog: http://watirmelon.com/2013/02/09/why-i-dont-like-jasmine/.

What are your thoughts on this?

Javascript is your ally for Selenium / WebDriver

First a background introduction / discussion

When you run into issues or limitations for Selenium, particularly WebDriver, whether it be in Selenium itself or a browser specific limitation, javascript is your ally.

Though javascript isn’t all power and limitless, there’s much that can be done with javascript provided you know what to do with it and how to use it.

With Selenium RC, you have access to the browserbot object that gives you javascript-based access to windows and the HTML Document Object Model (DOM) via Selenium. You can do web search on Selenium browserbot to learn more about it or find usage examples. One of my posts show how to use it in interesting ways (under the DOM examples): Special element state validation with Selenium and CSS and DOM

With Selenium 2 / WebDriver, you have even more power with javascript as you’re given direct access to javascript with the only security restrictions being that of the browser itself (e.g. no system file access, no cross domain AJAX calls w/o workarounds), whatever you can do with javascript, you can execute with the WebDriver JavascriptExecutor. Again, you can web search for some examples.

Now the goodies:

Unfortunately, most Selenium users are not well versed in javascript (not front end web developers), and fortunately, Selenium is good enough 80-95% of the time that you don’t need to resort to javascript. But for the 5-20% of the time where you need a workaround, javascript might be it. So first, it is good idea to learn javascript programming so it can help you be better with Selenium. Outside of Selenium, it also helps you better debug/troubleshoot front end web application bugs.

In this blog post, I’ll list some useful javascript related content links that apply towards Selenium. And I’ll update this post as I find more. Sort of acting as a central repository of useful Selenium javascript workaround hacks. Since it’s not easy scouring the web for such solutions. Feel free to contact me with additional hacks you find.

NOTE: to understand how these hacks work, you have to learn more about javascript…

List of useful Selenium workaround javascript hacks

Verifying images – “really” verifying an image is rendered/displayed on the browser, not just a broken link
http://watirmelon.com/2012/11/27/checking-an-image-is-actually-visible-using-webdriver/

Checking scrollbars – checking whether an element (div, iframe, textarea, etc.) that holds other content has scrollbars (vertical and/or horizontal). The solution here is for Selenium RC in Java, but can be adapted for WebDriver and other languages. A good javascript learning exercise on how to adapt it.
http://selenium-automation.blogspot.com/2010/10/selenium-automation-problems.html

Manipulating (or clearing) HTML5 local & session storage
https://gist.github.com/anonymous/3972875
Can add/remove, check for data, etc. but in terms of automated testing, we’re more likely just wanting to clear storage cache like clearing cookies. The above link is a Java example for session storage, but can be adapted to other languages, and local storage has the same API, just window.localStorage instead. And read this for more info on working with HTML5 local & session storage: http://diveintohtml5.info/storage.html

Drag & Drop using javascript instead of native interaction / Actions API
http://ynot408.wordpress.com/2011/09/22/drag-and-drop-using-selenium-webdriver
In case the native actions drag & drop fails to work for any given browser, or where not available like SafariDriver

Mouse over using javascript instead of native interaction / Actions API
http://code.google.com/p/selenium/issues/detail?id=2067, comment #60 & other related comments (before & after that are relevant). This method useful when the native method fails for any given browser, and currently is only option for SafariDriver.

Mouse click using javascript instead of native interaction / Actions API or WebElement.click()
See the mouse over link above for code. You then just need to modify the code to replace “mouseover” with “click” and “onmouseover” with “onclick”. How is this useful? When the regular WebElement.click() method fails to cause the expected click action, and in case the native interactions / Actions API fails to work either (or where not available like SafariDriver).

Force setting value to a (Web)Element – sometimes, you can’t seem to manipulate a WebElement as action does not seem to take effect or it is not allowed because the element is not visible/displayed (e.g. hidden). One trick is to use javascript to set the value (e.g. value of a form input field, or value of a hidden input field that is actually used by web app and set via javascript when you perform action against some other element that’s usually near it in the DOM tree). For an example of this, see my blog post (WebDriver update at the end): Special element state validation with Selenium and CSS and DOM

Mouse wheel zoom action
http://stackoverflow.com/questions/6735830/how-to-fire-mouse-wheel-event-in-firefox-with-javascript
This currently doesn’t appear to be available in Selenium, even with the native interactions / Actions API? Or I overlooked it. Here’s a possible javascript solution. I haven’t fully tested whether it works or not. As the link is to web developer/application code, you do have to adapt it to work within Selenium. Not simply copy, paste, and run.

Highlighting elements
http://selenium.polteq.com/en/highlight-elements-with-selenium-webdriver

Getting a WebElement from x and y coordinate among other neat stuff too
http://www.slideshare.net/nzakas/javascript-apis-youve-never-heard-of-and-some-you-have
you might be able to get the WebElement from WebDriver API too, not sure, but here’s a way to do it from javascript, and more that might be useful for QA test automation purposes although I’ve not found actual need for them myself at present besides getting element from given coordinates.

more to come…

2/15/2018: Extracting image(s) off HTML5 canvas for validation/testing: Forgot to mention before, but using javascript, you can extract image/graphics off an HTML5 canvas as base64 encoded text of the binary image/graphic data. Using the scripting/programming language of choice that you use with Selenium, you can then base 64 decode the data into binary and save it as the corresponding image file type, or keep as base64 version for processing if you have tools that work with base 64 or that render base64 data into binary visualization. In terms of having it saved into file, you can then compare the image file against say a known image (file) to expect for validation, or run fuzzy logic image matching with OCR tools like Sikuli/Applitools, etc. For how to do that, you just need access to the canvas element in javascript (can pass in the WebElement reference via Selenium to JavascriptExecutor), and call this API: canvasElement.toDataURI(optionalImageTypeDefaultingToPng,optionalEncodingOptions)

Getting window handle on existing windows in javascript by use of window name

I came across this recently. Don’t recall if I found the basis for technique off the web or not (think I did, forgot the source URL), but adapted it for use in javascript code.

In rare cases, could also be used for Selenium/WebDriver, if you’re not using their built in window handling APIs.

//get handle to some existing window
var someWinHdl = window.open(null,”some existing window name if you know it”);

//get handle to self (current window)
var selfWinHdl = window.open(null,window.name);

granted getting the current window handle this way isn’t all that useful though since you could also just use “window.self”. But it is useful to gain access to an existing window if you know it’s “window.name” value.

This same technique can also be applied for opening new windows to keep a handle on the newly opened window, just provide a real URL instead of null to window.open().

With this window handle in place you could access it like any window object such as closing it, using the stored handle variable example above:

someWinHdl.close();

NOTE – update 7/8/14:

For this technique, the code will only work under these conditions:

• will only work for current window & any windows that were opened from current window. That means that if window A opened window B, then window B won’t be able to access window A with this technique, even if you knew window A’s name. Or if window B was opened separately/manually by the user as separate tab/window and it somehow has a name, window A won’t be able to access it. This is likely from browser security sandboxing. You can only access windows you control (e.g. current window and any that you opened via javascript from current window).
• window.name needs to be defined with a value (in the case of getting handle to self/current window) or the window name to another window (opened by current window) needs to be a valid defined window name, otherwise, it will just open a blank window and you won’t get a reference to the (correct) window handle. For current window or self, you can arbitrarily set the window.name value (e.g. to “testing123”) if it is undefined, and then you will be able to use the code above. For other windows currently open, we assume those windows were opened with a defined window.name (or that it is defined at some point) so that you can reference it. If the name is undefined (anonymous), then this code won’t work against other windows that were previously opened by the current window.
• The code here is generally not that helpful in regards to windows already opened since you get the window handle when opening the window with window.open() and storing the return value. However, sometimes one might not have stored that value in the original window.open() call, or for testing purposes, you don’t have that handle but you do have the window name. As such, you can make a repeat call with the presented code to retrieve the window handle without changing the current location/URL of that window (e.g. don’t cause it to navigate away or page refresh).

iOS mobile testing tips

These tips may be found elsewhere on the web already, I’m just archiving this info I put together a while back into a single repository in the form of a blog post.

Tips for both iOS devices and iOS simulator

Firebug for iOS

Set up a javascript bookmarklet to Firebug Lite, so that you can invoke the lite version of Firebug in mobile Safari on actual device or simulator. See Martin Kool’s article for details, if you don’t know how to set up javascript bookmarklets. NOTE: on simulator, you will lose the bookmark when you do an iOS “Reset Content and Settings…” command. No issue on device unless you specifically delete bookmark or do a factory reset.

View Source of a page

iOS 8+: install an iOS 8 extension like the one called View Source. There are several extensions to choose from. The View Source one may be best as most popular, well known. Or you can also choose the other option for older iOS versions.

iOS 8+, 7.1, and earlier : use this Javascript bookmarklet trick/tip, or this version called Snoopy. NOTE: on simulator, you will lose the bookmark when you do an iOS “Reset Content and Settings…” command. No issue on device unless you specifically delete bookmark or do a factory reset.

Tips for iOS simulator

Set up camera roll to be able to use photos and videos for testing

See: How to import photos into the iPhone Simulator — Aptogo. In general, drag & drop photo into simulator and then tap & hold to save image in simulator will add to camera roll for later use. See the linked article if you need to bulk import photos into simulator. NOTE: on simulator, you may lose the photos when you do an iOS “Reset Content and Settings…” command. No issue on device unless you specifically delete photo or do a factory reset.

For videos, instead of photos, see How do I get video into the iPhone simulator camera roll in iOS 4.2? – Stack Overflow. NOTE: on simulator, you may lose the videos when you do an iOS “Reset Content and Settings…” command. No issue on device unless you specifically delete video or do a factory reset.

Tips for iOS devices

Debugging mobile and tablet websites on iOS – on Windows or locally on device

Tips on mobile and tablet device testing shared usage and remote testing outside corporate network

Sharing bookmarks and photos across devices

If you use an Apple/iCloud account across the devices (This might work on simulator iOS 7/7.1+ if you sign in to iCloud from simulator and then do a Debug > Trigger iCloud sync, although I haven’t tested it):

The photos on each device (if not existing photos, then new photos taken) will be accessible across devices via the Photo Stream section of Photos app. And deleting photo locally in camera roll of a device won’t delete it from the Photo Stream (for that device and all other shared iOS devices), so you have to specifically delete from Photo Stream to delete across devices, the camera roll is treated separately from Photo Stream on each device, so you will notice copies of what you have locally on camera roll also in the Photo Stream that’s available across devices. NOTE: that the Photo Stream (for iOS 6-7.x) is temporary storage, and to make it permanent on each device, you have to save it to local system (device/simulator) before it expires in 30 days (I forget, but the system may copy the photos you’ve previewed to your device after expiration rather than let it disappear from the stream). The iCloud Photo library in iOS 8 is different though with it being permanent storage.

Creating a bookmark on one device will automatically sync (if enabled, by default) them to the other devices using same account, so you don’t have to set up bookmarks/bookmarklets individually on each device.

Recording actions against or test session of iOS physical device into a video

For iOS 8+ only, but see Record your iOS Test Session with QuickTime | Daniel Knott

Tips for simulating iOS device in browser via user agents, etc.

Device screen sizes for testing mobile and tablet views of a site and user agent tools

A/B testing and Selenium automation

Don’t know how many people have to deal with this particular topic. But I had to research and deal with it before, so thought I’d mention a few things and share some links.

Whatever the case, unless you need to specifically “test” the A/B testing system/framework in terms of how it allocates/routes users to one flow or another, statistical reporting, and other things, it is best & simplest in terms of UI test automation to just be able to go through both A and B flows during testing.

Some people have mentioned to disable A/B test (e.g. go with A/old/base flow always), or workaround A/B testing. You can’t always work around it, and unless you can simply or easily disable A/B testing, it is simply just going through one of the 2 flows alwasy. In which case, it’s technically not that hard to set up the test framework to be able to switch flows.

Usually the A/B test framework/system has a mechanism to assign flows including always forcing to a particular flow, but each framework/system implementation is different. You would utilize that capability to select the flow to test in Selenium. By the way, usually the implementation involves, cookies, javascript, or URL manipulation. The way I’ve done it so far is to define a table/spreadsheet that defines the flow for each A/B test of interest to specify to Selenium test which flow to force going through. You can have variations of the spreadsheet (different files, different Excel worksheet tabs, different SQL DB tables, etc.). It would also be helpful to build into your framework the ability to query for what the defined/specified flow is for going through so the test knows what to do, as well as query for what the actual flow (as returned by the site/system) is so as to confirm the automation has correctly force set the desired A/B test flow. How all that is done is dependent on your A/B test system.

As for how you implement handling the A/B test flows, it is easiest to tackle if you use page objects. I would recommend encapsulating all/most of the A/B flow checking within the page object class methods, so that the tests themselves and the test author doesn’t need to know the details. That way the test can implicitly support both A/B flows as it is written. The only case you detract from that is when the actual workflow changes between A/B flows (e.g. a functional/workflow change not a UX/visual/icon change) or when you need to verify specifically UX elements like images, icons, and text between the flows. In such case you would then put A/B handling logic in the test (or test utility/parent classes) itself.

Handling A/B flow is simply control and branching logic (if/else/switch/case statements) in checking what flow is to be traversed and handle appropriately based on the given flow.

A tip on handling UI element locators in A/B testing is that one can minimize creating additional locators for an A element and B element, by instead using multi-value locators. This does then force you to use CSS selectors or XPath instead of simply by ID, name, etc. These 2 types of location strategies support defining multiple values and as long as one value matches, then you have found a match for a WebElement.

CSS: a flow value, b flow value

XPath: a flow value | b flow value

as you see in example above, the multi value separator in CSS is a comma and in XPath is the horizontal pipe character.

Also when handling A/B testing within page object methods and attempting to not change a test to support both flows, that means you typically modify a page object method to internally check what flow it should go through and handle it, so that the caller of the method (usually a test, or another page object method) does not need to know about A/B test.

Following my suggestions, it will be easier to manage the A/B testing in progress with minimal code changes across files. And the cleanup work will be easier (when you decide to stick with one flow 100 percent). Cleanup is simply then a removal of branching flows and the obsolete branches and deleting one of the values in the multi-valued locator variables/constants. The locator variable/constant (names) and the page object method signatures all remain the same, and tests themselves may require very little if any update at all.

Update 8/10/2021: I realize going through this post & its comments that I didn’t thoroughly cover the background context to why support both A & B flows as an option in test automation as opposed to simply always go through A or B flow, etc. among the other things discussed in this post. For this missing context, please read through the comments section, I think it gives more context there than trying to repost the comment discussion back into this main post.

Update 1/29/2019: and now here’s some links to topics on A/B testing that I have seen online that you might find useful:

http://elementalselenium.com/tips/12-opt-out-of-ab-tests

https://watirmelon.blog/2016/01/29/running-automated-tests-with-ab-testing/

http://sauceio.com/index.php/2011/07/easiest-ever-ab-split-tests/

https://www.kenst.com/2018/06/opting-out-of-a-b-tests-while-running-your-automated-tests/

https://groups.google.com/forum/#!msg/selenium-users/5t3vnPoUXzA/3glQ6kT9oZkJ

http://stackoverflow.com/questions/5269027/how-do-you-handle-testing-with-selenium-when-you-are-running-ab-tests

http://stackoverflow.com/questions/12078675/writing-integration-tests-against-external-resources-which-are-a-b-testing

Update 3/1/2016: it just occurred to me today, from reading a Selenium forum post, that this blog post of mine can also apply to a case where you have both desktop and mobile views for a website or web app (e.g. responsive design, etc.) or a case where you have a website and a native mobile app, and as such in both these cases there is shared logic (user/app/site workflow, element locators – even if the actual values differ, but the “logical” representation is the same such as login button on both web site and mobile app, etc.). Using the techniques defined here, you can share locator references/variables and share page object methods and have a single page object represent both mobile & web versions. For sharing of methods with branching logic, instead of checking which A/B flow to go through, you check what platform you’re on and go to appropriate branch logic based on that.

Debugging mobile and tablet websites on iOS on Windows or locally on device

I was doing some research and came across these useful tools one can use to debug and analyze mobile/tablet versions of websites/web applications in the case one doesn’t have a Mac handy to do the remote Safari debugging.

Remote Safari debugging on Windows

You can try option of using Telerik AppBuilder (Windows client) as a replacement on Windows for Safari debugger on Mac when remote debugging. There’s a nice blog post about the steps to do it in link below.

http://blog.falafel.com/Blogs/josh-eastburn/2014/03/04/ios-web-inspector-on-windows-with-telerik-appbuilder

The process of using it is much the same as with Safari remote debugging. Have the app open, connect device to Windows via USB, have Safari open on device, then browse device in the app, and open debugger in the app. The tool also will require you to create a Telerik account, and either load/use the default app or create a blank (javascript?) app just to use for debugging mobile Safari on your device.

The tool requires a license or you can use the trial, which becomes a starter edition afterwards. I think the starter edition will still allow you to do the debugging.

NOTE: using this method may require you to have the iOS device drivers to be able to interact with it from AppBuilder. You can get the device drivers by installing iTunes, but if you want to avoid that bloat, you can try to get just the drivers by searching online or downloading iTunes and extracting out the drivers from the installer so you only install the drivers rather than the whole iTunes package, both of which are not covered in this blog post.

Local device debugging on iOS/iPhone/iPad

You can also try these iOS apps below, you can find them in the iTunes App store. They give you built in developer tools like feature (right on iOS, no remote debugging needed) that mobile Safari doesn’t offer. However, they are not the native mobile Safari view/app, though I think it might use the internals of mobile Safari/Webkit to be close enough. These apps are useful when you don’t have a Mac or Windows PC around to do remote debugging with, so you can debug locally on the device through the apps.

MIH Tool – basic edition

HTTPWatch Basic – HTTP Sniffer and Debugger

I gave them a try and they’re at least better than the mobile Safari you get on iOS, unless one needs to target full mobile Safari compatibility. I’m guessing the pro/paid editions of those apps give you more/better features, I confirmed that was the case for HTTPWatch.

Update 7/30/14 – remote Safari debugging that is open source & cross platform

Found this today https://github.com/google/ios-webkit-debug-proxy. Works for Linux, Mac, and Windows support eventually to come. I haven’t tested this out myself though. However, except for tinkering with things, cross platform, and open source. I’d say this option is best when working under Linux. On Mac, better to go with the native support from Apple. And for now, Windows is better with the option presented above with an easy GUI for novices.

Update 8/8/2014 – another Safari remote debug for Windows

Adding on previous update, found a Windows port: https://github.com/artygus/ios-webkit-debug-proxy-win32

Thoughts on a Selenium interactive exploratory test and debug tool

I came across a nice Selenium-based tool recently called SWD Recorder. You can find more about it here or watch this video. But for me, the topic to discuss today is how it could be extended to offer features that I had been planning to build into a tool that would use the Selenium API. But since a similar tool already exists now, I can build off of it rather than create from scratch. But in case I don’t get to it yet or never do, I’m blogging my thoughts on what I would build upon this current tool.

The tool is great but I had been looking for specific features (to have), which don’t necessarily align with this tool’s original goal and thus I’d fork it for my own purposes. In general, I’ve been looking to build a cross browser Selenium interactive debugging tool that has the following features:

• A Selenium IDE or Selenium Builder like command executor. But cross browser compatible and doesn’t run as browser extension but separate tool that drives browser via Selenium APIs. Basically the tool provides a set of standard API commands to select from and user supplies the locator value and then you can click execute/run to execute the selected command against specified locator. I’m not looking for option to dynamically add a list of commands to execute like a test sequence/scenario/case or be able to save that to disk as a test case/suite file that Selenium IDE/Builder offers. Rather it would be more for exploratory test/debug session to see if certain Selenium commands work against specified element in given browser, etc. But those additional options I mention that are not of interest to me could be extended/added by other folks for a type of Selenium IDE/Builder that actually works across browsers (not just for running tests but recording actions or for manually defining/creating the action list/set). Examples of commands to execute: click, sendKeys/type, getText, getAttribute, drag & drop.
• Expand the previous bullet to also offer a list of javascript emulated/simulated action commands (not native actions/interactions API from Selenium but pure javascript events & emulation of actions) to execute. This would be nice addition that even Selenium IDE/Builder doesn’t offer I think. It serves as a way to see if javascript workarounds for some Selenium commands will work against given browser or not. Without such feature, the only way to test this is to actually wrap & executed the needed javascript code within your test framework to then test & try out, which also requires trial test runs of some test case with setup & teardown that takes time, or using a Python/Ruby shell to execute it but which requires you to type it up or load a pre-written library. But why waste all that time when you can test out quickly in a special debugger tool like this. Examples of javascript wrapped commands: drag & drop, mouse click, mouse over, mouse up, mouse down, scroll mouse wheel. See this blog post about javascript workarounds for ideas.
• Add alternate options for finding & testing locators. For example buttons to inject Selector Detector, Selector Gadget, and Super Selector among other similar tools without need to set up bookmarklets. Enhance the tool’s current WebElement explorer with option (by default or not) to generate a matching CSS selector (if applicable) for the given XPath that is generated. In this case, for Windows we’d have to have a pre-compiled binary of CSSify Python script for Windows, while on other OSes, we could perhaps just run the script natively. Or better yet, we instead screen scrap the CSSify public page/service to call the underlying “web service” (an HTTP POST call) to translate the XPath to CSS without havng to shell execute a Python script. This collection of goodies would then finally give Selenium users a cross browser element locator finder & tester tool to find elements (their XPath and/or CSS) and then modify the default XPath/CSS value and see if it still works, etc. A tool comparable to FirePath and Firebug. Granted it won’t be a perfect equivalent since it doesn’t quite show the HTML DOM source relative to the inspected element like Firebug & FirePath do at this point.
• Add alternate debugging options across browsers. For example, have button to inject & load FirebugLite without having to set up bookmarklet.
• Add (sort of) cross browser javascript error collector functionality for debugging with Selenium. Have buttons to inject the javascript error collection code snippet, and button(s) to check/display/retrieve the collected javascript errors (error count and the specific list of messages). Or instead of button(s) to check errors, it could be automatically dumped out in the tools display after the injection whenever error occurs. This functionality of course would not support javascript page load errors (only after page load) and would not persist across pages, have to manually inject on every page desired. Unless we enhance the tool by modifying the Selenium source code (.NET binding?) to auto inject on every page load. Now this might kind of seem pointless, but it’s sort of a cross browser solution as well as nice alternative to IE’s not so nice developer console or alert dialogs of javascript errors. It’s also a way to test out how well such a solution would work when you actually implement the same into your test framework without you have to do that first by evaluating it in an interactive debugging mode.
• Have a tab section where you can inject arbitrary javascript source files (via HTTP URL) into the current page. It’s a lot easier than having to manually write the javascript code snippet to inject the script element with src set to the HTTP URL then to execute that javascript snippet with Selenium command in say an interactive Python/Ruby shell. Just paste the URL to the GUI tool’s text field, then click inject script.
• Followng on previous bullet, also nice to have a section for a cross browser Selenium javascript console, whee you can execute any desired javascript code snippet via Selenium WebDriver’s JavascriptExecutor. You paste or type the code snippet in a textarea field and click execute. Any return value is cast as a String and dumped back in a results area for user to see in the tool. This provides a javascript console equivalent to the browser’s native developer tools but one in which the code is executed by Selenium rather than directly by the browser. It would be a nice way to test out whether certain javascript can be executed or whether it works well with Selenium before you actually code it into your test framework or test. I used to and currently do this over a Python interactive shell but it’s more simpler to do this over a GUI tool, especially for novice users.

Now from all the features mentioned above, to summarize, I’m looking to have an interactive GUI-tool based Selenium exploratory test & debugging tool. One in which you can test out code snippets, locators, and Selenium commands cross browser before you actually code it into test framework. Others might prefer the direct approach but I personally prefer interactive test & debug first as it is a lot more insightful and faster this way than to put everything in framework and a throwaway code test script that also takes much more time to execute through the test flow or having to set breakpoint in debugger and debug from that point. Having such a tool, you can easily combine manual & automated steps in one to see how things work. Such a tool is essentially a GUI version of what I talk about in previous posts:

How to debug test and try selenium code with interactive shells

Using selenium with interactive interpreter shells

A selenium IDE alternative for other browsers and another record playback method

Update 01/26/2014:

It recently occurred to me, that it would be nice to be able to inject jQuery and/or Sizzle into Selenium with such a tool via some buttons to click. To be able to test out whether can locate elements defined with jQuery syntax or for the non-standard CSS provied by Sizzle. This of course is only needed in the case of jQuery if the site under test doesn’t already use jQuery.

Update 04/14/2014:

I recently came across a similar too, this one is not .NET based but Java as a JAR file. Love it when the community comes up with new supporting 3rd party tools.

https://github.com/dmolchanenko/LookingGlass

Selenium WebDriver – extracting an image on page by use of take screenshot and cropping

Normally, if we wanted to get an image off the web page under test, we’d download it externally using the URL extracted off the image element source attribute. Unfortunately, in today’s AJAX based web apps, that doesn’t always work. Sometimes, the elements appear like images visually to the user but are rendered by javascript into DOM elements that are not “image” elements, composed of sprites, etc. on the server side.

In such cases, you have to go an alternate route…

The technique to do it is explained here:

https://groups.google.com/d/msg/selenium-users/DjoEjKwvIN0/sI-siW0gOA8J

and an example implementation in Java here:

https://groups.google.com/d/msg/selenium-users/8atiPIh39OY/Gp9_KEXnpRUJ

Side comment 1 – it would be nice if others in the community can contribute other language implementations to the Java source code example above so others can use it. Python, Ruby, C# perhaps? And a sprinkle of PHP, Perl…I may contribute code snippets when I have time.

After you’ve obtained the image, you can do whatever you want with it. Binary/MD5/SHA-1 hash compare the image against a known benchmark one?

For my case, that’s not sufficient as the image isn’t always a single image but a collection formed as one and dynamically generated. So we do some flexible fuzzy logic image comparison with our internal in house image comparison solution.

And on that topic, side comment 2:

I’ve noticed that taking screenshot then cropping to desired element location & size isn’t 100% cross platform compatible. So blogging this for reference to others and to see if anyone else have same issue. The problem here is that different browsers crop with slightly different coordinates and/or width/height ending up with cropped images that differ slightly across browsers. Now I haven’t tested across all browsers, but my findings reveal that Safari provides the most exact desired cropping, while Firefox and IE have slight variations that are not desirable. I didn’t test Chrome, nor mobile Safari with Appium.

With that in mind, you definitely can’t do hash/binary compare of the cropped images across browsers due to differences, unless you have a different benchmark image per browser. So with a single benchmark image, you’d need some flexible fuzzy logic image comparison technique. There’s multiple options here, but none readily available, other than maybe ones like Sikuli. The in house solution we use is based off ImageMagick and does the dirty work of calculating the parameters for the image comparison for you wrapped in a nice REST API.

Now anyone have experiences they can share around the topics discussed above?

Candidate interviews should include/have a debugging test

I had some experiences where some QA folks are not that quick or sharp at debugging problems in web applications, and automation test failures. Granted you should be intimately familiar with the application to better debug it, or with the automated test framework to know where and what failures happen.

But finding failure then tracking it back is a general principle that’s not that hard to do if you practice and do it enough. And in automation test failures, the exceptions or error message stacktrace is good starting point to backtrack against the test code. Combined with rerunning the test (in debug mode or not) and manually observing what happens, you can get an idea and track down the failure. But I’ve seen some folks not quite get the hang of it at tracking down the exact failures to an automated test. Or how to find further details of a failure when testing manually (by monitoring the AJAX, HTTP calls, or the javascript console logging of errors).

So it got me thinking that a “debug” test in an interview is useful for interviewing candidates. Purposely build or create a broken website/application and have the candidate find the cause or at least details to the failure than just that they seem something is broken functionally or visually in the UI (find out more why it is so). And for test automation. Same thing, purposely have a test that tests against old/obsolete functionality requirements that have since changed and/or a test that is valid but website/app is broken, hence test fails. Then have the candidate debug the test failure to figure out why the test fails.

Such questions will be useful in seeing how th candidate fares in actual work when debugging is needed. Or for QA candidates & manual/exploratory testing – to ensure the candidate can look into details on the failure and give that to the developer rather than just say I can reproduce and here’s what I see visually, I have no idea what might be causing failure underneath, you take a look.

What are your thoughts on this?