Integrating AutoIt, Sikuli, and other tools with Selenium when running tests in Selenium Grid

A true integration for tools like AutoIt, Sikuli, etc. with Selenium would be when they are able to run under Selenium Grid configuration just like Selenium. That would probably first require that they can run over WebDriver API, JSONWireProtocol as with AutoItDriverServer. But when Grid automatically selects which node to run your Selenium tests with, would it not do the same for AutoIt, except if you specifically request a certain capability set that let’s you know which exact node the test will go to. But anyhow, it might get confusing how to map/correlate which grid node runs the Selenium part of test vs the AutoIt/Sikuli part, etc. So maybe having AutoIt/Sikuli run under Grid configuration isn’t the whole answer either.

Regardless if AutoIt/Sikuli ran under Grid configuration or not, you could still deploy your Selenium tests that use AutoIt/Sikuli, etc. under Grid. I did blog about this a while back but never gave concrete examples. In StackOverflow tradition, people seem to want specific examples. So I’ve finally worked up a simple illustrative example.

To have AutoIt/Sikuli work with Selenium while it’s running under grid mode, you need to find a way to determine the node host that runs the Selenium tests at runtime, so that you can then call/run AutoIt/Sikuli code on the same target node host, so that everything runs on the same correct machine (rather than the dilemma novices encounter where Selenium executes on the remote node but AutoIt/Sikuli runs on the localhost, causing mismatch and test fails).

The easiest example solution is to extract out the node host information from the Selenium Grid API by providing the WebDriver session ID that you first need to extract. Once you have that information, you can then use various ways to remotely execute AutoIt, Sikuli against the node host that you’ve extracted. All this is presented in this Github gist:

https://gist.github.com/daluu/3b4746f6b672c49f7e8f

of all the various ways in the sample gist, I personally prefer calling AutoIt, Sikuli over WebDriver API, where possible rather than to resort to using PSExec.exe, SSH, etc. I’d go for web service route as alternative to WebDriver API. Sticking with PSExec.exe type option as last resort. But you are free to pick whichever option works best for you.

WebDriver API and JSONWireProtocol is not just for web and mobile applications testing, it can be for desktop too!

And so, here’s a blog post about just that. I’ve found that the WebDriver API in general, with regards to common location strategies of ID, name, class, tag, and kind of XPath, along with element manipulations–such as click, type/sendKeys, get/set text, get attributes, mouse operations, key up/down, taking screenshots, finding elements & validating properties (enabled, visible, selected)–all that being common across web, mobile, and desktop.

Selenium/WebDriver started off for web applications. Then came along Appium, ios-driver, etc. which expanded it to the mobile space, first iOS then Android, and now somewhat even Windows Phone.

But there has been very little in the desktop area. The first for it was Appium for Mac. And we’ve seen nothing else since. Though if you consider unofficial Selenium/WebDriver-like APIs, then maybe there’sa few more: Twin and sikuli-remote-control. But now, I’ve worked out some proof of concept prototypes (based off the old Appium Python implementation) to give you more options to desktop automation using WebDriver API and for Windows, not Mac!

https://github.com/daluu/AutoItDriverServer
https://github.com/daluu/AutoPyDriverServer

Check them out. Pretty interesting. The AutoIt one has a Selenium integration demo showing automation of AutoIt and Selenium together against websites using 2 drivers – a “web” driver and an AutoIt driver. They’re not Selenium Grid compatible yet, something to look into for the future. But at least they work for remote deployment. Though there is a way to make it work unofficially with Selenium Grid deployment of Selenium tests (but where these desktop UI servers are just not officially part of Grid as nodes) – this solution will be presented in a future blog post.

And last, I had wanted to complete with a three’s company big bang, but I couldn’t get Sikuli working just yet, maybe in the future…

https://github.com/daluu/SikuliDriverServer

So try them out, submit feedback, send pull requests with enhancements and bug fixes, etc.! 😉

P.S. all this made possible by the great work of others. Like the Appium team for the server base I used. And if/when I work on the Java server version – the Selenium team or ios-driver team, and .NET server version – Jim Evans for his Strontium server implementation. As well as the work of those who build great free or open source tools like AutoIt, Sikuli, AutoPy.

Update 6/5/2015: since my post, I found a new solution that’s better and more recent than Twin for desktop Selenium-style UI automation – Winium.

Update 9/27/2021: forgot to cross-reference update this post when I made a newer post regarding SilkAppDriver.