Archive | Automation RSS feed for this section

UI test automation of virtual app/browser delivery to local desktop

7 Jul

This thought just came to mind as I read through forum posts recently regarding automation scenarios like the following:

  • a web app or site is delivered to the user via a browser presented within a browser. e.g. via Citrix virtualization solutions delivered over a browser for desktop & browser-based apps.
  • a web or desktop app delivered to the user remotely via virtualization but made to feel as if it was installed, running, and deployed locally on the local machine, delivered via tools like MS Application Virtualization Client, Citrix, etc.

How do we test for these apps/cases, particularly say if we wanted to test against the exact deployment setup of how they are used? It would seem from the current infrastructure, this is likely a manual testing activity. And for automated testing, we are relegated to simplifying the test setup and assuming the simplified setup will behave the same as actual deployment, and just cover any differences with manual testing.

I bring this up as I know not of any existing automation solutions, commercial or free/open source, but particularly the latter, where these kinds of special deployment setups are supported for automation testing. As these deployments are delivered through virtualization technology, unless the test automation tools are integrated in that virtualization delivery (e.g. hosted and running in the same context as the app that’s delivered over virtualization rather than locally on the client receiving the virtualized app), the test tools won’t be able to detect and automate the virtual app. That integration is likely not feasible or easy to do due to platform isolation and security restrictions and sandboxing. It’s not quite the same as just running everything (the app under test & the automation tools) on a remote host and accessing it via remote desktop/VNC to see it in action.

Perhaps this isn’t much a concern for the community/industry until more and more people use this type of app delivery infrastructure where it is ubiquitous like mobile devices/apps are today. Still, something to ponder about for the future of testing. Wonder what is to come in this space for test automation. Or have I just been ignorant and we already have some tooling for this today?

Advertisements

A page object representing both desktop and mobile views or website and mobile app

2 Mar

It just occurred to me today, from reading a Selenium forum post, that a past blog post of mine can also apply to the following cases:

  • you have both desktop and mobile views for a website or web app (e.g. responsive design, etc.)
  • you have a website and a native mobile app that offers similar service/functionality

and as such in both these cases there is shared logic (user/app/site workflow, element locators – even if the actual values differ, but the “logical” representation is the same such as login button on both web site and mobile app, etc.).

Using the techniques defined in that other post, you can share locator references/variables and share page object methods and have a single page object represent both mobile & web versions. For sharing of methods with branching logic, instead of checking which A/B flow to go through, you check what platform you’re on and go to appropriate branch logic based on that.

Also on some specifics, I haven’t tested this type of implementation before, but in theory should work out even for cases where you have website and a native mobile app:

  • You just have to use the right driver instance in the page object methods (or instantiate the page object with the right driver instance), etc. based on the platform (e.g. Appium/ios-driver/etc. vs Selenium WebDriver).
  • For shared locators, XPath may work best, using the multi-valued locator approach with pipes “|”. That is because I know Appium (at least) supports XPath based locators. Not sure about CSS selectors. So, I’m hoping/assuming Appium supports the multi-valued XPath functionality. Don’t know about the other tools lke ios-driver, etc. If this doesn’t work out, you would need separate locator references/variables then.

Some of you might not agree, but this is one way to do it with code reuse via some manageable complexity. The alternative is to have separate page objects with separate methods and locators. That keeps things simpler but give you extra files, extra code, and some redundancy when some of that code and locators are kind of similar. In the end of course, choose whatever works best for you.

To dockerize your tests or not?

16 Sep

I recently started working with Docker. And Docker is the rage these days, with people trying to deploy and run their systems/applications as Docker containers. And how to test these new deployments, etc.

As a matter of fact, even Selenium is made available in Docker containers for test infrastructure:

https://github.com/SeleniumHQ/docker-selenium

https://github.com/elgalu/docker-selenium

It got me thinking, should we as test automation specialists take things further and also dockerize our tests (and test framework, tools, etc.) as a package? Something like pull down a docker container that has all your test environment preconfigured (Java, maven, Eclipse, JUnit/TestNG, etc.) for you to easily run functional/black blox/regression/integration/API/UI tests, etc. rather than set up your localhost environment with the right software and pull down test code from source control then run, or use a fat bloated virtual machine with all this preconfigured.

I made a post to StackOverflow for this but have yet to receive any feedback. Some folks upvoted my post though. Do you as my blog reader have any thoughts you’d like to share here or in my SO post?

Update 11/21/2015: Well, it’s nice to see an instance where Robot Framework is being dockerized. Searching online for “docker Robot Framework” brings up a few more results.

Update 02/12/2016: Came across these posts about Docker + (Ruby-based) BDD: Dockerizing BDD : Presentation at #BDDX15 Conference LondonDockerizing Cucumber-BDD and Ruby Friends

To re-invent the wheel or not? And testing frameworks…

16 Sep

Was reading a slide presentation today from another blog post: http://www.slideshare.net/abagmar/automate-across-platform-os-technologies-with-taas. Brought back memories of earlier work I did of defining a test automation framework: http://yadiytaf.sourceforge.net. I still think my YADIYTAF specification is a nice read. Although I never did create a reference implementation of the spec. I ended up prototyping what I needed at the time against the existing RobotFramework (RF). From that effort, I ended up expanding the functionality/feature set of RF by expanding it’s interoperability with Java, .NET, Perl, PHP via their remote library server interface implementing servers in those languages. So RF is a lot more powerful now since I first evaluated it years back.

In the end, makes me think, should one re-invent the wheel or not at times? Or see what’s already available and build upon that. I guess it depends on what works for you or not.

Complex test scenario example to truly evaluate test frameworks?

6 Jun

This came to mind in a discussion I read a while back, so thought I’d mention it. Not sure if I already mentioned it or not. But the general way we evaluate test frameworks, is rather simple. Or supposedly our system/application is not so complex to test after all, at least in terms of interactions among different component types, which themselves need to be UI/functional/user acceptance tested. For example, these days, most systems to test involve a website front end and mobile app client front ends, both of which talk to a server back end. And the website and mobile app clients don’t often need testing where they interact with each other (or rather say you test something on one client and verify on the other for example). Instead we simply test them standalone, independent of each other (or maybe mock the other side). And/or we got lucky in that it happens that the different clients (web, mobile) happen to be able to use the same platform (e.g. Java, Selenium-API) so that we can use the same framework to test them.

But truly testing the capabilities of a test framework is when you have a complex scenario that involves multiple components that need to be tested in integration (whether you mock them or not), and for the sake of simplicity in this argument, let’s say if you had to mock them, you mock them in the same language as the original component (not mocking to the platform of the desired test framework, etc.). Thus in a truly complex scenario/world, you won’t be able to have all things in Java, .NET, or whatever.

One example showcasing testing a system that comprised of the following components would be ideal: a website or web application/client, an iOS mobile app client, an Android mobile app client, a desktop GUI client, a SIP/VoIP client, perhaps a terminal/console client talking to server back end, perhaps networking communication w/ server on backend (some type of TCP/IP or other network communication), REST API web services utilized by the web/mobile/desktop clients, some native Java/.NET/etc. APIs to call. And let’s assume each of these components are built with different technology stacks and/or the tools to test them (not the framework) use different technology stacks.

Such scenarios may void certain test frameworks out of the box, unless one doesn’t mind writing custom glue logic to bind the different testing (components, tools) together within the single test framework. But a few frameworks do have better support built-in that allow you to integrate different tech stacks and tools to the framework, granted it’s not necessarily plug & play, but it minimizes the amount of effort in glue logic to integrate. I’ll leave it as an open thought on what frameworks are useful for such complex scenarios and which aren’t so much.

Integrating AutoIt, Sikuli, and other tools with Selenium when running tests in Selenium Grid

22 Jan

A true integration for tools like AutoIt, Sikuli, etc. with Selenium would be when they are able to run under Selenium Grid configuration just like Selenium. That would probably first require that they can run over WebDriver API, JSONWireProtocol as with AutoItDriverServer. But when Grid automatically selects which node to run your Selenium tests with, would it not do the same for AutoIt, except if you specifically request a certain capability set that let’s you know which exact node the test will go to. But anyhow, it might get confusing how to map/correlate which grid node runs the Selenium part of test vs the AutoIt/Sikuli part, etc. So maybe having AutoIt/Sikuli run under Grid configuration isn’t the whole answer either.

Regardless if AutoIt/Sikuli ran under Grid configuration or not, you could still deploy your Selenium tests that use AutoIt/Sikuli, etc. under Grid. I did blog about this a while back but never gave concrete examples. In StackOverflow tradition, people seem to want specific examples. So I’ve finally worked up a simple illustrative example.

To have AutoIt/Sikuli work with Selenium while it’s running under grid mode, you need to find a way to determine the node host that runs the Selenium tests at runtime, so that you can then call/run AutoIt/Sikuli code on the same target node host, so that everything runs on the same correct machine (rather than the dilemma novices encounter where Selenium executes on the remote node but AutoIt/Sikuli runs on the localhost, causing mismatch and test fails).

The easiest example solution is to extract out the node host information from the Selenium Grid API by providing the WebDriver session ID that you first need to extract. Once you have that information, you can then use various ways to remotely execute AutoIt, Sikuli against the node host that you’ve extracted. All this is presented in this Github gist:

https://gist.github.com/daluu/3b4746f6b672c49f7e8f

of all the various ways in the sample gist, I personally prefer calling AutoIt, Sikuli over WebDriver API, where possible rather than to resort to using PSExec.exe, SSH, etc. I’d go for web service route as alternative to WebDriver API. Sticking with PSExec.exe type option as last resort. But you are free to pick whichever option works best for you.

WebDriver API and JSONWireProtocol is not just for web and mobile applications testing, it can be for desktop too!

19 Jan

And so, here’s a blog post about just that. I’ve found that the WebDriver API in general, with regards to common location strategies of ID, name, class, tag, and kind of XPath, along with element manipulations–such as click, type/sendKeys, get/set text, get attributes, mouse operations, key up/down, taking screenshots, finding elements & validating properties (enabled, visible, selected)–all that being common across web, mobile, and desktop.

Selenium/WebDriver started off for web applications. Then came along Appium, ios-driver, etc. which expanded it to the mobile space, first iOS then Android, and now somewhat even Windows Phone.

But there has been very little in the desktop area. The first for it was Appium for Mac. And we’ve seen nothing else since. Though if you consider unofficial Selenium/WebDriver-like APIs, then maybe there’sa few more: Twin and sikuli-remote-control. But now, I’ve worked out some proof of concept prototypes (based off the old Appium Python implementation) to give you more options to desktop automation using WebDriver API and for Windows, not Mac!

https://github.com/daluu/AutoItDriverServer
https://github.com/daluu/AutoPyDriverServer

Check them out. Pretty interesting. The AutoIt one has a Selenium integration demo showing automation of AutoIt and Selenium together against websites using 2 drivers – a “web” driver and an AutoIt driver. They’re not Selenium Grid compatible yet, something to look into for the future. But at least they work for remote deployment. Though there is a way to make it work unofficially with Selenium Grid deployment of Selenium tests (but where these desktop UI servers are just not officially part of Grid as nodes) – this solution will be presented in a future blog post.

And last, I had wanted to complete with a three’s company big bang, but I couldn’t get Sikuli working just yet, maybe in the future…

https://github.com/daluu/SikuliDriverServer

So try them out, submit feedback, send pull requests with enhancements and bug fixes, etc.! 😉

P.S. all this made possible by the great work of others. Like the Appium team for the server base I used. And if/when I work on the Java server version – the Selenium team or ios-driver team, and .NET server version – Jim Evans for his Strontium server implementation. As well as the work of those who build great free or open source tools like AutoIt, Sikuli, AutoPy.

Update 6/5/2015: since my post, I found a new solution that’s better and more recent than Twin for desktop Selenium-style UI automation – Winium.

technolinchpin

Inspire and spread the power of collaboration

Niraj Bhatt - Architect's Blog

Ruminations on .NET, Architecture & Design

Pete Zybrick

Bell Labs to Big Data

Seek Nuance

Python, technology, Seattle, careers, life, et cetera...

TELLURIUM

New Era of Test Automation

Der Flounder

Seldom updated, occasionally insightful.

The 4T - Trail, Tram, Trolley, Train

Exploring Portland with the 4T

Midnight Musings

Thoughts on making art

Automation Guide

The More You Learn The More You Play...!

The Performance Engineer

Code.Test.Tune.Optimize.

humblesoftwaredev

Thoughts related to software development

Yi Wang's Tech Notes

A blog ported from http://cxwangyi.blogspot.com

Appium Tutorial

Technical…..Practical…..Theoretically Interesting

LinuxMeerkat

I swear! Meerkats can do Linux

PacketsDropped

Requeuing the packets dropped in my memory.

Two cents of software value

Writing. Training. Consulting.