A/B testing and Selenium automation

12 Sep

Don’t know how many people have to deal with this particular topic. But I had to research and deal with it before, so thought I’d mention a few things and share some links.

Whatever the case, unless you need to specifically “test” the A/B testing system/framework in terms of how it allocates/routes users to one flow or another, statistical reporting, and other things, it is best & simplest in terms of UI test automation to just be able to go through both A and B flows during testing.

Some people have mentioned to disable A/B test (e.g. go with A/old/base flow always), or workaround A/B testing. You can’t always work around it, and unless you can simply or easily disable A/B testing, it is simply just going through one of the 2 flows alwasy. In which case, it’s technically not that hard to set up the test framework to be able to switch flows.

Usually the A/B test framework/system has a mechanism to assign flows including always forcing to a particular flow, but each framework/system implementation is different. You would utilize that capability to select the flow to test in Selenium. By the way, usually the implementation involves, cookies, javascript, or URL manipulation. The way I’ve done it so far is to define a table/spreadsheet that defines the flow for each A/B test of interest to specify to Selenium test which flow to force going through. You can have variations of the spreadsheet (different files, different Excel worksheet tabs, different SQL DB tables, etc.). It would also be helpful to build into your framework the ability to query for what the defined/specified flow is for going through so the test knows what to do, as well as query for what the actual flow (as returned by the site/system) is so as to confirm the automation has correctly force set the desired A/B test flow. How all that is done is dependent on your A/B test system.

As for how you implement handling the A/B test flows, it is easiest to tackle if you use page objects. I would recommend encapsulating all/most of the A/B flow checking within the page object class methods, so that the tests themselves and the test author doesn’t need to know the details. That way the test can implicitly support both A/B flows as it is written. The only case you detract from that is when the actual workflow changes between A/B flows (e.g. a functional/workflow change not a UX/visual/icon change) or when you need to verify specifically UX elements like images, icons, and text between the flows. In such case you would then put A/B handling logic in the test (or test utility/parent classes) itself.

Handling A/B flow is simply control and branching logic (if/else/switch/case statements) in checking what flow is to be traversed and handle appropriately based on the given flow.

A tip on handling UI element locators in A/B testing is that one can minimize creating additional locators for an A element and B element, by instead using multi-value locators. This does then force you to use CSS selectors or XPath instead of simply by ID, name, etc. These 2 types of location strategies support defining multiple values and as long as one value matches, then you have found a match for a WebElement.

CSS: a flow value, b flow value

XPath: a flow value | b flow value

as you see in example above, the multi value separator in CSS is a comma and in XPath is the horizontal pipe character.

Also when handling A/B testing within page object methods and attempting to not change a test to support both flows, that means you typically modify a page object method to internally check what flow it should go through and handle it, so that the caller of the method (usually a test, or another page object method) does not need to know about A/B test.

Following my suggestions, it will be easier to manage the A/B testing in progress with minimal code changes across files. And the cleanup work will be easier (when you decide to stick with one flow 100 percent). Cleanup is simply then a removal of branching flows and the obsolete branches and deleting one of the values in the multi-valued locator variables/constants. The locator variable/constant (names) and the page object method signatures all remain the same, and tests themselves may require very little if any update at all.

And now here’s some links to topics on A/B testing that I have seen online that you might find useful:








IDEs and fancy development tools can be bad for you

23 Aug

Especially for testers (at least those without a good developer/hacker type skillset), you become reliant on them. And when you want to do something outside of them, you are lost.

I’ve seen quite a few posts from testers about how to execute Java (Selenium/TestNG) tests outside of Eclipse, and they usually have maven set up with Eclipse too.

As someone who likes to have options & builds tools & utilities from (glue) scripts and custom mini applications, whether for testing or not, it irritates me to see those questions pop up.

Either the poster is ignorant & an idiot for not knowing how to search up the solution (one search may not give you the entire solution, but search up the components that will make up your solution individually and you can piece together the puzzle to solve), or they are plain lazy to just want someone to solve it for them.

Those who work with Java toolset should know how to compile Java from scratch on command line, how to execute maven from command line, how to execute JUnit and TestNG tests from the command line test runner.

Those who work with .NET/C# should know how to compile the code from scratch on command line, how to execute NUnit/etc. tests from command line test runner, and the similar equivalents to what you do in Java.

Those who work with scripting languages should know how to run code from files on command line, run code from interpreter via command line (in GUI interpreter mode or pure command line).

Those who work with any language should know how to use libraries/packages/modules, installing, referencing them, etc. all without the fancy IDE – just command line compilation, installation, etc. Should also know how to use methods & functions of libraries/packages/modules without auto-complete in IDE, rather simply by looking up quick references and API documentation instead.

Those who work on Windows should know how to work with batch files and the Windows task scheduler. Even better to learn WSH, WMI, VBScript and/or Powershell.

Those who work on *nix should know shell scripting and cron jobs at least.

Selenium page objects can be implemented outside code and for keyword driven testing

4 Aug

FYI, Selenium page objects don’t have to be implemented in code and can be adapted for keyword driven test frameworks, provided they have the right features to support page object structuring. One such framework for this is Robot Framework. If doing this from another framework or custom framework, one might have to build out the necessary missing support features.

Here’s an example implementation I did that is an approximate mapping or conversion of page objects in Java code to Robot Framework non-code based page objects, the example has links to other implementation examples for Robot Framework as well.


Safari 5 on Windows may not be able to perform multiple file uploads in parallel

8 Jul

Just wanted to point this out for those who do testing and test automation. Although we’re supposed to be going off Safari on Windows and test directly Safari on Mac. But I’m sure there are still some people who use Safari 5 on Windows to do interim/tentative testing due to lack of resources of a Mac or because it’s handy to have around.

As such, just wanted to point out that Safari 5 on Windows appears to have an issue with uploading multiple files in parallel via the HTML5-based multiple file upload support. At least in the implementation of a site I test.

You won’t run into this issue with Safari on Mac. So just something to be aware of.

Debugging mobile and tablet websites on iOS on Windows or locally on device

13 Jun

I was doing some research and came across these useful tools one can use to debug and analyze mobile/tablet versions of websites/web applications in the case one doesn’t have a Mac handy to do the remote Safari debugging.

Remote Safari debugging on Windows

You can try option of using Telerik AppBuilder (Windows client) as a replacement on Windows for Safari debugger on Mac when remote debugging. There’s a nice blog post about the steps to do it in link below.


The process of using it is much the same as with Safari remote debugging. Have the app open, connect device to Windows via USB, have Safari open on device, then browse device in the app, and open debugger in the app. The tool also will require you to create a Telerik account, and either load/use the default app or create a blank (javascript?) app just to use for debugging mobile Safari on your device.

The tool requires a license or you can use the trial, which becomes a starter edition afterwards. I think the starter edition will still allow you to do the debugging.

NOTE: using this method may require you to have the iOS device drivers to be able to interact with it from AppBuilder. You can get the device drivers by installing iTunes, but if you want to avoid that bloat, you can try to get just the drivers by searching online or downloading iTunes and extracting out the drivers from the installer so you only install the drivers rather than the whole iTunes package, both of which are not covered in this blog post.

Local device debugging on iOS/iPhone/iPad

You can also try these iOS apps below, you can find them in the iTunes App store. They give you built in developer tools like feature (right on iOS, no remote debugging needed) that mobile Safari doesn’t offer. However, they are not the native mobile Safari view/app, though I think it might use the internals of mobile Safari/Webkit to be close enough. These apps are useful when you don’t have a Mac or Windows PC around to do remote debugging with, so you can debug locally on the device through the apps.

MIH Tool – basic edition

HTTPWatch Basic – HTTP Sniffer and Debugger

I gave them a try and they’re at least better than the mobile Safari you get on iOS, unless one needs to target full mobile Safari compatibility. I’m guessing the pro/paid editions of those apps give you more/better features, I confirmed that was the case for HTTPWatch.

Update 7/30/14 – remote Safari debugging that is open source & cross platform

Found this today https://github.com/google/ios-webkit-debug-proxy. Works for Linux, Mac, and Windows support eventually to come. I haven’t tested this out myself though. However, except for tinkering with things, cross platform, and open source. I’d say this option is best when working under Linux. On Mac, better to go with the native support from Apple. And for now, Windows is better with the option presented above with an easy GUI for novices.

Update 8/8/2014 – another Safari remote debug for Windows

Adding on previous update, found a Windows port: https://github.com/artygus/ios-webkit-debug-proxy-win32

Testing with your own environment vs a shared environment

27 May

I do wonder, within QA among all the different organizations/companies, does QA generally test on a shared (test) environment, or have their own testbed environment to work with, or both?

It may be organization/industry specific, as certain fields/areas work better where you have individual testbeds/environments to work with and some are better with a shared environment. The contraints and decision are often based on:

  • resource efficiency/utilization – will people be more productive testing with their own environment or a shared one? Can the environment be shared easily for testing? Is it easy to deploy & manage the environment individually by anyone or is it easier to manage centrally as a shared environment? Cost to deploy individual vs shared environment
  • corporate policy – what are the organization’s IT policy/policies that affect environment setup for testing?
  • test data setup/sharing, reproducibility of bugs – whether better with individual environment or shared
  • codebase sync of test/release branch code with the environments with scheduling/agile in mind – easier with individual vs shared environment
  • hmm…any others that come to mind?

But what’s interesting to me though, is that regardless of the QA test setup, developers will usually have their own environment rather than a “shared” development environment. Why is that? More productive & effective to develop individually before you merge all code to a central codebase & environment for testing? But I could be wrong with this assumption as I’ve not worked at too many places but so far have noticed devs do get their own environments at the places I’ve worked. So perhaps QA should get their own environments too if devs get their own. A hybrid approach can be useful though, shared use environment and individual testbed environments, so you get the best of both, though it does cost more this way.

What are your thoughts on this? I personally prefer the individual environment or hybrid option. It’s nice to have more control of the environment that you test, as long as you are smart & responsible on how you configure and test the system.

From personal experience I can provide the following insight:

  • if the industry/organization deals with hardware, you are likely to get your own testbed environment to configure & manage & test. There may be a shared environment as well where multiple people use it, perhaps in real world type scenario (e.g. PBX phone system with all organization members having an account, phone, and access to that system to test and use what they build themselves)
  • if the industry/organization is about website and/or web applications/services, you are likely to get a shared test environment, individual might not be likely though is awesome if you do get that. The shared environment may or may not be managed by QA, it could be managed by IT, DevOps, or Release Engineering, etc.

Selenium page object modeling challenge – are you up for it?

11 Feb

Having seen some good & bad page object model code & especially after having read Clean Code book, I sure wish there was some book, presentation, blog post, etc. that went over in more detail about the page object model beyond the simple examples presented in various online articles and the official reference example.

Something that presents case studies of good implementations, bad ones, complex ones (ones that you’d expect novices would never implement or have trouble implementing), and examples/tips on how to fix bad implementations into good ones. For bad/complex ones, would be nice to cover things like:

  • shared page widgets, global (or somewhat global) headers, footers, etc.
  • templated/master/skeleton pages where other pages & sites inherit from (similar flow & UI but where the final sites may differ from one another in minor aspects on some flow or UI element implementation/definition like CSS/JS specification)

if such good one presentations already exist, please point me to them. If not, I urge my fellow Selenium bloggers (especially those that follow my blog), fellow Selenium users/contributors/developers, my blog followers/readers to try and take on this challenge (find such a presentation or come up with one to present online, at some meetup, a code camp, or upcoming official Selenium conference).

I would think such a presentation will greatly benefit the Selenium user community, particularly the novice users who lack the initial software development/architectural design skills to appropriately model page objects with the right level of abstraction.

I do understand such a task may not be easy as it requires finding/sharing code that might be confidential or intellectual property (corporate code) unless reworked as generalized code that could be shared publicly. Unless of course we can find some good open source Selenium test code that happens to be bad or complex (or good) implementations to show as case studies. We need to find and present such code in a central location (as presentation, etc.) because it’s labor intensive to scour the internet OSS codebase for Selenium code examples as case studies (for each person to do individually).

What are your thoughts on this? My ideal thoughts are we need a Clean Code type book for Selenium that covers page objects in detail. A sort of “Clean Selenium Page Objects” or “Clean Selenium Code” book.

On that note, maybe you might have code you can share to use for such a case study presentation? Or maybe you could try to get your organization to approve a generalized version of your Selenium page object code (not all of it but some example excerpts) to use for a public case study?

Mary Ann M-P's Blog

Selenium: My Take


Just another WordPress.com site


All about coding!

Watir WebDriver

the most elegant way to use webdriver with ruby


Automated Tests and other stuffs

The Guide On Asian Male Interracial Dating


Musings on Integration with Oracle Fusion Middleware


A blog on Test Automation, Technical QA, Testing Tools & Techniques, Test Architecture and more…


Making the QA World a Better Place!


Notes about testing, web development, etc...

Burdette's Test Blog

All about how I test software.

VBA 4 all

programming in VBA

Correct Toes

Align. Strengthen. Restore.

The Informed Healthnut

Science-based health information, recipes, and life!

Don't Make the Same Mistake Twice

best and worst development practices

Adventures in QA

Software Testing, Mobile Testing and Agile Development by Daniel Knott

Muses Of IT Engineer

The Learnings of a QA engineer


some nerd writing about computery thingies


Get every new post delivered to your Inbox.

Join 51 other followers