A/B testing and Selenium automation

12 Sep

Don’t know how many people have to deal with this particular topic. But I had to research and deal with it before, so thought I’d mention a few things and share some links.

Whatever the case, unless you need to specifically “test” the A/B testing system/framework in terms of how it allocates/routes users to one flow or another, statistical reporting, and other things, it is best & simplest in terms of UI test automation to just be able to go through both A and B flows during testing.

Some people have mentioned to disable A/B test (e.g. go with A/old/base flow always), or workaround A/B testing. You can’t always work around it, and unless you can simply or easily disable A/B testing, it is simply just going through one of the 2 flows alwasy. In which case, it’s technically not that hard to set up the test framework to be able to switch flows.

Usually the A/B test framework/system has a mechanism to assign flows including always forcing to a particular flow, but each framework/system implementation is different. You would utilize that capability to select the flow to test in Selenium. By the way, usually the implementation involves, cookies, javascript, or URL manipulation. The way I’ve done it so far is to define a table/spreadsheet that defines the flow for each A/B test of interest to specify to Selenium test which flow to force going through. You can have variations of the spreadsheet (different files, different Excel worksheet tabs, different SQL DB tables, etc.). It would also be helpful to build into your framework the ability to query for what the defined/specified flow is for going through so the test knows what to do, as well as query for what the actual flow (as returned by the site/system) is so as to confirm the automation has correctly force set the desired A/B test flow. How all that is done is dependent on your A/B test system.

As for how you implement handling the A/B test flows, it is easiest to tackle if you use page objects. I would recommend encapsulating all/most of the A/B flow checking within the page object class methods, so that the tests themselves and the test author doesn’t need to know the details. That way the test can implicitly support both A/B flows as it is written. The only case you detract from that is when the actual workflow changes between A/B flows (e.g. a functional/workflow change not a UX/visual/icon change) or when you need to verify specifically UX elements like images, icons, and text between the flows. In such case you would then put A/B handling logic in the test (or test utility/parent classes) itself.

Handling A/B flow is simply control and branching logic (if/else/switch/case statements) in checking what flow is to be traversed and handle appropriately based on the given flow.

A tip on handling UI element locators in A/B testing is that one can minimize creating additional locators for an A element and B element, by instead using multi-value locators. This does then force you to use CSS selectors or XPath instead of simply by ID, name, etc. These 2 types of location strategies support defining multiple values and as long as one value matches, then you have found a match for a WebElement.

CSS: a flow value, b flow value

XPath: a flow value | b flow value

as you see in example above, the multi value separator in CSS is a comma and in XPath is the horizontal pipe character.

Also when handling A/B testing within page object methods and attempting to not change a test to support both flows, that means you typically modify a page object method to internally check what flow it should go through and handle it, so that the caller of the method (usually a test, or another page object method) does not need to know about A/B test.

Following my suggestions, it will be easier to manage the A/B testing in progress with minimal code changes across files. And the cleanup work will be easier (when you decide to stick with one flow 100 percent). Cleanup is simply then a removal of branching flows and the obsolete branches and deleting one of the values in the multi-valued locator variables/constants. The locator variable/constant (names) and the page object method signatures all remain the same, and tests themselves may require very little if any update at all.

And now here’s some links to topics on A/B testing that I have seen online that you might find useful:

https://groups.google.com/forum/#!msg/selenium-users/5t3vnPoUXzA/3glQ6kT9oZkJ

http://stackoverflow.com/questions/5269027/how-do-you-handle-testing-with-selenium-when-you-are-running-ab-tests

http://stackoverflow.com/questions/12078675/writing-integration-tests-against-external-resources-which-are-a-b-testing

http://elementalselenium.com/tips/12-opt-out-of-ab-tests

https://speakerdeck.com/neurites/accessibilitytesting

http://sauceio.com/index.php/2011/07/easiest-ever-ab-split-tests/

Update 3/1/2016: it just occurred to me today, from reading a Selenium forum post, that this blog post of mine can also apply to a case where you have both desktop and mobile views for a website or web app (e.g. responsive design, etc.) or a case where you have a website and a native mobile app, and as such in both these cases there is shared logic (user/app/site workflow, element locators – even if the actual values differ, but the “logical” representation is the same such as login button on both web site and mobile app, etc.). Using the techniques defined here, you can share locator references/variables and share page object methods and have a single page object represent both mobile & web versions. For sharing of methods with branching logic, instead of checking which A/B flow to go through, you check what platform you’re on and go to appropriate branch logic based on that.

Advertisements

3 Responses to “A/B testing and Selenium automation”

  1. Quentin November 5, 2014 at 4:29 pm #

    Insteresting post, I disagree with it though.
    A test should have condition and expectations.
    If you click a button and get a page you do not expect, this could be concidered a bug.
    Also, A/B testing is only temporary, which means you have to mess with the automation code, possibly adding a lot of IFs, only to remove it afterwards. Which can (will) introduce bugs or flakyness.
    So I would say a better approach is to force your system to always go A or always go B.
    With the automation code, you can add parameters to the url or use cookies, and set up the A/B testing system to check these values and alway go one way if they are present.

    • autumnator November 5, 2014 at 8:29 pm #

      Thanks for the comment, appreciated. In response to your feedback:

      Yes, a test should have conditions & expectations. Implementing A/B test support is no different. It just depends on what the test is supposed to test. Bear in mind that some A/B testing changes UI or minor functionality so clicking a button takes you to same page. Only the page looks different, and the test in question might not be testing anything specific regarding the A/B change, only that it needs to go through some flow that requires clicking yet another button on the A/B changed page that will break if run under new B flow, but not in original A flow, because the button locator definition changed.

      So continuing from that example, one can either create a separate new test just to be able to handle B flow, don’t test under B flow (always under old A flow) and we manually cover B flow, or implement some A/B test support such that you can handle both A & B flows (in this example, just being able to click the needed button on page X for both A & B flows because we need to get to page Y for the test and both A & B flows eventually go to page Y and the test itself is concerned with testing page Y by way of going through page X and some other pages first). Which choice you take may depend on your organizations needs. Obviously the best option is always to go to one flow and not bother with the other, but you can’t always do that.

      It’s nice to force system to always go A or B. But sometimes you need to be able to do both if your organization requires it. Run tests under A flow and later B flow. So you either have separate specific tests for each or do code reuse with the type of A/B support I mention where there is commonality between the two. However, based on what I’ve found online and from your response, it looks like most organizations don’t have this complex need so they can easily just pick to automate for A or B flow only. Perhaps where I work, we’re more unique. I can’t divulge much details other than that our A/B tests that we run on the site may involve just UI changes, functional changes, or even both together. So we have need to support both flows, particularly the functional change tests, during the A/B test run duration period. In our A/B tests the new B flow is the target to roll out eventually but we play it safe in case we need to roll back to the old A flow, so as QA we need to be ready to support either case that we will end up with in terms of automation support. And there can be much regression coverage here to support, though most of that is for common code (user flow, shared locators with different values depending on A/B flow). Obviously, it would be nice for others to share how they do A/B testing in a bit more detail in case they have situation similar to ours. Because simply saying force go through just one flow with cookes, URL, etc. doesn’t really tell you much. Hence why I wrote this post to at least give readers a different viewpoint from the norm, even if it’s not a favored or common approach.

      Last, one can assume force setting A/B flow will always work after setting cookie, etc. so the automation framework doesn’t need any special utility code to check if it’s in the right flow. Which makes sense. But nice to have just in case, as the A/B test system can’t be guaranteed to work properly just as a site or software may have bugs. But of course, in such case, we can also just let test fail normally and manually analyze/debug the test why it failed (and end up finding that it didn’t go through right A/B test flow even though we forced it).

      I would also like to mention sometimes browsers may be finicky, causing problems for A/B or cookies as well. For example, Selenium issue 5503 gave us some trouble with supporting A/B (even force setting to just one flow) until we figured out the core problem because stupid Safari didn’t complain with error but instead just not work right. That was why we particularly added some feature to be able to check the A/B flow a test is in just in case.

Trackbacks/Pingbacks

  1. A page object representing both desktop and mobile views or website and mobile app | autumnator - March 2, 2016

    […] just occurred to me today, from reading a Selenium forum post, that a past blog post of mine can also apply to the following […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

ÜberDev

open notebook

a happy knockout mouse.

my journey into computer science

Perl 6 Advent Calendar

Something cool about Perl 6 every day

technolinchpin

Inspire and spread the power of collaboration

Niraj Bhatt - Architect's Blog

Ruminations on .NET, Architecture & Design

Pete Zybrick

Bell Labs to Big Data

Seek Nuance

Python, technology, Seattle, careers, life, et cetera...

TELLURIUM

New Era of Test Automation

Der Flounder

Seldom updated, occasionally insightful.

The 4T - Trail, Tram, Trolley, Train

Exploring Portland with the 4T

Midnight Musings

Thoughts on making art

Automation Guide

The More You Learn The More You Play...!

The Performance Engineer

Code.Test.Tune.Optimize.

humblesoftwaredev

Thoughts related to software development

Yi Wang's Tech Notes

A blog ported from http://cxwangyi.blogspot.com

Appium Tutorial

Technical…..Practical…..Theoretically Interesting

%d bloggers like this: