Polymorphic security measure could be a pain for Selenium if running tests with that enabled

7 Oct

Came across this site today:

https://www.shapesecurity.com/technology/

Interesting technology. But if the site to be tested uses real time obfuscation of the (form) element IDs, names, classes, etc. that could be a pain to automate tests against since you’d have to get a handle on the element location to perform the automation. Perhaps one would need or use a backdoor option to disable the obfuscation when running automation to test the site normally.

Either that or utilize an API provided by that solution perhaps that lets you real time map original element location identifiers to the obfuscated ones to use with Selenium at run time.

Be interesting to hear of anyone automate testing of a site that uses such technology.

Organizing tests by repetition or granularity?

24 Sep

I’ve seen this often in my job covering test scenarios manually as well as adapting those tests for automation. Have others come across same issue?

In terms of manual, and more correctly, exploratory testing, as a human tester, you can choose how you perform your tests and it is often logical to consolidate areas of testing (or in documentation parlance, “test cases”) around things that would otherwise be repetitive. e.g. re-using same test data to test slightly varying scenarios. Where workflow or sequence of test steps are common across several tests, you may as well run them sort of together like multitasking rather than separately one after another, repeating the common steps all over again.

One example is testing a workflow that happens to be mostly same across desktop, tablet, mobile. Only UX or UI is different. Do the test 3x? Or run test nearly 1x by testing all 3 at same time where/when possible using one test/path to do most of the workflow and verifying across all 3 paths at the critical points.

But that’s easy to do for exploratory (and undocumented manual) testing. But what do you do when it comes time to put this on pen & paper for others like an offshore team to execute said tests, who are one or more connections/layers away from the intimacy of the feature being test than you are? And then converting such documented tests to automation.

Do we strive to follow the human (exploratory test) nature/convention of consolidating the test coverage around the repetitive workflow and shareable test data minimizing amount of test cases and test scripts to create OR strive to keep granular individual specific tests for tracking/auditing purposes and simplify or attempt to idiot proof things, even though it will be in ways redundant in repeating test steps and generating more repeated tests data that could have been shared/reused otherwise?

One argument could be to do it granular since automation and/or hardware is cheap. But it turns out unless your site is 1990s basic HTML, repeating test steps can still be slow against modern websites/applications against modern browsers. And running tests in parallel helps with this but adds complexity and still can take long time once your test codebase grows big. So in some cases, maybe consolidating testing isn’t necessarily bad, you have to weight the pros & cons with short & long term automation coverage/infrastructure in mind.

Just wondering what experiences and approaches folks have taken in the real world.

Selenium page objects beyond pages like a cart object?

23 Sep

Selenium page objects is a design pattern to help you model test code better. But one doesn’t have to follow the full guidelines of the design pattern.

Some people have used it to model parts of pages as well (headers, footers, navigation, templates, widgets, etc.).  But perhaps it can be useful for more than that, and some people might have already done this or not, as I couldn’t really search anything up or don’t know what/how to search specifically for this. If people have already done this, they haven’t widely publicized it.

What I have found so far is this: https://github.com/cheezy/page-object/wiki/Indexed-Properties

It’s an interesting piece to review. I had this similar thought in mind recently and decided to blog about it:

A shopping cart page doesn’t really do much really. It contains cart items, and offers a visual call to action (click button) that then takes you to checkout. That along with standard site (header/footer/navigation) template actions (login, logout, links to other areas of site).

The core functionality in the shopping cart page really belongs to the cart items and what you can do with them. So in my mind, having the shopping cart page object manipulate cart item actions doesn’t seem quite appropriate for object oriented design.

For example, this would be how you might typically implement the cart page in basic page object model:

cartPage.updateQuantity(cartItemIndex, quantity);
cartPage.removeCartItem(cartItemName);
cartPage.getCartItemName(cartItemId);
//obviously access cart item by index, text string name of item, or by some unique ID

However, perhaps you can extrapolate the cart items outside of the page object to manipulate individually as a collection or set of related WebElements (name, quantity field, remove button, etc.) or for more advanced usage as an encapsulated cart item object model itself.

Both modeling options are presented here below (since I find it hard to list code in a “basic” WordPress blog)

https://gist.github.com/daluu/8e4c43d653c07eb0c57a

I would note that modeling cart items in such a way, while being more object oriented, can make implementing the cart page object (particularly the getCartItems method) and associated procedure to locate and group the related cart item elements together functionally more complicated. Because often, the web application will not have an easily implemented UI that has all the related elements easily related and uniquely identifiable to each other, especially on a cart page with N number of cart items.

Usually that may require sophisticated use of CSS and XPath patterns to locate and relate the set of elements for N cart items on the page. So this this whole approach is not something novice page object model and Selenium users can easily tackle. It takes time and skill to do. But worth trying out.

In the long run, I feel this type of approach is more maintainable, scalable, and makes the tests more readable. It just requires more thought in architectural design and more work upfront to implement. However, the complexity to implement could be reduced if you can get the developers to make the element locator values easily defined w/o resorting to custom CSS and XPath, and make it work for N cart items, and X related cart item elements (e.g. item color, item description, item this, item that, for every cart item)

If you ask me personally which cart item object model I prefer, it is the latter one that resembles a page object rather than the one that is simple a container of WebElements for a cart item.

What are your thoughts on modeling things or objects on a page like a page object. Cart item is the one that tends to come to mind, but there are others, for which they can be thought of as objects but not widgets, nor headers, footers, or navigation for page object modeling. Some other possible examples include a search result, a category listing, etc.

Also, please do inform me if you come across other articles about using page objects for things like cart items, search results, etc. where we’re not working with a page but some other object per se.

Selenium was jokingly meant as a cure for QTP? How effective has the cure been?

23 Sep

I had found that out some time ago in a social media post. Here’s a reference about it on wikipedia:

http://en.wikipedia.org/wiki/Selenium_(software)

How effective would you say the cure is?

I’d say making progress, but we’re not cured of QTP just yet…Appium + WebDriver + JSONWireProtocol is making that more achievable across all types of automation for the cure…right now we just lack a “WebDriver” for the desktop GUI. As we’ve got mobile covered (Appium, Selendroid, ios-driver, and other similar tools), and web was long covered by Selenium RC and WebDriver.

We kind of have for desktop: Twin project

https://code.google.com/p/twin/

https://www.youtube.com/watch?v=NiCCcf5HKQI

http://www.slideshare.net/videotuition/twin-automation-tool-automating-windows-applications-selenium-web-driver-grid

 

but it’s nowhere near the prominence of a household name as Selenium or Appium in regards to desktop GUI automation. I personally don’t know how well that tool works either as I’ve not tried it out yet. But until that tool gains prominence or a new project comes along, there will always be QTP and like commercial tools for the following:

  • desktop GUI automation
  • integrated or crossover automation that involves desktop + web/mobile
  • automation for things Selenium can’t control (HTTP authentication popup, file browse popup, download popups, etc. for which in OSS world you use tools like AutoIt or Sikuli with Selenium to bridge the gap)

I would say though that if I had to go with desktop GUI automation and wasn’t stuck on having to use an existing tool, I’d go for Twin and at same time improve it where and however I can.

A/B testing and Selenium automation

12 Sep

Don’t know how many people have to deal with this particular topic. But I had to research and deal with it before, so thought I’d mention a few things and share some links.

Whatever the case, unless you need to specifically “test” the A/B testing system/framework in terms of how it allocates/routes users to one flow or another, statistical reporting, and other things, it is best & simplest in terms of UI test automation to just be able to go through both A and B flows during testing.

Some people have mentioned to disable A/B test (e.g. go with A/old/base flow always), or workaround A/B testing. You can’t always work around it, and unless you can simply or easily disable A/B testing, it is simply just going through one of the 2 flows alwasy. In which case, it’s technically not that hard to set up the test framework to be able to switch flows.

Usually the A/B test framework/system has a mechanism to assign flows including always forcing to a particular flow, but each framework/system implementation is different. You would utilize that capability to select the flow to test in Selenium. By the way, usually the implementation involves, cookies, javascript, or URL manipulation. The way I’ve done it so far is to define a table/spreadsheet that defines the flow for each A/B test of interest to specify to Selenium test which flow to force going through. You can have variations of the spreadsheet (different files, different Excel worksheet tabs, different SQL DB tables, etc.). It would also be helpful to build into your framework the ability to query for what the defined/specified flow is for going through so the test knows what to do, as well as query for what the actual flow (as returned by the site/system) is so as to confirm the automation has correctly force set the desired A/B test flow. How all that is done is dependent on your A/B test system.

As for how you implement handling the A/B test flows, it is easiest to tackle if you use page objects. I would recommend encapsulating all/most of the A/B flow checking within the page object class methods, so that the tests themselves and the test author doesn’t need to know the details. That way the test can implicitly support both A/B flows as it is written. The only case you detract from that is when the actual workflow changes between A/B flows (e.g. a functional/workflow change not a UX/visual/icon change) or when you need to verify specifically UX elements like images, icons, and text between the flows. In such case you would then put A/B handling logic in the test (or test utility/parent classes) itself.

Handling A/B flow is simply control and branching logic (if/else/switch/case statements) in checking what flow is to be traversed and handle appropriately based on the given flow.

A tip on handling UI element locators in A/B testing is that one can minimize creating additional locators for an A element and B element, by instead using multi-value locators. This does then force you to use CSS selectors or XPath instead of simply by ID, name, etc. These 2 types of location strategies support defining multiple values and as long as one value matches, then you have found a match for a WebElement.

CSS: a flow value, b flow value

XPath: a flow value | b flow value

as you see in example above, the multi value separator in CSS is a comma and in XPath is the horizontal pipe character.

Also when handling A/B testing within page object methods and attempting to not change a test to support both flows, that means you typically modify a page object method to internally check what flow it should go through and handle it, so that the caller of the method (usually a test, or another page object method) does not need to know about A/B test.

Following my suggestions, it will be easier to manage the A/B testing in progress with minimal code changes across files. And the cleanup work will be easier (when you decide to stick with one flow 100 percent). Cleanup is simply then a removal of branching flows and the obsolete branches and deleting one of the values in the multi-valued locator variables/constants. The locator variable/constant (names) and the page object method signatures all remain the same, and tests themselves may require very little if any update at all.

And now here’s some links to topics on A/B testing that I have seen online that you might find useful:

https://groups.google.com/forum/#!msg/selenium-users/5t3vnPoUXzA/3glQ6kT9oZkJ

http://stackoverflow.com/questions/5269027/how-do-you-handle-testing-with-selenium-when-you-are-running-ab-tests

http://stackoverflow.com/questions/12078675/writing-integration-tests-against-external-resources-which-are-a-b-testing

http://elementalselenium.com/tips/12-opt-out-of-ab-tests

https://speakerdeck.com/neurites/accessibilitytesting

http://sauceio.com/index.php/2011/07/easiest-ever-ab-split-tests/

 

IDEs and fancy development tools can be bad for you

23 Aug

Especially for testers (at least those without a good developer/hacker type skillset), you become reliant on them. And when you want to do something outside of them, you are lost.

I’ve seen quite a few posts from testers about how to execute Java (Selenium/TestNG) tests outside of Eclipse, and they usually have maven set up with Eclipse too.

As someone who likes to have options & builds tools & utilities from (glue) scripts and custom mini applications, whether for testing or not, it irritates me to see those questions pop up.

Either the poster is ignorant & an idiot for not knowing how to search up the solution (one search may not give you the entire solution, but search up the components that will make up your solution individually and you can piece together the puzzle to solve), or they are plain lazy to just want someone to solve it for them.

Those who work with Java toolset should know how to compile Java from scratch on command line, how to execute maven from command line, how to execute JUnit and TestNG tests from the command line test runner.

Those who work with .NET/C# should know how to compile the code from scratch on command line, how to execute NUnit/etc. tests from command line test runner, and the similar equivalents to what you do in Java.

Those who work with scripting languages should know how to run code from files on command line, run code from interpreter via command line (in GUI interpreter mode or pure command line).

Those who work with any language should know how to use libraries/packages/modules, installing, referencing them, etc. all without the fancy IDE – just command line compilation, installation, etc. Should also know how to use methods & functions of libraries/packages/modules without auto-complete in IDE, rather simply by looking up quick references and API documentation instead.

Those who work on Windows should know how to work with batch files and the Windows task scheduler. Even better to learn WSH, WMI, VBScript and/or Powershell.

Those who work on *nix should know shell scripting and cron jobs at least.

Selenium page objects can be implemented outside code and for keyword driven testing

4 Aug

FYI, Selenium page objects don’t have to be implemented in code and can be adapted for keyword driven test frameworks, provided they have the right features to support page object structuring. One such framework for this is Robot Framework. If doing this from another framework or custom framework, one might have to build out the necessary missing support features.

Here’s an example implementation I did that is an approximate mapping or conversion of page objects in Java code to Robot Framework non-code based page objects, the example has links to other implementation examples for Robot Framework as well.

https://github.com/daluu/robotframework-simple-page-object-example

Fijiaaron

Testing around the world

Mary Ann M-P's Blog

Selenium: My Take

Let's BDD PHP

Just another WordPress.com site

muthutechno

All about coding!

Watir WebDriver

the most elegant way to use webdriver with ruby

AzevedoRafaela

Automated Tests and other stuffs

The Guide On Asian Male Interracial Dating

RedStack

Musings on Integration with Oracle Fusion Middleware

TestO'Matic

A blog on Test Automation, Technical QA, Testing Tools & Techniques, Test Architecture and more…

AutomationRocks.com

Making the QA World a Better Place!

jdhnet

Notes about testing, web development, etc...

Burdette's Test Blog

All about how I test software.

VBA 4 all

programming in VBA

Correct Toes

Align. Strengthen. Restore.

The Informed Healthnut

Science-based health information, recipes, and life!

Don't Make the Same Mistake Twice

best and worst development practices

Follow

Get every new post delivered to your Inbox.

Join 56 other followers