Effective automated functional test coverage equals pair testing and 2x resourcing

20 Aug

Unless you have rockstar QA/testers and/or developers in your organization who do an excellent job of test coverage, both manually and with automation, or you already plan testing activities well for worst case scenarios, then this article might be useful to you. In terms of planning well and excellent job of testing, I mean as in you have very little or no wishful automation goals that are significantly hard to achieve (e.g. need more manpower or people with more skills/experience, or need more funding for the projects), rather your goals are more like TODO tasks that you want to tackle now or sometime in the future.

For those with the wishful goals, taking aside funding issues for the moment (e.g. cost of test tools or limited budget in hiring staff) as well as expertise issues (e.g. we lack testers with any or good automation skills), one of your probable issues is in how you allocate resources. Because in terms of funding issues, in todays industry, at least for web and mobile applications, and even desktop (but not embedded and hardware), you have open source tool options to offset proprietary tooling costs. In terms of hiring budget, you just have to work out how you justify the ROI for hiring more staff or the right staff (the latter which is the expertise issue part too).

So back to the subject. Resourcing issue? Based on my QA career, my observations lead me to believe most organizations lack the proper assessment/understanding of the real testing work involved when you factor in automation through the chain of command (entry level staff through upper management). Either they are ignorant, or have wishful thinking / high expectations beyond reality. Automating functional tests is also an art itself to account for environment issues, flaky results, and how to efficiently develop and maintain test codebase in an architecturally sound manner (same way as the software product code) since automation code is not meant to be throw away spaghetti code. Add in the fast paced software development release cycle of agile today (and even traditional waterfall where testing & automation always goes last even with delays in the other phases of the project), you always have a burden to address both the initial manual, regression, & exploratory testing of the product as well functional test automation for future regression coverage to fit within project schedules.

Often due to the time it requires to automate and how organizations seem to find manual testing more cheaper and effective (throw more people in project to test, outsource the work to offshore, etc.), the automation portion gets skimped or passed off to offshore group, etc. And as a result automated test coverage and quality suffer.

The best approach is to budget and allocate for a project 2x the originally planned QA resourcing. Whatever your ratio of QAs to developers, double the QA number. Utilize that in a form of pair testing. That is one QA will do manual & exploratory testing (and any manual regression that can’t be automated) for the project, and the other will do the test automation for the project. Both do the work at the same time. In terms of agile process, both do their work in tandem as the developer does the code & design work. The automation QA will design and code up the interface for the automation before the code is ready (based on mockups, etc.). The manual QA will plan/write out the test cases to cover. When the code is ready to test, the manual QA does the testing, the automation QA then implements the automation code within the defined interface (e.g. fill in the UI element locators, fill in the test methods/functions for what it actually needs to do) and writes the automated test cases. The manual QA can also help write automated test cases if time allows. Then to avoid each QA doing the same type of work over and over again, rotate their roles for every project for proper knowledge retention as well as keep the work fun.

If you think 2x QA resourcing costs too much, look at it this way: if they can do the work on schedule, good. No delays. If they go ahead of schedule, you can take on more work/projects at least on the QA side. If this approach results in better automated test coverage & quality, great, as intended. The not so good case is you don’t get as much out of it as you intended but hopefully your automation deficit doesn’t grow so bad as it used to. If you find it is no better than before you used 2x resourcing and pair testing, then it means you either have the wrong QA staff or you are not managing them well (perhaps they’re great testers but they slack off and don’t do their best).

Bear in mind you have to hire the right QA people for this to work well. If you rotate their work, then they have to have well rounded skills. Good manual testing and good automation skills (don’t need both if not rotating roles). But funny most job postings implicitly if not explicitly requests for such skills anyways. But the funny thing is that those organizations seldom use 2x resourcing and pair testing to take advantage of such skillset, so the testers may be bogged down with manual work to do any effective automation, or they do automation most of the time and don’t get exposure to the manual testing aspects and might not then create effective automated tests (done well but bad test coverage).

It’s definitely worth a shot to try if you feel your automation strategy has a long ways to go in terms of meeting your utopian ideals. For those not in management, try to get your management to adopt this approach if you can. Let me know what you think.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Anastasia Writes

politics, engineering, parenting, relevant things over coffee.

One Software Tester

Trying To Make Sense Of The World, One Test At A Time

the morning paper

an interesting/influential/important paper from the world of CS every weekday morning, as selected by Adrian Colyer

RoboSim (Robot Simulator)

Visualize and Simulate the Robotics concepts such as Localization, Path Planning, P.I.D Controller


open notebook

a happy knockout mouse.

my journey into computer science

Perl 6 Advent Calendar

Something cool about Perl 6 every day


Inspire and spread the power of collaboration

Niraj Bhatt - Architect's Blog

Ruminations on .NET, Architecture & Design

Pete Zybrick

Bell Labs to Big Data

Seek Nuance

Python, technology, Seattle, careers, life, et cetera...


New Era of Test Automation

Der Flounder

Seldom updated, occasionally insightful.

The 4T - Trail, Tram, Trolley, Train

Exploring Portland with the 4T

Midnight Musings

Thoughts on making art

Automation Guide

The More You Learn The More You Play...!

The Performance Engineer



Thoughts related to software development

%d bloggers like this: