Tag Archives: Rants

On filing bugs to Apple

26 Aug

Unlike other public bug tracking systems, Apple’s one is closed to public viewing other than Apple and the reporter of the bug.

So bugs you report, others can’t see the status of. And as such, there can be likelihood of duplicate bugs being filed, etc. I assume Apple has good reasons to overlook potential duplicate bugs in favor of this closed access bug tracking system.

Amusingly, a public repository of Apple bugs is available at OpenRadar. It’s useful for the public community to look for existing bugs filed to Apple before filing their own, and check status of said bugs. There is one caveat though, OpenRadar does not pull data from Apple’s system. Therefore to get data into OpenRadar, one must file a “duplicate” bug to it for whatever one files to Apple.

As such, I would recommend those of us who deal with Apple bugs to:

  1. Check OpenRadar first before filing bug to Apple. If one exists on OpenRadar, just subscribe to and monitor that. Although one is free to file a duplicate to Apple just in case.
  2. If one doesn’t exist on OpenRadar, then file bug to Apple. And if the bug is not specific to your internal case (confidential/security/proprietary info, intellectual property, etc.), post a duplicate filing on OpenRadar. This allows the public to see the bug you filed. Also makes it easier for team members to view the bug for those teams that don’t use a shared/team Apple developer account or whose said accounts aren’t all under the same organization access.

Candidate interviews should include/have a debugging test

11 Oct

I had some experiences where some QA folks are not that quick or sharp at debugging problems in web applications, and automation test failures. Granted you should be intimately familiar with the application to better debug it, or with the automated test framework to know where and what failures happen.

But finding failure then tracking it back is a general principle that’s not that hard to do if you practice and do it enough. And in automation test failures, the exceptions or error message stacktrace is good starting point to backtrack against the test code. Combined with rerunning the test (in debug mode or not) and manually observing what happens, you can get an idea and track down the failure. But I’ve seen some folks not quite get the hang of it at tracking down the exact failures to an automated test. Or how to find further details of a failure when testing manually (by monitoring the AJAX, HTTP calls, or the javascript console logging of errors).

So it got me thinking that a “debug” test in an interview is useful for interviewing candidates. Purposely build or create a broken website/application and have the candidate find the cause or at least details to the failure than just that they seem something is broken functionally or visually in the UI (find out more why it is so). And for test automation. Same thing, purposely have a test that tests against old/obsolete functionality requirements that have since changed and/or a test that is valid but website/app is broken, hence test fails. Then have the candidate debug the test failure to figure out why the test fails.

Such questions will be useful in seeing how th candidate fares in actual work when debugging is needed. Or for QA candidates & manual/exploratory testing – to ensure the candidate can look into details on the failure and give that to the developer rather than just say I can reproduce and here’s what I see visually, I have no idea what might be causing failure underneath, you take a look.

What are your thoughts on this?

The irony of being a new hire vs an veteran at an organization

21 Sep

This is based on what I’ve experienced at two employers that I’d generalize the assumption. It is with respect to QA test automation, but likely applies to other fields and disciplines as well.

It is funny that I notice that beinga new hire, I have extra time on my hands while I ramp up and get familiar with the company and it’s products/services, and it’s internal development processes, etc. I could actually make use of such spare time to further the company’s software quality in terms of test automation coverage and infrastructure (my specialty). But unfortunately, at such a point in time, I lack the familiarity and nuances of the system & processes to know what is lacking and what needs improvement.

I’ve observed it takes at least a year at any given company to gain the insider knowledge to know effectively what needs improvement or what’s missing, etc. And as that time grows and you become a veteran, there is so much you can contribute with your knowledge but by now, you’re saddled with other tasks to ever have the spare time (that you had as a new hire) to make the very useful contributions that you could make. So those ideas/efforts get related to “backlog” status or technical debt.

My, it sure would be nice if you could find/get a new hire who is smart & gets up to speed quickly sometimes. Maybe then you can hire such a new person, and while the company lets him/her sink in to familiarize with the company, you pass along all your backlog stuff you’ve wanted to do and have the newbie do it, if he/she were smart enough to do the job.

Oh the irony…why can’t I have my cake and eat it too? There’s always some trade-off…

Anyone also experience the irony that I’ve experienced?

Effective automated functional test coverage equals pair testing and 2x resourcing

20 Aug

Unless you have rockstar QA/testers and/or developers in your organization who do an excellent job of test coverage, both manually and with automation, or you already plan testing activities well for worst case scenarios, then this article might be useful to you. In terms of planning well and excellent job of testing, I mean as in you have very little or no wishful automation goals that are significantly hard to achieve (e.g. need more manpower or people with more skills/experience, or need more funding for the projects), rather your goals are more like TODO tasks that you want to tackle now or sometime in the future.

For those with the wishful goals, taking aside funding issues for the moment (e.g. cost of test tools or limited budget in hiring staff) as well as expertise issues (e.g. we lack testers with any or good automation skills), one of your probable issues is in how you allocate resources. Because in terms of funding issues, in todays industry, at least for web and mobile applications, and even desktop (but not embedded and hardware), you have open source tool options to offset proprietary tooling costs. In terms of hiring budget, you just have to work out how you justify the ROI for hiring more staff or the right staff (the latter which is the expertise issue part too).

So back to the subject. Resourcing issue? Based on my QA career, my observations lead me to believe most organizations lack the proper assessment/understanding of the real testing work involved when you factor in automation through the chain of command (entry level staff through upper management). Either they are ignorant, or have wishful thinking / high expectations beyond reality. Automating functional tests is also an art itself to account for environment issues, flaky results, and how to efficiently develop and maintain test codebase in an architecturally sound manner (same way as the software product code) since automation code is not meant to be throw away spaghetti code. Add in the fast paced software development release cycle of agile today (and even traditional waterfall where testing & automation always goes last even with delays in the other phases of the project), you always have a burden to address both the initial manual, regression, & exploratory testing of the product as well functional test automation for future regression coverage to fit within project schedules.

Often due to the time it requires to automate and how organizations seem to find manual testing more cheaper and effective (throw more people in project to test, outsource the work to offshore, etc.), the automation portion gets skimped or passed off to offshore group, etc. And as a result automated test coverage and quality suffer.

The best approach is to budget and allocate for a project 2x the originally planned QA resourcing. Whatever your ratio of QAs to developers, double the QA number. Utilize that in a form of pair testing. That is one QA will do manual & exploratory testing (and any manual regression that can’t be automated) for the project, and the other will do the test automation for the project. Both do the work at the same time. In terms of agile process, both do their work in tandem as the developer does the code & design work. The automation QA will design and code up the interface for the automation before the code is ready (based on mockups, etc.). The manual QA will plan/write out the test cases to cover. When the code is ready to test, the manual QA does the testing, the automation QA then implements the automation code within the defined interface (e.g. fill in the UI element locators, fill in the test methods/functions for what it actually needs to do) and writes the automated test cases. The manual QA can also help write automated test cases if time allows. Then to avoid each QA doing the same type of work over and over again, rotate their roles for every project for proper knowledge retention as well as keep the work fun.

If you think 2x QA resourcing costs too much, look at it this way: if they can do the work on schedule, good. No delays. If they go ahead of schedule, you can take on more work/projects at least on the QA side. If this approach results in better automated test coverage & quality, great, as intended. The not so good case is you don’t get as much out of it as you intended but hopefully your automation deficit doesn’t grow so bad as it used to. If you find it is no better than before you used 2x resourcing and pair testing, then it means you either have the wrong QA staff or you are not managing them well (perhaps they’re great testers but they slack off and don’t do their best).

Bear in mind you have to hire the right QA people for this to work well. If you rotate their work, then they have to have well rounded skills. Good manual testing and good automation skills (don’t need both if not rotating roles). But funny most job postings implicitly if not explicitly requests for such skills anyways. But the funny thing is that those organizations seldom use 2x resourcing and pair testing to take advantage of such skillset, so the testers may be bogged down with manual work to do any effective automation, or they do automation most of the time and don’t get exposure to the manual testing aspects and might not then create effective automated tests (done well but bad test coverage).

It’s definitely worth a shot to try if you feel your automation strategy has a long ways to go in terms of meeting your utopian ideals. For those not in management, try to get your management to adopt this approach if you can. Let me know what you think.

Shorthand to XPath and CSS in developer console and javascript libraries but not WebDriver API?

15 Jul

This thought just came to mind while I blogged and commented online about finding elements by XPath and CSS using the DOM following DOM conventions (accessed/executed in javascript whether in browser developer console or with WebDriver JavascriptExecutor) like:

document.getElementById(‘someId’)

document.getElementByXPath(‘//someXPath’)

document.getElementByCssSelector(‘someCss’)

where some people pointed out you could just do

$x(‘//someXPath’) and $$(‘someCss’)

in the browser developer console (Firefox, Chrome) to do same thing. And yes, it is much shorter to type out.

And of course, you can do similar if your web application already uses jQuery or other javascript libraries, in which case you can execute their ways of locating elements with WebDriver’s JavascriptExecutor. If the web application under test doesn’t use them, the alternative if you want those location strategies is to inject the jQuery and like libraries into the page under test using JavascriptExecutor first before you can utilize them.

So it seems a bit interesting how some people prefer shorthand use in javascript library and in developer console, but how we don’t have much complaints or work done on the WebDriver side to mimic same changes on the web development/debugging side.

Like how document.getElementByXyz() is considered unnecessary extra work but driver.findElement(By.xpath()).click() is ok instead of targeting for something like driver.$x(‘//someXpath’).click() or driver.$$(‘some css’).click(). Granted in the WebDriver case, due to object oriented programming and instantiating classes, you’d always have a driver object that you can’t skip typing though you could shorten it by naming it “d”. But maybe we can also shorten the find by XPath and find by CSS methods with alias methods. And if there’s an argument of no, then one might also ask, since we can live with this with WebDriver, why complain that you can and should use shorthand in the browser console and jQuery?

I guess it is easier to code and debug when shorthand but standardization and consistency of APIs and naming to me is more important. Less to remember & learn, especially for novices. To me

document.getElementByXyz() clearly maps better to  driver.findElement(By.xyz()) than $x() and $$(), and vice versa. Same applies for the multiple elements version of the methods.

RobotFramework test libraries and remote servers can be used for much more for other purposes

12 Jul

I just wanted to mention that with regards to RobotFramework (RF), whether one actually likes and uses the framework or not, their specification of test library APIs and the remote library API / server specification, while not perfect or flawless, is thought out really well. Really well in that the test libraries and remote library servers can be reused or re-purposed  for other uses outside of RF, like for integration with other test frameworks. I can be wrong, but I don’t think there are many other test frameworks (though there are some) that offer the same type of reuse.

RF test libraries – in general (but does not apply to all libraries), are designed such that they are like class libraries, modular components, such that you can reuse them within frameworks that use the same language platform (Python, Java). You call the keyword methods as normal class library methods.

RF (generic) remote (library) servers – are even better. Their specification based on XML-RPC, while a bit outdated compared to say REST (on top of HTTP) or SOAP, is still very useful. Combined with RF test libraries served via the remote servers, you can easily integrate tools of other platforms/languages into your main test framework platform by exposing those other tools via a (XML-RPC based) web service, even when they natively aren’t web service enabled, as long as they are usable as RF test libraries (for the given remote server).

Cases in point:

  • I successfully integrated (for a prototype demo) calling Sikuli image recognition tool and AutoIt remotely via PHPUnit-Selenium framework such that they can be used in Selenium Grid tests, provided you knew which nodes a running Selenium test to execute Sikuli/AutoIt against. But that isn’t so hard to do if you set up the Grid configuration such that you’d always knew which nodes the tests would run on for specific tests. The integration required creating special PHP functions that make use of a PHP XML-RPC client to talk to Sikuli and AutoIt RF test libraries remotely via the remote server. In the case of Sikuli, it’s a Java library served via the Java remote server. In the case of AutoIt, as I didn’t need the full AutoItLibrary from RF, just a subset, I wrote a simple custom AutoIt library in Perl and served it via the Perl remote server.
  • The Sikuli Java library I wrote offers a command line interface to execute Sikuli API commands without writing a Sikuli script (to be run with Sikuli IDE). But it’s also a class library that simplifies the Sikuli API offering a simpler API on top of Sikuli for quick use and by novice users.
  • I created a RF test library to manage Hyper-V virtual machines, and as it is a .NET class library, it can be used outside of RF to easily manage Hyper-V with a simple API.

See the GitHub gist PHP example below on how I integrated some RF libraries and remote server with PHPUnit for testing. Note that it only presents code from the PHP side, not the RF remote server & library side, which a RF user/developer should already be familiar with. See resources section at end of post for some of the libraries and servers used in the gist.

I encourage users and developers of RF (and it’s related libraries, projects) to explore usage of test libraries and remote servers outside of RF.

I also encourage those evaluating RF but end up not using it to explore possibilities of partially using it by reusing the test libraries and/or remote servers in another framework, etc.

You may find you might have some other uses for them.

Reference resources to some RF test libraries and remote servers (mentioned in blog post or not):

Remote servers

.NET
http://code.google.com/p/nrobotremote/
http://code.google.com/p/sharprobotremoteserver/

Java
https://github.com/ombre42/jrobotremoteserver
http://code.google.com/p/jrobotremoteserver/

Ruby
https://github.com/semperos/robot-remote-server-rb

Perl
http://code.google.com/p/plrobotremoteserver/

PHP
https://github.com/daluu/phrrs

Test libraries

http://code.google.com/p/simplesikuli/

http://code.google.com/p/robotframework-hypervlibrary/

On contributing to Selenium open source projects

12 Jul

I think I recall seeing (in online Selenium Google Group, etc. discussions) some people interested in contributing to Selenium project and learning from it.

Obviously, one can supply patches for bugs fixes or implement new features for Selenium (the server or any of the supported official language bindings, Selenium Grid, etc.).

However, I think one area that is a ripe candidate for contribution and learning from it with respect to Selenium are the 3rd party unofficial language bindings, like the ones in PHP, Perl, VBScript, etc.

Some of the 3rd party bindings get no love compared to the official bindings and fall behind in support of WebDriver API functionality offered by the official bindings (it’s not that they can’t do it, work’s just not been done on them).

There are several benefits of contributing to these 3rd party bindings:

  • you learn more about the JSONWireProtocol that is at the heart of the WebDriver API. All the official language binding clients actually communicate to the “server” part of Selenium through that protocol. For local (not remote) WebDrivers in the official bindings, it’s just that the “server” part is all packaged together that you don’t need to start up a Selenium server JAR unless you are using RemoteWebDriver, Selenium Grid, or a 3rd party binding (which doesn’t have it integrated). So in essence you learn more about the internals of Selenium.
  • you might learn a new language, if those 3rd party bindings aren’t what you normally use.
  • you get to improve the 3rd party bindings for the language you like that was not deemed official (e.g. PHP, Perl, VBScript, etc.). Making it better means you can use it and others can too, making it more on par with the official languages. Such that one can’t say no more that they can’t use the 3rd party bindings because they’re incomplete, unofficial, or don’t have support for them.
  • you help promote awareness of these third party bindings, and the fact that you don’t have to use Selenium strictly in Java, C#/.NET, Python, or Ruby.

I myself am a contributor to the PHP WebDriver bindings on Google Code, and have also forked it on GitHub with some additions not merged back to main project. I will continue to contribute to that project when I have time, as it still has a lot of work remaining to make it a full fledged binding that matches what the full WebDriver API offers. FYI, I picked this PHP binding to contribute to (and use) because at the time I evaluated the PHP bindings, this one offered the best PHP API to WebDriver that most closely matched the WebDriver API as an API. The Facebook PHP bindings are not bad either, but it’s API veers too close towards the JSONWireProtocol requiring to know specifics of the protocol to use it well (not user friendly to the average user). I welcome you to do the same and maybe contribute the PHP project I am a member of, should you be thinking of contributing to Selenium open source project(s).

Update 07/20/2013: I forgot to mention earlier, another option for contribution is to use the WebDriver JSONWireProtocol to implement an interface to other test tools so that you can have a single language interface (WebDriver API) to automate with different tools. Currently, the WebDriver API is only implemented for Selenium on the web application side, and Appium & ios-driver on the mobile web/native applications side. There may be one or two others not so well known on the mobile side. Image if we extend that to other areas like desktop GUI automation, like how about using WebDriver API to control AutoIt or Sikuli rather than using their native scripting, API, and/or supported language interfaces. If you built a WebDriver API client/wrapper to them, you could use a single code Selenium codebase/language to control both AutoIt/Sikuli and Selenium, if you had a need to run tests that involved AutoIt or Sikuli. And that’s just mentioning two of the most popular desktop UI tools, there are others as well. Doing such contributions here will offer people more choice in tools and languages used beyond just web application testing, and you’ll also learn JSONWireProtocol, WebDriver API, how Selenium Grid works (if implementing Grid support like for AutoIt/Sikuli via WebDriver API), and how to integrate with other tools working with their APIs and languages.

Seek Nuance

Python, technology, Seattle, careers, life, et cetera...

TELLURIUM

New Era of Test Automation

Der Flounder

Seldom updated, occasionally insightful.

The 4T - Trail, Tram, Trolley, Train

Exploring Portland with the 4T

Midnight Musings

Thoughts on making art

Automation Guide

The More You Learn The More You Play...!

The Performance Engineer

Code.Test.Tune.Optimize.

humblesoftwaredev

Thoughts related to software development

Yi Wang's Tech Notes

A blog ported from http://cxwangyi.blogspot.com

Appium Tutorial

Technical…..Practical…..Theoretically Interesting

LinuxMeerkat

I swear! Meerkats can do Linux

PacketsDropped

Requeuing the packets dropped in my memory.

Two cents of software value

Writing. Training. Consulting.

@akumar overflow

wisdom exceeding 140 chars.

Lazy Programmer's Shortcut

Java, J2EE, Spring, OOAD, DDD & LIFE! .......all in one :)

Testing Mobile Apps

www.SoftwareTestingStudio.com

Photofocus

education and inspiration for visual storytellers