Archive | Rants RSS feed for this section

On filing bugs to Apple

26 Aug

Unlike other public bug tracking systems, Apple’s one is closed to public viewing other than Apple and the reporter of the bug.

So bugs you report, others can’t see the status of. And as such, there can be likelihood of duplicate bugs being filed, etc. I assume Apple has good reasons to overlook potential duplicate bugs in favor of this closed access bug tracking system.

Amusingly, a public repository of Apple bugs is available at OpenRadar. It’s useful for the public community to look for existing bugs filed to Apple before filing their own, and check status of said bugs. There is one caveat though, OpenRadar does not pull data from Apple’s system. Therefore to get data into OpenRadar, one must file a “duplicate” bug to it for whatever one files to Apple.

As such, I would recommend those of us who deal with Apple bugs to:

  1. Check OpenRadar first before filing bug to Apple. If one exists on OpenRadar, just subscribe to and monitor that. Although one is free to file a duplicate to Apple just in case.
  2. If one doesn’t exist on OpenRadar, then file bug to Apple. And if the bug is not specific to your internal case (confidential/security/proprietary info, intellectual property, etc.), post a duplicate filing on OpenRadar. This allows the public to see the bug you filed. Also makes it easier for team members to view the bug for those teams that don’t use a shared/team Apple developer account or whose said accounts aren’t all under the same organization access.

What the f**k to test?

23 Feb

Was reading my emails and some blog posts today and came across these:

http://lifehacker.com/5911385/seven-helpful-sites-that-tell-you-what-the-fk-to-do

https://github.com/soulwire/WTFEngine

it made me think, as a tester, it would be interesting to have/see a site about “What the f**k to test” or something similar. It could be useful to the entry level and novice testers just getting into testing profession. And it could be useful to all of us testers when we happen to feel lazy or have “tester’s block” not being able to test or decide how/what to test.

Or do we actually have something like that already? If so, let me know.

If not, anyone interested in helping build such a site/service? It would be a site that randomly decides to give you the tester a tip on what to test for your site/application/system (in general terms that are not specific to your site/application/system).

Chrome version upgrade can break Chrome extensions?

28 Jan

This is a short blog post today:

Who would have thought this, but some recent Chrome version upgrade can break the UI of your Chrome extension (by changing extension popup window size and font sizes). I only found out after reading a user review.

For reference to the particular issue: https://code.google.com/p/autosmsclients/issues/detail?id=3

and now I have to find time to go debug and fix this, just because Chrome decided it wanted to change things on how a Chrome toolbar UI renders, and without notifying it’s extension developers about this…sigh…

WebDriver API and JSONWireProtocol is not just for web and mobile applications testing, it can be for desktop too!

19 Jan

And so, here’s a blog post about just that. I’ve found that the WebDriver API in general, with regards to common location strategies of ID, name, class, tag, and kind of XPath, along with element manipulations–such as click, type/sendKeys, get/set text, get attributes, mouse operations, key up/down, taking screenshots, finding elements & validating properties (enabled, visible, selected)–all that being common across web, mobile, and desktop.

Selenium/WebDriver started off for web applications. Then came along Appium, ios-driver, etc. which expanded it to the mobile space, first iOS then Android, and now somewhat even Windows Phone.

But there has been very little in the desktop area. The first for it was Appium for Mac. And we’ve seen nothing else since. Though if you consider unofficial Selenium/WebDriver-like APIs, then maybe there’sa few more: Twin and sikuli-remote-control. But now, I’ve worked out some proof of concept prototypes (based off the old Appium Python implementation) to give you more options to desktop automation using WebDriver API and for Windows, not Mac!

https://github.com/daluu/AutoItDriverServer
https://github.com/daluu/AutoPyDriverServer

Check them out. Pretty interesting. The AutoIt one has a Selenium integration demo showing automation of AutoIt and Selenium together against websites using 2 drivers – a “web” driver and an AutoIt driver. They’re not Selenium Grid compatible yet, something to look into for the future. But at least they work for remote deployment. Though there is a way to make it work unofficially with Selenium Grid deployment of Selenium tests (but where these desktop UI servers are just not officially part of Grid as nodes) – this solution will be presented in a future blog post.

And last, I had wanted to complete with a three’s company big bang, but I couldn’t get Sikuli working just yet, maybe in the future…

https://github.com/daluu/SikuliDriverServer

So try them out, submit feedback, send pull requests with enhancements and bug fixes, etc.! 😉

P.S. all this made possible by the great work of others. Like the Appium team for the server base I used. And if/when I work on the Java server version – the Selenium team or ios-driver team, and .NET server version – Jim Evans for his Strontium server implementation. As well as the work of those who build great free or open source tools like AutoIt, Sikuli, AutoPy.

Update 6/5/2015: since my post, I found a new solution that’s better and more recent than Twin for desktop Selenium-style UI automation – Winium.

Organizing tests by repetition or granularity?

24 Sep

I’ve seen this often in my job covering test scenarios manually as well as adapting those tests for automation. Have others come across same issue?

In terms of manual, and more correctly, exploratory testing, as a human tester, you can choose how you perform your tests and it is often logical to consolidate areas of testing (or in documentation parlance, “test cases”) around things that would otherwise be repetitive. e.g. re-using same test data to test slightly varying scenarios. Where workflow or sequence of test steps are common across several tests, you may as well run them sort of together like multitasking rather than separately one after another, repeating the common steps all over again.

One example is testing a workflow that happens to be mostly same across desktop, tablet, mobile. Only UX or UI is different. Do the test 3x? Or run test nearly 1x by testing all 3 at same time where/when possible using one test/path to do most of the workflow and verifying across all 3 paths at the critical points.

But that’s easy to do for exploratory (and undocumented manual) testing. But what do you do when it comes time to put this on pen & paper for others like an offshore team to execute said tests, who are one or more connections/layers away from the intimacy of the feature being test than you are? And then converting such documented tests to automation.

Do we strive to follow the human (exploratory test) nature/convention of consolidating the test coverage around the repetitive workflow and shareable test data minimizing amount of test cases and test scripts to create OR strive to keep granular individual specific tests for tracking/auditing purposes and simplify or attempt to idiot proof things, even though it will be in ways redundant in repeating test steps and generating more repeated tests data that could have been shared/reused otherwise?

One argument could be to do it granular since automation and/or hardware is cheap. But it turns out unless your site is 1990s basic HTML, repeating test steps can still be slow against modern websites/applications against modern browsers. And running tests in parallel helps with this but adds complexity and still can take long time once your test codebase grows big. So in some cases, maybe consolidating testing isn’t necessarily bad, you have to weight the pros & cons with short & long term automation coverage/infrastructure in mind.

Just wondering what experiences and approaches folks have taken in the real world.

IDEs and fancy development tools can be bad for you

23 Aug

Especially for testers (at least those without a good developer/hacker type skillset), you become reliant on them. And when you want to do something outside of them, you are lost.

I’ve seen quite a few posts from testers about how to execute Java (Selenium/TestNG) tests outside of Eclipse, and they usually have maven set up with Eclipse too.

As someone who likes to have options & builds tools & utilities from (glue) scripts and custom mini applications, whether for testing or not, it irritates me to see those questions pop up.

Either the poster is ignorant & an idiot for not knowing how to search up the solution (one search may not give you the entire solution, but search up the components that will make up your solution individually and you can piece together the puzzle to solve), or they are plain lazy to just want someone to solve it for them.

Those who work with Java toolset should know how to compile Java from scratch on command line, how to execute maven from command line, how to execute JUnit and TestNG tests from the command line test runner.

Those who work with .NET/C# should know how to compile the code from scratch on command line, how to execute NUnit/etc. tests from command line test runner, and the similar equivalents to what you do in Java.

Those who work with scripting languages should know how to run code from files on command line, run code from interpreter via command line (in GUI interpreter mode or pure command line).

Those who work with any language should know how to use libraries/packages/modules, installing, referencing them, etc. all without the fancy IDE – just command line compilation, installation, etc. Should also know how to use methods & functions of libraries/packages/modules without auto-complete in IDE, rather simply by looking up quick references and API documentation instead.

Those who work on Windows should know how to work with batch files and the Windows task scheduler. Even better to learn WSH, WMI, VBScript and/or Powershell.

Those who work on *nix should know shell scripting and cron jobs at least.

Testing with your own environment vs a shared environment

27 May

I do wonder, within QA among all the different organizations/companies, does QA generally test on a shared (test) environment, or have their own testbed environment to work with, or both?

It may be organization/industry specific, as certain fields/areas work better where you have individual testbeds/environments to work with and some are better with a shared environment. The contraints and decision are often based on:

  • resource efficiency/utilization – will people be more productive testing with their own environment or a shared one? Can the environment be shared easily for testing? Is it easy to deploy & manage the environment individually by anyone or is it easier to manage centrally as a shared environment? Cost to deploy individual vs shared environment
  • corporate policy – what are the organization’s IT policy/policies that affect environment setup for testing?
  • test data setup/sharing, reproducibility of bugs – whether better with individual environment or shared
  • codebase sync of test/release branch code with the environments with scheduling/agile in mind – easier with individual vs shared environment
  • hmm…any others that come to mind?

But what’s interesting to me though, is that regardless of the QA test setup, developers will usually have their own environment rather than a “shared” development environment. Why is that? More productive & effective to develop individually before you merge all code to a central codebase & environment for testing? But I could be wrong with this assumption as I’ve not worked at too many places but so far have noticed devs do get their own environments at the places I’ve worked. So perhaps QA should get their own environments too if devs get their own. A hybrid approach can be useful though, shared use environment and individual testbed environments, so you get the best of both, though it does cost more this way.

What are your thoughts on this? I personally prefer the individual environment or hybrid option. It’s nice to have more control of the environment that you test, as long as you are smart & responsible on how you configure and test the system.

From personal experience I can provide the following insight:

  • if the industry/organization deals with hardware, you are likely to get your own testbed environment to configure & manage & test. There may be a shared environment as well where multiple people use it, perhaps in real world type scenario (e.g. PBX phone system with all organization members having an account, phone, and access to that system to test and use what they build themselves)
  • if the industry/organization is about website and/or web applications/services, you are likely to get a shared test environment, individual might not be likely though is awesome if you do get that. The shared environment may or may not be managed by QA, it could be managed by IT, DevOps, or Release Engineering, etc.
Seek Nuance

Python, technology, Seattle, careers, life, et cetera...

TELLURIUM

New Era of Test Automation

Der Flounder

Seldom updated, occasionally insightful.

The 4T - Trail, Tram, Trolley, Train

Exploring Portland with the 4T

Midnight Musings

Thoughts on making art

Automation Guide

The More You Learn The More You Play...!

The Performance Engineer

Code.Test.Tune.Optimize.

humblesoftwaredev

Thoughts related to software development

Yi Wang's Tech Notes

A blog ported from http://cxwangyi.blogspot.com

Appium Tutorial

Technical…..Practical…..Theoretically Interesting

LinuxMeerkat

I swear! Meerkats can do Linux

PacketsDropped

Requeuing the packets dropped in my memory.

Two cents of software value

Writing. Training. Consulting.

@akumar overflow

wisdom exceeding 140 chars.

Lazy Programmer's Shortcut

Java, J2EE, Spring, OOAD, DDD & LIFE! .......all in one :)

Testing Mobile Apps

www.SoftwareTestingStudio.com

Photofocus

education and inspiration for visual storytellers