Selenium page object modeling challenge – are you up for it?

11 Feb

Having seen some good & bad page object model code & especially after having read Clean Code book, I sure wish there was some book, presentation, blog post, etc. that went over in more detail about the page object model beyond the simple examples presented in various online articles and the official reference example.

Something that presents case studies of good implementations, bad ones, complex ones (ones that you’d expect novices would never implement or have trouble implementing), and examples/tips on how to fix bad implementations into good ones. For bad/complex ones, would be nice to cover things like:

  • shared page widgets, global (or somewhat global) headers, footers, etc.
  • templated/master/skeleton pages where other pages & sites inherit from (similar flow & UI but where the final sites may differ from one another in minor aspects on some flow or UI element implementation/definition like CSS/JS specification)

if such good one presentations already exist, please point me to them. If not, I urge my fellow Selenium bloggers (especially those that follow my blog), fellow Selenium users/contributors/developers, my blog followers/readers to try and take on this challenge (find such a presentation or come up with one to present online, at some meetup, a code camp, or upcoming official Selenium conference).

I would think such a presentation will greatly benefit the Selenium user community, particularly the novice users who lack the initial software development/architectural design skills to appropriately model page objects with the right level of abstraction.

I do understand such a task may not be easy as it requires finding/sharing code that might be confidential or intellectual property (corporate code) unless reworked as generalized code that could be shared publicly. Unless of course we can find some good open source Selenium test code that happens to be bad or complex (or good) implementations to show as case studies. We need to find and present such code in a central location (as presentation, etc.) because it’s labor intensive to scour the internet OSS codebase for Selenium code examples as case studies (for each person to do individually).

What are your thoughts on this? My ideal thoughts are we need a Clean Code type book for Selenium that covers page objects in detail. A sort of “Clean Selenium Page Objects” or “Clean Selenium Code” book.

On that note, maybe you might have code you can share to use for such a case study presentation? Or maybe you could try to get your organization to approve a generalized version of your Selenium page object code (not all of it but some example excerpts) to use for a public case study?

Thoughts on a Selenium interactive exploratory test and debug tool

24 Jan

I came across a nice Selenium-based tool recently called SWD Recorder. You can find more about it here or watch this video. But for me, the topic to discuss today is how it could be extended to offer features that I had been planning to build into a tool that would use the Selenium API. But since a similar tool already exists now, I can build off of it rather than create from scratch. But in case I don’t get to it yet or never do, I’m blogging my thoughts on what I would build upon this current tool.

The tool is great but I had been looking for specific features (to have), which don’t necessarily align with this tool’s original goal and thus I’d fork it for my own purposes. In general, I’ve been looking to build a cross browser Selenium interactive debugging tool that has the following features:

  • A Selenium IDE or Selenium Builder like command executor. But cross browser compatible and doesn’t run as browser extension but separate tool that drives browser via Selenium APIs. Basically the tool provides a set of standard API commands to select from and user supplies the locator value and then you can click execute/run to execute the selected command against specified locator. I’m not looking for option to dynamically add a list of commands to execute like a test sequence/scenario/case or be able to save that to disk as a test case/suite file that Selenium IDE/Builder offers. Rather it would be more for exploratory test/debug session to see if certain Selenium commands work against specified element in given browser, etc. But those additional options I mention that are not of interest to me could be extended/added by other folks for a type of Selenium IDE/Builder that actually works across browsers (not just for running tests but recording actions or for manually defining/creating the action list/set). Examples of commands to execute: click, sendKeys/type, getText, getAttribute, drag & drop.
  • Expand the previous bullet to also offer a list of javascript emulated/simulated action commands (not native actions/interactions API from Selenium but pure javascript events & emulation of actions) to execute. This would be nice addition that even Selenium IDE/Builder doesn’t offer I think. It serves as a way to see if javascript workarounds for some Selenium commands will work against given browser or not. Without such feature, the only way to test this is to actually wrap & executed the needed javascript code within your test framework to then test & try out, which also requires trial test runs of some test case with setup & teardown that takes time, or using a Python/Ruby shell to execute it but which requires you to type it up or load a pre-written library. But why waste all that time when you can test out quickly in a special debugger tool like this. Examples of javascript wrapped commands: drag & drop, mouse click, mouse over, mouse up, mouse down, scroll mouse wheel. See this blog post about javascript workarounds for ideas.
  • Add alternate options for finding & testing locators. For example buttons to inject Selector DetectorSelector Gadget, and Super Selector among other similar tools without need to set up bookmarklets. Enhance the tool’s current WebElement explorer with option (by default or not) to generate a matching CSS selector (if applicable) for the given XPath that is generated. In this case, for Windows we’d have to have a pre-compiled binary of CSSify Python script for Windows, while on other OSes, we could perhaps just run the script natively. Or better yet, we instead screen scrap the CSSify public page/service to call the underlying “web service” (an HTTP POST call) to translate the XPath to CSS without havng to shell execute a Python script. This collection of goodies would then finally give Selenium users a cross browser element locator finder & tester tool to find elements (their XPath and/or CSS) and then modify the default XPath/CSS value and see if it still works, etc. A tool comparable to FirePath and Firebug. Granted it won’t be a perfect equivalent since it doesn’t quite show the HTML DOM source relative to the inspected element like Firebug & FirePath do at this point.
  • Add alternate debugging options across browsers. For example, have button to inject & load FirebugLite without having to set up bookmarklet.
  • Add (sort of) cross browser javascript error collector functionality for debugging with Selenium. Have buttons to inject the javascript error collection code snippet, and button(s) to check/display/retrieve the collected javascript errors (error count and the specific list of messages). Or instead of button(s) to check errors, it could be automatically dumped out in the tools display after the injection whenever error occurs. This functionality of course would not support javascript page load errors (only after page load) and would not persist across pages, have to manually inject on every page desired. Unless we enhance the tool by modifying the Selenium source code (.NET binding?) to auto inject on every page load. Now this might kind of seem pointless, but it’s sort of a cross browser solution as well as nice alternative to IE’s not so nice developer console or alert dialogs of javascript errors. It’s also a way to test out how well such a solution would work when you actually implement the same into your test framework without you have to do that first by evaluating it in an interactive debugging mode.
  • Have a tab section where you can inject arbitrary javascript source files (via HTTP URL) into the current page. It’s a lot easier than having to manually write the javascript code snippet to inject the script element with src set to the HTTP URL then to execute that javascript snippet with Selenium command in say an interactive Python/Ruby shell. Just paste the URL to the GUI tool’s text field, then click inject script.
  • Followng on previous bullet, also nice to have a section for a cross browser Selenium javascript console, whee you can execute any desired javascript code snippet via Selenium WebDriver’s JavascriptExecutor. You paste or type the code snippet in a textarea field and click execute. Any return value is cast as a String and dumped back in a results area for user to see in the tool. This provides a javascript console equivalent to the browser’s native developer tools but one in which the code is executed by Selenium rather than directly by the browser. It would be a nice way to test out whether certain javascript can be executed or whether it works well with Selenium before you actually code it into your test framework or test. I used to and currently do this over a Python interactive shell but it’s more simpler to do this over a GUI tool, especially for novice users.

Now from all the features mentioned above, to summarize, I’m looking to have an interactive GUI-tool based Selenium exploratory test & debugging tool. One in which you can test out code snippets, locators, and Selenium commands cross browser before you actually code it into test framework. Others might prefer the direct approach but I personally prefer interactive test & debug first as it is a lot more insightful and faster this way than to put everything in framework and a throwaway code test script that also takes much more time to execute through the test flow or having to set breakpoint in debugger and debug from that point. Having such a tool, you can easily combine manual & automated steps in one to see how things work. Such a tool is essentially a GUI version of what I talk about in previous posts:

How to debug test and try selenium code with interactive shells

Using selenium with interactive interpreter shells

A selenium IDE alternative for other browsers and another record playback method

Update 01/26/2014:

It recently occurred to me, that it would be nice to be able to inject jQuery and/or Sizzle into Selenium with such a tool via some buttons to click. To be able to test out whether can locate elements defined with jQuery syntax or for the non-standard CSS provied by Sizzle. This of course is only needed in the case of jQuery if the site under test doesn’t already use jQuery.

Update 04/14/2014:

I recently came across a similar too, this one is not .NET based but Java as a JAR file. Love it when the community comes up with new supporting 3rd party tools.

https://github.com/dmolchanenko/LookingGlass

Robot Framework Test Automation book review

9 Dec

I recently volunteered to review a book about Robot Framework (RF) in exchange for an eBook copy. As a user & fan of RF, a previous technical subject matter reviewer for Packt publishing (publisher of the book), and one who loves free stuff, I volunteered to review the book to see what it had to offer. Here’s my review, reposted from my Amazon review below. And first, here’s a link to the book:

http://www.packtpub.com/robot-framework-test-automation/book

I think some of the reviewers may have been a bit harsh in their reviews, so I’ll be one of the few to provide a balanced assessment. However, do note that I’ve read most of the book but skimmed over parts of it and have not looked at the accompanying source code that you have to download from the internet. Therefore, the review is not a complete review of the whole content of the book including the source code.

This appears to possibly be the first book about Robot Framework (excluding the framework’s user & quick start guides), which I think is a positive thing regardless of the quality of the book. As a whole, the book is nicely written for someone not familiar with Robot Framework (RF) and for non-technical people. It provides a good introduction about RF, delves into some of capabilities and benefits of RF, and is a good stepping stone to use & learn more about RF. It is a good bridge or companion to the RF project’s existing User Guide and Quick Start guide, which provide more technical detail and information not covered by this book. For more advanced coverage of RF, look elsewhere, hopefully there will be such a book to cover that in the future.

Having said that, the book does have its downsides. As pointed out by others, there are some typos in the current/first edition of the book, not a whole lot, but a few here and there as I’ve seen. Understandable, but sadly that these were missed before publication. The organization of the book is decent, but could be improved, and the chapter titles don’t reflect well against the actual topics in the chapters, at least from a technical user for what they expect to see based on the title alone as the content vs title doesn’t quite fit descriptively. Only by reading the details of the chapter summary at beginning of chapter or in the Preface do you see what the chapter is really about (compared to its title).

I also found the book a bit lacking in some areas in terms of subject matter or content. The section about Data Driven testing and Behavior Driven Development (BDD) testing could have benefited from elaboration with some actual test case examples and/or code implementation (in the book, not as external accompanying source code samples). The BDD information I felt was too brief in coverage. In same area, the book mentions “DSL” without actually defining the abbreviation (DSL = Domain Specific Language), which is bad form in technical writing as we don’t expect the user to already know abbreviations. The book also mentions RF generically without mentioning which version of RF is being covered at the time of publication/writing.

I may be incorrect in assumption, but the author also might not be active with and well informed of the RF user and developer community. I say this because while the book does mention some common & useful test libraries, there are some omissions whether by choice or ignorance. For example: AutoItLibrary (for desktop UI testing with free AutoIt tool), ranorex-robot-library (for desktop UI testing with Ranorex commercial tool) SimpleSikuli (Java test library alternative version of Sikuli integration that was covered in the book), SSH library. Another thing the author missed was the fact that the Remote Library interface/API of RF provides for much more than what was mentioned in the book, particularly that it can be used to interface RF to other languages, tools, platforms not natively supported by RF (Python/Jython/IronPython) and as such to execute tests in those areas as well. For example: (pure/native) Java (as opposed to through Jython), pure .NET as opposed to via IronPython, Ruby, Perl, PHP, and more. Furthermore, the book mentions that only the Python and Ruby versions of the (generic) remote server are implemented for users to make use of for remote libraries, and for anything else, one would have to create it themselves, though it is not necessarily hard to do so. But in reality, as of this review (and at least about 1-2 years before it), there have been other (generic) remote libraries server implementations already available (Java, .NET/C#, Perl, PHP, Clojure, node.js), so it is truly really easy to use remote libraries and for other platforms. The author should have also put in some references that for more technical information and whatever not covered by the book, the reader can look to the RF user guide and discussion with the RF user/developer community via the online Google Groups, providing the links to them with the references.

So given all this, to an (or a more) advanced/technical user, the book is not very helpful, and for what you can get for free with online searching, the RF user/developer community discussion group, and the framework’s existing user guide, the price of the book may also seem extravagant. But I think it is a useful book that can be used to help try and convince upper management, non-technical business stakeholders to adopt RF, etc.

I’d like to end the review with some positive points at least. Unless one is well familiar with RF, there are some useful things that can be learned or be reminded of from this book. For me, I hadn’t noticed that RF offered a randomization feature when executing test cases and test suites. And the coverage of the test configuration/data files like variable files was a helpful reminder to me. Last, I like the fact (at least what is stated in the book) that Packt publishing has a “Packt Open Source Royalty Scheme, by which Packt gives a royalty to each Open Source project about whose software a book is sold”. Which would mean RF should get monetary donations from Packt for each copy of this book bought by someone. I wonder what happens in the case of refunds/returns by customers though. Does that get removed from the donation amount or Packt still passes that on to OSS projects for the “initial” sale.

Maybe a future edition of the book will be more improved. And/or look forward to seeing an advanced coverage version of the book for RF for more advanced/technical readers.

Selenium WebDriver – extracting an image on page by use of take screenshot and cropping

4 Dec

Normally, if we wanted to get an image off the web page under test, we’d download it externally using the URL extracted off the image element source attribute. Unfortunately, in today’s AJAX based web apps, that doesn’t always work. Sometimes, the elements appear like images visually to the user but are rendered by javascript into DOM elements that are not “image” elements, composed of sprites, etc. on the server side.

In such cases, you have to go an alternate route…

The technique to do it is explained here:

https://groups.google.com/d/msg/selenium-users/DjoEjKwvIN0/sI-siW0gOA8J

and an example implementation in Java here:

https://groups.google.com/d/msg/selenium-users/8atiPIh39OY/Gp9_KEXnpRUJ

Side comment 1 – it would be nice if others in the community can contribute other language implementations to the Java source code example above so others can use it. Python, Ruby, C# perhaps? And a sprinkle of PHP, Perl…I may contribute code snippets when I have time.

After you’ve obtained the image, you can do whatever you want with it. Binary/MD5/SHA-1 hash compare the image against a known benchmark one?

For my case, that’s not sufficient as the image isn’t always a single image but a collection formed as one and dynamically generated. So we do some flexible fuzzy logic image comparison with our internal in house image comparison solution.

And on that topic, side comment 2:

I’ve noticed that taking screenshot then cropping to desired element location & size isn’t 100% cross platform compatible. So blogging this for reference to others and to see if anyone else have same issue. The problem here is that different browsers crop with slightly different coordinates and/or width/height ending up with cropped images that differ slightly across browsers. Now I haven’t tested across all browsers, but my findings reveal that Safari provides the most exact desired cropping, while Firefox and IE have slight variations that are not desirable. I didn’t test Chrome, nor mobile Safari with Appium.

With that in mind, you definitely can’t do hash/binary compare of the cropped images across browsers due to differences, unless you have a different benchmark image per browser. So with a single benchmark image, you’d need some flexible fuzzy logic image comparison technique. There’s multiple options here, but none readily available, other than maybe ones like Sikuli. The in house solution we use is based off ImageMagick and does the dirty work of calculating the parameters for the image comparison for you wrapped in a nice REST API.

Now anyone have experiences they can share around the topics discussed above?

Candidate interviews should include/have a debugging test

11 Oct

I had some experiences where some QA folks are not that quick or sharp at debugging problems in web applications, and automation test failures. Granted you should be intimately familiar with the application to better debug it, or with the automated test framework to know where and what failures happen.

But finding failure then tracking it back is a general principle that’s not that hard to do if you practice and do it enough. And in automation test failures, the exceptions or error message stacktrace is good starting point to backtrack against the test code. Combined with rerunning the test (in debug mode or not) and manually observing what happens, you can get an idea and track down the failure. But I’ve seen some folks not quite get the hang of it at tracking down the exact failures to an automated test. Or how to find further details of a failure when testing manually (by monitoring the AJAX, HTTP calls, or the javascript console logging of errors).

So it got me thinking that a “debug” test in an interview is useful for interviewing candidates. Purposely build or create a broken website/application and have the candidate find the cause or at least details to the failure than just that they seem something is broken functionally or visually in the UI (find out more why it is so). And for test automation. Same thing, purposely have a test that tests against old/obsolete functionality requirements that have since changed and/or a test that is valid but website/app is broken, hence test fails. Then have the candidate debug the test failure to figure out why the test fails.

Such questions will be useful in seeing how th candidate fares in actual work when debugging is needed. Or for QA candidates & manual/exploratory testing – to ensure the candidate can look into details on the failure and give that to the developer rather than just say I can reproduce and here’s what I see visually, I have no idea what might be causing failure underneath, you take a look.

What are your thoughts on this?

The irony of being a new hire vs an veteran at an organization

21 Sep

This is based on what I’ve experienced at two employers that I’d generalize the assumption. It is with respect to QA test automation, but likely applies to other fields and disciplines as well.

It is funny that I notice that beinga new hire, I have extra time on my hands while I ramp up and get familiar with the company and it’s products/services, and it’s internal development processes, etc. I could actually make use of such spare time to further the company’s software quality in terms of test automation coverage and infrastructure (my specialty). But unfortunately, at such a point in time, I lack the familiarity and nuances of the system & processes to know what is lacking and what needs improvement.

I’ve observed it takes at least a year at any given company to gain the insider knowledge to know effectively what needs improvement or what’s missing, etc. And as that time grows and you become a veteran, there is so much you can contribute with your knowledge but by now, you’re saddled with other tasks to ever have the spare time (that you had as a new hire) to make the very useful contributions that you could make. So those ideas/efforts get related to “backlog” status or technical debt.

My, it sure would be nice if you could find/get a new hire who is smart & gets up to speed quickly sometimes. Maybe then you can hire such a new person, and while the company lets him/her sink in to familiarize with the company, you pass along all your backlog stuff you’ve wanted to do and have the newbie do it, if he/she were smart enough to do the job.

Oh the irony…why can’t I have my cake and eat it too? There’s always some trade-off…

Anyone also experience the irony that I’ve experienced?

Effective automated functional test coverage equals pair testing and 2x resourcing

20 Aug

Unless you have rockstar QA/testers and/or developers in your organization who do an excellent job of test coverage, both manually and with automation, or you already plan testing activities well for worst case scenarios, then this article might be useful to you. In terms of planning well and excellent job of testing, I mean as in you have very little or no wishful automation goals that are significantly hard to achieve (e.g. need more manpower or people with more skills/experience, or need more funding for the projects), rather your goals are more like TODO tasks that you want to tackle now or sometime in the future.

For those with the wishful goals, taking aside funding issues for the moment (e.g. cost of test tools or limited budget in hiring staff) as well as expertise issues (e.g. we lack testers with any or good automation skills), one of your probable issues is in how you allocate resources. Because in terms of funding issues, in todays industry, at least for web and mobile applications, and even desktop (but not embedded and hardware), you have open source tool options to offset proprietary tooling costs. In terms of hiring budget, you just have to work out how you justify the ROI for hiring more staff or the right staff (the latter which is the expertise issue part too).

So back to the subject. Resourcing issue? Based on my QA career, my observations lead me to believe most organizations lack the proper assessment/understanding of the real testing work involved when you factor in automation through the chain of command (entry level staff through upper management). Either they are ignorant, or have wishful thinking / high expectations beyond reality. Automating functional tests is also an art itself to account for environment issues, flaky results, and how to efficiently develop and maintain test codebase in an architecturally sound manner (same way as the software product code) since automation code is not meant to be throw away spaghetti code. Add in the fast paced software development release cycle of agile today (and even traditional waterfall where testing & automation always goes last even with delays in the other phases of the project), you always have a burden to address both the initial manual, regression, & exploratory testing of the product as well functional test automation for future regression coverage to fit within project schedules.

Often due to the time it requires to automate and how organizations seem to find manual testing more cheaper and effective (throw more people in project to test, outsource the work to offshore, etc.), the automation portion gets skimped or passed off to offshore group, etc. And as a result automated test coverage and quality suffer.

The best approach is to budget and allocate for a project 2x the originally planned QA resourcing. Whatever your ratio of QAs to developers, double the QA number. Utilize that in a form of pair testing. That is one QA will do manual & exploratory testing (and any manual regression that can’t be automated) for the project, and the other will do the test automation for the project. Both do the work at the same time. In terms of agile process, both do their work in tandem as the developer does the code & design work. The automation QA will design and code up the interface for the automation before the code is ready (based on mockups, etc.). The manual QA will plan/write out the test cases to cover. When the code is ready to test, the manual QA does the testing, the automation QA then implements the automation code within the defined interface (e.g. fill in the UI element locators, fill in the test methods/functions for what it actually needs to do) and writes the automated test cases. The manual QA can also help write automated test cases if time allows. Then to avoid each QA doing the same type of work over and over again, rotate their roles for every project for proper knowledge retention as well as keep the work fun.

If you think 2x QA resourcing costs too much, look at it this way: if they can do the work on schedule, good. No delays. If they go ahead of schedule, you can take on more work/projects at least on the QA side. If this approach results in better automated test coverage & quality, great, as intended. The not so good case is you don’t get as much out of it as you intended but hopefully your automation deficit doesn’t grow so bad as it used to. If you find it is no better than before you used 2x resourcing and pair testing, then it means you either have the wrong QA staff or you are not managing them well (perhaps they’re great testers but they slack off and don’t do their best).

Bear in mind you have to hire the right QA people for this to work well. If you rotate their work, then they have to have well rounded skills. Good manual testing and good automation skills (don’t need both if not rotating roles). But funny most job postings implicitly if not explicitly requests for such skills anyways. But the funny thing is that those organizations seldom use 2x resourcing and pair testing to take advantage of such skillset, so the testers may be bogged down with manual work to do any effective automation, or they do automation most of the time and don’t get exposure to the manual testing aspects and might not then create effective automated tests (done well but bad test coverage).

It’s definitely worth a shot to try if you feel your automation strategy has a long ways to go in terms of meeting your utopian ideals. For those not in management, try to get your management to adopt this approach if you can. Let me know what you think.

AutomationRocks.com

Making the QA World a Better Place!

jdhnet

Notes about testing, web development, etc...

Burdette's Test Blog

All about how I test software.

VBA 4 all

programming in VBA

Correct Toes

Align. Strengthen. Restore.

The Informed Healthnut

Science-based health information, recipes, and life!

Don't Make the Same Mistake Twice

best and worst development practices

Lets Run Trails

Trail Running Mum + Vegan + Writer

Adventures in QA

Software Testing, Mobile Testing and Agile Development by Daniel Knott

Muses Of IT Engineer

The Learnings of a QA engineer

c089

some nerd writing about computery thingies

Assert Selenium

Selenium Automation in a Right Way

Geoff Varosky's Blog

SharePoint, Office 365, and Other Musings...

Misadventures of a castaway coder!

Help.. Help.. Just trying to answer this call for help !

The Storyteller

rise of the machines...

WatirMelon

A 93% Software Testing Blog by Alister Scott

More Than Coding

Because machines still need humans

Follow

Get every new post delivered to your Inbox.

Join 30 other followers