Tag Archives: Frameworks

To re-invent the wheel or not? And testing frameworks…

16 Sep

Was reading a slide presentation today from another blog post: http://www.slideshare.net/abagmar/automate-across-platform-os-technologies-with-taas. Brought back memories of earlier work I did of defining a test automation framework: http://yadiytaf.sourceforge.net. I still think my YADIYTAF specification is a nice read. Although I never did create a reference implementation of the spec. I ended up prototyping what I needed at the time against the existing RobotFramework (RF). From that effort, I ended up expanding the functionality/feature set of RF by expanding it’s interoperability with Java, .NET, Perl, PHP via their remote library server interface implementing servers in those languages. So RF is a lot more powerful now since I first evaluated it years back.

In the end, makes me think, should one re-invent the wheel or not at times? Or see what’s already available and build upon that. I guess it depends on what works for you or not.

Advertisements

Complex test scenario example to truly evaluate test frameworks?

6 Jun

This came to mind in a discussion I read a while back, so thought I’d mention it. Not sure if I already mentioned it or not. But the general way we evaluate test frameworks, is rather simple. Or supposedly our system/application is not so complex to test after all, at least in terms of interactions among different component types, which themselves need to be UI/functional/user acceptance tested. For example, these days, most systems to test involve a website front end and mobile app client front ends, both of which talk to a server back end. And the website and mobile app clients don’t often need testing where they interact with each other (or rather say you test something on one client and verify on the other for example). Instead we simply test them standalone, independent of each other (or maybe mock the other side). And/or we got lucky in that it happens that the different clients (web, mobile) happen to be able to use the same platform (e.g. Java, Selenium-API) so that we can use the same framework to test them.

But truly testing the capabilities of a test framework is when you have a complex scenario that involves multiple components that need to be tested in integration (whether you mock them or not), and for the sake of simplicity in this argument, let’s say if you had to mock them, you mock them in the same language as the original component (not mocking to the platform of the desired test framework, etc.). Thus in a truly complex scenario/world, you won’t be able to have all things in Java, .NET, or whatever.

One example showcasing testing a system that comprised of the following components would be ideal: a website or web application/client, an iOS mobile app client, an Android mobile app client, a desktop GUI client, a SIP/VoIP client, perhaps a terminal/console client talking to server back end, perhaps networking communication w/ server on backend (some type of TCP/IP or other network communication), REST API web services utilized by the web/mobile/desktop clients, some native Java/.NET/etc. APIs to call. And let’s assume each of these components are built with different technology stacks and/or the tools to test them (not the framework) use different technology stacks.

Such scenarios may void certain test frameworks out of the box, unless one doesn’t mind writing custom glue logic to bind the different testing (components, tools) together within the single test framework. But a few frameworks do have better support built-in that allow you to integrate different tech stacks and tools to the framework, granted it’s not necessarily plug & play, but it minimizes the amount of effort in glue logic to integrate. I’ll leave it as an open thought on what frameworks are useful for such complex scenarios and which aren’t so much.

Robot Framework Test Automation book review

9 Dec

I recently volunteered to review a book about Robot Framework (RF) in exchange for an eBook copy. As a user & fan of RF, a previous technical subject matter reviewer for Packt publishing (publisher of the book), and one who loves free stuff, I volunteered to review the book to see what it had to offer. Here’s my review, reposted from my Amazon review below. And first, here’s a link to the book:

http://www.packtpub.com/robot-framework-test-automation/book

I think some of the reviewers may have been a bit harsh in their reviews, so I’ll be one of the few to provide a balanced assessment. However, do note that I’ve read most of the book but skimmed over parts of it and have not looked at the accompanying source code that you have to download from the internet. Therefore, the review is not a complete review of the whole content of the book including the source code.

This appears to possibly be the first book about Robot Framework (excluding the framework’s user & quick start guides), which I think is a positive thing regardless of the quality of the book. As a whole, the book is nicely written for someone not familiar with Robot Framework (RF) and for non-technical people. It provides a good introduction about RF, delves into some of capabilities and benefits of RF, and is a good stepping stone to use & learn more about RF. It is a good bridge or companion to the RF project’s existing User Guide and Quick Start guide, which provide more technical detail and information not covered by this book. For more advanced coverage of RF, look elsewhere, hopefully there will be such a book to cover that in the future.

Having said that, the book does have its downsides. As pointed out by others, there are some typos in the current/first edition of the book, not a whole lot, but a few here and there as I’ve seen. Understandable, but sadly that these were missed before publication. The organization of the book is decent, but could be improved, and the chapter titles don’t reflect well against the actual topics in the chapters, at least from a technical user for what they expect to see based on the title alone as the content vs title doesn’t quite fit descriptively. Only by reading the details of the chapter summary at beginning of chapter or in the Preface do you see what the chapter is really about (compared to its title).

I also found the book a bit lacking in some areas in terms of subject matter or content. The section about Data Driven testing and Behavior Driven Development (BDD) testing could have benefited from elaboration with some actual test case examples and/or code implementation (in the book, not as external accompanying source code samples). The BDD information I felt was too brief in coverage. In same area, the book mentions “DSL” without actually defining the abbreviation (DSL = Domain Specific Language), which is bad form in technical writing as we don’t expect the user to already know abbreviations. The book also mentions RF generically without mentioning which version of RF is being covered at the time of publication/writing.

I may be incorrect in assumption, but the author also might not be active with and well informed of the RF user and developer community. I say this because while the book does mention some common & useful test libraries, there are some omissions whether by choice or ignorance. For example: AutoItLibrary (for desktop UI testing with free AutoIt tool), ranorex-robot-library (for desktop UI testing with Ranorex commercial tool) SimpleSikuli (Java test library alternative version of Sikuli integration that was covered in the book), SSH library. Another thing the author missed was the fact that the Remote Library interface/API of RF provides for much more than what was mentioned in the book, particularly that it can be used to interface RF to other languages, tools, platforms not natively supported by RF (Python/Jython/IronPython) and as such to execute tests in those areas as well. For example: (pure/native) Java (as opposed to through Jython), pure .NET as opposed to via IronPython, Ruby, Perl, PHP, and more. Furthermore, the book mentions that only the Python and Ruby versions of the (generic) remote server are implemented for users to make use of for remote libraries, and for anything else, one would have to create it themselves, though it is not necessarily hard to do so. But in reality, as of this review (and at least about 1-2 years before it), there have been other (generic) remote libraries server implementations already available (Java, .NET/C#, Perl, PHP, Clojure, node.js), so it is truly really easy to use remote libraries and for other platforms. The author should have also put in some references that for more technical information and whatever not covered by the book, the reader can look to the RF user guide and discussion with the RF user/developer community via the online Google Groups, providing the links to them with the references.

So given all this, to an (or a more) advanced/technical user, the book is not very helpful, and for what you can get for free with online searching, the RF user/developer community discussion group, and the framework’s existing user guide, the price of the book may also seem extravagant. But I think it is a useful book that can be used to help try and convince upper management, non-technical business stakeholders to adopt RF, etc.

I’d like to end the review with some positive points at least. Unless one is well familiar with RF, there are some useful things that can be learned or be reminded of from this book. For me, I hadn’t noticed that RF offered a randomization feature when executing test cases and test suites. And the coverage of the test configuration/data files like variable files was a helpful reminder to me. Last, I like the fact (at least what is stated in the book) that Packt publishing has a “Packt Open Source Royalty Scheme, by which Packt gives a royalty to each Open Source project about whose software a book is sold”. Which would mean RF should get monetary donations from Packt for each copy of this book bought by someone. I wonder what happens in the case of refunds/returns by customers though. Does that get removed from the donation amount or Packt still passes that on to OSS projects for the “initial” sale.

Maybe a future edition of the book will be more improved. And/or look forward to seeing an advanced coverage version of the book for RF for more advanced/technical readers.

RobotFramework test libraries and remote servers can be used for much more for other purposes

12 Jul

I just wanted to mention that with regards to RobotFramework (RF), whether one actually likes and uses the framework or not, their specification of test library APIs and the remote library API / server specification, while not perfect or flawless, is thought out really well. Really well in that the test libraries and remote library servers can be reused or re-purposed  for other uses outside of RF, like for integration with other test frameworks. I can be wrong, but I don’t think there are many other test frameworks (though there are some) that offer the same type of reuse.

RF test libraries – in general (but does not apply to all libraries), are designed such that they are like class libraries, modular components, such that you can reuse them within frameworks that use the same language platform (Python, Java). You call the keyword methods as normal class library methods.

RF (generic) remote (library) servers – are even better. Their specification based on XML-RPC, while a bit outdated compared to say REST (on top of HTTP) or SOAP, is still very useful. Combined with RF test libraries served via the remote servers, you can easily integrate tools of other platforms/languages into your main test framework platform by exposing those other tools via a (XML-RPC based) web service, even when they natively aren’t web service enabled, as long as they are usable as RF test libraries (for the given remote server).

Cases in point:

  • I successfully integrated (for a prototype demo) calling Sikuli image recognition tool and AutoIt remotely via PHPUnit-Selenium framework such that they can be used in Selenium Grid tests, provided you knew which nodes a running Selenium test to execute Sikuli/AutoIt against. But that isn’t so hard to do if you set up the Grid configuration such that you’d always knew which nodes the tests would run on for specific tests. The integration required creating special PHP functions that make use of a PHP XML-RPC client to talk to Sikuli and AutoIt RF test libraries remotely via the remote server. In the case of Sikuli, it’s a Java library served via the Java remote server. In the case of AutoIt, as I didn’t need the full AutoItLibrary from RF, just a subset, I wrote a simple custom AutoIt library in Perl and served it via the Perl remote server.
  • The Sikuli Java library I wrote offers a command line interface to execute Sikuli API commands without writing a Sikuli script (to be run with Sikuli IDE). But it’s also a class library that simplifies the Sikuli API offering a simpler API on top of Sikuli for quick use and by novice users.
  • I created a RF test library to manage Hyper-V virtual machines, and as it is a .NET class library, it can be used outside of RF to easily manage Hyper-V with a simple API.

See the GitHub gist PHP example below on how I integrated some RF libraries and remote server with PHPUnit for testing. Note that it only presents code from the PHP side, not the RF remote server & library side, which a RF user/developer should already be familiar with. See resources section at end of post for some of the libraries and servers used in the gist.

I encourage users and developers of RF (and it’s related libraries, projects) to explore usage of test libraries and remote servers outside of RF.

I also encourage those evaluating RF but end up not using it to explore possibilities of partially using it by reusing the test libraries and/or remote servers in another framework, etc.

You may find you might have some other uses for them.

Reference resources to some RF test libraries and remote servers (mentioned in blog post or not):

Remote servers

.NET
http://code.google.com/p/nrobotremote/
http://code.google.com/p/sharprobotremoteserver/

Java
https://github.com/ombre42/jrobotremoteserver
http://code.google.com/p/jrobotremoteserver/

Ruby
https://github.com/semperos/robot-remote-server-rb

Perl
http://code.google.com/p/plrobotremoteserver/

PHP
https://github.com/daluu/phrrs

Test libraries

http://code.google.com/p/simplesikuli/

http://code.google.com/p/robotframework-hypervlibrary/

Pete Zybrick

Bell Labs to Big Data

Seek Nuance

Python, technology, Seattle, careers, life, et cetera...

TELLURIUM

New Era of Test Automation

Der Flounder

Seldom updated, occasionally insightful.

The 4T - Trail, Tram, Trolley, Train

Exploring Portland with the 4T

Midnight Musings

Thoughts on making art

Automation Guide

The More You Learn The More You Play...!

The Performance Engineer

Code.Test.Tune.Optimize.

humblesoftwaredev

Thoughts related to software development

Yi Wang's Tech Notes

A blog ported from http://cxwangyi.blogspot.com

Appium Tutorial

Technical…..Practical…..Theoretically Interesting

LinuxMeerkat

I swear! Meerkats can do Linux

PacketsDropped

Requeuing the packets dropped in my memory.

Two cents of software value

Writing. Training. Consulting.

@akumar overflow

wisdom exceeding 140 chars.