This came to mind in a discussion I read a while back, so thought I’d mention it. Not sure if I already mentioned it or not. But the general way we evaluate test frameworks, is rather simple. Or supposedly our system/application is not so complex to test after all, at least in terms of interactions among different component types, which themselves need to be UI/functional/user acceptance tested. For example, these days, most systems to test involve a website front end and mobile app client front ends, both of which talk to a server back end. And the website and mobile app clients don’t often need testing where they interact with each other (or rather say you test something on one client and verify on the other for example). Instead we simply test them standalone, independent of each other (or maybe mock the other side). And/or we got lucky in that it happens that the different clients (web, mobile) happen to be able to use the same platform (e.g. Java, Selenium-API) so that we can use the same framework to test them.
But truly testing the capabilities of a test framework is when you have a complex scenario that involves multiple components that need to be tested in integration (whether you mock them or not), and for the sake of simplicity in this argument, let’s say if you had to mock them, you mock them in the same language as the original component (not mocking to the platform of the desired test framework, etc.). Thus in a truly complex scenario/world, you won’t be able to have all things in Java, .NET, or whatever.
One example showcasing testing a system that comprised of the following components would be ideal: a website or web application/client, an iOS mobile app client, an Android mobile app client, a desktop GUI client, a SIP/VoIP client, perhaps a terminal/console client talking to server back end, perhaps networking communication w/ server on backend (some type of TCP/IP or other network communication), REST API web services utilized by the web/mobile/desktop clients, some native Java/.NET/etc. APIs to call. And let’s assume each of these components are built with different technology stacks and/or the tools to test them (not the framework) use different technology stacks.
Such scenarios may void certain test frameworks out of the box, unless one doesn’t mind writing custom glue logic to bind the different testing (components, tools) together within the single test framework. But a few frameworks do have better support built-in that allow you to integrate different tech stacks and tools to the framework, granted it’s not necessarily plug & play, but it minimizes the amount of effort in glue logic to integrate. I’ll leave it as an open thought on what frameworks are useful for such complex scenarios and which aren’t so much.