9 top open-source testing automation frameworks: How to choose

If you're thinking about building your own test automation framework, think again. You'll do very well in most cases to consider one or more of the open-source options available. That's because, in general, a framework is a set of best practices, assumptions, common tools, and libraries that you can use across teams. In most cases, you simply don't need to build one that's unique to your dev environment.

You just need a framework to help make your test automation code:

  • Reusable
  • Maintainable
  • Stable

I’ve seen some teams take these benefits to the extreme by building their own, elaborate automation frameworks from scratch. This drives me crazy, because they could have easily made use of existing open-source tools and libraries that would have met their needs without writing any code—and in most cases with better results.

Another area most folks forget to include when looking for a test automation framework is the ability to test APIs. I’m a big believer in testing functionality beyond the user interface (UI) whenever possible. And since integration/API tests bypass the UI, they tend to be quicker and much more reliable than GUI tests.

That's why it’s important to have a test suite that tests several different layers of your application. For that, you need tools that can test both the UI and non-UI components of your application.

So, before you fall into the “build your own framework” trap, be sure to check out these nine open-source test automation frameworks.

Gartner Magic Quadrant for Software Test Automation


If you need a Java-based framework that integrates with behavior-driven development (BDD) tools such as Cucumber and JBehave and lets you keep your test scenarios at a high level while accommodating lower-level implementation details in your reports, Serenity BDD (formerly Thucydides) might be the tool for you. It's designed to make writing automated acceptance and regression tests easier.

Serenity acts as a wrapper on top of Selenium WebDriver and BDD tools. It makes writing BDD and Selenium tests easier because it abstracts away much of the boilerplate code you sometimes need to write to get started. Serenity also offers plenty of built-in functionality, such as handling WebDriver management, managing state between steps, taking screenshots, running tests in parallel, facilitating Jira integration, and more—all without having to write a single line of code.

Serenity creates highly detailed reports. Out of the box, it creates living documentation that you can use not only to view your Selenium BDD test results, but also as documentation for your application.

It also has an extra annotation method, called @Step. When used correctly, @Step gives you an extra level of abstraction that makes your tests more reusable and maintainable.

Serenity recently added integration with REST Assured, the popular open-source Java library for testing RESTful APIs. It creates another wrapper around REST Assured that makes Serenity Selenium tests flow seamlessly. And Serenity's awesome reporting feature also includes REST information, which means you can easily see and validate all requests and response details in your test reports without having to explicitly add any extra logging code.

Serenity takes care of many things you would normally have to code from scratch if you were creating your own BDD framework. It’s one of the better open-source frameworks available, although I might be a bit biased: It’s the tool I use for my own day-to-day test automation efforts.

Serenity has also created a JavaScript version called SerenityJS. Since many front-end developers are also responsible for testing, a JavaScript version of an automated test tool is very helpful.


Compared to most of the other tools on this list, Cypress is a more developer-centric test automation framework that focuses on making test-driven development (TDD) a reality for developers. One of its design principles was to be able to package and bundle everything together to make the entire end-to-end testing experience pleasant and simple.

It has a different architecture than Selenium; while Selenium WebDriver runs remotely outside the browser, Cypress runs inside of it.

This approach delivers more consistent results, since it’s able to understand everything happening inside and outside the browser. It also gives you native access to every object without requiring you to deal with object serialization or over-the-wire protocols. Basically, you’re pulling your application into Cypress.

As a result, it can synchronously notify you of every single thing that happens inside the browser so that you have native access to every distributed object model (DOM) element.

Cypress also makes it easy to simply drop a debugger into your application, which in turn makes it easier to use the developer tools while you’re developing.

Robot Framework

If you want to use Python for test automation, you can’t go wrong with the Robot Framework. This mature product, created for testers, uses a keyword-driven approach to make tests readable and easy to create. And it includes many test libraries and other tools you can use. The Robot Framework is Python-based, but you can also use Jython (Java) or IronPython (.NET).

While the Selenium WebDriver library might be the most used external test library, Robot Framework can test things other than websites, such as FTP, MongoDB, Android, and Appium. In addition to all this open-source awesomeness, it has a lot of APIs that help make it as extensible as possible.

The keyword approach used by Robot Framework is great for testers who are already familiar with other vendor-based, keyword-driven test tools, making the transition to open source much easier for them.

If your team is mostly made up of testers, Robot Framework is a great option for your automation framework.


RedwoodHQ takes a slightly different approach from the other tools on this list. It creates a website interface that lets multiple testers work together and run tests from one web-accessible location.

You can write tests in Java/Groovy, Python, and C# to test web applications with Selenium, APIs, or databases using their web IDE, which also includes built-in Git integration.

You can leverage RedwoodHQ’s existing action keywords to start creating and modifying tests quickly. To create a test script, just find the action you want to perform, drag it into your test case, and enter the parameter values it expects.

The built-in IDE lets you create and modify test cases and actions as well as execute tests. You can also run your tests against predefined machines in parallel and see a running history of all your previous tests runs.

RedwoodHQ is tester-friendly and has many built-in features you can use to create complete test automation and test management solutions for your teams.


Developed for testing web applications, Sahi offers both an open-source and a pro version. It works as a proxy server that you can use within a browser. From the Sahi dashboard, you can launch the browser you want to test.

The Sahi controller guides you through the recording and capturing of the interaction you want to perform against the application. When you hover over any element in the Sahi browser, for instance, it shows you all of the actions you can perform against that element.

You can also use the controller to play back your script. Sahi’s record and playback functionality makes it easy to get started with test automation of simple HTML applications. But beware of relying on record and playback; this approach has proved to be a bad practice for creating maintainable, robust automation tests. It’s okay for helping create an initial test, but you’ll need to customize it to make it reliable and maintainable.

I’ve heard quite a bit about Sahi, but my first impression of it is that it’s not as powerful as some of the other options. That said, lots of folks rave about the Sahi pro version.

Galen Framework

If your automation efforts are focused on user experience design (UX) or layout testing, Galen Framework might be a perfect fit for your needs.

Created specifically for UX testing, Galen has a special syntax and rules you can use to describe and verify your application’s layout. It also lets you specify your browser size, then run tests against your application to verify layout specifications. Galen tests also generate detailed HTML reports with screenshots, and it includes a visual image comparison with a cool heat map feature.

You can write tests in a simple text file using the Galen syntax, JavaScript, or Java.


Gauge is one of the newest open-source test automation tools for Mac, Windows, and Linux, developed by the folks at ThoughtWorks—the same company that created Selenium.

Practitioners of TDD and BDD will appreciate Gauge’s focus on creating living and executable documentation.

Gauge automation tests, called "specs," are written in a simple markdown language syntax using Java, Ruby, and C# within developer IDEs such as Eclipse and Visual Studio. You can also extend Gauge’s functionality with its support for plugins.

Gauge is pretty new—it's still in beta—but if your team is into cutting-edge test development, you might want to give it a try.

Citrus Framework

Citrus is an open-source framework that will help you automate integration tests for pretty much any messaging protocol or data format. If you have a test flow that requires that your application interact with other services, then APIs and components using functional test tools such as Selenium can’t help.

If you are using any kind of messaging transport such as HTTP, REST SOAP or JMS, Citrus is the framework or choice to test messaging integration.

Citrus also integrates with Selenium. This is handy if you need to interact with a user interface and then verify a back-end process. For example, say you had to click on a "send email" button and verify on the back end that the email was received. Citrus can just receive this email or the JMS communication triggered by the UI and verify the back-end results, all in one test.

In this way, Citrus gives you true end-to-end test coverage that you can’t get with just a GUI test tool.


If you’re already using Cucumber and need to also test a REST API, Karate-DSL might be a perfect option for you. It’s been around for almost a year now, and more and more folks are adopting it for their API testing needs.

Since Karate’s REST test tool is built on top of Cucumber-JVM, you can run your tests, view reports, and leverage any other Cucumber functionality, just as with any standard Java project.

Time to choose!

That's just a small sample of the many open-source test automation frameworks available out there. By now you should be able to decide which ones best fit your needs.

Before you write that first line of code to create your own framework, make sure there isn’t a library or framework you can leverage. Stop wasting your time reinventing the wheel; check out these automation frameworks first. And for more detailed discussions, visit my automation testing blog.

These nine tools are my picks, but they're just a small sampling of the open-source test automation frameworks out there. What are your favorites? Let me know what you think I should add next time—and why—in the comments below.

Gartner Magic Quadrant for Software Test Automation 2017

Image credit: Flickr

Note: This is an updated version of Joe Colantonio's original, "6 top open-source testing automation frameworks: How to choose".

Topics: Quality