AI is the hottest topic in the tech industry, and there's no shortage of tools claiming to leverage AI for test automation. Given this trend, we decided to explore and evaluate several of these AI-powered tools to see if they truly deliver on their promises. Using our evaluation system, we assess each tool's accuracy, ease of use, cost, and integration with our existing systems.
We used a common e-commerce scenario, like shopping on eBay, to see how each tool handles real-world tasks. This article will summarize our evaluation process and share our thoughts on how well testRigor performs as an AI testing system.
In order to properly evaluate testRigor, we need to understand what it claims to do and how it works. According to their website, testRiggor turns high-level instructions into actionable test steps.
We evaluated the testRigor tool by starting with the signup process and then creating a basic test case using the well-known website eBay to perform typical user actions.
The testRigor signup process is straightforward, requiring only an email and password. We selected the free open source version of the product.
Using this version, all test cases will be subject to “The Unlicense” open source license, meaning they will be public and searchable on the internet.
Once you sign up, you are presented with a form to create your first test suite. The tool offers an AI-generated tests feature that leverages Generative AI to create tests. There are two options available:
First, I chose to “Generate a sample test with AI” and check the results.
This test script outlines a typical e-commerce user journey generated by the AI tool testRigor. It verifies the online shopping platform's search functionality, product selection, and checkout process.
Specifically, the script includes steps to:
By selecting the option to "Generate an actual test with AI," a new test is automatically created, added to the test suite, and executed. This feature is particularly impressive as it eliminates the need for manual coding—simply provide the site URL (e.g., https://www.ebay.com ) and the login details.
Additionally, testRigor generates a video of the test case and provides a 'tree' view. This view allows users to quickly see which areas of their application are being tested during each test run.
Now, we will create the same test case using the testRigor recorder plugin. This tool captures test cases in plain English by recording user interactions with the application. Note that the testRigor recorder works exclusively with the Chrome browser and is available as a Chrome browser extension that needs to be added to your browser.
The test case was created successfully; however, the generated code included many repeated steps that I did not perform while interacting with the page.
And when executing the newly created test case, it failed with the following error message:
For some reason, it didn’t include the first step of clicking on the “Sign in“ link. So I had to fix it manually and re-run the test case.
During the second run, the test successfully clicked the "Sign In" link. However, an interesting issue arose: the eBay website automatically changed its language from English to Portuguese (my native language). This caused the test to fail because the locators are based on the actual text values of the elements. For instance, the button text changed from "Continue" to its Portuguese equivalent, "Continuar."
This issue highlights a key benefit of using data-test-id attributes for locators instead of relying on text values. If locators were based on data-test-id attributes, the tests would remain functional regardless of the language changes on the site.
Of course, if testing the actual text values and ensuring they are correct in a particular language is a requirement, then it makes sense for the test to fail. Nonetheless, it’s something to keep in mind while using this tool.
After reviewing the test case and making the necessary adjustments, the test case worked as expected and passed.
Rating: 3/5
Strengths:
Weaknesses:
Rating: 4/5
Rating: 3/5
Pricing Models: testRigor offers a free open source version that is accessible without initial setup costs. The open source version is licensed under “The Unlicense,” meaning test cases are public and searchable on the internet.
Cost-Benefit Ratio: The free version provides significant functionality for no cost, but long-term costs could arise if opting for advanced features or privacy. The tool’s automation and AI features could reduce manual testing efforts, offering potential cost savings over time.
Vendor Lock-In: testRigor’s use of the open source version suggests that the tests are stored and managed within their system. If you decide to move to a different platform, exporting and migrating test cases might be complex, particularly if they rely on testRigor's proprietary features.
Rating: 2/5
CI/CD Integration: According to testRigor documentation, integration into CI/CD pipelines is straightforward with its pre-populated scripts. You can easily copy and paste these scripts into your existing pipeline. For Linux-based systems, a bash script is available, while a PowerShell script is provided for Microsoft systems.
Local Development: testRigor is primarily a cloud-based tool, which means its core features and functionalities are accessed through the cloud.
Code Storage: testRigor’s free version operates in the cloud, which may mean that test code and results are stored online.
testRigor is a valuable tool for non-developers, offering an easy way to create and manage tests without needing to code. Its user-friendly interface and AI-powered features simplify test creation and execution. However, it falls short of providing experienced developers with more control over test interactions and the underlying code-generation process. Although their website is functional, the user experience and look and feel could benefit from a more refined design; currently, it feels quite basic.
Even non-developers may encounter challenges, as the tool sometimes requires adjustments to address errors and handle more complex scenarios effectively. Additionally, the ability to export testRigor code to plain JavaScript, Python, or any other commonly used language for test creation would be a valuable enhancement. Integration with other tools like Playwright, Cypress, Selenium, or similar would also be beneficial.
While testRigor is progressing in the right direction, I would be cautious about relying on it for a large test suite with over 200 test cases, given the limited control it offers.
AI testing is still an emerging technology. Stay up to date on the latest trends and keep your eye out for the next installment in our Choosing the Best AI Testing Tool series. If you have questions about your own use case, feel free to ask them in our community Discord.