AI is the hottest topic in the tech industry, and there's no shortage of tools claiming to leverage AI for test automation. Given this trend, we decided to explore and evaluate several of these AI-powered tools to see if they truly deliver on their promises. Using our evaluation system, we assess each tool's accuracy, ease of use, cost, and integration with our existing systems.
We used a common e-commerce scenario, like shopping on eBay, to see how each tool handles real-world tasks. This article will summarize our evaluation process and share our thoughts on how well testRigor performs as an AI testing system.
Table of Contents
- About testRigor
- testRigor signup process
- Using testRigor to create test cases with AI
- testRigor Analysis and Scoring
- Final Verdict
About testRigor
In order to properly evaluate testRigor, we need to understand what it claims to do and how it works. According to their website, testRiggor turns high-level instructions into actionable test steps.
We evaluated the testRigor tool by starting with the signup process and then creating a basic test case using the well-known website eBay to perform typical user actions.
testRigor signup process
The testRigor signup process is straightforward, requiring only an email and password. We selected the free open source version of the product.
Using this version, all test cases will be subject to “The Unlicense” open source license, meaning they will be public and searchable on the internet.
Using testRigor to create test cases with AI
Once you sign up, you are presented with a form to create your first test suite. The tool offers an AI-generated tests feature that leverages Generative AI to create tests. There are two options available:
-
Generate a sample test with AI
-
Generate an actual test with AI
First, I chose to “Generate a sample test with AI” and check the results.
This test script outlines a typical e-commerce user journey generated by the AI tool testRigor. It verifies the online shopping platform's search functionality, product selection, and checkout process.
Specifically, the script includes steps to:
-
Performs the user login
-
Search for a product using the search bar.
-
Review the search results for relevance.
-
Select a product to view its details.
-
Add the product to the shopping cart.
-
Navigate to the cart and review the items.
-
Proceed to checkout, providing shipping and payment information.
-
Complete the purchase.
-
Verify the purchase in the user's purchase history.
By selecting the option to "Generate an actual test with AI," a new test is automatically created, added to the test suite, and executed. This feature is particularly impressive as it eliminates the need for manual coding—simply provide the site URL (e.g., https://www.ebay.com ) and the login details.
Additionally, testRigor generates a video of the test case and provides a 'tree' view. This view allows users to quickly see which areas of their application are being tested during each test run.
Recording test cases using testRigor recorder
Now, we will create the same test case using the testRigor recorder plugin. This tool captures test cases in plain English by recording user interactions with the application. Note that the testRigor recorder works exclusively with the Chrome browser and is available as a Chrome browser extension that needs to be added to your browser.
The test case was created successfully; however, the generated code included many repeated steps that I did not perform while interacting with the page.
And when executing the newly created test case, it failed with the following error message:
For some reason, it didn’t include the first step of clicking on the “Sign in“ link. So I had to fix it manually and re-run the test case.
During the second run, the test successfully clicked the "Sign In" link. However, an interesting issue arose: the eBay website automatically changed its language from English to Portuguese (my native language). This caused the test to fail because the locators are based on the actual text values of the elements. For instance, the button text changed from "Continue" to its Portuguese equivalent, "Continuar."
This issue highlights a key benefit of using data-test-id attributes for locators instead of relying on text values. If locators were based on data-test-id attributes, the tests would remain functional regardless of the language changes on the site.
Of course, if testing the actual text values and ensuring they are correct in a particular language is a requirement, then it makes sense for the test to fail. Nonetheless, it’s something to keep in mind while using this tool.
After reviewing the test case and making the necessary adjustments, the test case worked as expected and passed.
testRigor Analysis and Scoring
Accuracy of Generated Code
Rating: 3/5
Strengths:
-
testRigor creates test cases in plain English, which makes it easy to understand and use. It also provides AI-generated tests and a 'tree' view to show which parts of the app are being tested.
-
The tool makes testing simpler by automatically creating and running tests, so you don't need to code manually.
-
It handles forms well but might have issues if form fields or labels change due to language settings.
Weaknesses:
-
Sometimes, the generated code includes extra steps that weren’t performed during the test. It can also miss important steps, like clicking the “Sign In” link, which can cause test failures.
-
The code might have errors or incomplete steps, especially when the website's language changes, which can lead to test failures.
Ease of Use
Rating: 4/5
-
Setup: The tool is easy to set up, especially with the free open source version. You only need an email and password to sign up.
-
Learning Curve: testRigor is user-friendly with its plain English interface and AI-generated tests, making it relatively simple to use. However, it might take some time to get used to how the tool works with different languages and dynamic content.
-
Support/Resources: The tool provides very good documentation, which helps you understand and use its features effectively.
Cost-Effectiveness
Rating: 3/5
-
Pricing Models: testRigor offers a free open source version that is accessible without initial setup costs. The open source version is licensed under “The Unlicense,” meaning test cases are public and searchable on the internet.
-
Cost-Benefit Ratio: The free version provides significant functionality for no cost, but long-term costs could arise if opting for advanced features or privacy. The tool’s automation and AI features could reduce manual testing efforts, offering potential cost savings over time.
-
Vendor Lock-In: testRigor’s use of the open source version suggests that the tests are stored and managed within their system. If you decide to move to a different platform, exporting and migrating test cases might be complex, particularly if they rely on testRigor's proprietary features.
Integration Capabilities
Rating: 2/5
-
CI/CD Integration: According to testRigor documentation, integration into CI/CD pipelines is straightforward with its pre-populated scripts. You can easily copy and paste these scripts into your existing pipeline. For Linux-based systems, a bash script is available, while a PowerShell script is provided for Microsoft systems.
-
Local Development: testRigor is primarily a cloud-based tool, which means its core features and functionalities are accessed through the cloud.
-
Code Storage: testRigor’s free version operates in the cloud, which may mean that test code and results are stored online.
Final Verdict: 3/5
testRigor is a valuable tool for non-developers, offering an easy way to create and manage tests without needing to code. Its user-friendly interface and AI-powered features simplify test creation and execution. However, it falls short of providing experienced developers with more control over test interactions and the underlying code-generation process. Although their website is functional, the user experience and look and feel could benefit from a more refined design; currently, it feels quite basic.
Even non-developers may encounter challenges, as the tool sometimes requires adjustments to address errors and handle more complex scenarios effectively. Additionally, the ability to export testRigor code to plain JavaScript, Python, or any other commonly used language for test creation would be a valuable enhancement. Integration with other tools like Playwright, Cypress, Selenium, or similar would also be beneficial.
While testRigor is progressing in the right direction, I would be cautious about relying on it for a large test suite with over 200 test cases, given the limited control it offers.
Conclusion
AI testing is still an emerging technology. Stay up to date on the latest trends and keep your eye out for the next installment in our Choosing the Best AI Testing Tool series. If you have questions about your own use case, feel free to ask them in our community Discord.
quality engineering
