In wordlwide’s software testing field, when discussing test case management tools, notable names include TestRail, TestLink, Zephyr, and others. Armoury+, launched by KEENLITY, is becoming an increasingly significant option in the software testing field.

We will be conducting comprehensive comparisons between various tools and Armoury+, including pricing models, main features, version control, automation integration, user permission mechanisms, and customer support.

This time, we’ll compare with TestRail.

TestRail, released in 2010, has a longer history in the software testing market and has accumulated a large customer base. It maintains high visibility in the software testing field, and when budget allows, most testers might choose TestRail. KEENLITY’s founder Fabian has also implemented TestRail in his previous work experience, confirming that TestRail is indeed one of the well-performing test case management tools in the market.

However, in KEENLITY’s extensive case studies and enterprise consulting services, the most of companies are hindered by TestRail’s or other similar tools’ high prices, preventing them from implementing effective tools to improve testing efficiency. Even when implemented, companies often resort to shared accounts, which negates the advantages of personnel management and test records, merely serving as an expensive Excel replacement.

Armoury+ is a paid test case management tool developed by KEENLITY, founded in 2021 as a software testing consulting company. Following KEENLITY’s business philosophy of “Focus on Testing,” Armoury+ designs its workflow and features from a software test engineer’s perspective. In April 2024, it launched its major feature update – API testing, allowing testers to send API requests directly within the Armoury+ system, receive responses, and build assertions using built-in widget. This enables companies to establish API automation testing mechanisms and integrate them into CI/CD processes without needing to hire programmers, obtaining test results to determine subsequent actions.

Pricing Model

TestRail prices per user account, with a Professional Cloud user account costing $420 per year, approximately NT$13,000. If you need test case version control, priority support, or other enterprise features, you’ll need to upgrade to the Enterprise Cloud plan at $852 per user per year, about NT$27,000.

Armoury+ prices based on user tiers and feature usage, including API testing functionality. The Starter plan for 5 users costs $329 or NT$9,800 per year, averaging about $65 or NT$1,960 per user annually. Importantly, this price includes version control and customer support features that are premium features in TestRail.

Based on this calculation, we can create the following comparison table:

Regardless of team size, Armoury+ maintains a significant price advantage, offering affordable costs for tool implementation. Companies can evaluate plans and costs based on their usage needs, and testers have better negotiating power when discussing tool implementation budgets with their companies. There’s no need to worry about learning additional server skills or operational maintenance, nor bear the risk of system crashes.

Main Features

Comparing the test case management features that both tools share, there isn’t much difference. Both provide a test case pool (called Test Suite in TestRail and Group in Armoury+), node grouping, and formal Test Runs. Armoury+ offers additional tag functionality, allowing for distributed management of test cases.

In a common scenario where the same test needs to be performed in different sections, TestRail requires writing the same case in each section. In Armoury+, you can use tags to add multiple attributes to cases, such as “Must Test” to indicate required testing, “Frequent Error” to mark frequently problematic functions, or “Feature X” to show additional feature relationships beyond node connections, providing greater flexibility.

There are detailed differences between the two. For instance, when dealing with identical cases of different types (such as testing on different OS versions or regions), TestRail uses variables to automatically create multiple combinations within one Run. Armoury+ uses Run copying functionality, allowing users to create one complete Run and then copy it as needed.

In terms of management hierarchy, TestRail has Milestone as its highest level, while Armoury+ uses Test Run as the highest level. TestRail’s design philosophy is goal-oriented, allowing one goal to be associated with multiple Test Runs. Armoury+ adopts a completion-priority approach, where test engineers only need to focus on active Test Runs in the list.

Version Control

TestRail uses version comparison, allowing users to select and compare two cases. Armoury+ uses a timeline approach, presenting version history from new to old. This design difference reflects TestRail’s developer-oriented thinking versus Armoury+’s software testing mindset.

Taking Git as an example, the widely-used version control tool’s ‘git diff’ command shows code differences between versions, which TestRail’s approach resembles. However, test case changes differ from code changes, being driven by product changes. Therefore, Armoury+ adopts a timeline model to clarify product evolution.

Both designs have pros and cons, and users can choose based on their needs.

Automation Integration

Automation testing is crucial in software testing, with most companies hoping to cover traditional manual execution with automated tests. A common challenge in automation development is “how to effectively present test results,” leading to separate reports for manual and automated testing. Most test management tools now include automation integration features, allowing automated test scripts to call via SDK or API and submit results, eliminating the need for separate integration or custom report formats.

In terms of test case management automation, both tools are comparable.

However, Armoury+ adds API testing functionality and can trigger automatic test execution via API. This provides an additional option for CI/CD test integration: when backend deploys new code, the CI/CD pipeline can trigger Armoury’s Collection or Request level API tests, determining if the new API version has errors based on test results, reducing reliance on manual testing for backend coverage.

User Permission Mechanism

The user permission mechanisms differ significantly. TestRail includes extensive permission settings, including roles, groups, projects, and custom permissions, offering great flexibility from a project management perspective. However, from a testing management viewpoint, this might be an obstacle, as system administrators need to spend considerable time ensuring permission compliance.

Armoury+ has built-in roles (administrator, supervisor, designer, tester, and viewer) with role-based permissions rather than project-based control. This significantly simplifies system administrator time costs in managing members, and each member clearly understands their permissions based on their role.

For example, with 10 testers but two independent projects requiring separate access, TestRail typically requires purchasing 10 user licenses and managing individual access permissions, with ongoing management needed for personnel changes. With Armoury+, you could purchase two Starter plans, getting two independent systems with 5-user quotas each. In this scenario, TestRail would cost about NT$130,000 (Pro) or NT$260,000 (Enterprise), while Armoury+ would only cost NT$19,600, a 6-13x difference.

We should consider whether project-level visibility restrictions are necessary when all testers are from the same company. If the concern is unauthorized data modification, Armoury+ provides Audit Log functionality in all plans, ensuring all test case changes are tracked, eliminating modification concerns.

TestRail’s Audit Log, available only in the Enterprise plan, is accessible only to system administrators. Armoury+’s Audit Log integrates with test reporting features, showing both who modified cases and overall progress status.

Customer Support

The customer support approaches differ notably. Based on previous TestRail Professional Cloud experience, issues are handled through Support Tickets or forum discussions, with priority support experience not discussed here.

Every Armoury+ user account has a customer service chat box, providing direct communication with support staff and guaranteed 24-hour response times, comparable to TestRail’s standard support.

For training, TestRail only offers online training courses in the Enterprise plan or through authorized training institutions at additional cost. Armoury+ includes KEENLITY’s original consulting services for various software testing topics, including Armoury+ system training. The money saved from TestRail can be invested in professional consultants for testing process transformation, yielding greater improvements than system implementation alone.

As a new market product, Armoury+ continuously listens to user needs, with paid users able to suggest desired features and influence KEENLITY’s development direction.

Armoury+’s Disadvantages

Currently, Armoury+’s main disadvantage is its lack of third-party ecosystem integration.

As a relatively new product, KEENLITY’s primary focus is building core software testing functionality. Based on KEENLITY’s enterprise experience, integrating testing tools with other team tools (like Jira, RedMine, Figma) doesn’t effectively improve testing efficiency and is often driven by other team members wanting to see testing status. Following KEENLITY’s philosophy, given the absolute advantage in account pricing, they encourage all team members to have Armoury+ accounts to participate in the complete testing process, rather than treating testing as just a project component.

Additionally, common enterprise features like SAML, OAuth, or OpenID Connect aren’t on Armoury+’s roadmap. In the short term, KEENLITY has no plans for developing integrations with other systems, focusing first on completing most software testing features before addressing this area.

Conclusion

In comparing main features, while both Armoury+ and TestRail focus on test case management, Armoury+ wins with its complete API testing functionality. Software testing encompasses many domains, with test case management being fundamental to manual testing, and API backend being crucial for most web software companies. After purchasing TestRail, developing API testing would require additional tools like Postman or Insomnia, creating another cost barrier. Armoury+’s implementation solves both test case management and API testing needs, with ongoing feature development focused on software testing.

For those seeking to improve testing efficiency with limited budgets, Armoury+ is the clear winner, offering the lowest cost among paid testing systems with similar functionality and providing Chinese language support.

With sufficient budget, TestRail remains an option, but if you have API testing needs or plan to develop API testing, you can’t ignore Armoury+’s built-in API testing functionality. If you need to train your testing team and want to improve and align testers’ skills and thinking, you can combine Armoury+ implementation with KEENLITY’s consulting services to enhance team capabilities.

Want to take your software testing team to the next level? Try KEENLITY’s Armoury+ for test case management and API test management. Register now for a one-year free trial quota, and contact us for a two-month full-feature trial.