For a long time, I've searched for a test case management suite that is a fit for a performance test group. It's been difficult, because most of the solutions out there fall into two categories: functional test systems that have some sort of performance add-ons, and huge suites that do "everything".
For what I needed, a functional test system is a waste of money, and usually has very little that is focused on performance testing. In many cases, the workflow is so rigid that you have to click into every test, or it's geared to a black-and-white pass/fail mentality. For our situation, clicking into all of the tests one by one would drive engineers insane. We have a very well developed automation system that performs the tests for us - we just need a way to provide a roll up of things that are a problem, not every file size and test.
A lot of these solutions also look like they were written in the early 1990s. Still using weird tree controls and flat file databases that you can't access with other tools. Many of them are also very expensive, and come with other features that very few people seem to use - test execution from within a test, feature and customer management, etc. My rule of thumb is to look at what a solution or company provides, and if it is a swiss army knife, chances are you won't like it because it will do many things OK, but not the things you care about really well.
Every 2-3 months I search for a solution, and a month ago, I found TestRail by Gurock Software. After downloading a trial (super fast and easy - can't say that about many of the other folks in this market!) and installing it, within about a day I knew it was the solution for me.
It has all of the basic things you would expect - test cases, suites, runs, milestones, and more. But it doesn't get too deep in the details so that it takes months of data input before you can see results. That's probably one of the best things about TestRail - you can get started and start seeing results immediately.
One of the features that people here like is the ability to select a group of tests, and change the status with a single dialog. This allows them to run a complete set of tests that last 20 hours, select all but the failed ones and mark them passed, input the link to the results in our custom database - all with a minimal amount of clicks. Then they can tag the one that did fail in the same amount of short clicks.
TestRail also has very simple but powerful integration with bug trackers, which we setup in about 5 minutes. This allows us to file new bugs directly from the TestRail interface, and tag the TestRail result with that ID quickly.
My all time favorite feature though has to be Test Plans. This is a big chore for us with our old systems, and required lots of revisions and planning. If we wanted to change one, it meant hours of work and validation. In TestRail, you can create a test plan, add test suites to it (you can include everything from the suite, or only specific tests) and then use what they call "configurations" to add n-number of runs depending on the configurations you select. For instance, we use this to add about 15 test suites to a Test Plan, then we have a bunch of different hardware platforms and concurrencies we test, so we add those as configurations. But, each test suite we've added doesn't need all of those configurations, so we can select the different variations for each and TestRail creates a test run for each that is part of the test plan and can be seen in one unified view.
If you want to modify the configs in the future, simply go in and change it, and TestRail will tell you what it's going to add/delete/change in terms of Test Runs. Very efficient for building out Test Plans for different stages of our release process. This has saved us a huge amount of time!
The TestRail forums and developers have been extremely responsive - usually the same day I've gotten responses to my questions and feature requests. They have a very good roadmap of features coming too.
The only issue we have is lack of authentication support - right now you have to create local accounts and passwords for folks to use the tool. That is on their roadmap as well, and will definitely speed the adoption of this tool along here for sure.
I'm extremely happy with TestRail and highly recommend anyone looking for a Test Case Management system to review it's features online with their tour, and download a trial version as well. It's the best system I've seen in years - it goes deep enough to show you what you need to know as a tester and manager, but doesn't add hours of work to your day to get the data.
Recent Comments