Network Performance Testing

Automation framework rewrite in Perl

Here at work we have embarked on a big project to rewrite our entire automation system for better support and to add more flexibility for new features. The framework started out over 5 years ago, written in TCL and was only targeted to run batches of tests on IXIA.

Since then, it has expanded a huge amount. It now fully automates IXIA, Spirent, several custom Linux based harnesses, switch and VLAN configuration, DUT setup and provisioning, error detection, and a central database and ASP.net web application eitih dynamic graphs and charts.

The entire framework was still being written and maintained in TCL and the team really wanted to change as much as possible to a new language. We considered a number of languages, including python, which was a front runner along with Perl. One big requirement is that the new language must have good TCL integration since IXIA and Spirent both require TCL for the automation we need.

Python had very poor TCL support, while Perl has quite nice integration. So our choice was made, and the team has been sprinting doing development to convert as much as possible.

We are also taking this opportunity to re architect a lot of the system to provide faster provisioning, to parallelize a number of setup processes to decrease time to test, and to centralize a lot of system functions.

A lot of people have asked why we don't use a commercial product for this. I've posted on this before, and the opinion still stands. We have never found a solution that allows the flexibility we need. Most of them are very functional test focused, or hide the statistics and raw data in a way that would require a huge amount of code for us to get at. In other examples, some of the promising solutions from test vendors are changing to rapidly and would require a big hit every time an upgrade comes up.

I think in a year or two some of these solutions will be ready for us to consider. For now it's back to Perl. I will post an update when we're closer to being done, and if possible, an architecture diagram.

How is the Network Performance Testing industry doing?

Just curious for those of you who are out there reading, what you think of the current market and industry?  I've seen a lot of innovation in the last 2 years, and the pace doesn't seem to be slowing.  Acquisitions, new markets, new hardware/features/software seem to keep coming - all good things.  

Compare this to 2-3 years ago where it was still the two big players battling it out, and most of the customers out there were stuck in the middle.  

It doesn't seem like everyone is still playing that game.... now it's more about the value that the solutions add, and less about who has what fancy feature.  It's more about interoperability, ROI and green initiatives, and there are plenty of new folks on the block that actually have mind and market share with key customers.

I think it's a good thing in many ways - especially the innovation and maturity for sure!  But with the good always comes the bad - less of the smaller guys - now you have to be reasonably big to even break into the market - and a host of other smaller things.

What are you seeing?  What do you think are the good and bad changes in the last 2 years?  Post your comments below...

Test Case Management by TestRail

For a long time, I've searched for a test case management suite that is a fit for a performance test group.  It's been difficult, because most of the solutions out there fall into two categories:  functional test systems that have some sort of performance add-ons, and huge suites that do "everything".

For what I needed, a functional test system is a waste of money, and usually has very little that is focused on performance testing.  In many cases, the workflow is so rigid that you have to click into every test, or it's geared to a black-and-white pass/fail mentality.  For our situation, clicking into all of the tests one by one would drive engineers insane.  We have a very well developed automation system that performs the tests for us - we just need a way to provide a roll up of things that are a problem, not every file size and test.

A lot of these solutions also look like they were written in the early 1990s.  Still using weird tree controls and flat file databases that you can't access with other tools.  Many of them are also very expensive, and come with other features that very few people seem to use - test execution from within a test, feature and customer management, etc.    My rule of thumb is to look at what a solution or company provides, and if it is a swiss army knife, chances are you won't like it because it will do many things OK, but not the things you care about really well.

Every 2-3 months I search for a solution, and a month ago, I found TestRail by Gurock Software. After downloading a trial (super fast and easy - can't say that about many of the other folks in this market!) and installing it, within about a day I knew it was the solution for me.  

To start, it has a very modern interface, very similar to some other project management solutions out there.  Using flash for graphs, and the most modern controls for things like updates, refreshes and the like, it just feels like you are being more productive using the system.

It has all of the basic things you would expect - test cases, suites, runs, milestones, and more.  But it doesn't get too deep in the details so that it takes months of data input before you can see results.  That's probably one of the best things about TestRail - you can get started and start seeing results immediately.   

One of the features that people here like is the ability to select a group of tests, and change the status with a single dialog.  This allows them to run a complete set of tests that last 20 hours, select all but the failed ones and mark them passed, input the link to the results in our custom database - all with a minimal amount of clicks.  Then they can tag the one that did fail in the same amount of short clicks.

TestRail also has very simple but powerful integration with bug trackers, which we setup in about 5 minutes.  This allows us to file new bugs directly from the TestRail interface, and tag the TestRail result with that ID quickly.  

My all time favorite feature though has to be Test Plans.  This is a big chore for us with our old systems, and required lots of revisions and planning.  If we wanted to change one, it meant hours of work and validation.  In TestRail, you can create a test plan, add test suites to it (you can include everything from the suite, or only specific tests) and then use what they call "configurations" to add n-number of runs depending on the configurations you select.  For instance, we use this to add about 15 test suites to a Test Plan, then we have a bunch of different hardware platforms and concurrencies we test, so we add those as configurations.  But, each test suite we've added doesn't need all of those configurations, so we can select the different variations for each and TestRail creates a test run for each that is part of the test plan and can be seen in one unified view.

If you want to modify the configs in the future, simply go in and change it, and TestRail will tell you what it's going to add/delete/change in terms of Test Runs.  Very efficient for building out Test Plans for different stages of our release process.  This has saved us a huge amount of time!

The TestRail forums and developers have been extremely responsive - usually the same day I've gotten responses to my questions and feature requests.  They have a very good roadmap of features coming too. 

The only issue we have is lack of authentication support - right now you have to create local accounts and passwords for folks to use the tool.  That is on their roadmap as well, and will definitely speed the adoption of this tool along here for sure.

I'm extremely happy with TestRail and highly recommend anyone looking for a Test Case Management system to review it's features online with their tour, and download a trial version as well.  It's the best system I've seen in years - it goes deep enough to show you what you need to know as a tester and manager, but doesn't add hours of work to your day to get the data.

Moved to TypePad and new posts coming

After a bit of a hiatus, I've moved www.perftesting.org over to TypePad, and will be picking back up the daily blog posting.

I've had a lot of issues with my other blogging software, and not enough time to fix them so I could continue posting regularly.  With the move to TypePad, all of those issues are behind me, and I can focus on writing instead of systems administration!

One note - the move to TypePad has broken all of the links that are embedded to other articles/posts on the site - TypePad does permalinks differently than Wordpress (old blogging software).  I'll be working through these over the next few weeks to clean them up.  For now, if you're looking for something and can't find it or there's a broken link, try the search box at the top of each page.

Thanks for your patience during this move!

Interop 2010

I've gotten a lot of invites and information from folks heading to Interop this year.  I hope to have a few posts this coming week on the press releases, announcements, and demos that are going on at Interop.  It looks to be a pretty interesting week for announcements and news!

Been quiet lately...

Hi folks, Lots of stuff going on lately which have prevented me from posting much.  Have had a few comments from some of you wondering what's up...  I will be posting a lot more frequently from now on, and look forward to your comments and interaction. Since I last posted, there have been a few noteworthy items happening... A couple of press releases from IXIA - one on the first field trial of 100GB ethernet by AT&T and IXIA.  Another regarding them joining NTAF, which I will post about later. Spirent's Blog has really taken off and has a lot of good posts on it as well. More to come...

Test Vendor Alliances - going somewhere, or just talking?

Having been a member of the TesLA Alliance, I am surely putting myself out there for some criticism with this post, but I think it needs to be said. I have been a strong proponent for years that test vendors standardize even a small portion of their interoperability APIs so that us end-users can utilize generic frameworks to interact with the multitudes of devices we have in our labs.  I've watched the companies I've worked for, as well as many others, struggle with hiring purpose-trained, highly customized, developers that end up building massive frameworks that cost almost as much to maintain as the benefit they bring.  For the longest time, I assumed none of the vendors would be interested in something like this because they used eachother's inabilities and lack of this-that-or-the-other-thing against eachother, including interoperability. Then along came TesLA - sponsored by IXIA, who I consider to be the leader in the market for the last 4 years or so.  I was ecstatic to see their goals and be a core member of the group.  Unfortunately, not all of the vendors joined TesLA, and some of the key ones, like BreakingPoint and Spirent, were obviously not going to anytime soon.  This made it far less interesting to me as those vendors are key players that I would want to be part of the more global framework. Spirent had an offering here as well, which has mysteriously disappeared. New on the scene is the Network Test Automation Forum which has a bunch of new members, and folks from TesLA as well.  There's been little information on this new forum shared with the rest of the world, and frankly, I have little faith in any effort at this point given the stops and starts that the other two (TesLA and Spirent's) had up until now. Don't get me wrong - I loved the collaboration I saw from everyone in TesLA, and I think it was going in the right direction, but it felt like folks were too worried about making the standards so perfect for every vendor and every case, that we never really delivered anything usable to the end user.  I realize things like this take time, but I think the folks who have used this equipment have been patient enough for years, and someone really just needs to step up and deliver something in 2010. Where do you think we should be focusing efforts?  Standards?  Interoperability?  Or should someone just build a product that works with everyone else's stuff and leave it at that?

Spirent?

Hope everyone is having a good holiday season. I wanted to post more of a question than anything.  At my work, and in several consulting related situations, I interact with Spirent equipment for various testing needs.  In the last year, it has become progressively more difficult to get responses and information out of Spirent for a number of reasons.  This has ranged from support calls, to software fixes and questions, to sales inquiries. Are others seeing this? Spirent has a lot of equipment and customers, and I have been contacted by a number of existing Spirent customers with similar concerns over the last year.  Most people do not have the demands and requirements (at least those I talk to) that my company does for performance test vendors, and as such, haven't run into any of the problems we're experiencing, or at least not to the degree we are.  Another way of saying this - we feel the pain more than most, faster than most. I wanted to see if others are experiencing similar issues, or if there are things I am not aware of that should factor into this opinion.

What happened to the Performance Harness Project?

Some of you have emailed asking what has happened to the Performance Harness Project I started writing about a few months ago.  I wanted to give you all an update... The project started as something I was spearheading myself, and has since taken on a life of it's own.  It's now an official project with larger involvement, which means that it slows down a bit :)  I hope to have some more details in early January as we flesh out the new architecture we'll be using for the system. Developers on my team have spent the last month prototyping a new reservation and scheduling system that will allow us to control more devices from a central point within our automation.  This will include things like L1/L2/L3 switches, our own product, and various test harnesses from IXIA, Spirent, ANUE, and others.  Part of this framework will be used in this new harness.   What we foresee right now is a mixed harness - automated build testing during the day using our customized framework, and more complex customer scenarios during the night, using IXIA Test Conductor. We have set aside two brand-new IXIA Acceleron-NP cards for this project - a huge amount of test capacity (more than 7 times what we originally had dedicated to the project) and re-tooled some of the ANUE and Spirent equipment involved.  We also may be adding some larger 10GB switches to support the different devices we're planning on testing. More details in a few weeks!

Looking for a performance test engineer

Just thought I'd drop a quick note that I'm looking for a Test Engineer focusing on Performance Testing.  Check out the job description on my employer's site, and drop a note here if you have any questions. It's always hard to find folks who have test experience specifically related to performance, so I'm hoping readers of the blog who might know someone looking can pass this info along.  Thanks!