Network Performance Testing

Automating with Test Conductor

I've spent a lot of time in the last few days working with IXIA's Test Conductor product, which is the cornerstone to our Performance Harness Project.  I've used it in the past, and know the basic portions of it pretty well. This time around, we're focusing on two main use cases for this harness:

• Quick baseline tests between builds of product during the development process.
• Complex scenario testing that can be automated

For the first one, Test Conductor has a lot of great features that we've used in the past to help.  The most important being the pass/fail criteria that allows you to specify percentages of tolerance against a baseline.  It's really quite slick and has saved us a lot of time - first you build your tests and regressions, and set the pass/fail criteria to have a baseline tolerance for a particular value or set of values.  Then, you run your regression and tell it to collect baseline values.  Once that is done, you are ready to run against something and see if the performance has varied.  Simply start the regression and run it normally, and the pass/fail criteria will evaluate your baseline and give you the result. It's a super simple way to check if performance has changed without complex analysis, graphs, reports, or worse - manual examination.  And it doesn't have what many other solutions have - hard coded values and engineer input.  In our use case, we simply reset the baseline on the next build after we've validated things are OK, and get ready for the next build coming down the line. We've designed a pretty focused set of tests that covers key features so we can try to get coverage on as many areas as possible.  It's not the kitchen sink, just a quick validation point.  This enables our test engineers to wait for the result, which only takes a short while, before moving on to more detailed testing through our other large automation systems. The second use case is one we're hoping to take more advantage of with Test Conductor - complex scenarios that can be automated.  Why do I say "can be automated"?   Lots of scenarios are complex, and many of them are so convoluted that to automate them would take more time than it's worth.  We test many of these already and have great solutions for them.  What I'm looking for is new ideas, new IXIA tests from customer visits, and complex existing configurations that the various other devices we have in the harness, such as the Apcon, can assist with. So far I've come across a lot of features in Test Conductor that are going to make this far easier than I thought it would be.  In particular the functionality in Test Composer is going to be key to making all of this happen.  So far we've created procedures that allow us to gather basic information from our DUTs, switches, and other devices, review the information and make decisions on what to do within a test or regression - all in just a day or two.  And, without coding in TCL or some other language.  Key features we're using so far include DUT Configurations, Procedures, Response Maps, and more.  As we get specific examples tested and better documented, we'll post a couple here.

IXIA + Arista = 10GBASE-T

IXIA announced a test, which they will be showing at Interop this week, of Arista Networks' new 48-port 10GBASE-T solution.  They used their IxYukon 10GBASE-T solution, which can be scaled up to 96 ports per chassis. From a pure deployment scenario, I haven't been seeing a lot of folks using 10GBASE-T yet, but it is an interesting choice.  In thinking about fiber versus copper, most of the solutions I've seen have been focused on fiber, and most people have built the infrastructure to support that.  I can see 10GBASE-T being popular within an individual rack itself, as it is much easier to cable, and not as prone to problems with damage, etc. However, most of the articles I've read focus on folks re-using CAT5e/6 architecture for 10GBASE-T which is also an interesting idea.  I think we'll see this pick up over the next year or two as folks add more and more virtual computing clusters that require larger data connections, but I'm not sure how many people actually use it now... Anyhow, interesting test, and definitely an interesting switch from Arista.  For those of you at Interop, you should stop by IXIA's booth and check it out.  I'm not there this year, so it'd be great if someone can post here any of their thoughts on the test itself at Interop.

Leaders in IP Testing Announce the First Multivendor, Open Automation

A huge list of vendors today announced a partnership to collaborate on an open, multi-vendor solution for test automation, called TesLA. It seems to be led by IXIA, but there are some other big names along with them:

• Apcon
• Codenomicon
• Fanfare
• IXIA
• MuDynamics
• QualiSystems
• Shunra

It sounds too good to be true - a single framework for managing all of these types of devices from all of these vendors. I hope others will join as well so that we're not stuck with just one choice for some of these areas... I think this is a great idea, and I look forward to seeing the results of this sort of collaboration - I'm actually quite surprised that some of these folks are working together, since they have competing (somewhat) products. I am sure a solution like this won't work for everyone, and I am hoping folks in the alliance will consider the highly customizable nature of this sort of testing, and make sure their solutions keep that aspect in mind - many of us have needs to use our own external <insert whatever it is here> and will need a lot of flexibility in customizing any centralized tool to manage things. Full press release can be found here: Press Releases :: Ixia - Leader in IP Performance Testing