Network Performance Testing

Automating with Test Conductor

I've spent a lot of time in the last few days working with IXIA's Test Conductor product, which is the cornerstone to our Performance Harness Project.  I've used it in the past, and know the basic portions of it pretty well. This time around, we're focusing on two main use cases for this harness:

• Quick baseline tests between builds of product during the development process.
• Complex scenario testing that can be automated

For the first one, Test Conductor has a lot of great features that we've used in the past to help.  The most important being the pass/fail criteria that allows you to specify percentages of tolerance against a baseline.  It's really quite slick and has saved us a lot of time - first you build your tests and regressions, and set the pass/fail criteria to have a baseline tolerance for a particular value or set of values.  Then, you run your regression and tell it to collect baseline values.  Once that is done, you are ready to run against something and see if the performance has varied.  Simply start the regression and run it normally, and the pass/fail criteria will evaluate your baseline and give you the result. It's a super simple way to check if performance has changed without complex analysis, graphs, reports, or worse - manual examination.  And it doesn't have what many other solutions have - hard coded values and engineer input.  In our use case, we simply reset the baseline on the next build after we've validated things are OK, and get ready for the next build coming down the line. We've designed a pretty focused set of tests that covers key features so we can try to get coverage on as many areas as possible.  It's not the kitchen sink, just a quick validation point.  This enables our test engineers to wait for the result, which only takes a short while, before moving on to more detailed testing through our other large automation systems. The second use case is one we're hoping to take more advantage of with Test Conductor - complex scenarios that can be automated.  Why do I say "can be automated"?   Lots of scenarios are complex, and many of them are so convoluted that to automate them would take more time than it's worth.  We test many of these already and have great solutions for them.  What I'm looking for is new ideas, new IXIA tests from customer visits, and complex existing configurations that the various other devices we have in the harness, such as the Apcon, can assist with. So far I've come across a lot of features in Test Conductor that are going to make this far easier than I thought it would be.  In particular the functionality in Test Composer is going to be key to making all of this happen.  So far we've created procedures that allow us to gather basic information from our DUTs, switches, and other devices, review the information and make decisions on what to do within a test or regression - all in just a day or two.  And, without coding in TCL or some other language.  Key features we're using so far include DUT Configurations, Procedures, Response Maps, and more.  As we get specific examples tested and better documented, we'll post a couple here.

Shenick and Fanfare partner to build Integrated IP Communications Test Solution

Shenick announced today that they are partnering with Fanfare to add full plug in support to the Shenick diversifEye™ product.  According to the press release, this will allow users of Fanfare's iTest product to control and manipulate per-flow testing from the Shenick product. A lot of the press release is focused on Service Providers and wireless markets, which is an area that Shenick focuses on more than many of the other players.  Direct from the press release: "NGN and LTE services and applications bring with them increased quality of service (QoS) and quality of experience (QoE) demands. Per-flow test and measurement is the only mechanism to guarantee performance at these granular levels enabling providers to pinpoint application and individual user issues accurately" I think it's great that this support has been added on both ends so that users of iTest can take advantage of existing or new Shenick equipment.  I hope there will be more details about how deep the integration goes in the coming weeks. One interesting thing is the fact that both Fanfare and Shenick are part of the TesLA Alliance - I wonder how much the proposed TesLA standards have influenced this solution, and if we'll see more of this in the coming months since the standards are getting closer to being approved...

TesLA Webinar: Automating the Provisioning and Configuration of Test Architectures

The second in a series of webinars by TesLA is happening this week.  I'm excited because I'll be speaking on the challenges with automating test architectures, and how I believe TesLA will help solve these challenges.  In particular, I'm looking forward in discussing interoperability standards and the challenges now with automating different devices all in one test architecture. Stop by and join us this Thursday at 10AM PST - link to webinar below. Hope to see you there! Webinar Details: Thurs., May 14 @ 10 a.m. PST - 2nd Installment of TesLA Webinar Series, Steve Mitchell of F5 (TesLA CAB member) Register ahead of time at: https://www2.gotomeeting.com/register/886587307

Pretty quiet lately...

Not a lot of new news out there in the last few weeks.  Have been looking around diligently for something new and exciting, but it seems folks are hunkering down for a while. At work, we're deep into figuring out lab and test bed automation, and reviewing a bunch of solutions from various vendors.  I hope to have more details for a posting in a week or two...

TesLA Alliance Quarterly Meeting

I'm on the Customer Advisory Board for the TesLA Alliance and we had our second quarterly meeting this last week.  I missed the first meeting due to being sick, so it was great to finally meet the rest of the TesLA Members and the Customer Advisory Board as well. I am really excited about all of the new things that we're planning to do, and very appreciative to be involved with the entire alliance.  The caliber of all of the members and board is top-notch, and there are many very exciting dicsussions going on that will forever change this industry in the positive. Why is it so exciting to me? Well, first you have a collection of Members that span the performance, functional, and lab automation arenas.  Some of these vendors are bitter rivals, yet they met together, and all agreed on many things that will help to not only unify the market, but provide far more benefits to end customers.  There are even more vendors that are interested, and attended the first part of the meeting to see what it was all about. Second, you have the goals of the alliance, which are exactly what I've been looking for.  Defined and published standards.  Framework to allow vendors to communicate between each other.  Completely open to all vendors.  An alliance focused on all of these things, and not on building their own product. Third, the momentum behind this is amazing, and the amount of progress that has been made in the last few months shows that this is being done at the right time, and the right place.  So many people agree that the industry needs this ASAP. Finally, the customer advisory board - my colleagues are all top-drawer folks from varying parts of the industry.  One of the members was adding up the experience we all have together and we came up with something around 80 years!  We're the reality check for the rest of the alliance to make sure that what they're coming up with meets customer expectations.  It's a privilege to work with these folks on such a critical standard for our industry. What's next? I would like to see even more vendors participate in the alliance.  There are some noticeable key players that are still playing politics, when they should really just put those aside and get on board. There's a lot more planned, and a lot of new things happening every week.  Check out the Activities Calendar, or subscribe to the RSS feed for the site.  I'm looking forward to the ongoing participation, as well as the new folks who will be joining.

Gale Technologies - silent?

Does anyone know what's happened with Gale Technologies?  Their news page has been empty for months, and the only other updates are a few customer stories... Many of you will remember that Gale was formed by the merger of QuikCycle and Edentree which I wrote about back in October.  Both were pretty major players in lab and test automation.   I sure hope we'll see something new from them soon! UPDATE: They are very much still alive - I bumped into some of them a few days ago at the TesLA Alliance Quarterly Meeting.

Test Automation Standard

Some really good news today - the TesLA Alliance announced the first draft of a test automation standard.  I'm really excited to see this at the TesLA quarterly meeting coming up next week, which I will be attending. TesLA is also hosting their first of a series of webinars starting this week.  The first one is entitled "Open Testing Standards Emerge: Enabling Seamless, Shareable and Platform-Independent Automation" and features HP ProCurve's Vince Mendoza who is part of the TesLA Customer Advisory Board.