QA Automation

Why should I have more than one runtime environment?

I think there is a real misunderstanding about the importance of the runtime tool.  Why would you want to have more than one tool?  How many should you have?  These are all questions that can be easily answered, but first we need to understand your work environment.  I will take you through a simple example for a small company that develops software.  For argument sake, our company creates software that tracks sporting statistics.  We will call them ACME Trakker. 

Let’s assume that ACME Trakker has two full time QA Authors.  These are the people who write the automation scripts.  They also might write the actual test cases.  Their product ACME Trakker has 300 test scripts in their automation suite.  The average test takes 30 minutes to run.  The tests include variations:  They are capable of running against Oracle, SQL Server and MySQL.  They also support the major browsers.  Do I really need to spell these out? The time it takes to run all these test take 150 hours.  That is 6.25 days!  They have about 75% of their application ready for automation.    Not BAD!

Facts Case 1

·      300 Automated Tests

·      150 Hours to run all automated tests

·      One Runtime tool

·      Two authoring stations.

If they have one runtime tool setup in their lab, it will take six days to run through the entire regression suite.  This may not seem like that big of a deal.  Let’s assume we find one critical error during the testing on day five.  The development group investigates the error and reports it is a major change that will require full regression testing to verify the fix.  Ouch!!!!! We now have been through 8+ days of testing. 

Facts Case 2

·      300 Automated Tests

·      150 Hours to run all automated tests

·      Three Runtime tool

·      Two authoring stations.

We will assume a balanced load on all the runtime tools.    Our new testing time is …. 2 days.  Now that same error is we found earlier is a worst case 4 days to fix and retest. 

As a product manager I sure like the last result a little better.  I am will be able to spend the left over time fixing those cosmetic issues that take manual testing to find.  Plus, these errors cause me a great deal of embarrassment and do not help with first impressions.  Sales!  It really is about sales.

There are many reasons why you should run from a runtime environment.  These runtimes can be setup in virtual machines (VM). I can setup multiple VM on a single PC.  First you would have to understand the load your test put on a machine. The VM can be torn down quickly and reloaded with little effort.  Remember VM or machines in the QA lab are usually clean.  I do not encourage testers run their scripts from their Authoring desktops.    If you are like me, I surf the web and load applications all day.  (Boss it is part of my job.  I am always trying to find great ideas!) 

Take a look at your lab and the number of pc’s you use to test your regression tests.  Let me know how long your testing takes and if one more machine or environment would make your life easier.  Finding 4 hours might just give you time on Friday for Margaritas.