QA Automation

Data Driven Testing

One of the many strengths of automated testing is its ability to data drive your scripts. This ensures that you have a wide coverage of inputs for your tests.  This also allows you to create data sets that can be used repeatedly with verifiable results. QA Wizard Pro allows you to read from both external data (e.g., SQL Server, Oracle, Excel…) as well as the ability to use internal datasheets. These are useful when your corporate DBA does not have time to create a quick data table for your tests.  

Data driving your scripts offer two great advantages. One key advantage is that scripts can quickly iterate through data that is required to complete transactions in the application. You can enter customer data with wide variations. This allows for a deeper and more thorough testing of your product.  
 

The other critical step in data driven testing is your scripts ability to verify the result of the data entered. You can create data sets that contain the expected value or values. Since each row in a data set may have a different outcome, you can checkpoint your scripts to validate the results. You can also create data driven checkpoints to validate that lists, grids, or combos are populating with the correct data. Manually inspecting the accuracy of data in a list is often time consuming and monotonous. Use your internal or external datasheets to verify.  

 

        The benefits of data driving your scripts are endless. Hey the money savings are great,    but making your job a little less tedious — Priceless. 

Why should I have more than one runtime environment?

I think there is a real misunderstanding about the importance of the runtime tool.  Why would you want to have more than one tool?  How many should you have?  These are all questions that can be easily answered, but first we need to understand your work environment.  I will take you through a simple example for a small company that develops software.  For argument sake, our company creates software that tracks sporting statistics.  We will call them ACME Trakker. 

Let’s assume that ACME Trakker has two full time QA Authors.  These are the people who write the automation scripts.  They also might write the actual test cases.  Their product ACME Trakker has 300 test scripts in their automation suite.  The average test takes 30 minutes to run.  The tests include variations:  They are capable of running against Oracle, SQL Server and MySQL.  They also support the major browsers.  Do I really need to spell these out? The time it takes to run all these test take 150 hours.  That is 6.25 days!  They have about 75% of their application ready for automation.    Not BAD!

Facts Case 1

·      300 Automated Tests

·      150 Hours to run all automated tests

·      One Runtime tool

·      Two authoring stations.

If they have one runtime tool setup in their lab, it will take six days to run through the entire regression suite.  This may not seem like that big of a deal.  Let’s assume we find one critical error during the testing on day five.  The development group investigates the error and reports it is a major change that will require full regression testing to verify the fix.  Ouch!!!!! We now have been through 8+ days of testing. 

Facts Case 2

·      300 Automated Tests

·      150 Hours to run all automated tests

·      Three Runtime tool

·      Two authoring stations.

We will assume a balanced load on all the runtime tools.    Our new testing time is …. 2 days.  Now that same error is we found earlier is a worst case 4 days to fix and retest. 

As a product manager I sure like the last result a little better.  I am will be able to spend the left over time fixing those cosmetic issues that take manual testing to find.  Plus, these errors cause me a great deal of embarrassment and do not help with first impressions.  Sales!  It really is about sales.

There are many reasons why you should run from a runtime environment.  These runtimes can be setup in virtual machines (VM). I can setup multiple VM on a single PC.  First you would have to understand the load your test put on a machine. The VM can be torn down quickly and reloaded with little effort.  Remember VM or machines in the QA lab are usually clean.  I do not encourage testers run their scripts from their Authoring desktops.    If you are like me, I surf the web and load applications all day.  (Boss it is part of my job.  I am always trying to find great ideas!) 

Take a look at your lab and the number of pc’s you use to test your regression tests.  Let me know how long your testing takes and if one more machine or environment would make your life easier.  Finding 4 hours might just give you time on Friday for Margaritas.  

How quickly are you going to adopt WPF?

Microsoft has released the new version of Silverlight. This product was formally Presentation Foundation Everywhere (WPF/E). I am very excited about this technology and what it will offer the consumer. I am not real concerned with what our community (Software Development) reaps from this technology.

I think most companies are finally going to break away from the traditional hyper linking websites that they have been creating. User are finally going to be able to use the web with the same rich interface that is offered on their MAC or PC. Web sites today are so boring. I cannot drag and drop to my shopping cart or favorites list. It is hard to offer a rich search function on the web.

This blog is not a place for me to spew love or hate for a company and their technologies. Adobe will be equally impressive with their implementation of Flex. The reason I bring this up is I am interested in knowing how quickly companies are going implement some form of this technology. I have my own selfish reason. I buy a lot on the web and the fourteen pages to check out is eating into my free time.

Give me an idea when your company will implement Flex or SilverLight Technologies.

Compliance

Manual testing can be a time consuming and often laborious task. Companies that perform manual testing struggle to have consistency amongst different browser and operating systems. Manual testing can often take too long and does not guarantee repeatability.

Automated testing can improve several areas of your QA department. A benefit that most companies miss in automated testing is compliance. Manual testing does not automatically confirm nor deny that the test has been run. Using a test case management tool and automated testing application your organization benefits from traceability. You have reports that show what you have done, how you went about testing and the results of each step. This is a laborious process for a manual tester.

Every day companies face more and more regulations. Many companies think of automation as a way to save money or improve time to market. Automated testing is a great way to prove your work and show that your product has quality built into it.

What do I do?

I am always asked how I would organize my scripts and workspaces. I am also asked to explain how and when I record my scripts.

Before I start automating I outline the test I want to automate. I then try and break down the test into bite size pieces. If I have a test that requires me to login to my application and create a new contact, I try and view it as two tasks or two scripts. The first task or script would be the login. I first create a new script and record the login process into it. Most tools encrypt your password so no worries. Once my Login script is recorded I then save it as a separate file and store it in an area for either the test or utility scripts.

Creating a new contact sounds easy but my test requires that I iterate through a dataset stored in the company database. My first step would be to record the new script. I enter some sample data into all the fields that I may be testing. This allows the tool to record my entries. After a successful record, I then replay my script. I get it working just the way want it. I am now ready to setup my dataset. Each tool its different, but QA Wizard allows me to select the datasheet I want to iterate. I change the typed text statements and enter the column locations for each controls data point. I run it to make sure it works.

All I have left is to make an executable test. I create a batch adding the two scripts I created. I then send my batch to the scheduler or runtime tool and away we go.

I always try and break down my tests into the smallest bite size chunks. This allows me to reuse them and to make changes without destroying the main areas of my test.

Planning for Success

All too often QA is an after thought in the project planning stage. I have taken over many projects that are going the wrong direction or have already turned the corner heading for disaster. One of the many problems that I detect is that QA was never included in the planning phase or the design phase. I receive comments from team members that boggle the mind. “They can plan when we know what it will look like.” This is just plain wrong. I was going to use the word stupid, but I thought it might be too strong.

Planning for success requires a solid team approach. That means that QA is involved with the design phase, as well as, including them in the requirements phase of the project. This statement is even more critical if you have an Automation department. In order for automation to be successful, you must plan for success.
Some areas that are critical to your planning are:

• Identify areas that require larges amount of data entry.
• Discover areas that may have repetitive testing with iterative data.
• Look for areas that may require data validation.
• What area of the application has complex business rules?

These areas all require a great deal of planning in order to be successful with automated testing. Which of these areas that I have identified above is QA not involved? I would argue they should be involved in all of them. My experience tells me that you will save over 50% in script and manual test development if QA can develop their scripts early, even against a prototype.

Quality and Perception Management

Most of you are old enough to remember the late 70’s and early 80’s. Some days I feel like everyone around me was born in the 80’s. The reason I bring up this era is not because of big haired rock bands or Van Halen (Rocks). It is to discuss quality. Many American companies were being criticized for their quality standards. This perception of reality hurt many of these companies and they lost huge market shares. The consumer’s perception was simple, “big companies do not care about quality”.
What does this have to do with the quality of your software? As a product manager for a commercial product I am always concerned with our initial image. A product should have good manners. Similar to what I tell my sons; use your manners when you are over (insert friend’s) house. People download and review your product daily. Quality is not just about how few defects you prevent going out the door. It is also about establishing the right image for the product.

I have seen products that work well, but the spit and polish is lacking. A product without polish will be hurt by first impression. You only get one shot at first impression.

Ask the Big Three automakers about first impressions. They suffered through some horrific times; at least their customers buy a new car every 3-4 year. In the software industry, you almost never get the opportunity to win back a customer. How much does it cost you to fix “Perception.” My guess? It is a lot cheaper to fix the reality.

Conversation with a customer

I spend a lot of time talking to customers. I think it is one of the best parts of my job. I enjoy talking to all my customers; even those that want to perform open-heart surgery on you with a blank DVD.
I recently had a conversation with a customer that shocked me. I have a set group of questions I always ask. I almost always get the same grouping of answers. Here are some samples:

• Tell me a little about your company’s commitment to quality?
• Describe your companies QA department?
• Why did you choose to buy a Quality Automation tool?

This last question is always answered one of two ways. Most customers answer with we want to reduce our manual testing on repetitive tests. Great reason! Another answer I always get is; we want to get our product to market faster, so we wanted to automate the repetitive tasks. Another great reason!

This customer’s answer really set me back on my heels. He said, “I have a very high turnover in my QA department. Once I started to automate, the testers felt like they were no longer drones pushing buttons.” This was one of the most enlightening conversations I have had with a customer. Not because they use automation to launch the space shuttle (not really), because they used it to keep employee turnover cost down. How much does it cost a company when a tester leaves? If you are a small company (1-2 testers), how important are your testers?