Q&A

How likely is it for guidance systems to evolve over the development period of a particular device? Does a company have to go back to older data and documentation to meet newly instituted guidance later in the development cycle?

(Avellanet – Cerulean Associates) Here is the answer that the data gives since about 1980 or so. Number one, it is very frequent that a guidance associated with your type of device will be developed while you are in the process of developing your device. If I had to lay odds based on everything that I have seen, I would say you probably have a 99.9% chance of a guidance coming out. It may only be relevant to a small portion to your device or development, but there is simply no data to go back and say that this guidance came out and only impacted the labeling on this device as opposed to wholesale design changes, etc. To be conservative, I would say this is going to happen without question.

Here is the exception: if your device has some miraculous cloud appear and you can get it done in less than three to four months or even a year, from idea to actual submission, you may be able to escape having to meet newly instituted guidance. In general, from the moment you hear someone at the FDA say we should do this (such as at a conference), you have about a three-year window before you can expect that guidance to come out. The exception being the infamous social media guidance and the revisions to Part 11 compliance. If your device development through submission takes less than a year you have a good chance of not dealing with it.

Now, how likely is it that a company has to go back to older data and documentation and meet newly instituted guidances that are released late in the company’s device development? Essentially, you’re going to have to do the same thing that I did and everyone else has to do. You are going to have to take a calculated risk. And the risk is this: if it is a guidance document, let’s say something associated with current device safety impacting exactly what you are about to submit, you are kidding yourself if you think that FDA reviewers are not going to have been involved in at least helping to review that guidance. Therefore, they will be looking at your submission for adherence to those guidelines. If, however, it’s a guidance on early preclinical development activities and it came out long after you completed the early stages of development, then you have to make a risk assessment. You need to ask yourself how likely it is that the guidance will impact your submission to the FDA if you don’t tackle at least some elements of the guidance. I’m not saying go back and re-do tests. I’m suggesting, however, that you decide if you should we draw out elements of what you did way back when so you can show the FDA that you were thinking about this type of thing way back when. You have to make that risk judgment, and that’s the best you can do.

Is it possible to get a copy of the sample project used in the webinar?

(Nicholson – Seapine Software) The sample project will be available at the beginning of the year, in the TestTrack 2012 release. At that point, it will be included in TestTrack as a sample project that you can use..

What level of administration resources are required to manage a system like TestTrack?

(Nicholson – Seapine Software) Initially, you’ll need a few days for training and configuring TestTrack to match your business compliance processes.. Once those activities are completed, it depends on how much you want to tweak TestTrack or how much your development process is changing. TestTrack is easy to administer, and many companies don’t even need a full-time TestTrack administrator.

What operating systems does TestTrack support?

(Nicholson – Seapine Software) TestTrack supports Windows, Mac OS X, and Linux.

Can outside design partners and other third-party users access a TestTrack sections set up just for them?

(Nicholson – Seapine Software) Yes, using TestTrack’s comprehensive security groups, you could configure specific viewing capabilities with read-only access.

The FDA wants to see validation of software used in the quality system, how you can help us validate TestTrack?

(Nicholson – Seapine Software) We can provide assistance through our Professional Services team. We have also worked with other consultants and partners that do this for a living. It’s really your decision on how much involvement you need. We also provide reports out of TestTrack that assist with these documented validation efforts.

Should the Investigative Review Board (IRB) play a role in traceability and documentation?

(Avellanet – Cerulean Associates ) I assume that you are talking about the investigative review board in terms of clinical testing and design. I think in that concept, one of the things that we want to talk about is that you want to view the IRB almost as an outside design partner. Thus, their role would be really to just review design documents. I am not necessarily assuming or thinking that they are going to go in and do testing. Certainty no IRB that I have been associated with wants to do that. But they do want to review documents. Giving them read-only access to any type of automated system is a great way to go about it because what’s going to happen is they are going to create their own sets of internal IRB documentation.

You also want to be careful about including that internal IRB documentation in your design history file. In a lot of cases it’s not necessary and, if the agency has actual questions, they will ask you to provide the specific contact information for the IRB (and point them in that direction). So, let the IRB be responsible for their own documentation. The exception being once your device has been on the market for years, you may want to pull the actual records from the IRB and then incorporate it into a big archived design history file that goes into your archive history and stays there for its full retention period. If you want to make sure your FDA recordkeeping is appropriate, think about finding an outside expert in “FDA records management” or “FDA document retention” (those are good internet search terms).

I presented two breakout sessions: “When to Ship” on Tuesday afternoon and “Measuring Technical Debt with Load Testing” on Wednesday morning. The sessions were videotaped, and if the links are posted I’ll provide them in an update to this post.  I was also interviewed on video by the conference chair about why I was participating in the conference and what I was getting out of it.

I sat in on a variety of sessions, the most compelling being a discussion of testing anti-patterns for Agile projects, and how those anti-patterns could be overcome. The speaker was Bob Galen, an Agile consultant and author of Scrum Product Ownership. Some of the anti-patterns included were that developers and testers didn’t work together, and that certain automation tools and practices were too rigid to adapt to change. Bob’s point was that once these anti-patterns were recognized, they could be overcome to make Agile testing a success.

Microsoft also had a well-attended session in which the topic was a highly marketing-oriented talk on Microsoft Test Professional and Team Foundation Server. While product descriptions and demonstrations are normally frowned upon, the audience appreciated the opportunity to see a Microsoft approach to automated testing and test case management.

The keynote talks included “Test Automation: Past, Present & Future” by industry veteran Dorothy Graham, and “Preventing Zero Day Attacks Through Automated Security Testing” by Joe Jarzombek of the Department of Homeland Security.  Graham spoke about trends from the past, how they were manifesting themselves today, and what scenarios testers were likely to face in the future. The security talk was geared largely toward the inevitability of threats and vulnerabilities, and the need to understand the nature and characteristics of both in order to successfully test application security. As might be expected by a talk given by a government security expert, it was heavily geared toward a regulatory understanding of the problems, but the definitions and characteristics of threats and vulnerabilities are useful in any security testing practice.

Possibly the most important thing I learned at this conference is that Agile and automation are inseparable. Agile methodologies are supposed to make full use of automation to accelerate repetitive tasks and to track data that is too cumbersome or time-consuming to record manually. Too often we interpret Agile as automation-unfriendly, but that’s an oversimplification. The goal is to choose tools and automation opportunities that don’t require more effort to set up and maintain than they save. Tools that are easy to learn and use can go a long way toward accelerating even Agile processes.

Peter Varhol will be presenting When to Ship: Using Automation to Determine Application Deployment Readiness at 2:45 p.m. on September 27. During this presentation, Peter will provide measurable criteria for determining when an application is ready to ship. He will also discuss how to create objective measures of completeness and how to collect data and track progress toward that end. Peter will explain how testers and quality assurance professionals can work with application stakeholders to contribute to both the definition and automated measurement that determines if an application is fit for its intended purpose.

Peter will also present Measuring Technical Debt Using Load Testing at 10:30 a.m. on September 28. This session focuses on the role of technical debt and how to assess technical debt with frequent use of load testing. Peter will provide a protocol for using load testing as a part of the testing process, and using it to assess the architecture and coding quality. He will also provide a mechanism for testers and developers to collaborate in decisions surrounding design and coding practices.

Can’t make the conference or want a sneak peek? You can view the PowerPoint slide decks that accompany the talks right now:

• When to Ship: Using Automation to Determine Application Deployment Readiness
• Measuring Technical Debt Using Load Testing

About the Conference

The Verify/ATI Conference 2011 is organized by Verify and the Automated Testing Institute. It will be held on September 26-28, 2011 at the Crowne Plaza National Airport Hotel in Arlington, VA, this conference provides a platform for attendees to learn strategies, techniques and best practices from peers and leaders in their field.

Who Should Attend

This conference is beneficial for software test engineers, software test automators, application developers, project managers, product managers, business analysts, and security engineers.

To do so, I started by creating the following Windows .bat file:

@echo off

set QAW_VirtualMachinePath="C:\Users\peterv\Documents\Virtual Machines\Windows XP VMWare\Windows XP VMWare.vmx"
 set QAW_RunScriptPath="C:\Program Files (x86)\Seapine\QA Wizard Pro\QAWRunScript.exe"
 set QAW_WorkspacePath="C:\Users\peterv\Documents\QA Wizard Pro Workspaces\VMTest\VMTest.qawwspace"
 set QAW_TestScriptPath="C:\Users\peterv\Documents\QA Wizard Pro Workspaces\VMTest\Sample Windows Script.qawscript"
 set QAW_ShutdownScriptPath="C:\Users\peterv\Documents\QA Wizard Pro Workspaces\VMTest\Shutdown Computer.qawscript"
 set QAW_ReportDir="C:\Users\peterv\Documents\QA Wizard Pro Workspaces\VMTest\VM Results"
 set QAW_RemoteMachineName=192.168.248.128
 set QAW_CleanSnapshot="Clean State Snapshot"
 set QAW_SnapshotName="Script Error %DATE% %TIME%"

REM **********************************************

echo Running VM.
 start "VM" "C:\Program Files (x86)\VMware\VMware Workstation\vmware.exe" -x -q %QAW_VirtualMachinePath%

echo Waiting 2 minutes for the VM to start up.
 ping -n 120 127.0.0.1 >NUL 2>&1

echo Starting QAW Test Script
 set QAW_TestScriptSuccess=yes
 %QAW_RunScriptPath% %QAW_WorkspacePath% %QAW_TestScriptPath% /reportdir %QAW_ReportDir% /remote %QAW_RemoteMachineName%
 IF ERRORLEVEL 1 set QAW_TestScriptSuccess=no

echo Shutting down remote machine.
 %QAW_RunScriptPath% %QAW_WorkspacePath% %QAW_ShutdownScriptPath% /noreport /remote %QAW_RemoteMachineName%

echo Waiting 2 minutes for the VM to shutdown.
 ping -n 120 127.0.0.1 >NUL 2>&1

IF %QAW_TestScriptSuccess%=yes GOTO SkipTakingSnapshot
 "C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe" snapshot %QAW_VirtualMachinePath% %QAW_SnapshotName%

:SkipTakingSnapshot

echo Reverting to base snapshot.
 "C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe" revertToSnapshot %QAW_VirtualMachinePath% %QAW_CleanSnapshot%

This example batch file uses VMware, but you can configure it to use a virtual environment of your choosing. You can also select the script you want to run in the virtual environment. The sample script and shutdown script that I use in the following example can be found in this VMTest QAWizard Pro Workspace.

You need to set up your virtual environment to ensure QA Wizard Pro will be able to run scripts on it. To do so you will need to:

• Set up the virtual machine to automatically log in to Windows. I used Windows XP in this example.
• Install QA Wizard Pro and the appropriate licenses.
• Have QAWRemote start when the virtual machine starts up. You can find QAWRemote.exe in the “Program Files\Seapine\QA Wizard Pro” folder. To have it start automatically, you can create a shortcut to it in your “Startup” folder in the Windows Start Menu.

After I completed the set up tasks, I made a VMWare snapshot of the machine with the name “Clean State Snapshot”. This allows me to reset the virtual machine to a known starting state each time I run a test.

Running the above batch file causes the virtual machine to start up:

Once started, the script is sent remotely to the virtual machine. It runs to completion and then sends back a report file with the results of running the test.

After the script completion, I run another script to force close any open programs and shut down the remote machine.


In the case of a previous error, the host machine then takes a snapshot of the virtual machine so I can view the state of the machine that the error occurred in. It then sets the virtual machine to the “Clean State Snapshot” that I saved when setting up the virtual machine, which ensures the machine is in a known good state the next time this script runs.

The resulting report from the script run is saved to the host machine. It can be copied to a different destination or emailed to others that are interested in the results. You can modify the above batch file to work with different virtual machine environments. Also, since the scripts are running in an environment separate from their host machine, you can use Windows Task Scheduler to run the batch file overnight, even while the host computer is locked.

Seapine ALM can send automatic notifications. Surround SCM and TestTrack can notify users when something happens under specific circumstances. In fact, Surround SCM 2011.1 introduces enhancements that provide more dynamic notifications, such as notifying the last user that checked in files, the user selected in a custom field, and more.

If notifications are not set up properly, you risk spamming your users with emails that are not of interest and just make noise. Before you know it, users set up a rule to route emails sent from Seapine ALM to a folder that will rarely be reviewed.

While the mechanics of notifications in TestTrack and Surround SCM are different, the following high-level best practices apply to both.

Harness the Power of Notifications

Waiting for a user to send an email about something that happened introduces unnecessary risk. If you need to manually send an email to let someone know that you assigned a defect to them, the time between the assignment and sending the email is time that the item is not worked on. What if there is a fire that you need to put out? More time that the item is assigned and the assignee is unaware of it.

If you have specific SLAs that require addressing items in a short timeframe, you cannot afford to use manual notifications.

Once you have configured rules for notifications, let Seapine ALM do the work for you and help you eliminate the gaps in your process that are caused by relying on manual intervention.

Select the Right Condition

Some notifications only make sense under specific conditions. Maybe a defect needs to be a certain priority, a requirement needs to be further along in the review process, or a file needs to be reviewed by a development team lead for possible bugs.

Both TestTrack and Surround SCM provide ways to test for a specific condition. TestTrack takes advantage of filtering functionality, while Surround SCM lets you to specify pre-conditions for a specific rule. Both help you ensure that the notification only happens under the conditions that make sense.

Surround SCM Preconditions for Notifications

Trigger on the Correct Action

Be Selective About Recipients

Control the Message

In Conclusion

In my previous life as a solutions consultant, I recall proposing using notifications to clients. They would either embrace the suggestion or look at me like I had just asked them to join me in a risky business venture. If they did not want to implement notifications, it was usually because of a bad experience with another system that did not have the flexibility that Seapine ALM offers and resulted in all their users receiving emails about everything and anything.

With Seapine ALM, this does not need to happen. You have the flexibility to ensure that emails are only sent to the right person, under the right conditions, and with the correct information.

Automate testing processes and activities.

In one sense, Agile methodologies were a reaction to the over-tooling of traditional methodologies. Many Agile proponents believe that too much time was being spent operating and collecting data for tools instead of focusing on the actual work to be done.

But automation is essential in many aspects of Agile testing. Automation enables the easy collection and analysis of data that enable testers and other project team members to make decisions about the quality and completeness of work. And testers need to make use of automated defect tracking to make sure a feature is retested when a defect is found, even if the defect is fixed right away.

Automated functional and regression testing even have roles in Agile. Even if you don’t think you’re going to run a test more than once, it’s important to record it and place it in the automation suite. You never know when you’re going to need to execute it again, possibly in response to other changes or in the regression suite.

But, perhaps, the most important role for automation in Agile is recording and tracking user stories. Forget about the three-by-five cards; it’s tedious both to keep them up to date and to share them across the team. User stories have to be online. Even though they aren’t as formal and complete as requirements, user stories change, and they need to be shared.

The key to reconciling automation and Agile is to choose your spots—that is, automate the activities that offer significant advantages over a manual approach. You also have to use automation tools that integrate well into your work approach, and don’t require more training and maintenance than they deliver in benefits.

It’s natural to be skeptical about the use of automation. Automation isn’t a goal in itself, especially in Agile methodologies. Rather, you examine your testing processes and activities, and automate the tasks and tests that can benefit from speed, data collection and analysis, and sharing.

Try…Catch statements

With the Try…Catch statements, you can run a block of statements in a script and continue playback in the Catch block of conditions if an error occurs. You can use a number of error statements, such as Err.Description, with the Catch statement to return error details. You can print error information or even send it to your defect tracking tool.

The following screenshot includes a basic script that uses the Try and Catch statements. You’ll also notice a Finally statement in the script. This statement allows you to always execute a statement if the script statements in the Try block passed or failed.

Try and Catch Exception

On Error statement

The On Error statement sets the error handling behavior of a script, function, or subroutine. In this script example, the On Error statement indicates that QA Wizard Pro should try to execute the next script step if it runs into a step that cannot execute. You could also use On Error Goto 0 to stop the script execution when the error is encountered.

On Error Exception Handler

Both exception techniques will be used in different script scenarios. My personal favorite is the Try and Catch statement because of my programming background. Check out the ‘Improving QA Wizard Pro Script Resiliency’ webinar recording to learn more about both exception handling techniques.

Download video in mp4 format

Q&A

Are both the catch and finally blocks required after the try block, or can we use only one?

Neither is required but you must use at least one of the two.

Are escaped errors logged in QA Wizard Pro reports?

Yes.

• Configure scripts to continue running when unexpected errors occur with On Error statements.
• Build advanced error recovery logic into a script with Try…Catch blocks.
• Script examples that demonstrate how error handling insulates scripts from application changes.

If you’re new to QA Wizard Pro or want to learn more about advanced script error handling, this event is for you!

Watch the Webinar Recording!

Download video in mp4 format