If anyone tries to convince you that the second kid is easier than the first, they have it backwards. So, here’s a belated usability test sequel.
Part 2: In-Test
Our first tester sat down with the testing goals, made a new QA Wizard Pro script, and just started recording. Our whole setup to look for troubles while trying to record were bypassed in less than a minute. He was well underway to completing the entire first goal in just a few minutes. We kind of looked at each other. “That means it’s intuitive, right?” Shrugs for a response.
Upon hitting the grid view of the script however, there was some trouble. It was nothing we either expected or were looking for, and I frantically started writing down as many observations as my hand would allow. We knew that there were some minor issues in this area, but the extent of how those affect actual users is lost until you see it in person. We had a hard time to keep from offering help, but we held our tongues. Eventually he got the script how he expected it.
Script > Run. Script runs, all looks good, first goal accomplished. Or so we thought.
Our tester saw it run, it flew by quick and seemed to end, but he had no proof. The script run ended without error, but also without any other message. A good 20 minutes later after re-running the script in many different ways and beginning to consult the help files, we decided that we had to intervene to keep the process moving. This was a surprising find for us, we always had found QA Wizard Pro playback to be quick and easy. The reports were always available and ready if you wanted to save the results.
So again we gathered a large amount of great information, and again it was not what we had set out to test. Record in the new fashion, the way that all the developers hated, seemed to work naturally in the test. So, we decided that we would run a second test to get a data point with the record set up the old way.
Our second tester took a little time starting up and looking over things, but took to recording just as quickly as the first. It seems that both the old organization of recording and the new organization were just as intuitive.
Watching the second tester work with the script in the grid view was mostly an exercise in us nodding our heads. All the troubles we saw from the first tester were reinforced by the second one, just in case we missed it. A few other informative finds later, and the second test was complete.
A summary of in-test lessons learned:
- You’ll want to help the testers along, resist. Unless the tester gets to a point where they are not likely to make any further progress.
- You may go in expecting to see data in a particular area. You probably will come out with data from lots of other “known good” areas.
So what did we decide to do with record and the other results? Refresh often for part 3. Hopefully with a smaller posting delta than the gap from part 1 to part 2.
Share on Technorati . del.icio.us . Digg . Reddit . Slashdot . Facebook . StumbleUpon
