The Seapine View

We recently released version 2010.1 of TestTrack, which contains many new exciting new features. One new feature that I would like to give some love to is Item Mapping Rules. This feature will benefit users who have more than one TestTrack applications, as it allows you to configure how field values are mapped from one application to another.

Continue reading…

Recently, a customer asked the following question: “How can you filter the list of test runs based on the test variants it contains?”

The reason the question came up is because, while you can display a “Test Variant” column in the Test Runs list window, you can only filter the column between runs that have variants and those that do not.

Continue reading…

As a consultant, I have been exposed to many different development methodologies and processes. In most cases, assignments are performed by a person that is aware of staff resources and distributes tasks according to the availability of these resources.
Continue reading…

Being able to easily and quickly segregate data by various criteria is a requirement of any product that claims to be “easy to use.” For example, product management is interested in commonly reported issues and feature requests during product planning, but nearing the end of a release cycle they’re focused on critical issues that could delay the release. QA wants to see test runs yet to be executed or those that need to be re-run and development wants to see approved requirements that are ready for implementation. All of these scenarios create the need for distinct and re-usable “views” of the artifacts in TestTrack. This is where TestTrack’s tabbed views can make life much simpler for you as a user.

Continue reading…

I review a lot of UI design lately, and this is the thought that most often comes to mind:

“A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.”
    ~Antoine de Saint-Exupery

A quick SAT-style analogy:
“Our software is bug-free.” : QA tested ::
      “Our software is easy-to-use.” : Usability tested

Some good UI classes today, likely best explained by the slides. Jeff Patton also gave a good analogy for UX testing. Looking at a parked car, can you tell how fast it can go? Sure, with a bit of knowledge about cars, you can make an educated guess, but it’s still just a guess. Usability is like that, you can look at a product and guess how well a user can use it to accomplish a task, but you’ll never really know unless you see them try it.

He also made mention of Donald Norman’s Emotional Design which had some good points to it, and I’ll have to read that book next. In it, it’s mentioned that perceived ugly things are labeled by users as “difficult to use”. Users will even report bugs in the software and use those to claim that it’s of poor quality. A “pretty” UI will still get users to ask for fixes, but they’ll be much more forgiving of them.

Jared M. Spool also gave a good talk on what causes software to be labeled “intuitive” and steps to take to get there. A lot of his talk was theory and results of his testing, with some nice visuals on the slides. Unfortunately, the slides are not yet available.

The long-awaited conclusion…

Part 3: Post-Test

Our usability testing regarding recording seemed inconclusive. Both the old way of recording and the new way were intuitive enough for our testers to breeze right by them. In the end, we ended up with a kind of hybrid approach that seemed to please both the developers and the testers. It turned out something like this:

This organization and wording of the menu items spell out the actions they cause as clearly as possible to (hopefully) avoid any confusion.

As for the grid view, we made what changes we could before release to address some of the more troubling issues our test users had with it. Even more changes are in store for the next one and we plan to look at some even larger-scale changes for future releases. Many of the changes would not have been as high a priority or looked at as closely without actually seeing a user struggle with them. It’s difficult to ascertain usability from a list of defects marked “UI”.

A summary of post-test lessons learned:

• Sometimes testing won’t reveal what you want it to. Either perform a varied test, or go with what data you did get.
• Reviewing the video can reveal even more than your initial notes. Be sure to check it.

If anyone tries to convince you that the second kid is easier than the first, they have it backwards. So, here’s a belated usability test sequel.

Part 2: In-Test

Our first tester sat down with the testing goals, made a new QA Wizard Pro script, and just started recording. Our whole setup to look for troubles while trying to record were bypassed in less than a minute. He was well underway to completing the entire first goal in just a few minutes. We kind of looked at each other. “That means it’s intuitive, right?” Shrugs for a response.

Upon hitting the grid view of the script however, there was some trouble. It was nothing we either expected or were looking for, and I frantically started writing down as many observations as my hand would allow. We knew that there were some minor issues in this area, but the extent of how those affect actual users is lost until you see it in person. We had a hard time to keep from offering help, but we held our tongues. Eventually he got the script how he expected it.

Script > Run. Script runs, all looks good, first goal accomplished. Or so we thought.

Our tester saw it run, it flew by quick and seemed to end, but he had no proof. The script run ended without error, but also without any other message. A good 20 minutes later after re-running the script in many different ways and beginning to consult the help files, we decided that we had to intervene to keep the process moving. This was a surprising find for us, we always had found QA Wizard Pro playback to be quick and easy. The reports were always available and ready if you wanted to save the results.

So again we gathered a large amount of great information, and again it was not what we had set out to test. Record in the new fashion, the way that all the developers hated, seemed to work naturally in the test. So, we decided that we would run a second test to get a data point with the record set up the old way.

Our second tester took a little time starting up and looking over things, but took to recording just as quickly as the first. It seems that both the old organization of recording and the new organization were just as intuitive.

Watching the second tester work with the script in the grid view was mostly an exercise in us nodding our heads. All the troubles we saw from the first tester were reinforced by the second one, just in case we missed it. A few other informative finds later, and the second test was complete.

A summary of in-test lessons learned:

• You’ll want to help the testers along, resist. Unless the tester gets to a point where they are not likely to make any further progress.
• You may go in expecting to see data in a particular area. You probably will come out with data from lots of other “known good” areas.

So what did we decide to do with record and the other results? Refresh often for part 3. Hopefully with a smaller posting delta than the gap from part 1 to part 2.

I recently got the chance to perform some usability tests on QA Wizard Pro. Besides the large amount of useful information we pulled out of it, it also conveniently gives me something to write about. I’ll split the information up into a few parts containing some of the more useful lessons learned.

Part 1: Pre-Test

We didn’t arbitrarily decide to run general-purpose usability tests, we had a specific goal in mind. We had recently discussed (at length) and then changed the way scripts were recorded. Out of the available options our choice seemed to be the clearest available, so I continued and implemented it. Most (maybe all) of the QA Wizard Pro developers hated it to varying degrees. Still, they were willing to concede if it’s really what the customers wanted. But we had no way of really knowing what the customers would want, or what potential customers (as first-time users) might think of it. Or did we?

It was suggested that we could just grab some Seapine employees who had never used QA Wizard Pro before, and have them be usability testers. It sounded like a great idea, so I brought it up and we decided to make it happen. I asked around, trying for some prime candidates. Ideally, we wanted someone from QA (as they would be our target audience) who had never used any version of QA Wizard before. Unfortunately, all of the QA department had gone through training on each Seapine product, so all of them had had some experience with it. So, we ended up with a couple developers.

Next up, how do we get them to test what we want? The original script had something like “Record a script where…”, but we decided early that that was too much information. We wanted the users to find the menu items by themselves to see if the wording, placement, etc. of the menu items was appropriate. We created goal-oriented tasks instead asking the user to accomplish something using the product.

Lastly, we wanted some way to record what was going on with the screen, and ideally anything that our test user said as well. For the sake of speed and convenience, a software solution seemed ideal, so I took a look at Camtasia. QA Wizard Pro however also does recording, so I was a little worried about Camtasia recording the user using QA Wizard Pro recording a recording. Seemed like a lot of recording all going on at once. Luckily, a little testing showed that my worries were unfounded and it all would work smoothly together.

With all of that ready, a fresh install of QA Wizard Pro and Camtasia on a computer, and two willing candidates, we were set to test.

So, in summary, lessons learned from the pre-test:

• Don’t make tasks that lead the user exactly where you want them to go. Make goal-oriented tasks and see how a user would actually try to use your product to accomplish those goals.
• Recording the screen and sound proved quite useful. We used Camtasia, and it worked perfectly.

So, did the usability tests get us what we were looking for? Stay tuned for part 2.