The Seapine View

The long-awaited conclusion…

Part 3: Post-Test

Our usability testing regarding recording seemed inconclusive. Both the old way of recording and the new way were intuitive enough for our testers to breeze right by them. In the end, we ended up with a kind of hybrid approach that seemed to please both the developers and the testers. It turned out something like this:

This organization and wording of the menu items spell out the actions they cause as clearly as possible to (hopefully) avoid any confusion.

As for the grid view, we made what changes we could before release to address some of the more troubling issues our test users had with it. Even more changes are in store for the next one and we plan to look at some even larger-scale changes for future releases. Many of the changes would not have been as high a priority or looked at as closely without actually seeing a user struggle with them. It’s difficult to ascertain usability from a list of defects marked “UI”.

A summary of post-test lessons learned:

• Sometimes testing won’t reveal what you want it to. Either perform a varied test, or go with what data you did get.
• Reviewing the video can reveal even more than your initial notes. Be sure to check it.

If anyone tries to convince you that the second kid is easier than the first, they have it backwards. So, here’s a belated usability test sequel.

Part 2: In-Test

Our first tester sat down with the testing goals, made a new QA Wizard Pro script, and just started recording. Our whole setup to look for troubles while trying to record were bypassed in less than a minute. He was well underway to completing the entire first goal in just a few minutes. We kind of looked at each other. “That means it’s intuitive, right?” Shrugs for a response.

Upon hitting the grid view of the script however, there was some trouble. It was nothing we either expected or were looking for, and I frantically started writing down as many observations as my hand would allow. We knew that there were some minor issues in this area, but the extent of how those affect actual users is lost until you see it in person. We had a hard time to keep from offering help, but we held our tongues. Eventually he got the script how he expected it.

Script > Run. Script runs, all looks good, first goal accomplished. Or so we thought.

Our tester saw it run, it flew by quick and seemed to end, but he had no proof. The script run ended without error, but also without any other message. A good 20 minutes later after re-running the script in many different ways and beginning to consult the help files, we decided that we had to intervene to keep the process moving. This was a surprising find for us, we always had found QA Wizard Pro playback to be quick and easy. The reports were always available and ready if you wanted to save the results.

So again we gathered a large amount of great information, and again it was not what we had set out to test. Record in the new fashion, the way that all the developers hated, seemed to work naturally in the test. So, we decided that we would run a second test to get a data point with the record set up the old way.

Our second tester took a little time starting up and looking over things, but took to recording just as quickly as the first. It seems that both the old organization of recording and the new organization were just as intuitive.

Watching the second tester work with the script in the grid view was mostly an exercise in us nodding our heads. All the troubles we saw from the first tester were reinforced by the second one, just in case we missed it. A few other informative finds later, and the second test was complete.

A summary of in-test lessons learned:

• You’ll want to help the testers along, resist. Unless the tester gets to a point where they are not likely to make any further progress.
• You may go in expecting to see data in a particular area. You probably will come out with data from lots of other “known good” areas.

So what did we decide to do with record and the other results? Refresh often for part 3. Hopefully with a smaller posting delta than the gap from part 1 to part 2.

I recently got the chance to perform some usability tests on QA Wizard Pro. Besides the large amount of useful information we pulled out of it, it also conveniently gives me something to write about. I’ll split the information up into a few parts containing some of the more useful lessons learned.

Part 1: Pre-Test

We didn’t arbitrarily decide to run general-purpose usability tests, we had a specific goal in mind. We had recently discussed (at length) and then changed the way scripts were recorded. Out of the available options our choice seemed to be the clearest available, so I continued and implemented it. Most (maybe all) of the QA Wizard Pro developers hated it to varying degrees. Still, they were willing to concede if it’s really what the customers wanted. But we had no way of really knowing what the customers would want, or what potential customers (as first-time users) might think of it. Or did we?

It was suggested that we could just grab some Seapine employees who had never used QA Wizard Pro before, and have them be usability testers. It sounded like a great idea, so I brought it up and we decided to make it happen. I asked around, trying for some prime candidates. Ideally, we wanted someone from QA (as they would be our target audience) who had never used any version of QA Wizard before. Unfortunately, all of the QA department had gone through training on each Seapine product, so all of them had had some experience with it. So, we ended up with a couple developers.

Next up, how do we get them to test what we want? The original script had something like “Record a script where…”, but we decided early that that was too much information. We wanted the users to find the menu items by themselves to see if the wording, placement, etc. of the menu items was appropriate. We created goal-oriented tasks instead asking the user to accomplish something using the product.

Lastly, we wanted some way to record what was going on with the screen, and ideally anything that our test user said as well. For the sake of speed and convenience, a software solution seemed ideal, so I took a look at Camtasia. QA Wizard Pro however also does recording, so I was a little worried about Camtasia recording the user using QA Wizard Pro recording a recording. Seemed like a lot of recording all going on at once. Luckily, a little testing showed that my worries were unfounded and it all would work smoothly together.

With all of that ready, a fresh install of QA Wizard Pro and Camtasia on a computer, and two willing candidates, we were set to test.

So, in summary, lessons learned from the pre-test:

• Don’t make tasks that lead the user exactly where you want them to go. Make goal-oriented tasks and see how a user would actually try to use your product to accomplish those goals.
• Recording the screen and sound proved quite useful. We used Camtasia, and it worked perfectly.

So, did the usability tests get us what we were looking for? Stay tuned for part 2.

I try to not regurgitate information found elsewhere too often, but this is one of the more frequently asked UI questions that I get that actually has a reasonably clean answer.

When do I use ellipsis on buttons and menu items?

A: When the item’s action requires additional information from the user to be completed.

For example, “Save As…” has ellipsis because in order to save, the user must input the file name to save to. “Save” does not have the ellipses because no confirmation is needed.

That one was easy.

How about “Properties”? No ellipses here, because clicking on the item will immediately perform the action of showing properties. Sure it does it in another dialog, but the dialog is actually the result of the item’s action.

Similarly, “Options” does not have ellipses, because it immediately displays the options to the user.

They get harder.

“Delete” or “Delete…”?
Hmm, the delete doesn’t happen right away if we prompt for confirmation. But then again, a confirmation prompt isn’t really asking for more information, it’s just confirming what the user already chose.

Apple’s offical stance on ellipses says that dangerous actions with confirmations should use them. A delete action seems dangerous enough.

Microsoft’s stance on ellipsis in Windows says to not use them for commands that may result in a confirming message box. Seems to apply here, though Microsoft’s wording is vague.

My own stance? I don’t see a confirmation as getting any additional information, it’s just there as a stopgap for accidental clicks on actions that you cannot undo. If the application can undo, then (like Apple suggests) you should probably avoid using pop-up confirmations in the first place. Either way, I would still just call it “Delete”.

UI doesn’t have to be visible to be useful.

Problem: A largish dialog with multiple lists, each with a sizable amount of data. Sometimes all of the data in a particular list isn’t visible and the user wants it to be.

The dialog in question is the redesigned Attach to Defect dialog in Surround SCM v5.0. Our standard of having the dialogs resizable still wasn’t flexible enough for some of our users. In this dialog, sometimes you really just want to see all of the information about the defects that you’re going to attach to. All that information is in the upper-left list. Iteration #2 (numbers not to scale) of the dialog added some splitters around the buttons in the middle, offering a way to resize individual pieces of the dialog. Wow, was that ugly.

Why not just make the splitters not visible then? They’ll still function, you’ll still see the cursor change on a mouseover, and you can still resize the sections. It will “feel” like you’re grabbing the edge of the group box and using that to resize the area.

That’s what’s in the current release.

“Haha, yeah, ninja splitters.”
“Yeah, you have to call it that.”
“Call what?”
“The class. CSCMNinjaSplitter. Do it.”
“But-”
“Do it.”
“Okay.”

Of course the first post is #0.

Those understanding of UI reading this probably recognize the issue. Programmers have a view of the world, and users often have a very different view. In coding world, programmers start counting from 0 so it’s only natural to pass on that information to the user as-is. The unfortunate truth is that users see that information a little different, and expect to see it their way. Seeing data through the eyes of a programmer only serves to annoy them.

That seemed the best way to start off my blog, and a reasonably accurate foreshadowing of where I’m heading with it. i.e.:

1. talk about UI stuff
2. offend as many people as possible
3. spell most words correctly

And I’ll try not to criticize only the programmers; I think Mr. Larson said it best: