Technical Report Project: 
Setting Up a User Test


ENGL 404: Technical Writing

Fall 2002
Dr. Jeff Jablonski

home | course info | projects | schedule | resources | jablonski


Project Overview

Choosing a Good Site

Deliverables

User Testing   

Design Plan Report

Progress Report    

Final Report   

Group Evaluation


Download Memo Template

This page presents the background information on usability and the user testing process that you will need to complete your Design Plan Report memo. It includes several links to more information and some sample user test data sheets. At the end is a more detailed list of online resources on usability testing and Web site design.

What is user testing?  
"Usability" generally refers to how well a tool or technology helps someone perform a task. Web site usability deals with effective user-interface design. The Usability.gov site defines Web site user testing the following way:
Usability testing encompasses a range of methods for identifying how users actually interact with a prototype or a complete site. In a typical approach, users — one at a time or two working together — use the Web site to perform tasks, while one or more people watch, listen, and take notes (http://usability.gov/basics/index.html).
The following online resources provide a good introduction to usability testing basics: What is the process for user testing?
Usability experts generally agree on a 5 to 6 step process for usability testing:
  1. Determine the goals of the study
  2. Develop a profile of typical users
  3. Write the user tasks
  4. Conduct user tests
  5. Evaluate the data
  6. Implement or recommend changes
Step 1: Determine the goals of the study
Your evaluation should target common tasks that users are expected to perform on the Web site. Identify the purposes or aims of the site. What are the intended uses of the site? Why do people come to the site? What tasks might they attempt to perform? Break these uses down into primary and secondary tasks. For example, a travel Web site’s primary function is airline booking, a secondary task is finding information about travel destinations.

Step 2: Determine the user profile
The goal is to match the people you user test with the target user profile. Identify typical users’ attributes: Who will come to the Web site? What is their age, gender, education level, etc.? What will be their knowledge level about the Web site and its purposes? What will be their knowledge level about the Internet in general? What other Web sites are related to this and will the users be familiar with them? If you’re using 404 students as your user group, justify how they fit with the user profile. 

Step 3: Write the user tests
User testing involves observing actual members of the target group using the Web site and interviewing or surveying them at key points (before, during, and after they use the site) to gather their feedback. Most experts agree that a well designed test of only a small group, between 5 to 10 people, reveals any major usability issues with a Web site (see Jakob Nielsen’s “Why You only Need to Test With 5 Users” http://www.useit.com/alertbox/20000319.html)

Once you’ve brainstormed a list of primary and secondary uses, translate them into a list of tasks or scenarios you will ask your users to perform during the tests. There are two basic types of user tasks:
  • Directed tasks. These are specific, close-ended tasks related to primary or secondary purposes. For example, to study the usability of an airline’s Web site, you might ask the user to  “Book a flight from Cleveland to Phoenix for your Christmas vacation. You want to leave on X date and return on Y date.”
Examples:
  • Undirected tasks. These are good for general evaluations/assessments (a good choice if you notice lots of problems with a Web site). Undirected tasks are more open-ended. They are scenarios designed to mirror how intended users might typically use the site, such as: “You’re at home, browsing the internet, interested in finding the latest news about Afghanistan. You decide to use the latest search engine being promoted by your ISP.” Once the user is finished with the task, you can collect their opinion about general and specific issues (see the examples below for general Web site evaluation criteria)
Examples:
After you have developed your user tasks/scenarios, decide on a method for collecting user feedback. Choose either a written protocol (distributed to user who writes own feedback) or an interview protocol (answers recorded by researcher) or a combination of both.

Make sure you plan on asking enough of the right questions before, during, and after your user tests:
  • Pre-test questions: Used to collect basic demographic information and possibly information related to purposes of your Web site (e.g., if it's an auto manufacturer's Web site, ask users if they have ever visited such a site before). Example pre-test questions:
  • Age?
  • Gender?
  • Previous Computer Experience?
  • How often do you access the World Wide Web?
  • For what length of time do you typically use the WWW?

  • For what purposes do you use the WWW?
  • Have you ever used to the WWW to investigate products you wish to buy?
  • Have you ever purchased anything over the WWW?
  • Have you ever purchased anything you saw advertised on the WWW?

  • In-test questions: Can be used to prompt users' reactions to specific tasks:
  • What are you doing now? Why? 
  • What is the first thing you’re attracted to? 
  • What is [particular feature, e.g., "background color"] like? 
  • NOTE: some of the post-test questions below might work better as in-test questions, given during or immediately after specific tasks. See also D’Hertefelt’s article on “obtrusive observation” techniques below

  • Post-test questions: Prompt users' overall reactions to the test and Web site:
  • What was your general impression of each page?
  • What problems did you have using the page and why?
  • What do you believe caused this problem?
  • What, if anything, helped you find the information you were looking for?
  • What, if anything, made finding the information more difficult?
  • What information would you like to see on the page that was missing?
  • Was the page easy to use?
  • Rate the pages you saw--which was best?  Give three reasons
Lastly, check to see if you want to use any other methods for collecting data:
  • Think-aloud protocol. Records what users say while performing tasks and verbalizing decision making process involved in performing a certain directed or undirected task.
  • Observation. Records users actions, e.g., where user clicks, where they move mouse, what their page-to-page navigation is (can use images of site/page to diagram). You can also record how much time users take on each task and the whole test.
The following are samples of user test protocols (or plans for conducting user tests) used by students and professionals:
  • Student Samples
  • Professional Samples
Step 4: Conduct the tests
This is where you get to actually see how users perform your tasks. Keep Keith Instone’s advice in mind as you conduct your tests: “watch and learn” (see http://www.webreview.com/1997/04_25/strategists/04_25_97_1.shtml). That is, don’t interfere with the users as they attempt to perform the tasks. Avoid giving verbal or visual cues that might influence the user’s interaction with the Web site. Do not give them any background information or help them if they get stuck. That’s one thing you’re looking for, if they get stuck. Take notes—write down what they say, what they get stuck on, what links they use, how much time it takes to perform tasks, etc. Ask follow up questions about what they had trouble with, their overall opinion of the site, etc. Use the questions from the protocol you developed beforehand. At the end of the test, you can demonstrate for users what they had trouble figuring out. 

Step 5: Evaluate the Data
The most important skill of the usability consultant is the ability to recognize patterns (see Jakob Nielsen’s “Becoming a Usability Professional” http://useit.com/alertbox/20020722.html). As researchers and usability analysts, it’s your team’s task to make sense of the results of your user testing. You should first compile your findings (one of the sections of your final report will be a “findings” section). You can group your findings by tasks and by answers to protocol questions.

Once you compile the data, you’ll want to look for significant patterns: what did most users have the most difficulty with? How severe were their difficulties? How satisfied were they with aspects you were testing? Once you start to ask questions like this, then you can begin to brainstorm solutions or fixes (your final recommendations). As you start to discover patterns in the results of your user tests, consult Web site design principles for expert opinions about how to improve these elements.

Keith Instone’s “First User Test” article is an excellent discussion of the findings and recommendations of his evaluation of the Internet Travel Network Web site http://www.webreview.com/1997/05_30/strategists/05_30_97_8.shtml.

Step 6: Implement or Recommend Changes
Most usability consultants can implement recommendations firsthand. Your team will have to instead write up your recommendations in a credible and persuasive technical report. Again, Keith Instone’s “First User Test” article is an excellent example of how detailed your recommendations should be (notice that Instone makes three recommendations, all of which are linked to a more detailed discussion http://www.webreview.com/1997/05_30/strategists/05_30_97_8.shtml).


Online resources

Usability Testing
Web Design


home | course info | projects | schedule | resources | jablonski

<http://complabs.nevada.edu/~jeff_jablonski/404/procedures_project.html>