A usability test method employed to generate early design evaluation by assigning a group of users a series of paper-based tasks that represent the proposed product interface and including participation from developers of that interface.
A systematic group evaluation of a design in which usability practitioners serving as walkthrough administrators guide users through tasks simulated on hard-copy panels and facilitate feedback about those tasks while developers and other members of the product team address concerns or questions about the interface.
Outcomes and Deliverables
Benefits, Advantages and Disadvantages
Reduces test-redesign-retest cycle by generating immediate feedback and discussion of design problems and possible solutions while users are present. Can provide early performance and satisfaction data before costly design strategies have been implemented. The group atmosphere encourages collaborative, constructive comments from users, developers, and other members of the product team.
Generates valuable quantitative and qualitative data on users’ actions by way of written responses. Product developers present during the session gain appreciation for common user frustrations or concerns about design.
A fixed sequence of hard-copy panels limits the simulations that users can perform (no browsing or exploring). Alternative paths for the same task are not explored. Product developers might not feel comfortable hearing criticism about their designs. Because the walkthrough is dependent on all users finishing each task before discussion can begin, the session can feel laborious.
Participants and Other Stakeholders
Who Can Facilitate
The walkthrough administrator (generally an experienced usability practitioner) must be able to guide users through tasks and facilitate collaboration between users and developers. It is usually best to avoid having a product developer/designer do it, as they tend to get defensive or move into “teach” mode rather than collecting user data.
Suitable for 6-10 representative users and products with linear tasks.
All users must complete each task before discussion and next task can begin, potentially affecting participants’ understanding of the design’s flow.
Data Analysis Approach
The users’ responses on their test packets, plus the answers to post-scenario surveys, can provide useful quantitative data to review how many completed tasks correctly and what percentage of users responded positively to the design. Also valuable are the qualitative data in the form of redesigns recommended by the users and the questions that both users and developers ask.
Walkthrough results provide developers and other product team members with constructive feedback about how to improve or modify design.
Costs and Scalability
People and Equipment
Approximately three hours are required to carry out the walkthrough. It usually takes two or three days to develop the test tasks and build the test materials.
Allowing multiple visually handicapped users who use screen readers in the same room likely would not work.
Video conferencing capabilities can be used to allow communication between users and developers who are located at different sites or different countries.
Walkthrough administrator should avoid characterizing users and developers as being on different “sides.”
Ethical and Legal Considerations
Pluralistic usability walkthrough involves the same legal considerations as for any user-based testing. These considerations include informed consent and non-disclosure agreements. To these, we add the concern that developers might not have had the same training in ethical treatment of test participants as usability practitioners have. Advise developers in advance to give positive regard to every user comment and approach users with courtesy and respect.
The usability practitioner must prepare the product developers and designers to be thick-skinned and to treat the users with respect.