Lessons from Starting a User Testing Program

The term document testing refers broadly to a cluster of techniques used to assess the accuracy, appeal, and appropriateness of a document. Appropriateness includes characteristics such as accessibility, effectiveness, and usability. Document testing is widely acknowledged as a critical aspect of developing and producing usable texts--from operator's manuals to on-line documentation, from form letters to in-house guides for service representatives (Ramey, 1989; Dumas & Redish, 1993). Researchers urge a broad view of testing--one that encourages testing throughout the design and development processes, incorporates a balanced representation of assessment strategies, and considers the context in which a document is used (Dumas, 1989; Jeffries & Desurvire, 1992; Sullivan, 1989). Researchers confirm that a well-designed, on-going testing program has numerous tangible and intangible benefits that include saving time and frustration in document design and development; saving money in service calls, maintenance, training, and revisions; and improving customer service as well as the organization's image (Guillemette, 1989; Potosnak & Koffler, 1986; Redish, 1994). CATEGORIES OF DOCUMENT TESTING Three categories of document testing--text-based, expert-based, and user-based (Schriver, 1989)--are distinguished by the way in which the information is collected and the strengths of each method for providing information about document quality and usability (Sullivan, 1989). Text-based testing concentrates on the words and sentences of a document, focusing on local-level features of a document and then drawing conclusions about characteristics such as reading level. Text-based testing includes checklists as well as readability tests and computer programs to assess structural and stylistic features. Expert-based testing gathers information from professionals who provide formal and informal peer reviews, technical reviews, editorial reviews, and document design reviews. Such testing is useful for assessing the technical accuracy of a document as well as the selection of supporting evidence and level of detail for the intended audience. Reader-based or usability testing gets information about a document's usability directly from the users in two ways: after they have finished reading a document (which is called retrospective testing) and as they read and use a document (which is called concurrent testing). Retrospective testing includes such methods as questionnaires, interviews, focus groups, comprehension tests, and reader-feedback cards. Retrospective user-based testing often provides useful information from actual users. However, the feedback should be used cautiously because reader's memories are not necessarily accurate, and the information they provide is often vague. Several kinds of concurrent testing can be especially useful. * Observations: A test administrator watches a reader read a document, usually in order to perform a task. For example, the tester can observe the amount of time and apparent ease (or difficulty) a reader or user has in locating information or performing individual tasks or sequences of tasks. * Think-aloud protocols: A reader is asked to think aloud--that is, to say aloud all comments, reactions, and opinions, which are tape-recorded and often transcribed. * Reading protocols: A reader is asked to read aloud as well as comment on the text. In a cued-reading protocol, a reader is asked to stop at marks (cues) in the text and comment--for example, provide a gist of content and/or predict what's to come. For all types of protocols, tape recordings or transcripts can be examined to identify patterns of problems. * Co-discovery: Two (or occasionally more) users are asked to work together to discuss and solve a problem or complete a task, such as installing computer software. …