HE URISTIC E VAL UATION
6 2 2: Eva l u a t i o n o f Systems an d S e r v i c e s
M a r c h 9, 2 0 0 4
T he T - Mo b i l e G r o u p
6 2 2 T M o b i l e @ c t n g . u m m u . u m i c h . e d u
E ric Coo k
S h inya Kasuga
I l-hwan K i m
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
AIM Functionality on x105: Usability Challenges & Proposed Solutions, Exams of Information Technology
An evaluation of the aim functionality on the x105 phone, focusing on usability issues and proposed solutions. The evaluation covers heuristic violations related to input modes, system response time, inconsistent methods for canceling and backing-up, new terms introduced, and visual display. Proposed solutions include improving color schemes, integrating a help system, and revising error messages.
Typology: Exams
Pre 2010
1 / 21
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download AIM Functionality on x105: Usability Challenges & Proposed Solutions and more Exams Information Technology in PDF only on Docsity!
HEURISTIC EVALUATION
6 2 2 : E v a l u a t i o n o f S y s t e m s a n d S e r v i c e s M a r c h 9 , 2 0 0 4 T h e T - M o b i l e G r o u p 6 2 2 T M o b i l e @ c t n g. u m m u. u m i c h. e d u E r i c C o o k S h i n y a K a s u g a I l - h w a n K i m
1. EXECUTIVE SUMMARY
Product and Goals:
This document describes a heuristic evaluation of the AOL Instant Messenger (AIM) software of the Samsung x105 cellular phone. These AIM functions on the x provide a compact and portable method of receiving and transmitting short text messages and pre-programmed emoticon images with other users of the AIM network. The goals of this heuristic evaluation process were to evaluate the usability of the system and provide developers with a prioritized list of issues to address in future revisions. Ultimately, this will assist in crafting a more effective and more marketable system.
Methods:
Heuristic evaluation provides an efficient and “low cost” method of usability assessment. Investigators apply a pre-existing set of general principles and guidelines to a system, and then rank the issues identified. For this report, a set of twenty-eight heuristics developed by Judy Olson were utilized, along with a 0-4 ranking scale created by Jacob Nielsen.
Findings:
This evaluation identified and prioritized nine problem issues. In order, the most severe of these were:
- System response time is unacceptably high for some operations.
- Input modes create unexpected results, and do not match user’s mental task model.
- Text input via keypad is physically inconvenient and inefficient.
- Inconsistent methods for canceling and backing-up through menu levels.
- New terms introduced that are unclear and unfamiliar to desktop IM users. Solutions were proposed for each issue, and are included in the body of the report.
Summary:
This heuristic evaluation succeeded in its stated goals. The evaluation team identified both problem issues and success areas, and this generated detailed information to assist developers in addressing these issues.
3. METHODOLOGY
3.1 Heuristics & Severity Ratings Heuristic evaluation is defined by Jakob Nielsen as a “discount usability engineering method” (1993, p 160) in which several testers individually apply a given set of criteria to an interface based on predetermined scenarios, and then combine their findings. By doing so, Nielsen has demonstrated that a high percentage of common usability concerns are identified. While the optimal number of evaluators is generally given as four or five, the illness of one investigator on our evaluation team limited the number to two during this session. Nevertheless, we feel confident that we have identified the major of primary usability concerns present in the product. For our evaluation of the AIM functions of the Samsung x105, we applied a set of twenty-eight heuristics developed by HCI researcher Judy Olson. These may be grouped into eight primary categories: # Heuristic Category 1 Correspondence to the way the user thinks about the task 2 Consistency 3 Menu command structure 4 System response time 5 Visual display 6 Error recovery 7 Help 8 Documentation and training released Utilizing this set of heuristics provided us with a broad checklist for identifying, categorizing and evaluating a wide set of usability concerns. The full specifications of the individual heuristics are quoted in Appendix A of this report. (Olson, 2004).
The system was evaluated against each heuristic by each evaluator. When issues were identified, their severity was ranked along Jakob Nielsen's rating scale for usability problems: Rating Definition 0 I don't agree that this is a usability problem at all. 1 Cosmetic problem only: need not be fixed unless extra time is available on project 2 Minor usability problem: fixing this should be given low priority. 3 Major usability problem: important to fix, so should be given high priority. 4 Usability catastrophe: imperative to fix this before product can be released. Utilizing this standardized scale allowed for a clear comparison across disparate heuristics, a convenient way of prioritizing our findings, and a consistent means of combining the results of both investigators. 3.2 Scenario: The evaluation was framed via a scenario and persona developed during an earlier SI 622 assignment (Cook, 2004, p. 4-7). Briefly, the persona user is Grady, a thirty-four year-old technical writer from Austin, Texas. Grady is an experienced and daily IM user on the desktop. While on an extended business trip to his company’s California office, he finds himself cut off from his usual IM and chat outlets by the office’s aggressive firewall. He borrows a Samsung x105 phone from an officemate, and attempts to login and IM with it for the first time. This scenario allowed us to step through all the AIM functions of the phone as a new user would, and in particular helped illuminate difficulties that might arise from conceptual and vocabulary mismatches between desktop IM clients and x105’s IM client.
**4.2 Specific Problem Areas
Issue Severity
Heuristic Category Specific heuristics violated** 1 System response time is unacceptably high for some operations. 4 System response Time #19, # Details: During evaluation testing, multiple system responses were measured to take longer than the recommended two second maximum; for some actions, response time ranged as high as 11 seconds. This is a severe problem, disrupting the user’s concentration and flow of usage, and impacting negatively on user enjoyment of the system. The primary appeal of IM messaging (as compared to email or SMS messaging) is its ability for near synchronous communication, but the lengthy response times destroy this perception of synchronicity. The designers obviously attempted to mitigate this problem by providing animated status screens, which show that the action is still in progress. However, these status screens give no indication of the remaining time to completion of the action, leaving the user to wonder if their phone has locked up while displaying the “busy” animation. Example: The following table illustrates sample system response times derived during our evaluation sessions. These response times can vary depending on signal strength, network load, and other external factors, but these samples clearly illustrate the degree to which this heuristic is violated. Action Sample Response Time (in seconds) Signing On 11 Signing Off 4 Send IM > 1 Refresh List 7 Settings 3 Add Buddy 8 Delete Buddy 5 Decline/Accept IMs 6 Alert Me 7 Away Message 6 Proposed Solution: Ideally, system response time for all actions would simply be reduced to less than two seconds. However, this is an unfeasible proposal, since the delay is likely caused by cellular network response times out of control of the designers responsibly for the AIM client. A less appealing, but still acceptable solution would be to add a status meter for all actions. Recurrent network speed tests could be performed and averaged in the background during an AIM session, allowing the client software to display reasonable approximations for time-to-completion of each action.
# Issue Severity Heuristic Category Specific heuristics violated 2 Input modalities do not match user’s mental task model. Default “T9” text entry mode provides unexpected results. 3 Correspondence^ #1, #2, # Details: Once the user logs onto the AIM network via the x105, their primary and most recurrent task will be to write and send instant messages. To accomplish this via the standard phone alphanumeric keypad, the system provides four distinct input modes: T mode (a predictive shorthand input mode popularized by SMS text messaging systems), Upper/lower-case character modes, Number mode, and Symbol mode. There are multiple issues resulting from this use of input modes. First, breaking input entry up into specific modalities clearly does not match the user’s mental task model. The typical user does not think of typing characters, numbers and punctuation as distinct categories of action; they simply think of the message they want to convey. Forcing them to change between these categories slows down input, and introduces undesirable cognitive load as they try to maintain the message in short-term memory while navigating between input modes. Ultimately, this modal approach unfairly forces the user to address the problem of a restricted user interface in a way that best serves the system, rather than the altering the system to best serve the user’s processes. “Avoid Modes,” states Nielsen bluntly. (1993, p. 146). These issues are compounded for new users. Modes are changed via the right softkey on the phone while performing IM text entry, but aside from the proximity suggestion of the current mode indicator in the lower right hand corner, this relationship is not at all clear. To make matters worse, on all other screens, the right softkey performs another action – moving the user up one menu screen level! (See issue #4 for more specifics about this inconsistency.) Finally, the default mode for input is set to T9 mode. While this predictive input mode may be useful to experienced users and those with prior SMS experience, it is extremely unclear for new users what this function is actually doing. For example, in normal text entry mode, to type the word “Hi”, the user would press the shift button, then press the 4 button twice, moving past the “G” character, and selecting the “H” character. The user would then press shift again to disengage uppercase, and press the 4 button three times, moving past the “g” and “h” characters to select “i.” In T9 mode, pressing the 4 button once selects the “g” character; pressing it twice causes the system to immediately display the word “Hi” on screen. Convenient if the behavior is expected, but for a user that is unaware of the existence of such a mode, placing them into this type of interaction by default borders on cruel. In addition, the labeling of the mode as “T9” is completely obscure to new users, therefore preventing them from attempting to debug the bizarre input appearing on their screens. (This labeling confusion is addressed in issue #5.)
keyboard on the phone. These are problematic solutions, however, since they involve major (and expensive) restructuring of the physical form factor of the phone. The system does attempt to alleviate some of these input difficulties by providing a shortcut menu of common phrases and the predictive T9 input mode. The shortcut menu is too limited to be of much use however, and is too disruptive to the user’s input process in its current implementation. The T9 mode may be of use to some users, but it does not provide enough feedback in its current implementation and is obscure to new users, as previously indicated. Including a better help and tutorial system for the T mode is a feasible solution, if still not entirely satisfactory. # Issue Severity Heuristic Category Specific heuristics violated 4 Inconsistent methods for canceling and backing-up through menu levels.
Consistency, Menu/Command Structure #4, #5, # Details: As mentioned in issue #2, the right softkey is used primarily to go back to the previous menu level. However, when the user is on a screen that allows for entering text (such as “Add Buddy” or “Send IM To…”), the right softkey changes function and instead becomes a mode selection key. This inconsistency prevents the user from building a clear mental model of navigation through the system. It should be noted that there is a global Cancel/Correction button build into the phone interface, which will successfully cancel the user out of any task. However, pressing this key does not return the user to the previous action or menu lever, but rather exits them from the AIM client entirely. Despite being consistent in its behavior, this button (perhaps more appropriately labeled the “Exit/Quit” button) does not provide an appropriate level of sub-task cancellation, and therefore also does not satisfy the need for this function. Example: Right softkey “Up menu level” function becomes text mode select on text-entry screens.
Proposed Solution: Provide separate, consistent and clearly labeled “Cancel,” “Quit” and “Up Menu” buttons. These buttons should not used for any secondary functions, so an “Input Mode Select” button will also be required if modal text entry is preserved. (See issue #2 for more details concerning input modes.) # Issue Severity Heuristic Category Specific heuristics violated 5 New terms introduced that are unclear and unfamiliar to desktop IM users 3 Consistency #1, # Details: Much of the text utilized in the x105’s menus and labels is clear and similar to that found in desktop IM clients, such as “Buddy List”, “Send IM To...” and so on. However, two new terms could be a source of serious confusion for many users. These are “T9” and “Conversation.” T9 indicates a predictive text input mode, and may be familiar to users of SMS text messaging systems. However, the label is unclear and unintuitive to all other users, and not adequately explained in the system. (See issue # for more details about T9 mode.) Conversation indicates an ongoing message exchange between the user and another individual on the AIM network. A conversation can be persistent across different message sessions, and may be viewed as being somewhat analogous to a combination between a text buffer and a specific message window on a desktop system. Functionally, it is convenient, as it affords the user a means of maintaining several IM sessions simultaneously and having their contents preserved even if the phone is closed or loses its connection to the network. However, this is a new concept and term for desktop AIM users, and it is not adequately explained in the system. To make matters more confusing, “Conversations” shows up as a primary category on the Buddy List screen, along with “Online” and “Offline.” The persistent nature of Conversations in the system means that the user’s buddies may be represented in both this menu and either Online or Offline simultaneously. Proposed Solution: The distinction between Conversations and the Online/Offline submenu labels is not sufficiently clear. Our proposal is to remove Conversations as a separate category entirely. Instead, we suggest two new icons be implemented in the Buddy List: one to indicate an active IM session with the given buddy, and another to indicate a saved text buffer for that buddy. If designed appropriately, these icons will provide a clearer indication of this information and will accompany other existing status icons that appear on that screen (such as those for Alert, Decline, Away, etc).
Example: Proposed Solution: This error message should be revised to clearly state both the problem and a suggested resolution: “Buddy name XXXX not found. Double check the name and try again.” # Issue Severity Heuristic Category Specific heuristics violated 8 Help menu functions only available in one location and are not especially helpful. 2 Help #24, # Details: The existing online help system is available only via a special help menu, and this menu is only accessible from one location in the highest level of the AIM client’s menu tree. Once the system is accessed, the information provided is terse and not especially helpful. Proposed Solution: An expanded help system should be integrated into the AIM client. Technical writers should be allocated to the development project to craft more useful and contextual help text. Help functions should be accessible via a specific Help key, available from all screens.
# Issue Severity Heuristic Category Specific heuristics violated 9 Menu vocabulary grammar is inconsistent. 1 Menu/Command Structure
Details: Menu commands are not formulated using a consistent grammar. In various menus throughout the system, examples can be found of Verb-Object constructions, Adjective- Noun constructions and single Nouns. Examples: Examples of inconsistent grammar across menus. Proposed Solution: Though this specifically violates heuristic #14, it is a relatively minor concern. In some cases, the labeling would be made less clear and less concise by trying to force it into a consistent grammar (e.g. “Buddy List” would not be made more effective by being changed to “Access Buddy List”). It is mentioned here primarily to draw the developers’ attention to the principle, and suggest that they evaluate all menu commands to consider whether they would be clarified by changing the grammar used.
Example: # Success Item Severity Heuristic Category Specific heuristics supported 2 Error messages are attention grabbing and consistent. 0 Visual Display # Details: Aside from the lack of clarity in one error message (identified in issue # 7), error messages are clear, consistent, and presented in way that captures the user’s attention and assists them with resolving the problem. Error messages appear overlaid upon the current screen, and stay visible for approximately 4 seconds. They cannot be accidentally dismissed by the user. Examples: Sample error message screens. Header information Information block Softkey functions
# Success Item Severity Heuristic Category Specific heuristics supported 3 Frequently used commands are always listed on the top of menus. 0 Menu/Command Structure
Details: The designers have carefully placed the most frequently used commands on the top of the menus in the system, giving the users quick and easy access to the functions they are most likely to use in any given menu. Other menu items appear to be sorted effectively based on predicted frequency of use. Examples: Three sample menus with primary/frequently used functions listed on top.
6. REFERENCES
Cook, E. (2004). SI622, Assignment #3: Personas & Scenarios. Unpublished paper. Nielsen, J. (1993). Usability Engineering. Academic Press, San Diego, CA. Olson, J. (2004). Quick Methods: Checklists, Heuristic Evaluation, Cognitive Walkthrough. Unpublished classroom slide presentation.
7. APPENDIX A: JUDY OLSON HEURISTICS
Correspondence to the way the user thinks about the task
- The terms the software application uses (what it calls the objects and what it calls the actions) should be as close as possible to the words users normally think of in conducting the task
- If new words must be used, they should be metaphorical and concrete.
- The order in which the user performs the subtasks should correspond closely with the order the software requires them to be performed. Consistency
- Actions should be performed in similar ways on the objects in the application (e.g., all objects should be deleted in the same general way).
- The way the user "backs up" or cancels an action should be the same, no matter where the user is in the interaction.
- The closer the command structure is to the structure of commands in other well-known applications, the easier the new one is to learn. Visual Display
- Readability of fonts increases if they are presented in both upper and lower case, serif fonts, variable width strokes, and with appropriate separation.
- Related items should be grouped spatially; unrelated items should be separated. Gestalt principle of Proximity.
- Color, highlighting, and special fonts such as bold and italics should be used to indicate similar meaning for those things that cannot be spatially grouped. Gestalt principle of Similarity.
- Screen areas should be used consistently for similar functions. e.g., error messages always displayed in the same, easy to see place.
- Display familiar items in familiar ways. e.g., music is shown as musical notation, spatially related items are shown in a spatial display.
- Error messages should be presented in way that captures the user's attention. e.g., flashing, reverse video, color, accompanied with a tone.
Menu/Command Structure
- For new or casual users, commands presented on menus are easier to learn and remember than commands that have to be typed in without a prompt. For experienced, dedicated users who know the command set well, commands entered with quick hand motions are preferred.
- Commands should be formulated with a consistent grammar: The order most closely associated with commands in English is Verb-Object-Modifier ("put-that-there"). [This is a very controversial point; many good, consistent systems have a grammar that is Object-Verb-Modifier; users find it easy to learn and easy to operate.]
- Frequently used commands should be easily accessible. e.g., frequent menu selections should be near the top of the menu.
- Related commands should be grouped on menus.
- If there are more than one way to do something, instructions should be clear about when each method should be used for maximum efficiency. System Response Time
- Responses to keypresses or mouse movements should be within 100 msec of the act.
- Normal computed responses (e.g., responses to a request for a font change of the document) should take place within 2 sec, if they are part of an ongoing task.
- The only times when system responses can be longer than 2 sec and not disrupt the user's cognition is when they occur at the end of a task, and the user is alerted that the time will be long. Showing how long that delay will be is helpful to the user to decide whether to engage in a secondary task during the time. Error Recovery
- Error messages should be clear statements of both the problem and how to recover from it.
- All actions should have a natural reverse, accessed through an "undo" command.
- Actions that have severe consequences should be prompted with "are you sure?" e.g., actions that delete an entire file should be prompted. Help
- Help messages should be accessible from any point in the dialog.
- Help should help. It should be written specific to the context in which the help was requested. It should be written sensitive to the task the user was engaged in. Documentation and Training
- Training should be specific to the tasks the user is most likely to be engaged in.
- Documentation explains both what the functions are in the system and how to use them in common tasks.
- Documentation should be accessible through a good keyword index (in the user's vocabulary) and through tabs, headers, and a table of context that makes the grouping of functions obvious.