R7: Discount Techniques

I liked the cooperative evaluation described in the book the best. Basically cooperative evaluation is performed by having the participant do some task on the system and describing their thinking aloud. Additionally the participant and the observer can ask each other clarification questions. I liked this technique for a couple reasons. First of all, I like that information from the study comes from both watching the user and writing down their thoughts. This can generate data that is less likely to be misinterpreted. Secondly, having the observer and the participant ask questions can promote an environment where the system is evaluated in at a deeper level. I think this would avoid finding only superficial usability problems and can yield useful leads on how to fix the more important problems found.

The cooperative evaluation technique seems ideal for testing the usability of software systems (as opposed to physical artifacts, although it could work for them too). I have done something somewhat similar at work before. I made a website that would be used by other employees. I sat down with a co-worker to teach them how to use the site, and during the processes they pointed out some things that did not make sense or that they wished would work another way. The suggestions were very valuable and helped create a better site. Cooperative evaluation seems very similar but instead of discussing while I teach them how to use the site, I would instead ask them to try to do something on the site and discuss as they go along.

“Zero users give zero insights”. I agree that the best way to evaluate a design is to involve the actual users (>1) who match the targeted user population, as evaluators. Similarly, it is also important to limit the users to a number such that the results are unique and saves time and effort. Early and rapid iteration of the design evaluation during system development is the core theme of the discount (low-cost) techniques. Although, heuristic evaluation may not be the best method for design evaluation, I feel it is definitely the most usable method for working within the constraints of time and resources, as Nielsen appropriately says it helps in achieving “the good”. Additionally, the set of ten heuristic rules defined, broadly cover the essentials for designing a user interface by illustrating a metric for directed assessment.

In my future work and for our project, I would prefer using more than one method for evaluating the design to cover the breadth and depth of a product design. Primarily, we can select three evaluators who can independently unearth the problems in design; these evaluators can be participants from our interviews, who are already using an existing personal finance management tool. Simultaneously, we can use a simplified version of co-operative evaluation; to promote active conversation between the user and the evaluator to offer clearer insights to the usability problems in design. Furthermore, we can send selected participants an annotated worksheet for evaluation that is built upon the Neilsen’s heuristic principles containing the severity ratings encountered to study the frequency, impact, persistence and market impact of the usability problems encountered. This worksheet will serve as a ready reference for a quick-debugging of the design. Moreover, it will be helpful to re-conduct extreme user interviews, this time in presence of a prototype, to understand the flaws and accomplishments of our design via canvassed interpretations of an expert and a novice.

Among those discount techniques mentioned by Nielson, I like the heuristic evaluation technique best. There are some reasons for this. First, that technique is no costly. We only need a few users to participate the evaluation, and involving more users will even be a waste. Sometimes we can also let the same users come back and evaluate again after we improve the design. Second, we don’t need to directly decide what and how to improve our design after the evaluation, and doing so may lead to some serious problems afterwards. Because we cannot rely on user evaluation, and there are too many uncertain factors in it. What we can do in heuristic evaluation is to compare the feedback from users and the heuristics and brainstorm again with other designers and users, and elicit some reliable results.

In my own projects, I can go the same ways as the process in Nielson’s passage. But a more convenient way to use the method is to refer to those heuristics regularly when we design the system. We can avoid possible bad design at the first place and save the time and efforts to correct them. Also, heuristics are not golden rules. They just provide us some ideas about good design. So we don’t need to stick to them, which will also hinder our progress. We can also try to reduce the number of subjects for the evaluation, since we cannot pay for usability test for some course projects. At last, we can combine this heuristic evaluation with other techniques to complement each other.

I think the best way to evaluate a product must involve some users. I was surprise about the fact that a product only need to test with 5 users (no single user test). And the article suggests that we could run as many small tests as we can afford. I believe the data based on the formula. Usability Testing, based on user participation, is the approach I like to assess a product. Users can test the prototype or a product, while we observe, record and analysis related data to find insight problems. But still, the methods involve users are phases on the end of the process. So I also like heuristic evaluation and cognitive walkthrough for the early evaluation. Heuristic evaluation is an approach of expert analysis, it’s flexible and cheap. Even heuristic evaluation is not accurate; the future work can offer complementary assessments (involve users).

Personally, I would like to use various evaluations to cater to different needs for my feature work. For the final project, I would like to use heuristic evaluation and field studies for usability testing. For heuristic evaluation, the group members can critique the project to find out potential usability problems independently. In the book, author mentions, with five evaluators, about 75% of the potential usability problems can be discovered. It’s a good discount technique for us to evaluate problems before involve more participators. And field studies for usability testing; we can see how it works exactly. I want to observe our project in action in user’s work environment, to gather complementary insights.

I think it’s quite interesting to know the usability test model that only 15 users are enough to discover all the usability problems. While in the article, Jakob Nielsen suggests to use the budget more efficiently. He would prefer to use the funding to recruit 15 representative customers and have them separated to three iterative test with 5 users each rather than waste them all on one test. He explains the idea of usability test aims to find problems and to fix problems. Using 15 users in one test would only be able to document 100% problems but not to solve them. While 5 users could help us to find 85% problems and we could save the other ten users in next user test to evaluate our problem fixing work. Due to the budget limiting, this iterative design and evaluation process is better for the goal of design and usability test. Having each individual users test the interface alone is heuristic evaluation. Only after all evaluations have been completed are the evaluators allowed to communicate and have their findings aggregated. This procedure is important in order to ensure independent and unbiased evaluations from each evaluator.

In my experience working as information system engineer in France Telecom. I think we unintentionally adept the 3-5 users testing model in order to save budget. We had no theoretical support back to that time, but we did understand when the test users amount increase, the budget increase while the problems finding slope decrease. It’s also at that time, we didn’t isolate the test subjects and I think it introduced a lot of confounding that most of problems they found are replicate.

Heuristic evaluation, developed by Nielsen, is the evaluation technique that I like best. Compared with other discount techniques, heuristic evaluation has several advantages. First, it is a relatively cheap approach. According to Nielsen’s study, three to five evaluators are enough to get the best result, covering 85% of the usability problems. Second, it is flexible. Because we can obverse what the user does during the test, we cab find out some potential usability problems. You can also decide the number of iterative usability tests based on your budget. Moreover, with iterative design, we can investigate some usability problems of fundamental structure, not only surface-level problems. Third, it has 10 general heuristics principles as guidelines to indicate the nature of interface design, which helps to discover and explain the problems that are found. Lastly, this approach is easy to use, and is useful in the early design stage to find most interface problems at relatively low cost.

In our class project, this approach is feasible to conduct a usability evaluation. It is easy to learn and use by following the rules given by Nielsen. We can have three to five users test the interface alone; discuss and analyze all the usability problems that are found; and compare them with the 10 principles. According to the results, we could probably discover and address most of the interface problems in the early design stage.

I like heuristic evaluation best. It can cover most of usability problems by involving several evaluators, so it can assure the breadth of evaluation; it has severity rating system, this can help us to decide the priorities and find both major and minor usability problems; it can be used in the early stage, so it can help us to find and fix problems cheaply and timely. Also, it is not intrusive and does not require high-level expertise. In addition, Nielsen’s ten heuristics give us a clear checklist, so it is relatively easy to use.

I would like to use heuristic evaluation together with other evaluation methods to complement each other. As for heuristic evaluation, I will do it with multiple evaluators to cover usability problems comprehensively. I will use the list of recognized usability principles to help me find problems, and try to draw into other possible usability principles to evaluate the system. Severity ratings will also be used so that all the evaluators can access the problems they have found independently, and we can give suitable priorities to the problems.

I like the heuristic evaluation best. I think it is really fast and easy for us to evaluate the design using Nielsen’s ten principles. In the principles, I really love “Visibility of system status”, which gives me a lot of reflections. One thing I immediate thought of was the design of the console. I have lots of chance to use the console when I am programming, and often I find that it’s really annoying to use it because I cannot tell or be sure what’s going on in my computer. Therefore, sometimes it’s really hard for me to evaluate my program in the console. I think that’s a point we have to consider in our design. The design might have a lot of great features, but we have to go through the evaluation to make sure if it is really useful. By following the principles from Nielsen, we not only provide simpler way for users to evaluate, but also help us rethink the features and structures of our design.

As a result, in our project or in my future design, I might focus on making users participate more in the design—they should not only learn how to use it, but also understand what’s going on after every interaction. Some good examples are the status bars used in every operating systems, progress percentage of uploading/downloading, the “loading…” notifications used in web browsers or games, and the spinning color ball on the mac. In programming, we often want the program to be as fast as possible in order to give users nicer experiences. However, I realized that the design should be truly “responsive” to users. It’s great if some operations can be done immediately, but if some operations need to take some time to finish, we have to let the users know. By the heuristic principles, I got the idea that the performance of the design does not depend on its features, speed, or layout, but on how the users “feel”, which can be easily found if we have heuristic evaluations.

I feel that the heuristic evaluation technique is the best discounting technique. It can be used to evaluate the design specification at an early stage. This will help in avoiding a lot of potential usability problems. It can also be used evaluate prototypes, story boards and even fully functional products. So it is flexible enough to be used at various stages of design and development of the product. In heuristic evaluation a small group of evaluators examine the interface and check if it complies with usability principles. If only a single evaluator examines the interface, he might end up missing out a lot of usability problems in the interface. But if there is a group of people, each person may think differently and might identify a variety of issues in the interface. That is why it is always better to have a group of people evaluate the product. Also there are 10 heuristics provided in the book, which help the evaluators to figure out issues. Another reason why I prefer this method is that it is cheaper when compared to other techniques.

It is recommended that about 5 evaluators evaluate a product. This makes it ideal for using this technique to evaluate the projects that we work on in class, especially if it is a group project. 2 – 3 groups can combine to mutually help each other in testing the product. This method is proving to be very useful for our current project related to an online portal for buying/selling items. We were able to find some issues related to navigability and consistency in our design.