If you inspect a prototype or product and systematically check whether it adheres to a set of heuristics, you are conducting what is called a heuristic inspection or heuristic evaluation. It is a simple, effective, and inexpensive means of identifying problems and defects and is an excellent first technique to use before moving on to more costly and involved methods such as user observation sessions.

It is usually best when a heuristic evaluation is carried out by an experienced usability specialist, but heuristic evaluations can also be very effectively when they are conducted by a team of individuals with diverse backgrounds (for example, domain experts, developers, and users).

To conduct a heuristic evaluation, you should choose several scenarios for various tasks that a user would perform. As you act out each of the steps of the task flows in the scenarios, consult the list of heuristics, and judge whether the interface conforms to each heuristic (if it is applicable).

Jakob Nielsen introduced the idea of heuristic evaluations, and his 1994 list of ten heuristics, reproduced below, is still the most commonly used set of heuristics today (Nielsen, 1994, p. 30):

An obvious weakness of the heuristic inspection technique is that the inspectors are usually not the actual users. Biases, pre-existing knowledge, incorrect assumptions about how users go about tasks, and the skill or lack of skill of the inspectors are all factors that can skew the results of a heuristic inspection.

Heuristic inspections can also be combined with standards inspections or checklist inspections, where you inspect the interface and verify that it conforms to documents such as style guides, platform standards guides, or specific checklists devised by your project team. This can help ensure conformity and consistency throughout your application.

One way of building up a mental model in a user’s head is to provide documentation such as a training manual or tutorial. The vast majority of users will never read documentation, however.

For enterprise systems in an organizational setting, you can have your users sit through a training program, but the effectiveness of corporate training varies. The motivation of employees to pay attention and learn is often low, and for training on complex applications, classroom sessions without hands-on practice are essentially useless if you want staff to understand and retain the material.

Some users have the benefit of being able to watch other users use the application, and this can be a very effective way of learning the basic concepts and understanding how to perform tasks. Having an expert nearby whom the user can ask for assistance is also very helpful.

But without any training, documentation, or opportunities to watch and ask other users, the only way a user can figure out how to use the application is to simply start using it. She will learn how to operate the application by trial-and-error. The visual presentation of the application’s user interface provides cues as to how to accomplish tasks, and the behavior of the application provides feedback on whether the tasks are being performed correctly. By exploring and experimenting with the application, the user gradually builds up a mental model of how the application works, and, with time and experience, the user’s mental model will (hopefully) increasingly approximate the designer’s mental model.

To use the terminology popularized by Donald Norman in The Design of Everyday Things, the conceptual model in the designer’s mind is called the design model. The user’s mental model is simply referred to as the user’s model. And the presentation and behavior that the product’s user interface exhibits is called the system image.

And so to design a usable and learnable product, then, the designer’s challenge can be viewed as aligning the design model and the system image, and structuring the system image in such a way that it accurately portrays the design model and enables the user to develop her own user’s model that closely approximates the designer’s model. As the completeness and correctness of the user’s model increases, the user’s skill at operating the application will approach that of the application’s designer.

Structuring the system image to make an application learnable and understandable is tricky, and the fundamental aim of this blog and the upcoming book Designing Usable Apps is to explain how to do this by means of design principles and techniques, and usability testing and evaluation techniques.

One traditional approach to designing and building software products was the waterfall model, where requirements are first gathered and written up in specifications documents. These are then handed off to designers, who create designs and write design specifications. These are then handed off to developers, who build the product. The product is finally handed off to testers, who verify that the product matches the specifications.

This sounds fine in theory — it’s a very logical, rational decomposition — but for large, complex products, this approach never really seems to work very efficiently. For complex projects, it’s never possible for analysts and designers and programmers to get everything correct and complete on the first try, and so waterfall projects inevitably tend to break down into a chaotic scene of documents being passed back and forth between groups for correction. And since software projects span months or years, it’s very frequent that the requirements will change during the course of the project, meaning that by the time the product is finally built and released, it may no longer actually meet the needs of the users and stakeholders.

An effective way of bringing some order to this chaos is to recognize that complex analysis, design, and development work is never done completely or correctly on the first attempt; it takes many iterations of reviewing, revising, and testing to get it right.

An iterative approach to design and construction breaks the project into many short, structured cycles of work. In each pass around the cycle — each iteration — the work products get better and better and more complete. An advantage to this approach is that you get a basic functioning version of the product available for testing very early on in the project, and this early product can be used to discuss and further refine requirements with the project stakeholders.

Attempts to illustrate an iteration of the design cycle usually end up looking something like this:

The Design Cycle

This diagram is unsatisfying, though: it suggests that the activities are separate and take place sequentially, and this is not always the case. There is often constant, fluid switching between the different activities, and team members will usually be working on different activities simultaneously in parallel.

In addition, the nature of different products can enable various different design approaches:

• For products with formal processes and very specific externally-imposed requirements, such as a tax calculator, requirements analysis and specification usually have to be figured out fairly thoroughly before detailed design can proceed.
• On the other end of the spectrum, products such as games have no real requirements — just about anything goes, design-wise — and so requirements analysis virtually disappears.
• Most products fit somewhere in the middle, and requirements analysis and design proceed together in a tightly meshed fashion. Sometimes requirements aren’t formally recorded at all, and instead the design is simply continually adjusted to match the new learnings about how the product should work. So in these cases, the Understand requirements and Design activities merge together.

And for products that lend themselves to rapid prototyping, often no formal design documentation is ever recorded. The prototype is the representation of the design, and so the Design and Build activities merge together.

The User-Centered Design approach recommends that you involve users in requirements gathering, and in the usability testing and evaluation of designs, prototypes, and the actual product.

In other blog posts, we’ll take a closer look at the activities in the design cycle. We’ll examine requirements analysis and validation, the process of design, prototyping, evaluating designs and prototypes, and conducting usability testing.

• Interviews with users
• Interviews with managers and other stakeholders (if applicable)
• Interviews with subject matter (domain) experts
• Questionnaire surveys
• Observing users…
• using your product or prototype
• performing their tasks without your product, either with a competitor’s product, or by doing the task manually without a technology-supported solution
• Analyzing website analytics data: User demographics, frequently-searched keywords, heatmaps, usage patterns, etc.
• Literature review: Academic research, third-party market research reports, etc.
• “Documentation archaeology” from past projects

Some user characteristics for your product might be chosen “by design” when you define your target market and product positioning strategy. For instance, by targeting a product towards children, you’re intentionally restricting the age range of the target user group to a certain range.

If you are making any assumptions about your users and target market, it is a good idea to test your assumptions. But research costs time and money, and so especially in smaller projects, you will have to weigh the cost-benefit ratio of improving the quality of the information you’ll be using for future decision-making.

A persona or user persona is a brief textual description of a fictional character who is representative of a stereotypical user in a user segment. The persona describes some invented personal details about the character, explains in general terms what they do with the product, discusses the context or environment in which they use the product, and mentions some of the problems, frustrations, and concerns that they might face.

You should invent a fictional name and a descriptive title for each persona; for example, Carol Jones, insurance adjuster, or Bob Davis, middle-aged man wishing to book a family vacation. To make the persona more tangible and memorable, you might also include a photograph (perhaps selected from a stock photo repository).

For example, a sample persona for a new word processing product might be as follows:

Example persona: Alice Smith, aspiring first-time novelist

Alice is 29 years old, divorced, and has a two-year-old daughter. She has a bachelor’s degree in English Literature from a state university, and currently works full-time as a marketing assistant for a major pharmaceutical firm. Alice sees herself as a very artistic person, and dislikes the fact that her corporate job offers no real opportunities to express herself creatively. She is particularly passionate about reading historical fiction, and so she now wishes to pursue her long-time dream of writing her own novel in this genre.

Because of her job and child-care responsibilities, Alice tries to get her writing done in small blocks either early in the morning, or late in the evening after she has put her daughter to bed. On weekends, she enjoys writing while sitting in her favorite coffee shop. She is frustrated it takes her a lot of time to get warmed up and into the “flow” of writing, and because she only has short windows of time to work on her book, she doesn’t feel that she is making very good progress.

Alice is currently using Microsoft Word on her MacBook laptop, but she is finding it difficult to organize her notes, plan out her plot structure, and generate an outline. She has character and plot notes and multiple chapters spread across multiple Word documents, as well as hand-written notes in a notebook and on various loose scraps of paper, and switching between these is becoming a hassle. She feels she could be more productive if she could just get a better handle on organizing all of her project materials.

She would like to have her novel published by a big-name publisher, but she knows that this is difficult for a first-time fiction author. She is considering self-publishing but has not had an opportunity to research what is involved, and she is particularly worried about the costs involved. She would also like to make her novel available on e-book readers like the Kindle, but she is unsure of how to convert her manuscript to an e-book format and get it listed on major booksellers’ sites like Amazon.com.

The advantage of personas is that they help your team develop a shared understanding of the requirements of each user segment, and they encourage empathy with the end users of your product. By bringing to life someone from each user segment, with a human name and face, designers can design specifically with that particular user in mind. When designing a task flow or a screen, you might ask, “How would Alice want this information presented?” or “How would Alice react in this situation?”

Personas can also help focus usability tests and evaluations. For preliminary evaluations of a prototype, tests could be carried out by somebody pretending to act in the role of the persona, or at least with that persona in mind. (Of course, it is preferable to involve real users from the user segments, but this is not always possible.)

You can also use the personas to prioritize the requirements. For instance, one release could concentrate on the core features needed by the Alice persona, and then next release could concentrate on another persona.

Personas are not without criticism; many people find the invention of fictional names and personal details to be highly gimmicky. Trying to represent a diverse user segment with a single persona runs the risk that the persona doesn’t accurately represent a large percentage of the members of the user segment. And if you try to write up personas without ever actually investigating or speaking to real live users, these completely hypothetical personas may have no relation to reality.

The following is a list of some of the characteristics you might want to know about each user segment. Not all characteristics are relevant for all types of product — some may only be appropriate for software used at a workplace, for instance.

• Age
• Gender
• Educational background
• Language and culture
• Computing skills
• Physical abilities and disabilities
• Domain-related knowledge and skills (e.g., accounting knowledge for an accounting application)
• Job experience and competence
• Place in the organizational hierarchy
• Attitudes, motivation, and morale
• Persistence, patience, confidence, problem-solving ability, curiosity, ability to deal with change, etc.
• Frustrations and problems relating to the user’s tasks or activities
• General sources of stress or anxiety (e.g., deadlines, performance targets, workplace competition)

Additionally, you may also give some though to the context in which your users will use your product:

• Physical environment (e.g., home, office, factory, vehicle, oil rig, on-the-go in an urban environment, etc.)
• Social environment (position within organization, relationship to other groups, political and interpersonal factors, degree of freedom, influence in decision-making, etc.)

Every individual is different, of course, and there is a risk of creating generalized stereotypes that doesn’t accurately describe many users in a user segment, but it is still worthwhile taking the time to think about each characteristic. For some of these characteristics, you might describe an approximate range and an average. For instance, bank tellers at a particular financial institution might range in age from 20 to 50 years of age, with the average age being 28.

For each user segment or role, you may want to write up a brief profile, which can then be reviewed and discussed with your project team. A profile for a user segment can be presented simply as a bulleted list with characteristics described in point form. Or, you might use a matrix to compare user segments side-by-side. Alternatively, you can represent user segments by means of personas, which we’ll discuss in the next post.

End users are the people who will use the product in a hands-on manner a day-to-day basis.

The buyer is the person (or committee) who makes the final decision to purchase the product.

For enterprise software such as a call center management application, executives make the decision to purchase and install the system, but these executives usually never operate the software themselves.

Similarly, educational computer games for young children are not purchased by the kids themselves. Children may influence the purchasing decision, but it is the parents or other adults who buy the product.

Products purchased as gifts are another example of where the buyer is not the end user.

So while user experience design and usability testing must focus on the needs of the end users, your product must also simultaneously appeal to the potentials buyer, or it will never sell. Market research to understand buyers’ expectations and concerns is important and can strongly influence the scope and design of your product.

Websites and enterprise software use roles to restrict access to functionality to different user segments. For instance, a web-based discussion forum will assign most users the “contributor” role, which permits them to post and reply on message threads, but some selected users will be assigned the “moderator” role, which gives these users the additional ability to edit and delete posts.

Other products may be general-purpose tools that can be used for various purposes, or may offer a wealth of features that users may use in various combinations. These products don’t have explicit roles. For example:

• Word processors may be used to write letters, business documents, essays, reports, novels, technical books, journalism articles, diary entries, webpages, and so on, and the needs of the user for each case might be different. You’d also expect a word processor to be used by a very diverse user community — students, professionals, and home users, with varying ranges of educational attainment, and with different physical impairments.

• Spreadsheets are used for many different purposes by financial analysts, data analysts, accountants, software developers, project managers, and others.

Understanding what user segments and roles are relevant for your product is an important early step, as the requirements of users in each group will drive the design of your product.

The environment

Some textbooks recommend setting up a formal usability testing lab with one-way mirrors and multiple cameras, and insist upon highly structured sessions with a full team of facilitators, observers and recorders. If you can afford this, then great, but these ideas make user observation seem more complicated and mysterious than it really is.

Not only are formal laboratory environments expensive, but they can make participants feel uncomfortable. Being watched by a team of people and recorded by cameras will make participants nervous, as if they’re performing for an audience.

To make people feel more relaxed, you can get great results simply sitting one-on-one with a participant in front of a laptop in a neutral and comfortable setting like a coffee shop. If you’re building a product that will be used in a particular environment, try to do the session in that environment: If you’re building enterprise software, sit down together at your users’ desks. If you’re building software for police officers on patrol, schedule meetings with officers in their police cars. Not only are people are more likely to open up and discuss their opinions more freely when they’re in familiar surroundings, but you’ll also get a feel for their environment and any distractions.

Depending on your goals and budget, you may consider recording the interaction with screen recording software. Possibly, you might also consider setting up a camera to record the user’s body language, facial reactions, and their physical interactions with input devices. But you need to be aware that people act differently when they know they are being recorded. While recording a session offers the convenience of replaying and reanalyzing the recording as many times as you like, you should also not underestimate the amount of time it will take to review and analyze a batch of recordings.

Choosing goals for the session

Decide in advance what kind of data or learnings you are aiming to get. For instance, you may want to:

• Determine what percentage of users are able to carry out a task successfully
• Find places or situations where users get confused, hesitate, or don’t know to proceed
• Find places or situations where users tend to make the most errors
• Collect impressions and suggestions from users on what works well and what could be improved
• Collect judgements from users on the value, usefulness, attractiveness, and usability of the product
• Get feedback on how your product compares with competing products

If you collect metrics such as the number of errors made or the average time taken to perform a task, you can compare statistics across different batches of testing sessions to test whether changes to the product have actually led to measurable improvements.

Running the session

When you start a session with a participant, welcome them and briefly explain what you’re aiming to accomplish. If you will be conducting audio, video, or screen recording of the session, or if any personally identifying data will be collected, it is customary to have the participant understand and agree to this by signing a consent form.

Because they are being observed, participants often feel that they are being tested or quizzed, and participants can often become embarrassed and ashamed when they make a mistake or can’t figure out how to do something with the product. Reassure subjects that you’re not testing or evaluating them personally; instead, you’re testing and evaluating the product, and because the product is not yet perfect or complete, the goal is to find flaws and opportunities for improvement in the product. Explain that if the participant makes an error or gets stuck, it’s not their fault; rather, it’s a signal that the product needs to be improved.

How you run the session depends on what data you’re intending to collect. Typically, you will ask the user to accomplish one or more goals, and you’ll observe them as they explore the product and try to figure out how to how they go about doing it. Make sure you explain the goals or tasks clearly, but at the same time, try not to give too many clues as to how to do it (“leading” the user).

Users will often ask for help or seek acknowledgement that they’re on the right track, asking questions such as, “I am doing this right? Do I click here? What’s the next step?” How you offer assistance is up to you. When the user makes a false step, you may be tempted to jump in right away, but it’s better to observe and how and when the user detects the error and how they recover from it.

You might choose to ask participants to “think aloud” — that is, as they try to figure out how the product works, they should try to vocalize their inner thoughts: “I want to do a search for something. Where can I do a search? I don’t see a search box anywhere. Normally it would be up in this corner over here. Maybe there’s something under one of the menus? No, there’s no Search menu. Maybe under the Edit menu? I see a Find command, but is that what I want?”  This kind of ongoing dialogue can provide useful insights, but many people find it uncomfortable and unnatural to do this. As well, if you’re trying to take notes by hand, you’ll never be able to write fast enough.

You can ask for critiques and suggestions at various stages, but also realize that not every user is in a position to give appropriate or good advice on issues like screen layout or interaction design.

Recording notes and observations

You will want to have a notepad handy where you can keep a log of the user’s actions, results, comments, questions, any long pauses indicating confusion, and so on. You should also keep a tally of errors and mistakes. If you see patterns emerging — users getting stuck at a certain point, or asking how to proceed — make a note and keep count of how many other participants encounter the same difficulty. To save time, consider preparing a template or chart you can fill out, and develop a list of short abbreviated codes to use to refer to recurring situations.

If you’re working with a high-fidelity prototype or the actual product, you might also use a stopwatch to time how long it takes a user to complete certain tasks. However, accurate timings are difficult if you’ve asked the user to “think aloud”, if questions and discussions are taking place, or if the user is pausing to let you take notes. Using a stopwatch will also put pressure on users, so again, reassure the user that it’s not a race and you’re not testing their personal performance.

If you find that your note-taking slows down the session, you may consider having another person join to take notes so you can concentrate on facilitating the session. But having multiple people managing the session can sometimes be distracting and overwhelming, and it can be unprofessional when the team members haven’t prepared and rehearsed their coordination ahead of time.

Afterwards

Be sure to thank your participant for their time and feedback. You might also ask participants to fill out a questionnaire afterwards. This gives you another chance to collect feedback (and it might give participants more time to think through their responses). If you ask for satisfaction ratings on, say, a scale of 1 to 10, you can collect quantitative data that you can compare with other batches of testing sessions.

Analyzing and communicating results

After running a batch of sessions, consolidate your notes and review any recordings. Tally and calculate any metrics, and compare any statistics to previous runs. By analyzing your notes and data, you can find problem areas, for which you can then recommend potential solutions. Put together the results and recommendations in a brief report for review in your project team.

If you have a consumer product intended for sale to the general public, you’ll need to make sure that your user tests involve a matching diversity of people representative of your target audience.

If you have a specialized niche product, your pool of potential users may be small, but your potential subjects will be more motivated to participate, as your product promises to solve their particular problems and is tailored to their needs. You may have a harder time finding enough suitable users in your local area and so you may need to resort to “distance testing” via teleconferencing and screen-sharing software. Industry publications, professional associations, and discussion forums catering to your target audience can be useful for recruiting suitable participants.

Here are some ideas for sources of potential users:

• Friends and family
• Employees in your development team or department
• Employees from elsewhere in your organization
• Your existing customers
• Subscribers to your company/product newsletter or blog
• People at trade shows and conventions
• People in your professional network
• People recruited through advertisements

Recruiting your friends, family, and coworkers is often easier than other methods, but statisticians call this convenience sampling, and convenience samples introduce biases into your results. Your participants may not accurately represent the cross-section of users who will actually be buying and using your product, and so you may draw incorrect conclusions from your observations and usability tests.

If you solicit participants from the general public via advertising, you’ll usually need to offer an incentive, usually cash. But be aware that this can attract a certain kind of people. And there are many people whom you may want to reach but who will never respond: Not many high-powered lawyers earning $300 per hour will take an hour or two out of their busy schedule for a $50 gift certificate, for instance. And introverted individuals are less likely to sign up for usability testing sessions. Again, the main point here is that you need to make sure that the people you’re recruiting are a reasonable sample of your target audience, and if you suspect your sample is not representative, then you need to be aware of potential biases.

Recruiting participants and scheduling meetings can be time-consuming, so you may want to delegate this to an assistant. You also need to plan for the fact that a shockingly large percentage of people will not show up to their appointments. Reminder phone calls the day before the appointment can help, though.

How many users do you need for a usability testing study? One or two participants is too little (though better than nothing); ten is sometimes too many as you’ll usually see patterns emerging by then. Five to eight is a good target to aim for.