Jasmine Christensen

How do we measure design quality? This question has been asked at every company where I’ve led design, and it comes not only from designers but from stakeholders and executives as well. Solutions range from “we’ll know it when we see it”, to principle-based evaluation, to detailed and rigorous product and design reviews. Other suggested solutions have included surveys like NPS (Net Promoter Score), CSAT (Customer Satisfaction Score), or custom user surveys like “How delightful do you find our product?”

Each of these solutions is problematic. Internally-led evaluations may be biased by internal motivations or blindspots. Surveys may not be able to isolate design quality, or they might add an unnecessary burden to the user.

A few years back, I was leading a design and research team at a company that was newly obsessed with John Doerr’s philosophy and application of OKRs. We found ourselves measuring everything, including design quality.

The evaluation mechanism we landed on was an expanded set of product and usability heuristics based on Jakob Nielsen’s 10 Usability Heuristics for User Interface Design. We came to the conclusion that while there may be bias in a design team evaluating their own work, it’s a no-brainer that designers should know what “good” is, as critique is an essential part of our practice.

I worked with a few teammates to develop and iterate on a set of heuristics mapped to our general breakdown of product design: product thinking, UX design, and UI design. An additive benefit was that we looked at designer qualifications and skills in similar groupings, so as we were evaluating quality, we could easily have input to the question “where has the designer exhibited strengths or areas of opportunity in this project?”

To evaluate, a small group of us sat down after every product launch and did a lightweight review of the work that was shipped. We answered a simple yes or no to each question and implemented a formula to give a quality percentage score (100 for yes, 0 for no).

While of course, this wasn’t a a perfectly precise conclusion, we logged resonant variants from project to project and found it useful to have the language to give feedback to designers as well as share a state of quality with non-designers.

Here are the heuristics.

1. Product thinking

User problem

• Is the user problem to be solved clear?
• Does the research at hand support the proposed direction?
• Does the data at hand support the proposed direction?
• Does the product or feature solve the user problem stated in the product brief?

Audience

• Is the product or feature solving the problem for the target audience?
• Does the product or feature avoid negative tradeoffs for the target audience?
• Does the product or feature have minimal negative impact on a global audience/other audiences?

Value

• Does the product or feature bring clear added value to the user’s experience?
• Does the product or feature avoid removing value in other areas?

Design principles

• Does the product or feature guide without overwhelming?
• Has the product or feature been pressure tested?
• Does the product or feature champion quality?

2. UX Design

System status

• Is it clearly evident where the users are in the experience?
• Is it clear to users what has happened previously and what will happen next?

Clear task at hand

• Can users easily find what they are expected to do?
• Do the users know how they are expected to accomplish the task(s)?

Straightforward and ease of use

• Does the task(s) at hand require minimal effort to learn?
• Does the task(s) at hand follow a logical order?
• Does the task(s) at hand avoid unnecessary friction?

User expectations

• Does the task(s) at hand follow the user’s mental models?
• Does the product or feature follow common interaction patterns?
• Is the product or feature flexible or customizable to meet the needs of different user groups?
• Is the product or feature cohesive with other relevant product or feature systems?

Error handling

• Do users have a clear path when something goes wrong?
• Is error messaging clear, descriptive, and concise?
• Have edge cases been considered?
• Do users have access to help?

3. UI Design

Consistent standards

• Is the design of the feature or product accessible?
• Does the design of the feature or product use the give design system?
• Does the UX writing in the design follow content strategy standards (appropriate voice, tone, style)?

Recognition over recall

• Does the design of the feature or product make objects, actions, and options visible?
• Are instructions or information easily retrievable?

Familiarity

• Does page layout, hierarchy, etc or the feature or product feel familiar to other highly used features or products?
• Does the design of the feature or product follow appropriate UI conventions for respective platforms?

Craft

• Does the design of the feature or product tightly follow standards?
• Does the design of the feature or product feel best in class?
• Is the design of the feature or product clear and minimalist when possible?
• Does the design of the feature or product allow for delight when appropriate?

As a note, these were customized to our beliefs on design quality, and might vary based on context. For example, design principles were the principles that we established as a design team, and we focused on how people learn as we were working with students.

While these worked for us, we acknowledged that it wouldn’t scale much past our small team; for example, a team with hundreds of designers shipping daily wouldn’t likely have the bandwidth for manual reviews. However, as designers learn to moderate their own craft in different areas, we believed that these would still reinforce quality, and that implementing these formally or informally would still lead to quality output and outcomes—feeding into measurements of product and feature success—for both the company and our users.

Thanks to jennie § yip\ and Casey Callow for the partnership in designing these!

‍