Kevin Cheng  

Is Usability Repeatable?

October 29th, 2004 by Kevin Cheng :: see related comic

When one hires a carpenter, one can expect similar results between a any skilled carpenters. They all have a basic set of tools: hammers, saws, screwdrivers. All of which are used to help them do their jobs right every time. Usability has a pretty impressive toolkit as well: contextual inquiries, think aloud testing, personas, interviews, focus groups, etc. So if someone hires a competent usability consultancy, they should expect mostly the same results as the next, right?

Depends who you ask. According to Jared Spool: Wrong. According to Eric Schaffer: Right.We attended UIE’s UI9 conference a couple of weeks ago and got to hear a lot of interesting speakers. One of the biggest topics that emerged was Spool’s keynote (the same presented to BayCHI and to be presented shortly in Boston’s SIGCHI) and Schaffer’s counter in his workshop Institutionalizing Usability.

Spool cites a number of interesting pieces of evidence, beginning with a look at what are considered by many to be the more “usable” companies: Apple (iPod) and Google. In each case, a critically acclaimed piece of usability has emerged (though I personally would debate the iPod now that I’ve tried it). Yet Apple is closing its usability labs and Google has currently 6 usability analysts. Contrast this with the four hundred or so strong usability team Microsoft has and one has to start wondering whether there’s a correlation between usability budget and usable products.

Even stronger evidence exists in Rolf Molich’s Comparative Usability Evaluations (CUE) tests. Four different tests have been conducted thus far but the theme is the same: a number of participants, all usability professionals or teams of professionals, evaluate the same application or website independently and report their results back. In all cases, there was an alarming lack of overlap in the issues uncovered. For example, in CUE-2, nine international usability teams conducted studies on Hotmail.com and not one issue was found by all nine teams.

Schaffer on the other hand, believes the methodology is now at a level such that people can be trained to a certain minimum and yield repeatable results. He believes it so strongly, in fact, that he has created a certification for usability professionals called Certified Usability Analyst. While some at his workshop attempted to probe his contradictions to Spool’s stance and evidence, Schaffer cited his 400 CUA certified professionals as evidence - or at least of support.

Both sides have much more to say on the subject and I encourage you to go and listen to, and debate with, either of them if you have the opportunity. The questions remains: is usability a discipline that has repeatable results or do companies just have to roll the die and hire the right company? If so, what defines a “good” usability company?

Maybe we should ask Eric. He has a CUA.

27 Responses to “Is Usability Repeatable?”
dave wrote:

This is all qualitative. That means that there is too much room for differentials (differentials are not errors). The story should be, that you are hiring an expert set of extra eyes to make up for your personal deep involvement as a designer/creator. This is not the same as copy editing, it is more akin to just editing, where things like style come into play.

I betchya if we had all 400 of Eric’s certified usability professionals run tests on the same solutions, there will be a noticeable amount of variation. Maybe less, b/c they are using the same methodology.

I would contend though that Eric’s methodology when compared to Jakob’s or Jared’s (I mean he doesn’t even have a J in his name) will garner very different results. I don’t think that if we did a 3-way “contest” with these 3 “gurus” that we would ever say that if there are differences that one is not quite as “expert” as they claim (or their press claims) they are.

noah wrote:

Well, do you consider usability an art or a science? If it’s an art, it’s subjective and fluid. If it’s science, it’s objective and reproducible. Methodologies can be scientific but unless you are in a vacuum with no connection to visual design–and by that extension, human perception (hey, psychology isn’t even a science)–the application of those processes have to be expressive.

“If so, what defines a ‘good’ usability company?”

I would suppose the same things that define a good design collective.

veen wrote:

Oh whatever.

Usability is a tool. One of many. In fact, it feels more like the 10mm box-end wrench of a well-stocked tool chest. You may turn to it often, but in the larger context it’s pretty insignificant.

I could care less if usability findings are repetable. Quantitative estimates of efficacy are speculative at best. Who cares if you only find 20 percent of the usability problems with your product in one round of testing? Fix those things. Right now. Go fix them. Then do a few more sessions. Fix some more stuff.

Test. Fix. Test. FixTestFixTestFixtestfixtestfixtestfix.

Forever.

Eventually the long tail of “found issues” will spread out so far that you’ll be wasting your time trying to achieve microscopic improvements. Then you can relax.

But until then, keep looking. And use everything you possibly can to help you. Except maybe “certification.”

Josh Orum wrote:

I wonder how many experienced carpenters would claim to (i) use the same tools and (ii) generate the same results / quality of results even with the same tools.

Kevin Cheng wrote:

Josh: If you ask for a carpenter to build you a bookshelf, they will get you a bookshelf. Will it be identical to the next? No. Will they use exactly the same tools? No.

If you ask a usability firm to find you the serious issues (not ALL the issues - just the ones deemed serious) they’d also use their own tools (as Molich’s study permitted) and each will find a different subset of them. If I was a business owner who wanted to fix a website, I’d find that really disconcerting.

So Veen says, let’s just iterate forever until all the serious ones go away which is great if you’ve got a ton of time and money. Again, I want to be able to hire a company and say “find me the top issues and recommend fixes” and while the end result doesn’t equate to perfection, it should have some level of guarantee.

Incidentally, if it was even vaguely in question, I also think certification is pretty far from the answer.

Is Usability a craft or science? Well that’s actually what Spool’s topic is around. He feels we need to become an engineering discpline and we don’t have the science - the theory - to do that yet. He feels that if we worked as a craft, it would ultimately hurt us.

I don’t quite agree with Spool but where I can certainly see a problem is that businesses want some guarantee on their results, whether that’s realistic or not. Sure, you can hire a crap usability company and crap is still crap no matter what but at the moment, anyone can go out there and point out 20 random issues - call 5 of them serious and call it a day. Is that really what usability is?

Column Two wrote:

Is usability repeatable?

Kevin Cheng has written a blog entry, asking: is usability repeatable? To quote: When one hires a carpenter, one can expect similar results between a any skilled carpenters. They all have a basic set of tools: hammers, saws, screwdrivers. All…

noah wrote:

To quote Bender, “I can guarantee you anything you want.”

Companies want assurances that they are applying their money wisely in usability, but this is mostly addressed by the assignment process of the company itself. They are (or at least, should be) researching firms based on past assignments, determining best-fits from meetings, and so on.

Yes, anyone can do sub-standard work — the contractors that my co-op hired to fix our roof laid a new one over the old one instead, ruining it further. The message the Board got from the experience was that they needed to improve their selection process, and not that all roofing companies were corrupt and/or incompetent.

Is usability as utilitarian as fixing a roof, or is it more? I have to believe it is more. It would depress me greatly if the field wanted it to be that impersonal. Where does innovation go, then?

Ron Zeno wrote:

The issue isn’t whether something is art, science, engineering, etc. The issue is simply what are the practitioners, educators, and researchers doing to go to ensure their work and claims? Most practitioners and educators don’t even appear to understand the issue. Researchers who publish in peer-reviewed forums are at least held to some standards designed to ensure their claims. See: Toward a Reliable and Valid Usability Testing Reliable & Valid Usability?

I think both Eric and Jared are correct, it’s just that Eric and Jared wouldn’t and don’t agree on what a competent usability consultancy is.

As for Jeffrey’s (veen) viewpoint on how testing works, I can only say that if he can actually do what he claims, others certainly cannot. See: Fix an Incomprehensible User Interface?

MeriBlog : Meri Williams' Weblog wrote:

Halloween, Experiments and Beard-Certified Gurus

This has been linked all over the show, but it’s still important to note that 1049 federal rights depend on marital status. I wonder if anyone’s put together the number here in the UK — I know that some…

Rick wrote:

It seems to me that companies seeking “repeatable results” are companies that do not employ a user-centered process from the outset of product design. And, as a result, they attempt to make up for their inadequacy by building in lots of usability testing at the end of the product design cycle.

If a business asks a firm to find the ten biggest usability problems after a product has been built, then they’re asking the wrong question at the wrong time, and they will never get a satisfactory answer.

These types of businesses have an engineering mindset–a quantitative mindset. It works for programming and development (programming languages, after all, are logical). It will never work for interaction design or usability testing because human interaction occurs on an emotional level in chaotic environments–just think of the typical household or office.

Further, to say that we need to employ more quantitative techniques ignores the softness of the science we practice–the imprecise, inaccurate, and emotional nature of the subjects we study. Sure, human beings are capable of intelligent, rational thought, but even the smartest programmers (a seemingly logical bunch) yell at their monitor and smack their mouse when they get frustrated at a product that’s not working as they expect it to.

To be successful as a discipline, we must do something that also seems illogical: quantify the business value of using qualitative techniques and user-centered processes to create products that empower uses and meet business goals.

Kevin wrote:

My question is about the example, so nine different firms found a number of problems with Hotmail and they were all different from one another. Does that say that usability findings are not repeatable, or does it say there are a bunch of things wrong with hotmail, to the point where it is possible to find that many different issues?

Kevin Cheng wrote:

Does that say that usability findings are not repeatable, or does it say there are a bunch of things wrong with hotmail, to the point where it is possible to find that many different issues?

Both. In fact, one of the encouraging parts of the study is that even in high budget systems, usability analysts are still able to identify problem areas. However, shouldn’t you expect SOME overlap between these? If they were all to identify the top 10 issues, wouldn’t THAT yield some overlap? Or would you propose MS or whoever else (it’s easy to poke fun at MS but really, we’re talking about anything here) to have to hire all nine teams just to get a list of the serious issues?

Kevin D wrote:

Ok, admittedly, I can see why some overlap would be expected. Largely, this seems like a type of physics envy that psychology has fought for years. I am not sure but maybe we are going to enter an era where we have an practical side of usability and a scientific side. (maybe analogous to counseling vs research in Psychology). Scientific findings produce theories which infuse counseling methodologies, but generally counsellors do not run experiements on their clients to find evidence supporting a theory, or do they? (My analyst says I crazy, so I just listen to the voices in my head, these days.)

Ron Zeno wrote:

Because there have been multiple studies, with different products and different testers conducted by different researchers, all finding complementary results, no it isn’t just hotmail (or large products, or web sites, etc).

In fact, one of the encouraging parts of the study is that even in high budget systems, usability analysts are still able to identify problem areas.

That’s one possible interpretation, but certainly not the only one. The results may be due to usability analysts having a very large variation in their competency to identify problems, which is not very encouraging. The interpertation I favor is that usability analysts are using methods that are superficially the same, but are actually very different.

Jared Spool wrote:

As one of the J’s (I’m the adorable one), I’d like to add my $0.02 to this conversation, though Kevin’s done a good job of representing my view.

I actually don’t care too much whether usability practice is craft or engineering. It does seem to me, however, that, right now, we’re acting like a craft but selling our selves as engineering, which hurts us.

A craft is made up of craftspeople. Each craftsperson has a different level of skills. The quality of the results will depend upon the skills of the craftsperson you hire. You want better quality, you hire a better craftsperson.

Engineering uses engineers, all of whom have equivalent skills. The quality isn’t dependent on the individual hired — it’s dependent on the basic skills and knowledge of engineering. When you want better quality, you raise the minimum bar to be an engineer.

In my experience, Usability, when sold, whether by consultants or internally, sells itself as being engineering. It doesn’t matter who you hire — the fact that you’re now investing in usability will guarantee improved results. I’m not saying that’s how it works. I’m saying how it’s sold by the supporting institutions and communities, such as CHI and UPA.

However, the members of the practice act as it’s a craft. They never question when two different teams have completely different results, giving reasons like, “It’s qualitative! Of course you’ll see differences. Differences aren’t bad!”

But we sell usability practice as if it’s akin to radiology — a function that, given the same inputs, should produce the same outputs, no matter what. Imagine you were feeling pains in your thorax and had 10 different radiologists look at your x-rays. Would you be happy the idea that each radiologist gives you a completely different diagnosis? (”It’s Qualitative! Live with it!”)

Most people want their radiologists to make the exact same diagnosis as the best radiologist out there. They don’t want differences.

And, in my experience, when management invests in usability, they expect that the people they invest in are going to produce the exact same results as the best in the field.

If the community of usability practitioners wants to continue going forward as a craft, that’s completely ok with me. After all, there are many noble fields of craftspeople.

But, let’s be honest with our clients, and more importantly, honest with ourselves. If we’re going to continue as a craft, where the skills of the best matter, then let’s acknowledge that and tell everyone that it really does matter which usability people you hire to get the job done. Let’s help them understand how to choose. Let’s build apprenticeship and mentoring programs to create more master craftspeople.

On the other hand, if we want to be engineers, then we need to start understanding why, when 5 teams look at the same product with the same users, they get completely different results. We need to invest in building techniques that do produce reliable results. And we need to focus on a mechanism (maybe certification, but maybe not) that ensure that anyone claiming to do the work will produce the same results as the best in the field.

That’s my perspective on this issue.

Jared

reed wrote:

I guess you’ve never seen a carpenter really screw up a job then…

Ron Zeno wrote:

Worse than having a carpenter really screw up a job is finding later that the carpenter was unqualified for the job, or even worse, that the person was not a carpenter at all.

Craft vs engineering doesn’t matter. Usability is neither. Whether usability becomes one or the other or something else depends on what I mentioned previously: What are the practitioners, educators, and researchers doing to ensure their work and claims?

Jared’s (incorrect) definition of a craft allows all craftspeople in a given craft to be incompetent and provides no means for determining the competency of craftspeople. I certainly don’t think usability is such a “craft”, and I certainly don’t want it to become so.

Jared Spool wrote:

I think it’s awfully unfair to characterize my definition of craft as all craftspeople being incompetent. On the contrary, in a free market, the better craftspeople tend to develop a following and a brand, allowing them to capitalize on their skill and constantly raising the standard through competition.

I would assume that, if the usability community adopts the craft model, the competition would be open and strong, thereby highlighting the various important qualities that will emerge for the master craftspersons to shine and prosper.

Only in a very unlikely scenario of no competition or comparison would Ron’s assumed scenario even be a liklihood. Ron has nothing to worry about.

Ron Zeno wrote:

I’m sorry Jared felt I was being unfair. Because Jared’s earlier definition of craft allows for incompetency of all craftspeople, it is incomplete and incorrect. In response to my issue of ensuring work and claims, the definition allows for no response - nothing is done to ensure work and claims. Know of any crafts where nothing is done to ensure work and claims? Can there be a craft that does nothing? I’d argue against it, or at least point out that such a definition of “craft” is meaningless or useless.

Jared’s now added a competitive marketplace to the picture, and the ability to compare results as well. That brings us back to the issue of repeatability. When results are compared, they are very different and have little in common. What does that mean? That analysts have a very wide rage of competence, that they are doing similar-looking but inherently different things, or perhaps some combination of the two?

Terry Sullivan wrote:

I think there are really two distinct issues arising here:

1) Is “repeatable” usability a core requirement for cost-effective evaluation?

I submit that the answer is a provisional “no.”

What matters most from a business perspective is whether one receives a satisfactory return on investment (ROI) from the evaluation method(s) applied by the evaluator(s) applying them.

If I have _N_ usability issues scattered randomly among _M_ interface components, will “breadth-first” testing produce the same results as “depth-first” testing? Probably not. And it’s a fact of life that evaluation coverage is rarely adequate, and almost never exhaustive. Evaluation, like design, is almost always an exercise in optimization-within-constraints.

But if ROI is comparable between the “breadth-first” and “depth-first” approaches, then their lack of overlap is mostly moot. And if ROI isn’t comparable, then the core issue is evaluation quality, not lack of repeatability. Which leads to the second issue…

2) Does practitioner skill impact usability evaulation results?

I guess my first thought is: Why exactly would one expect usability to be exempt from practitioner quality issues? The only tasks I know of where (most) everyone is equally good are entirely mechanical in nature (say, running a vacuum cleaner, or tying shoelaces).

But in every task requiring ever moderate skill, individual differences abound.

To cull from the examples used above: A carpentry hack is not interchangeable with a fine cabinet maker. Highly skilled radiologists frequently “see” things that others miss. In software development circles, it is an axiom that the best developers are at least 10 times better than the worst ones. Even in engineering, marked differences exist among practitioners (which is one of the reasons that engineers often work in teams). Hell, the last dentist I went to needed three tries before he fixed my tooth properly. (And even now, I’m still suspicious…)

Strengths differ; skills differ. And so long as they do, quality of results among practitioners will differ, too. The more complex and challenging the task, the greater the impact of practitioner skill on the eventual outcome. In every human endeavor, talent is a scarce resource. Egalitarian urges aside, it’s hard imagine why anyone would truly expect usability to be different.

Ash Donaldson wrote:

Something that hasn’t popped it’s head up here is the question “What kind of usability testing are we talking about?”

There’s a very big difference between formative and summative usability testing yet many practitioners I’ve come across haven’t even heard of them. Thus people often confuse the two types.

Formative testing is less formal and more qualitative. Summative testing is more rigorous, taking into account controlled and uncontrolled variables, environment, sample (users in this case) and procedure. It’s conducted in a similar fashion as a scientific experiment to allow for reproducible results. There’s even a standard for the test report (ANSI/INCITS 354-2001: Common Industry Format for Usability Test Reports) that some of us have been working to fast track to an International Standard.

As far as I can tell, the quoted examples were “comparing apples and oranges”. None of them seem to have used the same rigorous procedure by trained professionals to compare metrics (timings, error rates, assists, etc), identified problems (points that caused errors and required assists), or subjective feedback (answers to standardised questionnaires), using a comparable sample population (the same user demographics - including cultural background).

The CUEs couldn’t possibly evaluate whether usability evaluation results were reproducible. They could, however, highlight the fact that different cultures will find different difficulties in a one-size-fits-all interface, or more pointedly that people practising usability using different techniques will come up with different results. It’s actually quite a strong case for a standardised curriculum and certification of those that have passed the curriculum.

Jared Spool wrote:

Terry said, quite correctly, “Highly skilled radiologists frequently “see” things that others miss.”

This is true. But, if you show a simple case to ten of these “highly skilled” radiologists, they will come back with extremely similar diagnoses. If you sit with them while they are diagnosing, you’ll hear that they use very similar techniques and protocols for evaluation. Their reports will look extremely similar.

As CUE and other studies have shown, no such similarities exist in the practice of usability. It’s left up to the individual practitioner. And, when you ask more than one to evaluate the same interface, the differences are really quite scary.

In CUE-4, 17 very talented teams evaluated the same interface. These weren’t randomly picked teams. They were well respected folks. I’ve gotten to know many of them well over the years. I’d worked closely with several of them. And I have great respect for all of their work. (I regularly recommend their work to many of our clients.)

So, from these top teams, I would’ve expected that we’d see a lot of overlap on the problems. The combined teams, discovered a boatload of problems, including many critical issues that, could lead to application failure — things the app’s designer agreed needed to be addressed immediately. However, 61 of those critical issues were only discovered by a single team. Each of the 17 teams discovered at least one critical issue that nobody else saw.

The CUE-4 application wasn’t a particularly complex interface. It was a one-screen hotel reservation system. It was far simpler than most of the interfaces we work on regularly. Yet, these highly skilled usability practitioners didn’t come close to producing similar results based on their highly skilled evaluations.

If the client wanted to discover and fix all of the critical issues, which team should he hire? He couldn’t hire just one — he’d have to hire all 17 to ensure he succeeded.

By allowing this to be the state of the art, are we saying it is unreasonable for him to expect to fix every critical issue in his application? As a professional, would you say to that client, “Well, I can only guarantee that I’ll find, at most, 10% of your critical problems. You’ll want to hire other people too.” Do you think clients will be happy to hire 17 teams, just to ensure they got the job done?

What happens when clients start to discover that hiring the best professionals doesn’t ensure that some critical, grief-causing issue will go undetected? Do they start to put less credibility in the practice of usability? Do we wonder why large organizations, such as Microsoft, would rather ship products they suspect are unusable just to see what clients complain about, than to try to identify and fix the problems before launch?

This isn’t a matter of making sure you get someone who is at the top of the field. This is a problem that is much more insidious than we ever suspected. If you have the best usability folks all evaluate the same interface, you still get radically different results. Why should anyone believe anything we say?

That’s why this is such an alarming issue. What are we doing about it?

Carolyn Snyder wrote:

Ron asked whether there are any crafts where nothing is done to ensure work and claims. Astrology comes to mind :-) . And more seriously, so does therapy, where reputable practitioners generally do not provide guarantees because so much of the outcome depends on what the client is willing/able to do. I tell my clients up front that testing will not find all the problems, but from a practical perspective it will find enough important ones to fill the time they can devote to making changes.

I’m becoming pretty convinced that usability is fundamentally a craft, and that those who practice it well have developed intangible skills in areas such as understanding the company’s business goals, learning how methods can bias results, and thinking through the consequences of recommendations. These skills are learnable, but they’re far less visible than usability tests and reports. By focusing on the tangible outputs, as the CUE studies have done, I think we’re missing most of the story.

The recent NIST workshop on reporting methods underscored this for me, as I realized I was less interested in simply seeing what other people put in reports as I was in understanding when and why they chose to do things as they did. Each workshop participant carried in their head a large number of unspoken “rules” about what works well in a given situation. There is a wealth of untapped knowledge here.

I also think there are some useful areas to research about usability testing itself. What accounts for the variability among practitioners of usability testing? Is it the users (we’ve all had those outliers :-) , the tasks, the facilitation methods, how the issues are prioritized, etc.? Of course, the answer is, “All the above and more,” but it would be great to start to tease this apart.

And even if there comes a day when we understand all this, usability testing will still be a craft in any situation where the circumstances are evolving rapidly, as is the case with the web and many software applications. The concept of repeatability only applies if you can control all inputs except the one you’re trying to measure. Carpenters and radiologists have a better shot at doing that because the underlying laws of physics and the human body aren’t changing out from under them every few months.

Terry Sullivan wrote:

Jared Spool wote:
The CUE-4 application wasn’t a particularly complex interface.

At the risk of seeming contrary:

  1. Multiple test subjects in CUE-4 vocally disagree with the above statement.
  2. Regardless of “screen count,” any interface that presents 300 unique opportunities for user error is (by any meaningful quantitative definition I know of) not “simple.”

As regards the CUE-4 results in particular, I think whether one sees the glass as “half empty” or “half full” is entirely dependent on one’s point of view:

  • Fact: Most errors were discovered by multiple teams.
    (Fully 62% of all errors reported were reported by at least two teams.)
  • Fact: There was a full order-of-magnitude difference (highest -vs- lowest) in terms of resources applied to the task. Large differences in test coverage (and therefore nonidentical results) are virtually guaranteed. (Conversely, had identical results been achieved, then we’d all be decrying the astonishing level of wasted effort.)
  • Fact: Unnecessary imprecision in the error classification criteria allowed the “same” error to be discovered/reported multiple ways, and to be assigned a different priority by different teams. There was far too much room for evaluator subjectivity in classifying errors and their severity. “Repeatability” simply cannot be measured (let alone achieved) in the absence of rigorously defined, unambiguous classification criteria.
  • Fact: There was an artificial cap placed on the number of reportable errors.
    The impact of this experimental artifact cannot be underestimated. If indeed 61 “critical” errors exist*, then the arbitrary upper limit of 50 reportable errors guarantees that one of two conditions must inevitably occur:

    • Fully 18% of the critical errors will go unreported by all 17 teams. (Thus allowing for 100% overlap among the participants’ reports… but this is desirable???)
    • There will be less than 100% overlap among the critical errors reported.

    Again, this says nothing about the teams, their methods, nor their results. It’s entirely an artifact of arbitrary experimenter fiat.

* I say “if” only because I am far too lazy to cull through 800+ error reports trying to identify some relatively tiny subset. I’m willing to take 61 as a working figure because the methodological confound remains at issue so long as the “true” number of critical errors exceeds 50.

I want to be clear: I’m not suggesting that the state of the usability art cannot be improved. But the CUE-4 data are far too ambiguous to permit any definitive conclusions about the root cause(s) of differences among evaluation results.

(The part of CUE-4 that I personally find most troublesome is the fact that there was so little consensus regarding the “top 5 usability problems” in the interface itself. But again, with at least 61 different candidates to choose from, there are plenty of “critical errors” to go around.)

Ron Zeno wrote:

I’m going to use as an example, Carolyn’s in-jest suggestion of Astrology as a craft. There are a couple of interesting things to note about astrologers:

They often give disclaimers (sometimes required by law) that their work is “for entertainment purposes only”.

The most noticeable (and perhaps only) difference between a professional astrologer and anyone else is that the professional is skilled at deluding others.

Ash Donaldson wrote:

If this area is purely a ‘craft’, how does anyone know if they’re right? Do you just ‘feel’ that you designed it correctly? If so, isn’t a software engineer’s design just as valid as a usability professional’s?

A well designed study involving humans won’t be confounded. Ask anyone who has studied psychology, HCI, Human Factors, Activity Theory, or any of the social and cognitive sciences… ;)

Currently, this area of practise lacks accountability (”It’s a good design because, ummm, well because I said so, and I’m a professional - so there!”). We have tools available from the sciences to validate our hypotheses - we should use them.

I’m not arguing that Interaction Design is purely science, but that it has its roots both in craft and science. In my mind, unless you are a specialist (usability test facilitator (psych), visual designer (art), information architect (information/library science), software engineer (IT), etc), practitioners should have a balanced toolbox of science, engineering and art, ie. a good understanding of Human Factors and HCI, knowledge of the technical aspects of the artefacts they are designing, and art theory/studio skills.

I understand that a few institutions are now providing this basis of study (even here in Australia!) and applaud their efforts.

Eric Schaffer PhD CUA CPE wrote:

The issue is how we as usability professionals will do business. Are we going to be a scattered set of small consulting firms and lone practitioners inside of companies? Forget it. To make the world usable we need to do this work on a large scale. To do this right we need systematic staff selection, training, and certification. We need methodology and tools and quality assurance. We need hundreds of practitioners, with global reach and offshore operations. We need to run usability like the other successful businesses today. NOT like a carpenter. Like a furniture factory. This does not mean we can’t be creative, strategic, and cutting edge. In fact, from my experience doing this at HFI, it opens a whole world of opportunities. It is effective and good fun.


Leave a Reply


OK/Cancel is a comic strip collaboration co-written and co-illustrated by Kevin Cheng and Tom Chi. Our subject matter focuses on interfaces, good and bad and the people behind the industry of building interfaces - usability specialists, interaction designers, human-computer interaction (HCI) experts, industrial designers, etc. (Who Links Here) ?