Kevin Cheng  

Diary Studies: Too Much Information?

May 21st, 2004 by Kevin Cheng :: see related comic

When I mention diary studies, I often expect people to either give me a blank stare or have some adverse reaction akin to “blech! that will take too much time/money!” The reality of it is that the methodology isn’t as time consuming nor is it all that complicated. Sometimes, we might get a little more information that we bargained for but that’s rarely a bad thing.

Let’s look at what a diary study is and where these misconceptions might come from.

What It Is

First of all, what is it? For the purposes of this discussion, I’m talking purely about the technique of before-after diary studies. I was going to link an appropriate article but can’t seem to Google one so here is my meagre attempt. The process goes something like this:

  1. Interview participant and record their plan for the day.
    • When do they plan to do things
    • How important are those tasks?
    • Fill in the gaps in between tasks mentioned if there are any (”What are you doing between these two meetings?”)
    • Check for any scheduled items which show more concrete sense of their schedule
    • What items need to be done but have no time set? When about will those be done?
    • Why are certain tasks rated as high importance?
    • Are there contingency plans for certain things?
    • Are there tasks that should be broken down into subtasks? (”How long does it take to prepare dinner?”)
  2. Have participant fill out diary during the day.
    • They should fill out start and end times of activities
    • They should fill as much detail as they are comfortable with but obviously, the more the better
  3. Interview participant at the end of the day
    • Let them speak first and listen to what details they convey as these are the details they feel are important to them
    • Ask about discrepancies in time (”How come you arrived later than you planned?”)
    • Ask about discrepancies in the details (”What made you go to this meeting instead of the other one?”)
    • Again, fill in the time gaps and separate tasks appropriately (”How long did you spend e-mailing before heading out to lunch?”)
    • The main question to ask is Why (”Why did you decide not to go to this event?”)
    • However, questions like Where, Who etc are obviously important as well (”Who did you run this meeting with?” “Where did you spend 15mins struggling with the photocopier?”)
    • In particular, ask about high priority items and scheduled items that were missed.
  4. Analyse the data (see below)

Obviously, these bullets are quick and dirty and by no means a complete guide nor do I claim to be an expert in the field but it’s a good primer to get this discussion going.

What Do You Do With It?

Read it like a novel and send a collection to your publisher as memoirs.

OK, maybe not. Initially, one would think simply having this data is sufficient in understanding the users better. This point is partially true. In asking the right kinds of questions in the post interview, you gain an understanding of what their real priorities are and what sort of external forces might cause them to change their perceived working (or living) habits.

In understanding where the participant’s priorities are, you also get to understand what aspects are important to them. This understanding leads to the possibility of determining what are relevant metrics for the participant.

For example, you may find something quantitative (usually time based) as relevant like “how many papers I marked in an hour”. However, you may also find that some measures are related to less concrete dimensions, such as quality. For example, perhaps the teacher participant didn’t read through some of the papers thoroughly but marked them anyways. By mentioning the incomplete nature (remember that what they mention is what’s likely to be important to them), they place a value on the quality of their markings.

So now we can say, “maybe we should measure the quality of the markings.” Rather vague, but it’s something that can be solidified over a few iterations. As the ethnographer becomes more experienced, they can go further by creating combinations of the metrics to see what is truly important. In this case, the number of papers marked and read thoroughly in an hour.

Metrics are important in understanding if the system you’re designing is actually effective and improving a task or process but what to measure on isn’t always clear. So aside from understanding what the participant is doing and why, we can use the studies to find out how to measure the things that actually matter.

Too Much Information?

What is most deceiving about diary studies is that a lot of information is seemingly irrelevant to the reason you’re conducting the test, and thus, you may choose not to collect the data. For example, maybe you’re looking into how students use computers at school and thus, only conduct a study for the school day. If you conducted the study for the full day, however, you might find that the reason they don’t use the computers at school is because they have their own computers at home and prefer the setup to the school’s or perhaps the school doesn’t have the necessary software installed.

Too Much Time?

On the part of the ethnographer, this study really only takes about an hour to an hour and a half for a single participant and perhaps another two hours for analysis. That kind of time is on par with usability tests (preparation and execution) or contextual inquiries.

So It’s Perfect?

Of course, no technique is ideal. Doing one day of test on a participant can of course lead to some anomalous data. A schedule breaker may occur early in the day like a planned event being cancelled. While the information on how such a cancellation affects the participant is interesting, it may not be as useful as a “typical” day. Thus, as with any other technique, more data means less risk of outliers.

Also, the technique requires a certain level of experience to ascertain the useful bits of information. It’s easy to be distracted by other interesting pieces of information which may have no bearing on your particular design. Moreover, it’s really hard to find the pieces that are relevant to your design because they are not always apparent.

Diary studies are not for everyone, but they’re incredibly useful when you want to understand more about the users, their metrics, their context or the activities which they perform - either directly related to your system or not. You might be surprised which ones are! Sometimes, the participants themselves will find out more about themselves as well, saying things like, “hey, I didn’t even realize I was so prone to interruptions.”

8 Responses to “Diary Studies: Too Much Information?”
Column Two wrote:

Diary studies: Too much information?

Kevin Cheng has written an entry on diary studies, in which plenty of practical information is given. To quote: Diary studies are not for everyone, but they’re incredibly useful when you want to understand more about the users, their metrics,…

Kevin Lee wrote:

I have run a few diary studies with both traditional method (paper) and practical method (palm pilot). I found that participant’s answers were greatly influenced and affected by what the medium they were asked to carry. Palm users were by far “proactive” in providing “unexpected” information while paper diary users were not.

I have not had a chance to use the smart phone with a video cam but one of my colleagues in Asia told me it works “fantastic.” Providing the most trendy medium as main diary study helps participant to increase their own confidence as well as proactive in the study..rather than passive and collect the cash. This device also allows participant to snap instances without forcing them to elaborate in length.

Diary study is effective in understanding user’s pattern without shadowing them. However, the validity of information should be justified and analyzed carefully based on participant’s motivation and willingness to establish a partnership with facilitators.

Bob wrote:

While merkin in a nice word, I think my favourite must be cran. Not as in R, but as in the slightly more (Monty) Pythonesque 37.5 gallon of herring. Slubberdegullion’s not too bad either.

Peter Van Dijck's Guide to Ease wrote:

OK/Cancel: Diary Studies: Too Much Information?: (in the comments) “I have not had a chance to use the smart phone with a video cam but one of my colleagues in Asia told me it works “fantastic.”…

Moi wrote:

Err, lads… You know that Merkin also (online) slang for an American, right?

Jonathan wrote:

WHOA!! That was one Google search that was NOT work-safe.

Jeff Axup wrote:

Interesting overview, but I’m wondering about step 1. Why is there emphasis placed on planned versus actual behavior? Is that relevant for the interface design? Diary studies I’ve seen focus more on actual reports of behavior and post-hoc explanation of that behavior. I suppose it might be useful information if you were designing a scheduling app, but for informing most interface designs I’d think it more appropriate to focus on real situated behavior and personal perception of activities.

Kevin, do you know what palm app was being used? Were they using thumb-keyboards? I could see rapid entry of verbose comments being a problem unlike paper.

Kevin Cheng wrote:

Jeff, Before and After studies include the “Before” step to understand how the participant sees themselves. Think of how oyu do user interviews, and take those results with a grain of salt because you know that they answer the questions with how they see themselves, not as how they may actually be.

In a contextual inquiry, you might find the participant stated previously that they communicate primarily by e-mail but then in observing them, find that they are interrupted often.

Before and After serves the same effect (without following the participant) and more. We can understand from the plan:

  • how they perceive their own working style to be
  • what kinds of things affect their plans
  • what things are perceived to be important and which can actually be dismissed
  • how they adjust to changes
  • how long they think they take to do things as opposed to how long it actually takes

Some of these points help understand the user but are difficult to apply directly. Others help you build the users’ mental models.

Leave a Reply

OK/Cancel is a comic strip collaboration co-written and co-illustrated by Kevin Cheng and Tom Chi. Our subject matter focuses on interfaces, good and bad and the people behind the industry of building interfaces - usability specialists, interaction designers, human-computer interaction (HCI) experts, industrial designers, etc. (Who Links Here) ?