As I’ve said before, garbage in, garbage out. From Rob Walker’s Consumed

Recently, Stardoll did a study of its own, polling United States users about their brand preferences. Apparently they saw real-world brands on the same plane as the half-dozen or so invented brands that exist only within the site. (Some respondents even made the – clearly impossible – claim that they wear the strictly digital Goth-style brand Fallen Angel to school.)

These sorts of stories always crop up in market research and business case studies. And they are wonderful because they illustrate the depth of meaning the products, services, brands, and stories we create can be to the people that consume them. So meaningful that they will conflate pretend brands online and tangible experiences offline. Wow, we marvel, that tells you how great our stuff is; they will lie about it.

But the flip side to that is that if you are going to ask people what they think and do and want, you better have a way of triangulating their responses against other data. If you don’t know more about the person than their response, how can you contextualize it? If you don’t know what they are really saying when they answer the question – if they understood the question or are answering it in the way you intended – then you must be very careful in what you conclude and how you act on those answers.

From our Design Research Methods class, some observations from an interviewing exercise.

The scenario was to conduct interviews in order to uncover opportunities in helping people to manage food, meals, nutrition, etc.

The question asked was
What are the challenges you face in meal preparation?

Of course, that question is flawed because it presumes that there are indeed challenges. This was evident when the respondent struggled with how to answer outside the frame of the question.

An alternative might be
Are there any challenges in meal preparation?
which is is more open-ended.

But better still is
What are your feelings about the experience of meal-preparation?
since it doesn’t put the label challenges into play. It would be important to understand the labels the person being interviewed places on the different aspects of their experience, and to use their terminology to probe further.

Also worth noting is that the original question came right off the sample interview guide I distributed. Sometimes the interview guide is a tool to document “questions you want answers to” rather than “questions you want to ask”; doing fieldwork involves a lot of translation back and forth between the two.