Posts tagged “research methodology”

ChittahChattah Quickies

Meet Google’s search anthropologist [SF Chron] – While the article still has a bit of the wow-technology-companies-use-social-science-to-watch-people-use-stuff wide-eyedness we see in every popular press piece, I was intrigued by the nice exploration of the gulf between what some people reveal they need and what design changes make it into the product. It’s not a one-to-one match and the article speaks to that reasonably well.

Google has hundreds of millions of users, each with different needs, working styles and levels of search competence. Every change for one subset – like those who occasionally use advanced search – comes at a cost for others – like the vast majority of people who never use it and don’t want it cluttering up the main page. Striking the right balance for the greater good requires listening to the data – and, of course, to the users themselves. “That particular interview didn’t finish off the painting,” Russell said. “But every interview helps fill in a little bit more of the canvas.”

Why Storytellers Lie [The Atlantic] – “Lie” is a perfect headline-grabbing word and it probably pays to read the piece with a less judgmental take on what people tell us. There are many situations that are lies but in research it’s our job to seek a number of possible truths and understand why what we hear may not always be the same as what we identify as true.

Sinister as that may sound, therapy likely helps many of us feel better at least in part because it encourages us to become less truthful autobiographers. As studies have shown, depressives tend to have more realistic-and less inflated-perceptions of their importance, abilities, and power in the world than others. So those of us who benefit from therapy may like it in large part because it helps us to do what others can do more naturally: to see ourselves as heroes; to write (and re-write) the stories of our lives in ways that cast us in the best possible light; to believe that we have grown from helpless orphans or outcasts to warriors in control of our fate…We should remember how much we all have a tendency to fictionalize, whether we realize it or not. We like stories because, as Gotschall puts it, we are “addicted to meaning”-and meaning is not always the same as the truth.

Clickers Offer Instant Interactions in More Venues [NYT] – This continues to be an almost-trend; the desire/opportunity/ability to “like” stuff IRL (“in real life”) the way we do on Facebook (see a previous example here).

The delighted shouts from middle-schoolers and seniors alike suggest that neither group is accustomed to having its opinions solicited. But with a clicker, “suddenly their voices are important,” said Professor James Katz, the director of the Center for Mobile Communication Studies at Rutgers. “If people feel their opinions really count, they’ll be happy and likely to give more opinions.” The dynamic of social comparison – understanding where you stand relative to your tribe – is also a draw. Clicker software satisfies that curiosity by immediately displaying a bar graph of responses in the room. “This is a new form of transparency for crowd psychology,” he said. He added some cautions about using clickers, also called audience response systems. In a society in which checking the crowd’s opinion becomes the norm, Professor Katz said, taking risks or relying on one’s instincts may be devalued. “Those who want to strike out in new directions and challenge the sentiments of a crowd, like artists and writers, have an additional burden with this technology because they can know that no one takes comfort in their vision,” he said. “There goes the Great American Novel.”

The Wizard of Oz Focus Group – Footage from an early focus group for The Wizard of Oz. ‘Nuff said.

Embedding disabled. Drat! Just click on the link above, then.

What Makes People Share Information? [Mozilla UX] – The Mozilla UX team is doing a nice job at sharing their inquiries, their methods, their artifacts and their thinking behind all of ’em.

We’re starting another research study this week. We’re interviewing 8 users in their homes, for 90 minutes each, to understand how people define their online life. It’s purposely broad as we’re trying to learn more about how people discover and organize websites from both online and offline sources. It wouldn’t be a successful interview without some artifacts to help us collect this data, so we came up with a two fun activities – the timeline and “me in the middle”. At the beginning we’ll start with a simple timeline and have the participants walk us through their yesterday – what they did, where they where and we’ll prompt for what tools and devices they used – but that is just a way to get all the raw data on paper quickly. What we are really after is their stories.

Bringing truth to advertising

Questioning the nature of research [research-live.com] – Ogilvy Group UK vice chairman Rory Sutherland advocates for context-based research to inform advertising, which is mostly served now by traditional quantitative market or survey research methodologies. We are messy beings, and straight-forward research approaches yield neat numbers that have nothing to do with the reality of decision making. This is preaching to the choir on this forum, but the interview is chock full of quotables! In the end, he calls for research that gets closer to people, and for an experimental approach to developing marketing and advertising.

No-one in any research group would ever say, “If there are four brands of shampoo, I’ll buy the one that has most bottles on the shelf”, or “I’ll choose the one that’s on the third shelf up because it’s the one that doesn’t require much reaching down” or “I’ll look at the prices of three products and choose the one in the middle.” In reality, we use heuristics and shortcuts and cognitively miserliness like this all the time. The mistake that quite a lot of advertising methodologies make is assuming that brand preference translates perfectly into purchase behaviour. It’s also making the assumption, of course, that preference is formed in advance of behaviour. Quite a lot of evidence from both behavioural sciences and from neuroscience suggests that we act first and form our opinions in light of our actions.

I think the way we think we decide and the way we actually decide don’t have that much in common. The conscious rational brain isn’t the Oval Office; it isn’t there making executive decisions in our minds. It’s actually the press office issuing explanations for actions we’ve already taken.

I’m emphatically not downplaying the importance of fame, awareness, mental availability – whatever you want to call it. What I would downplay is detailed dissections of consumers stated reasons for adopting or planning to adopt a particular course of action.

Quite often, people within a group will pretend they are a maximiser, when most of our decisions are taken as satisficers. We always claim in the presence of others that we are great connoisseurs looking for the best value for money we can find, but most of the time we simply don’t have the mental energy for that. It would be an insane use of our mental resources in any case. So what we do is look for something that is pretty much guaranteed not to be crap. That’s why in some ways you can never quite make sense of the popularity of McDonalds. Everybody, whenever you talk about food, they’ll always talk bullshit about health and Italy and olive oil. But actually, when it comes to eating, what we want is a place that won’t rip us off, won’t give us food poisoning, the toilets will be clean, the service will be OK, and everything will be pretty good. Paul Dolan, the government’s wellbeing adviser, says: “Nothing is as important as we think it is when we’re thinking about it”.

ChittahChattah Quickies

The Saddest Movie in the World [Smithsonian.com] Describes the rigorous process of choosing clips that will reliably evoke various emotions for clinical research purposes, and how the use of movies to elicit unpleasant emotional responses is considered humane and ethical. It’s incredible that a Ricky Schroder scene from the rather obscure The Champ has been scientifically deemed sadder than, say, Bambi’s Mom dying or Old Yeller. Can’t argue with science! (But I’d bet that the first 5 minutes of Up would beat them all.) Another gem here: the two clips that are proven most effective in generating feelings of disgust – yes, I’m on about disgust again! – are an amputation and… Pink Flamingos!

The story of how a mediocre movie became a good tool for scientists dates back to 1988, when Robert Levenson, a psychology professor at the University of California, Berkeley, and his graduate student, James Gross, started soliciting movie recommendations from colleagues, film critics, video store employees and movie buffs. “Everybody thinks it’s easy,” Levenson says. Levenson and Gross ended up evaluating more than 250 films and film clips. They edited the best ones into segments a few minutes long and selected 78 contenders.

Scientists testing emotions in research subjects have resorted to a variety of techniques, including playing emotional music, exposing volunteers to hydrogen sulfide (“fart spray”) to generate disgust or asking subjects to read a series of depressing statements. They’ve rewarded test subjects with money or cookies to study happiness or made them perform tedious and frustrating tasks to study anger. “In the old days, we used to be able to induce fear by giving people electric shocks,” Levenson says. Ethical concerns now put more constraints on how scientists can elicit negative emotions. Sadness is especially difficult. How do you induce a feeling of loss or failure in the laboratory without resorting to deception or making a test subject feel miserable? “You can’t tell them something horrible has happened to their family, or tell them they have some terrible disease,” says William Frey II, a University of Minnesota neuroscientist who has studied the composition of tears. But as Gross says, “films have this really unusual status.” People willingly pay money to see tearjerkers-and walk out of the theater with no apparent ill effect. As a result, “there’s an ethical exemption” to making someone emotional with a film, Gross says.

ChittahChattah Quickies

The Writer of a Cat Food Commercial Confronts a Focus Group [McSweeneys.net] – This “Short Imagined Monologue” by Mike Gallagher has a little fun with the unfortunate power dynamics of a focus group, and the struggle those behind the glass can have accepting the feedback that they are asking (and paying) for. Fault must lie at the feet of the participants, who are clearly incapable of understanding!

I don’t mean to startle you by barging into your focus group like this. Everything’s cool. I’ve been watching you guys from behind the one-way mirror there and I thought I’d make my presence known. Normally I’m content to just chill behind the glass, make disparaging comments to my fellow observers, and eat handfuls of M&Ms. And while I know this is “highly irregular” I feel like I have to say something to help you better understand the overall messaging, the gestalt if you will, of the TV commercials we’re testing here today. Not to take anything away from Fiona, your lovely and talented focus group moderator. Hey, she’s doing a great, great job… I don’t think you people truly appreciate what’s at stake here. We, all of us, are tasked with “redefining” the pet food commercial from the bowl up… What’s concerning me is that my work may be too- strong for you groundlings. Too avant-garde. So this one time I’m going to explain it very slowly and in terms you can all understand. Like captions for the “Thinking Impaired”.

What’s in your garage?

My sister sent me this photo of my niece, enjoying a springtime adventure on her driveway with three items she found in the garage: a skateboard, a sled, and a plastic bat.

As we embark on our various projects, her example can serve as an inspiration to use the tools we have at our disposal in surprising combinations, to lead us in delightful new directions!

Reading Ahead: Photo Diaries

Reading ahead logo with space above

In addition to our in-person fieldwork, we often ask research participants to do some kind of task on their own. In past projects, these assignments have included a range of activities, from workbooks and journals with specific daily prompts, to fairly open-ended photo diaries.

For Reading Ahead, we asked people to do a short photo diary, and send us five or so digital pictures (before the interview session) that would help us get a sense of how reading fits into their lives.

Diaries like this accomplish several things. They give us access to a person’s world beyond what we might be able to get in just a face-to-face meeting. We’re able to see what they do in more locations, at more different times, and in more situations.

We probably won’t be there to see someone actually reading in bed before they go to sleep at night, but they might well ask a family member to take a picture of it for their photo diary, as Tracy did for this project.

Portigal-Consulting_diary1

Diaries also help us build rapport more quickly with the people we’re meeting, by giving us a common set of stories to begin the conversation with. There’s often a bit of back and forth dialogue between us and the participants during the assignment as well, which helps us establish a relationship.

Having some shared knowledge and possibly interaction prior to the interview means that when we are face-to-face, we can jump right in with that person at a deeper level, which can free up time in the interview session for exploring more areas of the research topic.

When you look at the pictures people sent us for this project, you’ll see that they’ve responded in a variety of ways. Some focused on objects, some took pictures of other people, and some photographed themselves or had other people photograph them. It’s useful to see the different ways people interpret the topic area, and the connections they draw. It helps us understand how each person sees the world, and can point us to additional lines of inquiry.

Below are the photo diaries for Reading Ahead.

Erica

Peter

Tracy

Julie

Reading Ahead: Props For The Field

Reading ahead logo with space above

As we get into actual fieldwork this week, we’ll be using (as is typical for us) a mashup of techniques. In addition to interviewing people, we’ll be watching how and where they read books or Kindles. But we’ll also want to get into a discussion of future possibilities. Reading Ahead is not about evaluating existing designs but instead getting inspiration and information that can drive future designs. So we don’t have anything to test, but that doesn’t mean we can’t use testing-like (“What do you think of this?”) activities. In this case, we’ve built extremely simple representations that suggest book and digital reader.

prototyping
Dan building a thing-to-hold for this week’s interviews

The idea (and we’ll learn what happens once we do a couple of interviews) is to have something neutral to put into people’s hands and let them gesture, sketch, or otherwise perform, so the activity of discussing the future isn’t just a verbal one. We’re going to ask people to draw on these props, and then we’ll have an artifact created by the participants themselves. Those artifacts can be compelling, and can also be a much more impactful symbol of what took place in that part of the interview.

We’ve never used this exact prop before, but in just about every project we’ll come up with a range of tools to use in the field. Our interviews are very open-ended so we could use either of the props to explore any emergent themes, depending on what we use them to talk about (From “How would you hold this?” to “Well, if it did come with your Happy Meal, draw what you’d expect to see on the main screen.”

For more, see Moving with a Magic Thing (PDF) and Design the Box

Reading Ahead: The Interview Guide

Reading ahead logo with space above

Before we go out in the field we write an interview guide (or field guide), a list of all the topics we want to cover.

Interview guides end up being very linear and structured, but the interactions we have in the field are looser and more conversational. We’ll let the way we pose our questions flow much more from what the person we’re interviewing is saying than from the sequence and phrasings of the interview guide.

Even though we know this will happen, we’ve still been working hard to hash out our questions ahead of time–even the basic ones. It’s like John McLaughlin said about jazz improvisation: you have to learn all the chords so that you can forget about them.

The interview guide is also something for everyone (including our clients) to look at, to make sure we’re all on the same page as we head into fieldwork.

Our interview guide for Reading Ahead is here.

Anonymous Responses Are Useless

From Garrick Van Buren’s Work Better Weblog

One of my current projects has a major survey component. The survey ends with:

If you’d be open to follow up questions, enter your email address below.

There’s about a 60 / 40 split on responses with emails and those without. The responses without email addresses have skipped questions, irrelevant answers, and are generally unusable. This is so much the case, that I’ve found it a better use of time to check for an email address first – then read the response.

It’s interesting that people comfortable with being contacted give useful answers, while those providing non-useful responses don’t provide a way to be contacted.

Conventional wisdom on requiring accountability has it backward. Accountable people want to be responsible for their actions. Those that aren’t don’t. Forcing it doesn’t change anything.

This is purely anecdotal of course, and may not generalize to other surveys about other topics, but nonetheless seemed an interesting data point.

Series

About Steve