I’ll be in London for the first half of September, and Peter Merholz
will be in town for a few days as well. We’ll be having a meetup on September 4th (coordinated by Andrew Harder, thanks so much), at John Snow at 19:00.
Looking forward to it!
Update: event is listed on Upcoming.
The other day we went to JumpSpace for a JumpTalk by Chuck Darrah about the work featured in his (along with James M. Freeman and Jan English-Lueck) new book Busier Than Ever!: Why American Families Can’t Slow Down
Busier Than Ever! follows the daily activities of fourteen American families. It explores why they are busy and what the consequences are for their lives. Busyness is not just a matter of personal time management, but of the activities we participate in and how each of us creates “the good life.” While numerous books deal with efficiency and the difficulties of balancing work and family, Busier Than Ever! offers a fresh approach. Busyness is not a “problem” to be solved-it is who we are as Americans and it’s redefining American families.
Chuck gave a compellingly accessible talk peppered with stories from their in-depth fieldwork (like something about of Albert Brooks’ Real Life, they spent a huge amount of time with these families, becoming unavoidably involved with their lives. So different than doing “an interview” as is typical in consulting work).
He handed me a copy of the book, so I’ve got some reading to do, but some cool themes/behaviors he told us about (and this is my scribbly documentation and doesn’t necessarily fully represent their work)
Ultimately, Chuck told us, it’s not about time, it’s about activities.
Meanwhile, a story that struck me a few weeks ago referred to a study (funded by the Alfred P. Sloan foundation, as was the Darrah et. al work above) about convenience food.
the researchers saw that convenience foods weren’t used as a time-saving substitute for the same dish made from scratch. Instead packaged foods offered a way for families to eat more elaborate meals than they would normally have time to prepare.
When families did cook from scratch, they ate simpler fare — like one-pot meals or stir-fry. In the end, dinner took about a half-hour to an hour to prepare, whether it was made from scratch or with convenience foods.
…
The study showed that meals with little or no convenience foods took 26 to 93 minutes to prepare. Meals that used a lot of convenience foods took 25 to 73 minutes to prepare. While convenience foods were time-savers on very elaborate meals, overall, there was no statistically significant difference in total preparation time.One difference that emerged was “hands on” time — the amount of time people spent slicing, dicing and stirring foods. Using convenience foods shaved about 10 minutes of hands-on time, but it didn’t make any difference in how quickly the food got to the table.
The study authors noted that the biggest time savings of convenience foods may be at the grocery store, where it’s faster to grab a frozen entree than to collect six separate ingredients to make the same dish from scratch. Grocery-shopping time wasn’t measured in the study. The average American spends about 22 minutes in the grocery store and shops about twice a week, according to the Food Marketing Institute.
Convenience foods also helped cooks offer a greater variety of dishes; cooks who made dinner from scratch offered three or fewer dishes. One family made a simple meal of sandwiches and edamame, using bread, cheese, greens and salmon and tomatoes. That meal took about a half-hour to prepare. Another family had a six-dish convenience-food meal of microwave barbecued ribs, macaroni and cheese, prebagged salad, bagged dinner rolls and a cookies and ice cream dessert. That meal also took a half-hour.
As Japan prepares to switch to a jury system, the government has been running mock trials to expose people to the idea. There’s a big cultural challenge, though.
“I think there is also the matter of how much he has repented,” one of the judges said. “Has he genuinely, deeply repented, or has the defendant repented in his own way? What’s the degree? I mean, some could even say that he hasn’t repented at all.”
Hoping for some response, the judge waited 14 seconds, then said, “What does everybody think?”
Nine seconds passed. “Doesn’t anyone have any opinions?”
After six more seconds, one woman questioned whether repentance should lead to a reduced sentence. “The way the defendant expresses himself and such, it could be viewed as someone who’s not good at it,” she said. “So there’s no way for us to know what is the degree of repentance from how he has repented in his own way.”
I’m reminded of our work with consumers in Japan; we got a lot of warning from people about what kind of answers the Japanese wouldn’t give us. I was somewhat nervous that things would tank entirely, but in fact, an open-ended conversational approach (with lots of follow up) worked out very well. The two situations (trials and design research sessions) are completely different; we didn’t represent the state, for one thing! Still, there’s a cultural layer to this story that many will no doubt recognize.
Transit Chief Plans to Ask Riders to Grade Subway and Bus Lines
Riders on each line will be asked to grade different aspects of service, including the cleanliness of cars and stations, safety and the responsiveness of employees.
He said he would also ask riders to list the three things that they thought most need to be improved.
“I want to know what passengers want,” Mr. Roberts said yesterday during a wide-ranging interview that touched on topics as diverse as dirty subway cars and his affinity for the poetry of Robert Frost.
“I think too often people sit around in offices like this and say, ‘O.K., I know better than the customer what it is they want and so this is what we’re going to do.’ I want the customer to drive the priorities.”
…He envisions cards that would be handed out to riders as they exit stations, and which they could fill out and mail in at no cost.
The impulse is good, but broken. Roberts realizes that the truth about riders/customers is not in his office but is “out there.” In the subway. With the riders. The real people.
So what does he do? He sits in his office and creates a piece of paper that will be given to those riders. The paper will be sent back into the office where people in the office will look at the paper and make decisions about what to do.
Why not go out of the office and talk to the riders while they are riding? Take that impulse, Roberts, and follow it to the next level!
I spoke recently at Shift, the IDSA conference held at RISD. I outlined seventeen ways to not suck at research:
1. Quit worrying about jargon
2. Think more broadly about which people you want to learn about
3. Garbage in, garbage out
4. Give other people the space to tell their stories
5. Follow up, and then follow up, and then follow up
6. Do you really want to use a survey? Probably not.
7. Collect and use natural language
8. Don’t forget that any research process with real live humans is hard
9. Breathe their air
10. Learning anything new requires rapport, and building rapport takes time
11. Finding insights requires pattern matching, creativity, synthesis
12. Personas are user-centered bullshit
13. Phil McKinney says “You’re probably not listening.”
14. Practice noticing stuff and telling stories (updated: read more here)
15. Do some improv
16. Embrace pop culture
17. Don’t forget about culture and social norms
The presentation was very well received, and I hope to share this material with another group before too long.
We’re doing a bunch of fieldwork these days looking at how people are using their software and hardware to accomplish some tasks. It’s interesting to see the difference in the flow between the part where we ask people to describe what they do and how they do it, and the part where we ask them to show us how they do it.
The discussion part is hard. It’s a bit abstract to explain a detailed behavior in absence of any props or artifacts. People work to give clear explanations and it takes a lot of follow-up to get the details.
Most of the same people light up when they are asked to show us (although some simply decide to show us without prompting).
But I find myself liking the “tell me how” part of the interviews better; the comfort level is lower, but the struggle to articulate is very insightful. Looking at how people describe things from memory isn’t wholly accurate in capturing their perceptions or usage, but it pulls out some neat contrasts.
And ideally, we’re trying to get stories, not factoids. The discussion (not the demo) is much better for stories. The participant takes over in the demo, and it becomes a semi-hurried list of “this works like this, and this other thing works like that.”
Of course, we’re doing both, and we need to be doing both, and some of the insights will come of the tension or differences between the two.
I think we’ll switch the order on today’s interviews, and maybe try starting with the demo, and then doing a discussion afterwards. I sometimes feel the demo requires time, and rapport, and trust, before we can safely ask for it (especially if the equipment in question rests within an inner sanctum), and so this is a bit of a leap for me.
Of course, there are few “right” answers in evolving one’s technique, it’s about building up a larger palette of approaches and making intelligent choices about when to switch around. I’m not at all unhappy about how the interviews have gone so far; they’ve been fun and fascinating, but I’m thinking hard about how to keep doing better.
This comment in the bad survey design thread got me thinking further about where/when/what to do with surveys. It’s not my primary tool so some of these reflections take me a little longer than someone who makes their living as a quantitative researcher, for example.
A tiny new restaurant opened in our tiny town of Montara – the Montara Bistro. I dropped by yesterday to pick up a menu and saw that they are already looking for customer feedback.
So to some folks, this is a survey. But it’s next to useless.
Why? Their questions are not too bad, but they are conversational questions, and should be presented that way. They are the basis of a conversation. Handing someone a sheet of paper (with no room to fill in any response) and asking them for essays is ineffective. It’s not fair. These are the questions they want answers to, but sometimes you have to ask a series of questions to get that information. And you can’t decide ahead of time which questions to ask. You have to ask a question, listen to the response, and then choose your next one. You can’t do that on a piece of paper. You need to have real people talking to each other and exploring the issues that way.
Not to mention that the restaurant has been open for a day or two, and there’s a presumption of an in-depth relationship that hasn’t really been built yet. What do I think of the Bistro Vision? Ummm, I don’t care.
I love what this artifact tells you about the company; that they really want to get a dialog going. They don’t have the tools in place to do it yet. Maybe it’s backed up by the way they interact with customers who come in; I don’t know. But this won’t work at all.
And I think this sort of inquiry is what a lot of design students are doing; identifying some open-ended (i.e., requires the respondent to write sentences) questions and sending them out by email. Some people will respond. Some may even write a lot. But you can’t follow up unless you send out another email. And then it’s just a conversation.
As with everything you “send out” who it gets sent to is a factor. Sending something to 3 friends is a very different approach than something that is quantitative in nature.
Look at this artifact from a recent project (created by our partners, not us):
This contained 31 questions, only a few open-ended ones. There’s randomization where needed (so you can filter out order-effects, where the first or last item might be picked more frequently in a list), and a large enough sample so that results can be processed to lead to conclusions – comparisons between different factors (this is the stats part I’ve been talking about).
Attitude toward technology meets Age
Purchase habits meets Region (with Age)
Stores shopped meets Region (with Age)
etc.
Tons of work and tons of math goes into creating tables (that then get interpreted) like
As Paul Hogan (sorta) said “That’s not a survey, now that’s a survey!”
I hope this brings a bit more clarity to the discussion.
They suck for us, the people who respond to them. Who wants to participate in something like this? (recorded after my conversation with a United rep to resolve the ticket cancellation I blogged the other day)
It’s not unusual at the conclusion of conversational bit of research (something interactive, interpersonal, and listening) for the person being interviewed to thank the person running the interview…at first you might think “you’re thanking ME? But I got great info from you…” but the fact is that an interview done well is a pleasure to participate in.
These surveys aren’t really interested in you; they’ve already forced you to think and respond in a completely unnatural and awkward fashion.
Sure, surveys do have their place and I’m being pejorative here only to highlight the bad user experience being created by this sort of research.
One of our recent clients, MediaMaster
just launched their product, that “lets you store all your music on the internet and play it from any internet-connected device.”
Their path from idea to launch has been a fascinating one (and I don’t know most of it, I’m sure). They came to this with a hand-coded technology to rip (via a CD jukebox) many albums in sequence, sort of a mass-scanning technology for CD ripping. But if you are going to rip the same albums over and over again, it’s time- and cost-effective to simply already have a copy of them already ripped and rather than rip, why not just check liner bar codes for proof of ownership and download the songs already on hand? And since the songs are already online, why not tie it to purchase of a new CD, and why not keep the music online permanently?
They dealt with a crazy mess of technological afforandances, changes of behaviors, retail and other partnership challenges and on and on. We did an online survey with some concepts, and then took concept boards out into homes to talk to different types of hypothesized target customers.
The product development process deals with a lot of moving targets, but startup folks deal with an excess of that challenge, collating input and constraints from so many quarters, it must make them crazy.
I’m excited to see the product launch, and to see where they’ve ended up with it, given where we were at during those rounds of research. I don’t know their business model, since the service is free right now. It’ll be fun to watch what happens with it and see if they make it succeed.
There’s tons of good stuff on business/manufacturing/processes/marketing/company culture/innovation in the fantastic article about Toyota from the Sunday NYT magazine. I’ve picked just a bit to share here.
Toyota’s chief engineers consider it their responsibility to begin a design (or a redesign) by going out and seeing for themselves – the term within Toyota is genchi genbutsu – what customers want in a car or a truck and how any current versions come up short. This quest can sometimes seem Arthurian, with chief engineers leading lonely and gallant expeditions in an attempt to figure out how to beat the competition. Most extreme, perhaps, was the task Yuji Yokoya set for himself when he was asked to redesign the Sienna minivan. He decided he would drive the Sienna (and other minivans) in every American state, every Canadian province and most of Mexico. Yokoya at one point decided to visit a tiny and remote Canadian town, Rankin Inlet, in Nunavut, near the Arctic Circle. He flew there in a small plane, borrowed a minivan from a Rankin Inlet taxi driver and drove around for a few minutes (there were very few roads). The point of all this to and fro, Jeff Liker says, was to test different vans – on ice, in wind, on highways and city streets – and make Toyota’s superior. Curiously, even when his three-year, 53,000-mile journey was finished, Yokoya could not stop. One person at Toyota told me he bumped into him at a hotel in the middle of Death Valley, Calif., after the new Sienna came out in 2004. Apparently, Yokoya wanted to see how his redesigned van was handling in the desert.
and
The way a farmer uses a truck is different from the way a construction worker does; preferences in Texas (for two-wheel drive) differ from those in Montana (for four-wheel drive). Truck drivers have diverse needs in terms of horsepower and torque, since they carry different payloads on different terrain. They also have variable needs when it comes to cab size (seating between two and five people) and fuel economy (depending on the length of a commute). In August 2002, Obu and his team began visiting different regions of the U.S.; they went to logging camps, horse farms, factories and construction sites to meet with truck owners. By asking them face to face about their needs, Obu and Schrage sought to understand preferences for towing capacity and power; by silently observing them at work, they learned things about the ideal placement of the gear shifter, for instance, or that the door handle and radio knobs should be extra large, because pickup owners often wear work gloves all day. When the team discerned that the pickup has now evolved into a kind of mobile office for many contractors, the engineers sought to create a space for a laptop and hanging files next to the driver. Finally, they made archaeological visits to truck graveyards in Michigan, where they poked around the rusting hulks of pickups and saw what parts had lasted. With so many retired trucks in one place, they also gained a better sense of how trucks had evolved over the past 30 years – becoming larger, more varied, more luxurious – and where they might go next.
Obu’s team, which drew on hundreds of engineers, ultimately produced a pickup model with 31 variations that include engines, wheelbases and cabs of different sizes. Design engineers, however, cannot simply create the best truck they can; they need to create the best truck that can be built in a big factory. In other words, Tundra’s design engineers had to confer with Tundra’s manufacturing engineers at every step of the way to create a truck – or 31 trucks, really – that could be assembled efficiently and systematically.