Transit Chief Plans to Ask Riders to Grade Subway and Bus Lines

Riders on each line will be asked to grade different aspects of service, including the cleanliness of cars and stations, safety and the responsiveness of employees.

He said he would also ask riders to list the three things that they thought most need to be improved.

“I want to know what passengers want,” Mr. Roberts said yesterday during a wide-ranging interview that touched on topics as diverse as dirty subway cars and his affinity for the poetry of Robert Frost.

“I think too often people sit around in offices like this and say, ‘O.K., I know better than the customer what it is they want and so this is what we’re going to do.’ I want the customer to drive the priorities.”

…He envisions cards that would be handed out to riders as they exit stations, and which they could fill out and mail in at no cost.

The impulse is good, but broken. Roberts realizes that the truth about riders/customers is not in his office but is “out there.” In the subway. With the riders. The real people.

So what does he do? He sits in his office and creates a piece of paper that will be given to those riders. The paper will be sent back into the office where people in the office will look at the paper and make decisions about what to do.

Why not go out of the office and talk to the riders while they are riding? Take that impulse, Roberts, and follow it to the next level!

I spoke recently at Shift, the IDSA conference held at RISD. I outlined seventeen ways to not suck at research:

1. Quit worrying about jargon
2. Think more broadly about which people you want to learn about
3. Garbage in, garbage out
4. Give other people the space to tell their stories
5. Follow up, and then follow up, and then follow up
6. Do you really want to use a survey? Probably not.
7. Collect and use natural language
8. Don’t forget that any research process with real live humans is hard
9. Breathe their air
10. Learning anything new requires rapport, and building rapport takes time
11. Finding insights requires pattern matching, creativity, synthesis
12. Personas are user-centered bullshit
13. Phil McKinney says “You’re probably not listening.”
14. Practice noticing stuff and telling stories (updated: read more here)
15. Do some improv
16. Embrace pop culture
17. Don’t forget about culture and social norms

The presentation was very well received, and I hope to share this material with another group before too long.

We’re doing a bunch of fieldwork these days looking at how people are using their software and hardware to accomplish some tasks. It’s interesting to see the difference in the flow between the part where we ask people to describe what they do and how they do it, and the part where we ask them to show us how they do it.

The discussion part is hard. It’s a bit abstract to explain a detailed behavior in absence of any props or artifacts. People work to give clear explanations and it takes a lot of follow-up to get the details.

Most of the same people light up when they are asked to show us (although some simply decide to show us without prompting).

But I find myself liking the “tell me how” part of the interviews better; the comfort level is lower, but the struggle to articulate is very insightful. Looking at how people describe things from memory isn’t wholly accurate in capturing their perceptions or usage, but it pulls out some neat contrasts.

And ideally, we’re trying to get stories, not factoids. The discussion (not the demo) is much better for stories. The participant takes over in the demo, and it becomes a semi-hurried list of “this works like this, and this other thing works like that.”

Of course, we’re doing both, and we need to be doing both, and some of the insights will come of the tension or differences between the two.

I think we’ll switch the order on today’s interviews, and maybe try starting with the demo, and then doing a discussion afterwards. I sometimes feel the demo requires time, and rapport, and trust, before we can safely ask for it (especially if the equipment in question rests within an inner sanctum), and so this is a bit of a leap for me.

Of course, there are few “right” answers in evolving one’s technique, it’s about building up a larger palette of approaches and making intelligent choices about when to switch around. I’m not at all unhappy about how the interviews have gone so far; they’ve been fun and fascinating, but I’m thinking hard about how to keep doing better.

I’ve written so much about surveys as of late and was so amused to receive one in the mail from the research arm of my HMO, Kaiser Permanente. The focus of the survey is Genes, Environment, and Health.

It’s a very sophisticated effort, with a two-page FAQ (some parts are interesting/amusing: Does this research involve cloning, or stem cells, or genetic engineering? and What do you mean by “genes” and “environment”?). See Page 1 and Page 2 for the full FAQ.

Here’s the last page, showing questions 30 through 37.

(click to enlarge)

This is a serious survey, it’s obviously been assembled by specialists in medical quantitative research, and has no doubt gone through human subjects approval/ethics guidelines, etc. I’d say this is an example of survey best practice. However, I’m not going to participate; the user experience for me is not pleasant; there’s very little upside (being asked to provide a saliva sample later on? Whoopee!) and although it’s funny out of context to look at question 36 (getting and keeping an erection (or hard-on) that is rigid enough…), it’s a not so interesting (or turn-off) to fill out a survey about that sort of thing.

I really want to say that the whole survey seems long, and hard. But that would bring the level of discourse way way way down on this blog and I pride myself on setting high standards here.

Recently we were interviewed for two different global studies on global innovation. Both the German Research Foundation or Deutsche Forschungsgemeinschaft and the Danish Ministry of Economic and Business Affairs and Centre for Economic and Business Research or FORA separately came through the Bay Area as part of as study of leading innovation/design entities in the US and around the world.

Both groups offered to share their results with me and so I’ll post more specifics here when I’ve got ’em. Meanwhile, it was fun to be interviewed instead of doing the interviewing, even if it was clearly a conversational, expert-interview type of thing, and not an ethnography (neither did a huge amount of following-up or probing, it was pretty much based on what I had to say). One interesting technique, though: the group from FORA asked me to reiterate some of my key points from our longer interview (recorded on audio) for a 5-minute summary video. They went over with each other, and with me, what points seemed the most salient, and then reinterviewed me into the video camera. The point was not to get more information, but simply to create an effective deliverable. I was not at all uncomfortable with it, indeed I felt very listened to in having a precis of my interview created on-the-fly. It reminded me that being able to communicate what is learned from the field is crucial; that methods could be innovated around the effective communication as much as the effective discovery.

This comment in the bad survey design thread got me thinking further about where/when/what to do with surveys. It’s not my primary tool so some of these reflections take me a little longer than someone who makes their living as a quantitative researcher, for example.

A tiny new restaurant opened in our tiny town of Montara – the Montara Bistro. I dropped by yesterday to pick up a menu and saw that they are already looking for customer feedback.

So to some folks, this is a survey. But it’s next to useless.

Why? Their questions are not too bad, but they are conversational questions, and should be presented that way. They are the basis of a conversation. Handing someone a sheet of paper (with no room to fill in any response) and asking them for essays is ineffective. It’s not fair. These are the questions they want answers to, but sometimes you have to ask a series of questions to get that information. And you can’t decide ahead of time which questions to ask. You have to ask a question, listen to the response, and then choose your next one. You can’t do that on a piece of paper. You need to have real people talking to each other and exploring the issues that way.

Not to mention that the restaurant has been open for a day or two, and there’s a presumption of an in-depth relationship that hasn’t really been built yet. What do I think of the Bistro Vision? Ummm, I don’t care.

I love what this artifact tells you about the company; that they really want to get a dialog going. They don’t have the tools in place to do it yet. Maybe it’s backed up by the way they interact with customers who come in; I don’t know. But this won’t work at all.

And I think this sort of inquiry is what a lot of design students are doing; identifying some open-ended (i.e., requires the respondent to write sentences) questions and sending them out by email. Some people will respond. Some may even write a lot. But you can’t follow up unless you send out another email. And then it’s just a conversation.

As with everything you “send out” who it gets sent to is a factor. Sending something to 3 friends is a very different approach than something that is quantitative in nature.

Look at this artifact from a recent project (created by our partners, not us):

This contained 31 questions, only a few open-ended ones. There’s randomization where needed (so you can filter out order-effects, where the first or last item might be picked more frequently in a list), and a large enough sample so that results can be processed to lead to conclusions – comparisons between different factors (this is the stats part I’ve been talking about).

Attitude toward technology meets Age
Purchase habits meets Region (with Age)
Stores shopped meets Region (with Age)
etc.

Tons of work and tons of math goes into creating tables (that then get interpreted) like

As Paul Hogan (sorta) said “That’s not a survey, now that’s a survey!”

I hope this brings a bit more clarity to the discussion.

In today’s New York Times

There are “brand aspirationals” (people with low incomes who are obsessed with names like KitchenAid), “price-sensitive affluents” (wealthier shoppers who love deals), and “value-price shoppers” (who like low prices and cannot afford much more).

The new categories are significant because for the first time, Wal-Mart thinks it finally understands not just how people shop at its stores, but why they shop the way they do.

Of course those segment labels are dehumanizing and unpleasant, but the source for this new understanding, years of in-depth studies with customers, must have been some very insightful research. Congrats to friends who I presume were the ones that actually did that research (even though they’ve moved on from Wal-Mart)!

Update: The friends disclaim any credit for this work!
Update2: This leaked PPT presumably explains their methodology.

I’m a believer in the power of authenticity (as well as the greater power of inauthenticity). Fake blogs (aka flogs) were a bit of a scandal recently. Predictions for 2007 are focusing on this as well.

Companies not acting in an authentic and honest way will be subject to the wrath of the newfound consumer voice.

The buzzword for 2007 will be authenticity and it will become a driving force for businesses.

But what is authentic? What is inauthentic? The eye of the beholder makes the final determination.

Here’s a story: we were doing in-home interviews last month, getting reactions to product packages in order to understand the important visual elements and cues (in order to inform an upcoming redesign). One package had a label with a jagged edge, meant to suggest torn paper. It didn’t look like real torn paper, it looked like a manufactured torn edge. Some people really liked it, but one person called it as unacceptably fake. He pointed to another packaging label that he had purchased, as this one had a more realistic-looking torn edge, where the paper was frayed and small threads and fibers were visible.

He was very clear that both of these edge treatments were done by machine; that no paper was torn by hand. The vernacular of the jagged paper was completely unacceptable. The more realistic (as he imagined it) frayed edge was the right way to do it.

It’s a bit of a post-modern take on authenticity, where it’s more of aesthetic that supports suspension-of-disbelief, rather than some extremely absolutely True and Real version. What does the way it looks let me comfortably accept into my reality?

How do you know what is authentic? How do you know that what you are creating or selling is authentic?

This past Friday was, well, User Research Friday.

Here’s the obligatory shots of backs of heads and a person and a slide. Comments on the whole thing follow the pictures.

I am so appreciative of all the work put in by the folks that organized User Research Friday; the constraints of the (un)conference problem are pretty extreme and they struck the best balance they could, given the effort put in (i.e., it was free, and all done by volunteers, and that’s appropriately going to limit what is being created; this isn’t TED). I’m looking forward to the next one, too.

It continues to be amazing what people can do in terms of throwing together an event with little budget, planning, advance notice, etc. And what goes with these unconventional events is a rethinking of the purpose of such a gathering.

We (and this is the collective we, as participants and organizers of events) are still not there yet; I haven’t seen one of these work to its potential (although the effort/payoff ratio is much better than a big expensive event, too, so part of the problem is common across events in general more than the specific approach; I’m more likely to (constructively) critical because these events are at least trying to rethink the approach). There’s a tension between the different goals that people come to these things with, and the way the event is configured to address those needs: content, discussion, and networking being the biggest ones I can suss out.

The content here was so-so. One presentation was a bald-ass sales pitch, complete with a pre-emptive slide for anyone who might disagree with the value of what was being sold, referred to as “that guy” – no one would want to be “that guy” would they? The ones that always asks those (eye roll) questions? Sheesh. Great to address the FAQs that come up, but no need to be such a dick about it. At least one talk went entirely over my head. Others shared some case studies in an informative and direct fashion.

Sadly much of the content dealt with workarounds for the constraints of business today. No time to go see customers, who are too far away and may be in a situation where we can’t go see them at the time of most relevance. Can we get someone else to go see them? Or can we put a piece of technology in place that can intermediate? In general, these are good solutions to real problems, but I fear I’m watching the field drift into a spot I’m not so crazy about. I realize this reflects the Bay Area/Silicon Valley thing and had I attended EPIC or AWF I wouldn’t be struck by the contrast. Any of our local events that are self-generated in terms of content suffer the same techno-drift (see DCamp, etc.).

Only one presentation was designed to elicit some sort of dialog (not that others presenters should have taken that approach; the format didn’t really support it).

The event offered little in terms of discussion. There are tons of people in the room, so as many questions as possible in the short session length were taken. But any large-number-of-participants event will rarely build into any new conclusion, it’s merely clarification after comment after clarification. There was to be a panel session (and I was asked to be a panelist) but the organizers decided to cut it. I am not sure why. Time? Lack of focus for a topic? Too much content? Of course, I wanted my fifteen minutes, so I felt bad and my perspective on the value of the panel is filtered strongly by my desire to have been involved in the panel. Breaks, if any, between presentations were brief and some folks no doubt were reviewing content with each other, while others were just chatting, queuing, and drinking (yeah, there was free wine and beer and eventually champagne)!

The networking was crammed into that time as well. I enjoyed having a printout of the signups ahead of time so (as an introvert, I guess) I could plan for who I could see that I knew; as well as having other brief chances to meet others.

The post-presentations networking (where more food and booze came out) was a bit disappointing; it was Friday so people left to go to their other lives fairly quickly. I imagined it running later than advertised, but it petered out earlier, so I was a bit bummed on that front.

I was struck by how much focus and interest there was on the presentations themselves; I pictured more hallway chatting going on adjacent to the talks, but we all gravitated towards the talks.

I’ve not been involved in anything more salon-like; smaller, more focused, with some intention to produce some result by the end. I’m not sure I’m ready to organize something myself, but I’m definitely interested in that, as a contrasting experience.

Charles Frith emailed a question that he hoped I could address here

Is ethnography more suited to shedding light and depth on peoples relationships with some products or services over others? Are there times when it’s completely unnecessary?

That’s a two-parter! I’ll take a shot at this all, but I hope that others will jump in to clarify, correct, disagree, or whatever.

Are there times when ethnography is completely unnecessary?! Absolutely. But your second part is about time and your first part is about category. In terms of time, in terms of the process of developing a product or service, there are different types of research that may be more or less appropriate at different points along that process. There’s no overall answer, for me, it always depends.

For example, who is the organization? What do they know about the category, or the customer, or the channel, or the market, at this point? Is it a new product in a new category, or a redesign of something they’ve been doing before that they are improving? These are all facets of “what do you need to know more about?” I guess.

The best times to do any sort of customer research (and sorry if this is obvious) is when you can take some action with the results. If there are decisions to be made and they need informing by a user/customer perspective, then try to do that research ahead of time.

Depending on how you define ethnography, you may wish to see it only as a discovery process, as something that happens early on to define needs so that you can create solutions to match those needs. But putting solutions – prototypes – back into that conversation as a way to further validate the needs or dive deeper into them is a great way to go. You can read a case study of ours here that relates how an early prototype was taken into homes in order to have a more meaningful discussion about what the needs were and how this product could address those needs. You can always hold back your artifact for some portion of your session; that’s a standard technique we use.

As far as categories of products and services go, I don’t see that as a big dividing line in terms of the usefulness of research (and I’m speaking about research in general, whatever methodology you want to consider, some way of bringing customer perspectives into your design and decision process). I’d point again to the organization and its process as the crucial gating factors.

And

Also, are there precedents for ethnographic research in Asia?

Wow – two opportunities to cite DUX conference papers in one blog post. This PDF

uses research in Japan (conducted by us, and by Adobe, over several years) as a template for considering research in global locations.

I think there’s two flavors here: Global companies coming to Asia to understand these markets, and Asian companies doing for their own domestic markets something like ethnography. For the first, absolutely. Japan, China, and India have all been high-profile destinations for user research (and in case anyone wonders, we are very keen to go back and do some more). For the second, I hear more about India, specifically, with an increasing drive by Indian companies to understand how their products and services will be received. A colleague in India does a regular audit of teenager attitudes for MTV India. I don’t know in much more detail the market for this sort of thing in any part of Asia, only the different anecdotes I’ve collected from various Asian colleagues.

Should I also re-plug CultureVenture at this point? For companies looking to get a handle on these other markets, this is our service to drive inspiration through immersion. Facilitated hanging out, if you will?

How’s that Charles? I hope you’ll add some follow up questions in the comments.