How to Grow and Thrive as A User Researcher

Check out an interview with me in How to Grow and Thrive as A User Researcher on the Adobe blog. An excerpt is below

What are the benefits of sharing career failures and mishaps rather than just successes?

We need to share both! Thinking about the field of user research, it’s important for practitioners to continue our development. Examining what went wrong (or what was different from what we expected) can highlight practices that might have avoided any particular mishap. But user research is so much about people and all their quirks, personalities, strengths, flaws, emotions, and so on — it’s what the work is about! There are inevitably surprises, and failures, and so another way to think about improving our skills is in accepting the lack of control, and even embracing it.

Researchers are often ‘selling’ the benefits of the practice to colleagues and stakeholders, and while I’m probably not going to lead with failure stories, it’s helpful to have a framework for considering them. ‘Failures’ are inevitable and while we work hard to prevent them, they are still coming for us, and reframing them as part of the messy people experience that we’re out there to embrace can help us discuss more realistically with our collaborators. There’s no reason any of us should feel alone with these experiences; as Doorbells, Danger, and Dead Batteries illustrates, they are part of this work.

Some well-known and very successful designers don’t do any user research. What do you think about that approach?

Let’s assume that’s true. It’s foolish to declare that the only way to innovate is through research. Even when we do research, there are many other factors at play that determine success. What does concern me is that approaches championed by ‘well-known and very successful’ individuals won’t succeed for everyone. It’s real swell that Steve Jobs (or substitute your favorite successful innovator) did it this way. But you’re not Steve Jobs!

Special thanks to Oliver Lindberg for the interview!

Snark about Bark

I’m rolling my eyes at this article about successful dog toy company, Bark. Specifically:

Bark’s design process begins with research. Packed in every BarkBox sent to 600,000 subscribers is a survey questioning dog parents about their beloved canine’s playing styles. The design team also gets anecdotal data from their Ohio-based customer service team who chat with with BarkBox loyalists about how toys were received. Based on user insights, Jensen and his team creates goofy toys that heighten a dog’s natural play style—chewing, fetching, or even destruction play.

“We know what a golden retriever in Kansas will like compared to a chihuahua in Seattle,” claims Jensen.

Some of what is grating is simply due to sloppy writing, but I am bothered by the hollow virtue-signalling around user research. Their methods are surveys – sent only to people who have purchased their product, and customer service reps – who are probably doing more troubleshooting than chatting with loyalists. The quote, then, implies a Big Data-style sense of insight across geography and breed, which is just untrue. They haven’t met any pet owners and they haven’t met with any pets!

I realize it’s expedient for PR to anchor your genius in customer-centricity, and I guess that’s a win. The founders are industrial designers and there would once have been a day when their innate brilliance would have been sufficient. But really, don’t cloak yourself in shallow methodologies and then claim you are doing everything based on research!

Customer Satisfaction at the Push of a Button


Image from HappyOrNot.com homepage

The New Yorker profiles HappyOrNot, the company that makes those kiosks to measure “satisfaction” (you may have seen them in airports or malls, they have a few colored buttons with different happy/unhappy faces). The article explains the history of the company and gives examples of organizations who are using the technology, what kind of data they are getting, and how they are acting on it.

On a large monitor on a wall of Javaid’s office, Jochym showed us several ways she’d devised to represent HappyOrNot data graphically, for presentation to other members of the organization. One was a stadium seating map on which the HappyOrNot terminals were identified by numbered, colored circles. “You can see all the terminals here, and you can move through the data hour by hour. The colors change as the ratings do.” The most consistently high-rated performer—even during the two most problematic periods for customer service, halftime and the fourth quarter—was a new vender, which, unlike most other venders, used the same, experienced work crew at every game.

How Louis Theroux asks questions

From 2016, here’s a great video by Ryan Hollinger analyzing how documentary filmmaker asks questions, builds rapport, uncovers information. This video doesn’t get into one of my personal favorites of Louis’s superpowers – that he ignores negative energy – discomfort, anger, superiority – from his interviewees. Rather than fleeing, he stays calm and continues to engage.

(update: video link corrected)

Listen to Steve on Helping Sells Radio

I had a really enjoyable conversation with Sarah E. Brown and Bill Cushard for their Helping Sells Radio podcast.

The episode is posted here and embedded below

Research is sometimes about not doing anything at all

“Sometimes the best thing you can do in user research is to sit on your hands,” Portigal said. For example when talking to someone and they use a product feature in a different way than you intended or mispronounce the name of your product. “Just let them talk, don’t correct them,” he advised.

By sitting back and letting the interview subject take control of the language they use and how they explain things, you’re giving them the power in the session. The power in any interview tends to lie with the interviewer, so by sitting back and letting them speak however they like, you’re giving it back to them.

The bonus lesson here is that interviewers are getting a first-hand look at the customer perspective and they can use that information to shift their own mindset. It can be hard figuring out what customers are thinking about or what they’re looking to get out of your products, so any interview you do with them helps with that. You are so invested in your product and so focused on it that it can be hard to step away and look at it from another view point. User research interview sessions give you that opportunity.

Highlights from my Ask Me Anything with What Users Do

I did an Ask Ask Me Anything with What Users Do recently. Here’s the recap they put together.

Any tips for interviewing users who work with highly confidential information? Any strategies for incentivising such an audience to be open about which problems they struggle with and how they use our software, when almost everything they do is supposed to be a secret? [Timi Olotu]

I think it’s always good to set expectations clearly when arranging for interviews; perhaps in that situation there might be concern – before even agreeing – about risk. I worked with a bank and they had a standard set of text they used in recruiting bank customers, along the lines of “We won’t ever ask for your bank balance or any information about accounts.” Perhaps that was required for regulatory compliance anyway but it served to be very clear about what we were NOT going to be asking about.

I think managing expectations is more important than incentive; but gaining access is also about understanding their dynamic or their relationship to you. Like, if they use your software, then they have a chance to give input and feedback.

I’d also add, I wonder what kind of anonymized experiences you can create, and I don’t mean that to be so fancy sounding, but how much of the interview could you do on paper, with wireframes.

I don’t mean give them a usability test, but can you give them a set of high-level scenarios and have them pick some and choose which to walk through using a sort of simulated or high level version of your experience.

You wouldn’t want to do that for the whole thing, but it could be a component of an interview.

When generating questions for interviews, how do you reduce bias from pre-existing hypotheses? [Amy]

I think bias is obviously a concern but it comes up so often as the main thing people are concerned about. This is totally biased work! We are humans who are the product of our experiences and meeting other people and exchanging the slipperiest of substances: words! How can we not have bias?

I think that having hypotheses is a great thing when going into research. I mean, you put it exactly right – hypothesis. That’s not a closely-held belief, or an aspiration, it’s an idea of what you think might be happening. That sounds like what we’re supposed to be doing.

If there is a belief or a hope or an expectation – be it implicit or explicit – that seems like something we want to get to in research.

Of course there’s a ludicrous way to do that. “Don’t you agree that it’s better now that we have this feature located in this part of the UI?”

That’s a biased form of inquiry, that’s almost abusive of your power. But trying to understand someone’s framework, expectations, preferences, experiences, mental model, etc. from an open and curious point of view, and having that curiosity informed by what you have been already considering about the problem space, sounds like good research to me.

You writings and talks have been a big influence on many folks working in the area of UX – but who are your influences and inspiration, and why? [Rob Whiting]

Our field is packed solid with great people. I keep thinking about some of the over achievers I get to meet and how accomplished they are.

I love Jess McMullin, he is one of the first people to start doing civic design, like YEARS ago, before it became such a big thing in so many parts of the world.

I am a big fan of Allan Chochinov (he wrote the foreword to Doorbells, Danger and Dead Batteries). He started a graduate program in design at SVA in NYC, called Products of Design that kind of takes a big picture look at what design can do. Mentor, friend, inspires me.

I think Kevin Hoffman is really inspiring. Funny, passionate about pop culture and knows everything, cares about people. His book about meetings is going to come out soon (I can say I saw the cover and it looks really cool).

It’s horrible to try and pick people as I’m using recency to come up with a list.

I’m trying to get user testing ingrained into my company’s process but it’s a struggle. It always seem to be the first element dropped when budgets are tight. Any advice on keeping it a part of the process? [Mike Mellor]

I think you’ve got the #1 FAQ about research… that it isn’t supported. I don’t think there’s a stock answer though. But you might wonder – or seek to understand – why is it being dropped? And why was it even being proposed or considered in the first place?

I mean, it’s one thing to say there’s no budget or time… but if someone chose to put it in to begin with, there was a narrative about its value.

If one could understand that better, one could propose an alternative or advocate, with a bit more information. I’m sure there’s some Rhetorical Studies model here I’m not expressing well, but understanding your adversary’s objections seems like one possible persuasive technique.

I’ve long said don’t advocate for the process “we have to do research” but for the outcomes “we have to make sure we understand this issue or this consequence will happen.”

IU059: Figure 9.4

IU060: Figure 9.5

If you want to get into a discussion of timing, you can use the above diagrams as inspiration – lets say the top one is sort of my gold standard, here’s what it takes to do it “all” – but if you want to do it more quickly, here’s how it’s going to look – it may be more or less appropriate but that way you can have a discussion about tradeoffs.

If we only have a day to find participants, for example, then we can’t be too picky, we can’t go beyond who we know right now at this moment. Maybe that’s sufficient.

So even though your question was about doing it – or not doing it – I think looking at ranges of commitments – where zero is in that range – and encouraging reflection on trade-offs – could be good. It’s not about what YOU need, it’s about what the work requires. So don’t take it on yourself. “I’m not allowed” “they won’t let me” – it’s about us, about our shared goal and your expert advice about how to reach that goal.

I’m wondering what strategies and methods you use to analyse data from your interviews. Would you recap after each interview and write down your observations when they are still fresh and then wait for all of them to be over to listen to transcripts? [Edyta Niemyjska]

You describe my preference pretty well. I separate the “processing the experience” and “processing the data.”

After an interview, I might do a debrief worksheet – but typically not. But at the end of each day, I write up a VERY quick paragraph or two about the interviews we’ve done.

It’s meant to be a storytelling exercise, it’s a forced analysis (take a large thing and pull out some smaller bits) – and it shares the fieldwork with the rest of the team. Here is a PERSON, they have a NAME, they own a THING, they told us an EXPERIENCE. So it helps me make a first pass at distilling and it gets people to think about these real actual people really quickly.

It’s not field notes.

I try to do it in just a few minutes, and do it stream of consciousness.

When we’re done with fieldwork, I like to sort of collate, very quickly pull together a topline – here’s what we think we’re hearing., what did you all think, what did you all hear. We started with questions and we have some thoughts, we have some weak signals, we have some things we’re excited about. Nothing about what to DO with this info, just where we’re at, at the moment.

And then, finally, let’s dig into the transcripts and see what actually happened.

What is the most effective way to ask simple questions to better understand where our users are coming from? Often, the users are ill at ease and want to “help” or are simply biased so they muddle the actual answer. [Karunakar Rayker]

Part of your question is about building rapport. People are often ill at ease at the beginning of a session. They want to do a good job and they don’t really understand what is going on, I mean not to be patronizing, they understand, but they don’t “get” what this exchange is meant to cover.

It’s one reason why super short interviews are challenging because it’s hard to get to a point in the relationship where you have established a smooth dialogue, where the person is not only comfortable but excited, reflective. That takes time, sometimes a huge amount of time – and people are unique, and the way we find a connection is unique to the combination of them and us.

The ways we have to contact with people ahead of time – before the interview – can be rapport builders. Maybe we have a quick phone call and let them ask any questions. Maybe we give them an exercise so we can see something about them. Exercises also prime people – it engages them in thinking about the topic so when we meet they aren’t coming up to it raw and fresh and new, it gets them involved.

Sometimes we make sure in our recruiting process that we screen out people who aren’t already meeting a certain comfort level – “the articulation screen” – if the person can’t answer a question from the recruiter (tell us about a recent experience you had going to the movies) for a few sentences, then they may not be the best participant for the study.

But assuming they are “articulate” – they may not be comfortable. So our job is to keep listening, to keep affirming. I do NOT mean “okay great! Cool! Wow!” etc. I mean listening, I mean, asking follow up questions, expressing interest, validating that their point of view is important because you give it time – that’s a harder way to validate the person because you want to do MORE, but when you do that enthusiastic thing you are actually pushing them to perform for you.

Finally, when you have an uncomfortable person YOU feel uncomfortable, you are sensitive to the cues that this person is feeling weird and I should probably do something different or leave. What if you could ignore those cues – which are about YOUR feelings? And just keep listening and focusing on them?

Can you share examples of what type of exercises you have the user complete when connecting with them prior to an interview? [Anne Jackson]

Come up with something that seems relevant, and so many different ways to go about it. But an approach can be ask them to take a picture of two different things, and send the pictures along with an explanation. Two, because it’s about examples of contrast.

Send us a picture of something in your neighbourhood that you think adds value to the experience people who live there have. And explain why. And send us a picture of something that detracts from the experience.

Send us something you’ve organized well. Send us something you wish was more organized.

These are kind of digging into the theme you suspect the session will get into.

The interview kicks off by getting them to tell you that story again!

It could be fill out a form and give a couple of examples, but the photo stuff can be fun. A screen shot, even.

I am currently in the process of introducing a lot of PMs and Engineers to customer interviews. We are also training a few younger designers to talk to their target users.

What are some basic strategies and tips to keep in mind when introducing non UX researchers to UX research? [Nachi Ramanujam]

I write – well, scribble – on the paper. I might draw a big circle around the quote that is interesting and then write my own thoughts, “Why does she do this” or “they don’t have alignment between their goals and their choices” etc.

This formative UX study is a bit more tactical but might be really helpful – it’s so well explained (not specifically transcripts but at least about analysis in general – I think less about synthesis – where we take small bits and put them into new ideas and frameworks – which is what I think we do with the big mess of annotations I’m producing).

My consulting guide to fieldwork is a one or two pager that is meant to help people do well when they are joining in (NOT LEADING) user research interviews. It is the most boiled down set of points I have. I think it’s like anything, the more you put in, the more you get out.

Here’s a 40 minute presentation that is about doing research. Do they have 40 minutes? http://portigal.com/speaking/ has a bunch of links to past talks so you can see where there are videos and slide decks.

I also do a workshop where I ask people to interview each other and then reflect on what worked and what did not work. Practice – in a safe place – not on a work problem but on a practice problem.

Listen to Steve on the Aurelius podcast

I had a great chat with Zack Naylor about user research for the Aurelius podcast. We talked about

  • Why you should be doing [more] user research
  • How to convince your stakeholders that user research is important
  • 3 approaches to building brilliant products and features (and which one is best)
  • Convincing your stakeholders and leaders to do (more) user research
  • What is a user research process to make sure you’re learning the right things
  • The difference between research analysis and research synthesis

It’s posted here and embedded below

Alexandra’s War Story: When One Door Closes

Alexandra Wills is an ethnographer working at Fuse by Cardinal Health, an innovation center in Columbus, Ohio. She told this story on stage at Midwest UX 2017.

I’ll never forget when I did ethnographic research for a project aimed at helping a car manufacturer learn what Millennials with small children really needed.

The project was hard. Taking on a project at the height of the Great Recession meant navigating a radical change in client engagement from what I had experienced since starting the work two years prior. “It’s Friday at 5 p.m. in Ohio and you want me in Los Angeles on Monday?” Okay. “We’re doing video diaries and in-home interviews and a post-interview ideation session with participants in two cities, all in two months?” Okay.

Added to all that, I had a nine-month-old and simply didn’t want to leave her for days at a time. Over the past few months of work, I had already breast pumped on an airplane and in dirty airport bathrooms. I had already begged flight attendants and fast food workers for ice to put in the cooler carrying pouches of my “liquid gold.” Did I mention it was my birthday?

At one point in the project, I was hanging out with a family in Austin who had a toddler. I knew nothing about toddlers. After all, I had a nine-month-old. Did I mention I am not a ‘kid person’?

We had just returned from running errands in their car. As we got out of the car, they were showing me some specific details about the vehicle. They had a Honda Element – the car with the interesting doors that open and close like a book. I was paying close, close attention to the parents and I had no idea that the little kid was right near me. So I closed the door. Suddenly, we all heard the kid screaming! His parents rushed to his side and looked him over, examining his hands. All I could do was yell impulsively, “I didn’t do it!” I was horrified. I thought, “I hurt a child! This child! A participant’s child! Oh noooooo this is bad. How am I going to fix this? What am I going to tell Melinda (my boss)?” To this day I don’t know if his finger got caught in the door, or if me closing the door just scared him.

There was no blood, no broken fingers. But inside, I wanted to die. I already felt plagued by my own mommy guilt and that feeling spread throughout my body like lava. So, not only did I feel like a horrible mom for leaving my kid, but here I was in Austin, making someone else’s kid cry. What a moment. Needless to say, any rapport I had developed in my time with the family evaporated in that instant.

I stopped recording, stepped back, apologized to the mom and waited for the parents to finish calming down their kid. I waited for them to say, “This is over.” They didn’t. Miraculously, they continued the interview, even if I could feel all their judgment the entire time as we wrapped things up. “Maybe I didn’t traumatize this family,” I thought insecurely.

The icing on the cake was that we used video to capture all our data, so not only did this happen, but my boss got to see the whole thing when she reviewed the video. Later in the project I mentioned the incident and she said, “Yeah I saw that.”

Series

About Steve