Podcast: Download

This episode of Dollars to Donuts features my interview with Vanessa Whatley, UX Director – Research & Documentation at Twilio.

For many years, I had anxiety and regret around not starting my career in the field that I’m in sooner because I felt very very lost stumbling through all of the different fields and roles, and only in hindsight do the dots connect. I’m better at what I do now because I learned the lessons in all of the different jobs. Even something like being an executive assistant, I was able to sit in on more senior leadership meetings, and I really early picked up on short attention span, How do you get your point across concisely, What do they care about? And I think that made me a better researcher right away, even as I was still learning the practice because it taught me something about communication…I think all of those little pieces along the way just shaped how I interact with people and I think has made me better at what I do today. Maybe just know that it’s all connected somehow. – Vanessa Whatley

Show Links

• Interviewing Users Workshop at Advancing Research 2024
• Interviewing Users,
  2nd Edition
• Steve on Darren Hood’s World of UX
• Vanessa on LinkedIn
• Twilio
• Segment

Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.

Transcript

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. As part of the upcoming Advancing Research Conference, I’m teaching a full day in-person workshop about user research. It’s March 27th, 2024 in New York City. This is a rare opportunity to learn about interviewing users from me in person. You’ll also have the chance to engage with other researchers at different levels of experience in an interactive environment. I’ll put the link in the show notes for the Advancing Research Conference with more info and information about how to register. If you know someone who would benefit from this workshop, then please pass this info along.

The newest version of my workshop makes use of the writing and rewriting I did on the very recent second edition of Interviewing Users, which you should absolutely buy several copies of.

Shortly after the book came out, I had a conversation with Darren Hood for his World of UX podcast. We got into the intricacies of asking questions. I’ll link to the whole episode, but I’m going to include an excerpt right here.

Darren Hood: And the topic of chapter six, the title is the intricacies of asking questions. And I love this because this is probably when I’m teaching people about research, it’s one of the outside of the classroom. When I’m talking to people about research, this is the topic for me that comes up the most. This one particular thing that you mentioned, and I’m going to read another excerpt from the book. And there’s a heading here, “There’s Power in Your Silence.” Oh my God, how many times have I talked to people about this? Steve says, “After you ask a question, be silent. This is tricky because you are speaking with someone you’ve never spoken to before. You’re learning about their conversational rhythm, how receptive they are to your questions, and what cues they give when thinking about an answer. These tiny moments from part of a second to several seconds are nerve-wracking.” And I love that because it’s one of the things that I see and people will ask a question. And it’s funny to watch people grind their teeth when the participant is silent. To watch people, hem and haw, the researcher, wants to help the participant and things get out of hand sometimes. I have seen people jump practically across the table to try to guide somebody because they just couldn’t stand the silence. But the title says it, the subheading there or the heading in the chapter, “There’s Power in Your Silence.” And so I’ll hand it over to you to elaborate on this topic.

Steve: Yeah, and I think you described some of the phenomenon pretty well. And there was a moment in this conversation, I think you, because we’re on video, even though we’re recording audio, we are in video and we’re looking at each other and nodding and doing all the, as best we can over video kind of feedback. There’s a point in which you said, “Oh, I see the gears in your head are turning, Steve, and I’m going to turn it over to you.” And I think, as a trained interviewer, as an experienced podcast, I was like, “You learn what that is.” And there’s been moments where I’m asking a question and I’ll just stop. I don’t need to finish my question. The person is ready to talk. And I’m going to ask a follow-up question. I’m going to ask dozens of follow-up questions.

So if the thing that they want to say is not exactly what I want to ask them about, it’s better for the whole dynamic to have them go and then me follow on and follow on and follow on as opposed to like, “No, no, no, wait, let me make sure you understand exactly what my question is so the information that you give me perfectly conforms to the parameters which I am articulating.” Like that’s not how research works. It’s this sloppy interface between people that kind of goes back and forth. And so you have to understand your role is to kind of draw that out of them. And so that hem and haw thing, or even trying to help them, as you say, I think is really important because if you can’t allow for that silence, the anti-pattern or the bad behavior that comes out is asking these run-on questions. And the run-on questions are deadly. And in those run-on questions, people start suggesting, so you’re going to ask what could be an open-ended question. Like what kind of microphone do you have for your video calls? But the run-on question is what kind of microphone do you have? Is that a USB mic or is that a Shure microphone or is there that part of your headset? Like you start suggesting possible answers.

Darren: Yep, yep.

Steve: And the motivation for doing that, I think there’s a lot of, you have to pay attention to yourself. It feels like you can kid yourself that you’re being helpful when you do that. I’m being helpful. I’m just showing them what examples are. But in fact, it’s because you are, and I shouldn’t say you, I should say we, this is, I’m in this all the time. It’s uncomfortable to stop and just say, what kind of microphone is that? For all the reasons in that quote that you described, like, I don’t know what’s going to happen. I’m going to lose space. I’m going to be seen as an idiot. That person’s not tracking with me. My boss is going to watch this video. There’s all this risk in that moment. It’s kind of like a little, it’s a little abyss that you’re kind of beeping into. But if you start suggesting things, it messes up the power dynamic. It says that for one thing, the participant is required to listen to the, to the interviewer going on and on and on.

Darren: Right.

Steve: And also starts to say over time that their answers should be, it’s multiple choice question. So you participant should be giving answers within the format that I have outlined. So you might think that’s ridiculous, Steve and Darren. If it’s none of those mics, the person’s just going to say, no, this is just an old karaoke mic that I brought up from the basement. They’re going to give you an answer that’s outside that list. And the first time they will, and maybe the second time they will, but eventually you are training them as to how to do a good job, which they want to do. They want to do a good job for you. And so they’re not being squelched from sharing their truth about microphones. They’re just trying to get through this interview and do a good job.

So the more you teach them indirectly what a good job looks like, in other words, one of the following, then the more you risk not hearing from them and not kind of getting stuff. And you don’t, we don’t realize the power we have over people despite being kind and self-deprecating and telling them at the beginning, I just want to hear from you. Just tell me your truth. And, you know, there’s no wrong answers. You can do all that. It doesn’t matter once you start training them what good looks like, then that’s what you’re going to get. So I think it’s, it’s hard. And the quote you read kind of explains why it’s hard and we trick ourselves that we’re helping. So that makes it harder. The risk in this, I think is significant because it accrues to changing the dynamic and the interview and changing what it is you’re going to hear.

Darren: Yes, yes, absolutely.

Steve: Again, that was Darren Hood’s World of UX podcast. Now let’s get to my conversation with Vanessa Whatley. She’s the UX director of research and documentation at Twilio. Vanessa thanks so much for being on Dollars to Donuts.

Vanessa Whatley: Thank you for having me.

Steve: It’s really great to have you here. So I’m going to just do my cliched opening. The only thing that’s really yet effective, I think, is just to throw it over to you right: away and ask you to give a bit of an introduction to yourself.

Vanessa: Sure. My name is Vanessa Whatley. I lead our research service design documentation team at Twilio. So my title is UX director, but I have all the UX functions outside of design is how I like to explain it.

Steve: And what’s Twilio for those of us like me that don’t know?

Vanessa: Yeah, great question. Usually people think it’s Trulia, like the real estate company. So we’re not that. Twilio is a communications API company, and specifically the portion that my team works on is second So that was a product that was acquired by Twilio a few years back, and we are essentially a customer data platform.

Steve: What companies or people or roles use Segment and what do they do with it?

Vanessa: Yeah, we are B2B, so lots of large as well small SMB companies use Segment to essentially get a better picture of their data. So a lot of times companies are collecting data in a lot of different tools, a lot of different systems, and then it’s siloed and they have a hard time reconciling. So if you are, you know, Steve on the mobile app and Steve on the website, and then you’re interacting by replying to an email, you might look like three different people. And so Segment really helps bring all of the data sources together, unify it into a single profile, and then from there it lets companies interact with you in a, I guess, a more intelligent way because they know who you are as a single person rather than three different IDs.

Steve: Can you briefly put Segment in the context of sort of the larger set of things that Twilio as a company does?

Vanessa: Yeah, so if you think about Segment as kind of the data layer, so that can be your foundation of you understand who your customer is really well, what they’ve done with you in the past, or even predict what they might do with you in the future. And then from the data piece, a lot of companies want to actually activate on that data. So they might want to send you a text message or send you an email, and so the rest of the Twilio portfolio kind of has more of the communications APIs so that you can go on ahead and like use that data to actually engage your customer. Another good customer example could be someone like a DoorDash where DoorDash is trying to connect a restaurant, a driver, and the person who ordered the food. Instead of them building everything from scratch in their app, the communication piece of me being able to text a driver without having to give my own number and the driver seeing my number and vice versa, you can use an API so that they can communicate without DoorDash as an app having to build all of that native into their platform.

Steve: But so if you’re a company like Twilio and just thinking about, you know, user research point of view, you’ve got those kinds of users that you’re describing in like that DoorDash example, the end user and the driver, the food purchaser and the driver. But you also have the — I guess it’s some kind of IT or development team that’s using Twilio tools to build that so that their end users or their drivers can all communicate. Are you — from a research point of view, where are you focused, if at all, on any of that?

Vanessa: Yeah, so on the Segment side, like I said, we’re more so the data piece that is powering a lot of different things. And so you’re absolutely correct. A lot of our customers end up actually being either the data team or engineering team within a company because they’re the ones that are essentially most likely collecting the data, manipulating the data so that there’s protocols and that it’s actionable.

And then ideally it goes all the way through to a business use case. So that might be a product manager or marketer then making decisions on that data set and deciding, okay, we want to run a marketing campaign or we want to analyze this cohort or this audience.

Steve: I want to go back to the beginning and you were describing sort of the structure a little bit or the areas of the organization that you’re focused on. And I’m sorry, you had a great catchphrase, which I should have written down because now I’ve forgotten it. Can you go back to that?

Vanessa: Oh, I was saying everything in UX outside of design. Yes.

Steve: Okay.

Vanessa: So there is a different person that manages our full design org, but content design, service design, technical writing, and research all sit within my team.

Steve: Can you say a little bit about the research team?

Vanessa: Yeah, so we currently have a team of about five and the makeup of that team has changed a lot over time. So we’ve grown with layoffs. We unfortunately lost a few people on the team but overall I think as the size of the team changed, our operating model has kind of shifted along with that. So we started out being a little bit more embedded and really aligning each researcher to a specific area or product area features. And then I would say last year we really decided to go all in on more foundational work and take a little bit less of our demand from product to try to answer like larger strategic questions. And now we probably sit somewhere in the middle where we do a mix of product work as well as foundational work.

Steve: I was expecting that embedded and centralized would be the sort of contrasting terms, but as you’re kind of relaying it, I kind of hear you. I think you’re contrasting embedded and foundational. And maybe you could explain what those endpoints look like as you’re kind of moving between them.

Vanessa: Yeah, I think you’re right. I never really thought about the fact that usually people say centralized. I don’t love centralized research orgs. I’ve worked in that manner before, but I’m a little hesitant to call it that with how we’re currently structured because oftentimes, at least my experience, I’m sure there’s many ways to do centralized, but in my experience that often means it’s a little bit more like an intake request type of situation where you act more like an internal consultant almost and you can cover a lot of different breadth.

And what I try to do with my team and why I call it foundational is a lot of times the researcher might have still had a focus area that they are stronger in or just cover an entire flow, like a user journey and sit within that part of the product. Or maybe they specialize in a set of personas, but they’re not necessarily bouncing around to any project that comes up and we’re managing bandwidth that way. It’s a little bit more driven by where we think product strategy is going to go and then trying to still align people to spaces that they can gain deeper knowledge in just because of the complexity of our space too. It’s really, really hard to bounce around and be the expert in the marketing persona, but then also the data engineer. And yeah, I think that’s why I use those two terms even though they’re not actually polar opposites per se.

Steve: So centralized might mean an intake process, which is challenging if that means that anyone gets assigned to anything. I’m kind of steamrolling over the nuance that you were depicting to kind of check and see. Because I feel like when you’re talking, there’s sort of a couple of aspects. One is like what projects are we going to do? And the other is who’s going to do them?

Vanessa: Correct.

Steve: So I don’t know. Are you — does the idea of intake, is that in itself limiting or something that you would try to avoid?

Vanessa: I think it’s a mix. We definitely talk to all of our stakeholders and try to understand what feature level work and even what foundational work do they want us to kind of produce or participate in, collaborate on. But I actually sit down with my team every quarter and we end up doing anywhere from like 90 minutes to two, three hours of brainstorming where I really encourage them. What are the gaps that you see? What are kind of like big strategic questions or areas where you feel like there hasn’t been enough emphasis or we’re not connecting the dots properly because I do think the risk of operating at the feature level means everything’s a little bit more siloed. And as we know, customers experience things in a series of steps or flows or have an entire journey they need to navigate. And so I just try to position my team so that it can really think at that level.

And I find that sometimes when we do more of the intake model from design and from product that they tend to focus on their scope and their area of focus which might more so sit at the feature level than it does cross-product.

Steve: Right. So it’s the proactive versus reactive aspect. So people that ask for help from research that aren’t — that don’t know about research as much as you do or your team does are going to ask for the problems to be solved that they think research can help with. But if your team brainstorms, here’s what we’re seeing, here’s where the gaps are, here’s how we can get ahead of what’s going on, then — so now I think the more we talk, the more I understand kind of how you — why you characterize that as foundational.

That it’s not reactive, feature level. And you haven’t said this, but I feel like when those questions come, sometimes they come late or when you do that intake model, there’s other ways that you could have helped if you were, like you said, reaching out to those stakeholders and talking about what they’re doing and how you can help them. And you’re saying that you’re kind of in the middle now, you’re somewhere in between if embedded in foundational or sort of endpoints, you’re kind of in the middle right now.

Vanessa: Correct. Because I think at the end of the day what I’m trying to balance for is impact. And so there are some, I guess, areas of the product or even feature level things that we know are very critical for us to get done this year. They’re highly complex. They need someone that is thinking in that space day in and day out. And so there are often one or two researchers that are embedded in those spaces. And then there’s broader strategic questions that maybe you’re not getting directly from the PMs.

Maybe that’s coming even from senior leadership where they’re thinking about the landscape and where do we need to go and broader, less scope, less defined questions. And so some of that will be, I guess, covered by us or we’ll just create bandwidth for so that we can operate at those different altitudes.

And then we’ve also kind of launched a bunch of internal programs so that the feature level work that does need to get done can still be supported. So we have office hours. We have rolling research. We encourage designers to do their own research or PMs to do their own research. I know that is often a hot topic in the industry. I think we try our best to make sure all of the things that would benefit from a researcher’s kind of attention actually gets that attention. But then we also try not to gatekeep due to the size of our team. We’re just not able to get to everything.

Steve: I want to follow up on that, but I want to just go back to the more you’re describing. I’m having another reflection, I guess, because you started off saying that you started off in one mode and then you kind of shifted to another and then you made another shift. And, you know, as people try to ask, like, what’s the right model, you know, to hear how over a fairly short period of time, you know, you have — you’ve iterated or evolved and that it makes me think that the answer — there’s so many — there’s so many it depends on, you know, what model to have, and it depends on your company, depends on your team. But also that these are things that change and that there’s no reason to pick one and stick with it, but to adapt as it sounds like you have to changing conditions and right, you know, in another X amount of time, you might go back to fully embedded because the company’s here or your — I guess other factors like your team or other changes in the strategy might lead you to choose a different model.

Vanessa: A hundred percent. I think those are some of the deciding variables. Team size. So at one point the team grew to 12 researchers, which is why we were able to embed because we had enough to go around almost. And then when we reduced in size, that meant, okay, we need to find the highest value areas. And then I think, yeah, company strategy, thinking about whether your product is in a place where it’s more stable or if we’re in a place where we have a lot of pressure to innovate. All of those things matter. And kind of I think I took into consideration on how can we work to still make it sustainable for the researchers too. Because of course we could have played the volume game and tried to crank out three to five projects every quarter.

But I’ve really been emphasizing let’s choose quality over quantity. And we need time for deep work and for thinking, even if that means you’re cutting down to one to two projects for the quarter and you’re spending much more time synthesizing across past work and doing more foundational or longitudinal work. And luckily we’ve been very much supported by our cross-functional partners and leadership because I know for some companies it’s like, no, everything needs to be tested before it ships. There’s many different reasons why you might get blocked from something like that. I think we’ve been able to answer questions and show areas where we should pivot because of how we choose to work and the types of insights and the clarity that that provides
that we’re continuing to see support and we’re not really being forced to play the volume game.

Steve: Is there anything you can point to that helped you establish that footing where there is that support?

Vanessa: I think a project I talk about internally often happened about a year and a half ago where it was my first time to really decide to go rogue a little bit and just grab two people on the team and say this is the area that I think we should investigate and publish research around. And then because I think with research sometimes there’s an ask or there’s like a push model. Like no one asked for this, but we’re telling them anyways. And so I think about a year and a half ago was the first time I did that for a larger scale project and we just found different avenues to communicate the information. Essentially the more we got it in front of the relevant stakeholders and leadership I think the more the problem became clear and people were bought in and then over time we also saw in one particular area like oh the quantitative data starting to support that story too because it was a newly launched product. And so we were almost ahead of the trend in being able to point out here’s some of the challenges we’re going to encounter. And I think just having a few case studies like that helped us prove the value and kind of earn the respect, get the traction for having that flexibility. Of course that could have backfired. It could have not been received well, but I think in that particular instance that to me was almost my personal proof point for we should keep going down this path and like helping also me gain confidence that this is the right path to take the team down.

Steve: The path in this case is referring to what kind of work and how you’re helping the company.

Vanessa: Exactly. The path is essentially choosing our own topics to go investigate even if no one’s asking for the work. And I mean there’s still ways like we don’t go away for three to five months and then come back and like ta-da we have something cool to show you. There’s still ways to gain buy-in along the way so we are doing our due diligence by crafting a proposal, shopping it around, seeing who at the company could be a good stakeholder to actually implement some of the changes that we’re suggesting. So I’m not saying just go rogue and hope it works out, but I guess it’s more so again the proactive versus reactive model like really taking ownership and saying we actually have things that we think are really important to go after and then advocating for that and pursuing it.

Steve: And in that first example, the first Rogue project a year and a half ago, I heard you talk about, you know, finding this area and choosing to spend resources and people’s time to go do it, but then you, I think also are highlighting, communicating that to some group of people to kind of highlight it. Yeah. Is there a way to sort of compare the proportion of effort in the — and I don’t know if it cleanly breaks this way, like doing that research and communicating that research?

Vanessa: Yeah, I don’t know if there’s a split. I will say for research that other people ask for, the effort and energy to advocate for it is way less. Like if you’re being pulled into something then people are organically just going to show up more, be interested in the findings. So I think there’s a lot more effort and design up front that goes into really making sure that what you do learn is strategically still a place that we can go. Because I think that’s the other thing you want to make sure of, right, is you don’t want to pursue a project and either everyone already knows the information or people didn’t know the information but were already locked into whatever plan when it comes to the product.

So I think the up-front work is really important and takes a lot more energy as well as the communication of findings because now you have to create your own forums and audience whereas other work just is very organic. Like yeah, you go to the product manager who owns this feature.

Steve: I’m hearing in your answer, and now the second time you’ve kind of explained this, that my question was flawed. My question was kind of about research and then communication, but you’re really emphasizing, even when it’s a study or especially when it’s a study of your own sort of discovery or advocacy, there’s that upfront due diligence. There’s doing the research and there’s the communicating. But you’re doing all those three pieces and I guess just to repeat what you’re saying, when it’s a rogue-style project, the upfront and the after part are significantly more effort.

Vanessa: Yeah, I would say so. And I’ve encouraged my team to do this across any project but I think we also do more work to design the artifact ahead of time to think about how this information could be presented or even just have something tangible for someone to react to before we kind of double down and we’re like, again, for foundational work it might be a much higher end than like testing a little feature and so we don’t want to be 15 plus interviews in and realize like, oh, this is not going to work out or people already know this.

Steve: So what kinds of things — you’re talking about what kind of output you create as a result of the research? What kinds of things might your team create as output?

Vanessa: Correct. Yeah, I think a lot of the traditional artifacts, so we do create a lot of decks. I think beyond decks we work in Figma a lot so we try to prototype different styles of outputs. So sometimes it might get really visual and we’re trying to bring in more graphs and charts or if we’re doing persona work, designing templates and stuff ahead of time to think about what data do we want to collect along the way. I mentioned we also have a service designer on the team so sometimes that is like a full-blown journey map where we’re bringing in all of the layers. We’re bringing in the product touchpoint and like the external guides and people like touchpoint when they’re talking to a salesperson or AE, account executive.

So I think we try to remain pretty open, sometimes try to get creative in terms of, you know, what the medium should be. Should it be pre-recorded? Should it be video? But I think, yeah, we really try to do some like content design ahead of time to also use that as part of the conversations when we shop around a project.

Steve: So to clarify, this is happening in the upfront portion of the project, so you’re thinking about what the output might be in order to have these conversations with people about this research, which you have identified as important, but they haven’t asked for. So what are you showing them? These are sort of — these are samples of what a deliverable might contain, but there’s no — there’s nothing in it because you haven’t done the research yet.

Vanessa: Yeah, either there’s nothing in it and it’s essentially like a template just to show like placeholders of the type of information we can show. But sometimes again we’re not starting from scratch. Like the reason we’re initiating this project is because we kind of have like sprinkled evidence popping up across the team, but we’ve never deliberately investigated this area. And so part of it is almost like an early synthesis of like we have a hunch because we saw this, we saw this.

And so sometimes the story just emerges from that already where at high level we can kind of come with an outline of like, okay, these are the group of people. We think they’re experiencing this problem. So of course that’s not like how you do research all of the time, but I think it could be very effective to do that some of the time, especially in an environment where it’s like people are looking to make decisions. If our research is just like, oh, that was cool and people move on, then we didn’t really do our job that well.

Steve: So it’s interesting. It’s almost like a — I don’t know, like a trailer for a movie or something. Like, here’s the decision you’re going to be able to make. Here’s where we are today. If we do this, then we can fill in these gaps. And so you’re not selling them research. You’re selling them the thing that they care about, the decision they need to make.

Vanessa: Correct.

Steve: Way back when I said I wanted to go back to something and then we got into this interesting thread, you said that part of what your team does is, you know, help other cross-functional partners and folks do their own research. And I think you said this is kind of a hot topic right now. But yeah, what’s been effective for you? What have you seen work well?

Vanessa: That’s a good question because sometimes it still scares me too. I think the things that have worked are especially stakeholders that have previously either had experience with research or have been close to previous work that we’ve done, tend to already have a better sense of like how to go about it. We’ve created templates just around like here’s how to write a study guide. And the ideal version of that would be they take a stab at it and then bring it to something like an office hours or talk to a researcher they know so that we can at least coach them a little bit on like are these the right questions and like is that the right set of people that you need to talk to? How do you recruit properly? Things like that.

And so I think if all of the setup goes well, I sometimes have like less concern about the actual sessions themselves if they’re able to just follow a script. I think the analysis piece then again becomes a little bit risky in terms of how people analyze, synthesize information. And I think we could probably do more there internally to like guide people through that process because I think it’s just yeah, challenging if you haven’t experienced it often enough or I’ve seen people overgeneralize or grab that quote that supports exactly what they want to push or need to do and kind of ignore the rest. So that’s why it’s definitely mixed, my feelings towards it. But at the same time, I have seen it also be very useful when it’s something we just can’t support or take on and they do have kind of the proper resources and guidance to get to some of those answers themselves.

Steve: size and availability of your team weren’t an issue, what would you do to help, you know, people who do research be effective with analysis and synthesis?

Vanessa: Oh, good question. I think I would probably be in the form of a workshop or just live debriefing. Multiple jobs go. We used to do a lot of that as in groups like in person with sticky notes and really go deep on that front. I think now that everyone’s remote, we try to do that more in digital tools. But I think having more guidance around how to approach that or even providing them with a framework or a plan on what notes do you want to capture, how do you properly set that up and capture them, especially the more participants you have. I think that’s where I’ve seen lots of people waste lots of time because they didn’t have a plan. They just went through, captured all the data, and then they’re kind of in a now what situation where it’s like, do I rewatch 20 videos? I’m like, please don’t do that.

And so I don’t know. I’ve joked with my team this week that I’ve equated research to party planning, but it’s like the more you plan up front on all the things that are going to be happening and go well and to have a schedule around it and have a plan for how to get to the end, I think the smoother it goes.

Steve: I really like party planning as a kind of a framing. And so yeah, analysis and synthesis is part of the party plan.

Vanessa: Yep.

Steve: I think it’s interesting to hear you and this is not meant to be presented as a disagreement or not, just a reflection that when you think about kind of how to help people move forward with that are not familiar with analysis and synthesis, you’re talking about the, I don’t know, like the tools and tactics of managing that data. That’s I think what I heard you kind of emphasize. I like that because those are things that can be described and be enacted. But there is this part to me of analysis and synthesis that feels, it scares me to, even though I do train people to do this, it still scares me because it feels like it’s creative. I’m even like sheepishly using that word and speaking to you, but it feels like it’s creative and a little bit magic and a little bit hard to describe.

Vanessa: 100%, which is why I still say it’s also the scariest part for me to let people go on that part of the journey. So I don’t know. Like when I was first entering research, I think a lot of things helped structure my thinking, at least, around the difference between hearing something verbatim versus having your own interpretation of it and kind of going back and trying to be a little bit more rigorous around what did you hear, what does it mean, and how do you navigate that? But I love hearing you talk about that it is a creative process because I think, yes, once all of the kind of analysis is done and the data is on the page, understanding what’s important, what story you want to tell, and how to put it all back together is, I think, an extremely creative process because regurgitating everything you heard is not going to work. Like you have to make it compelling and you have to find a point.

And I actually think that’s the thing that most junior folks struggle with is they want to share everything they learned. And like what are your top three things? Because in reality, we can only action probably one to two, if anything. So I’m a huge fan of pushing for prescriptive findings and having ideas around what should happen next.

Steve: Can you explain prescriptive findings?

Vanessa: Yeah. I think so. Rather than having a recommendation like XYZ needs to be clearer on this page or like users were confused by X, which is a little bit more just like describing what happened and where the problem lies, telling the rest of the team, here’s how I think we should solve it. Like, are you saying you should write something differently? Are you saying a human needs to help them? Are we going to introduce AI because that makes it better for them? Like, I think there’s so many different ways to kind of get into solutioning. And I mean, maybe one could argue by just presenting the problem, you’re leaving that solutioning piece open.

And in that case, I would say follow up with a workshop and get a room full of people together to do that in a more deliberate way. But if you’re limiting yourself to a readout and you have reasonable confidence that you actually know the next step forward, I ask people to just say it rather than hold back and just try to be more neutral.

Steve: Yeah.

Vanessa: Yeah, that’s kind of where I land on, being prescriptive. Curious if that resonates with you.

Steve: Yeah.

Vanessa: [Laughter]

Steve: I mean, I think that, you know, you’re kind of getting at what are some of the hot topics in research and I think do researchers give recommendations is a hot topic. At least for me, it feels something I’m sensitive about. But you’re providing some nuance here and you’re kind of giving some of the, it depends. And I think I’m hearing you saying like, if it’s clear what the thing is like what the solution is right.

Then, then yes, there’s no reason to hold back on that. I think where I, if I were to think about my own practice. I would, I agree with you. People are confused by X is, it’s just a description. But I, and again, we’re speaking very about generalities here but I think what I try to do is say, people are confused by X, because they understand this word to mean this and you’re using it in a way that means that.

Vanessa: Yeah.

Steve: So, you know, you can go further and say like if you want people to understand this, you know, the language has to line up. And I think that’s very different than change the label on this button from A to B so people know how to use X, I tend to not. And this is also, I’m a consultant I don’t have the same relationship with the product team that your folks do.

Vanessa: Yeah.

Steve: And I might be more likely to like, to like, give them the whole to decompose it a lot so that they understand, and maybe what to do is obvious. But I think I don’t ever know enough to say, I shouldn’t say ever I often don’t know enough or what all the possible solutions are what the roots of that are. And I want to sort of hand them. You know that thing about helping somebody telling somebody what to decide but making it feel like they’re the ones that are deciding it.

Vanessa: Yeah.

Steve: You know that’s easier for consultant or that’s maybe more appropriate for an outside person for in house in house person.

Vanessa: I love that, though. I think I’ve actually done a mix. Like, back when I was still doing a lot of research myself, I would probably initially land at kind of the fidelity that you’re talking about, of like being very specific about what’s wrong and how they’re misunderstanding it, and even the prevalence of the problem and just really making it concrete of, where are people getting hung up? And then I would just use a little idea, you know, bubble icon, and then separate that and be like, “What idea is this?” So that I’m still getting it in there, but they can at least anchor on, “Okay, this is the finding, and then I’m taking it one step further, but they’re not so tightly coupled.” Because I think if I would jump straight into, “Hey, you need to do this, this, this, and this,” and they don’t have the context behind it, one, it loses credibility, and then two, if I was off, now they don’t have the anchoring problem to actually address it differently. So I kind of like the combo.

Steve: I really like the putting the recommendation or the suggestion in a separate area the, the idea bulb, or the light bulb kind of call out. Because I, you know, I think sometimes what I’m trying to activate is to get them thinking about that transition between research and action like, and that there are, you know, I love your example of the workshop and we can always get the workshop so can we at least put this forward as, for instance, for instance, if we did this this would we’re not saying this is the only way but for instance this would. This is a way that you could use the tools of design or copy or whatever to solve this problem. I mean, I had an experience a year or so ago where, you know, I put some for instances into something for a client that I really wanted them to riff on what was possible because I just didn’t know what was possible.
And I was really torn between like, you know, as a researcher if somebody build something based on what you’ve recommended like, that’s a tremendous win. So I was really proud of that and, but also, I wasn’t right, it wasn’t a recommendation go to x it was like start thinking about solutions in this area. So I had this sort of mixed reaction, you know, in that experience.

Vanessa: That’s understandable, but I agree. I think that’s always like, counted as a success of like, “Hey, my idea made it into the product.” And ideally, due to our background and our proximity to customers, it’s like a pretty legitimate idea. It’s not like we just pulled it out of thin air. So I like that.

Steve: Maybe we’ll switch topics a little bit. You’ve, you talked a little bit about, you know, you’re just talking now about some ways that you practice in previous roles. You know, now you’re in a leadership position and thinking about right driving you’re advocating for the practice. But if we could rewind, however far back we want to go but you know, can you talk a little bit about, you know, what was your path to get to the role that you’re in now.

Vanessa: Yeah, my path was definitely not a conventional one, but I think we are in a field where that is often the case. So I think my first real entry into, I guess, being in a tech environment was after I quit my personal training job. So I had a anthropology degree from Berkeley as my undergrad degree, and I did not know what to do with that. I would go to job interviews, and they barely knew what that meant. Someone asked me if that means studying dinosaurs, and I was like, “Not quite. It’s actually the study of people and cultures.” So I struggled quite a bit after college and kind of had like this passion for fitness on the outside, was a personal trainer, and did not love the working hours and how that schedule tends to play out because you have to train when everyone else is not working.

And so I found a temp agency that had this office manager job at a startup called BrightRoll, and within a few months of me being hired by Yahoo. So that was my first exposure on talking to people that were working in product and UX and marketing because I had ambitions of other things but I just did not know what I wanted to do. And then fast forward, I actually worked at another company that got acquired by Capital One, and I was still kind of in an office manager role and then shifted to an administrative role. So I ended up becoming the executive assistant to a product VP and that was really where I figured out that UX is what I wanted to do because I essentially told them I wanted to learn the business. I wanted to be close to the work that was happening and they had a very heavy design thinking culture. So I was participating in trainings and design sprints and home visits and just like really being immersed in UX and decided that research felt a little bit more aligned with my undergrad experience kind of in anthropology.

And so about a year after doing that, still at Capital One, I was able to connect with a few folks in UX and get a research operation job. I wanted to jump straight to research but the head of the department said you need a master’s degree for that. So I ended up doing a lot of research operations which gave me a lot of proximity to research and pursuing my master’s at the same time. So I was, you know, doing a lot of the recruiting, writing screeners, managing our tools, like procuring new tools. So really it was like all of the research. I almost felt like a research assistant in many ways while I was getting my master’s in human factors and information design and then from there once I did have my degree I formally became a researcher at Capital One. So there I was working a little bit on the mobile app and then on the website Capital One dot com. So that was like a really fun time in my career and then after when I ended up at Google I switched very quickly from being an IC to kind of first managing a few contractors to managing a qual team to also managing folks that were quant researchers to now my role kind of expanding even beyond research.

So it all kind of fell in place very, very quickly as I yeah, kind of navigated my career and didn’t spend as many years doing IC work as I expected. Like a lot of the time when I was at Google I was kind of in a hybrid role so I was still conducting my own research, managing a few vendors that were doing kind of consulting projects for us and then had a team.

Steve: How did you learn the managing part you kind of moved into that and that’s a different skill set than what you had been got your master’s degree and I’m assuming. How did you learn that.

Vanessa: Yeah, I think very, very different skill set. But I think I was gravitated towards that. Like even when I was younger I think kind of leadership came natural to me like in high school I was like president of this club or captain of the volleyball team so I always kind of gravitated towards that and I was a tutor in high school for many, many years. So I think there was always the spirit of like helping people, coaching, collaborating that came natural to me and then on top of that I think on the job training of just like understanding how to navigate different situations.

I mean I don’t think I could have ever imagined the situations you find yourself in because of course there’s performance related stuff. There’s stakeholder related stuff, politics, and then there’s personal stuff and it all kind of comes together and then there’s the actual like looking at your team as a whole are we actually operating and performing in a way that is like helpful to the individual but also to the company and so I feel like there’s a lot of different variables to juggle, but I would say I kind of picked it up from like observing my former managers, realizing what wasn’t working for me, and then just like workshops, trainings, and on the job experience.

Steve: How did you distinguish between management and leadership were sort of using both those words in this conversation I wonder, do you have a definition or an explanation for either.

Vanessa: I mean I guess I tend to agree with like the ones you typically hear where it’s like anyone can lead or be a leader and then management is kind of described more as like a distinct set of facilities, so right now I’m using them pretty interchangeably but I know there are formal distinctions on how that works.

Steve: When you and I talked in anticipation of having this conversation. One thing that you brought up was both an inclusive research practice and things like inclusive hiring and I guess inclusive as a big term here, but it seemed like you had had some experience.

I had some experience with that and some perspective on that, not just a Twilio but I think throughout your career and I’d love to hear you illuminate some of what you’ve done and how you approach it.

Vanessa: Yeah. I’m glad we’re touching on this topic. I think kind of inclusive research, workplace, all the things has always been really important to me. Unfortunately I think in 2024 we still see lots of forms of discrimination whether intentional or not. I think it’s a big conversation again now that all of the companies are thinking about AI and thinking about okay how are these data sets, like where are they coming from, like how are models being trained, and a lot of it is always looking back, and so I think just throughout my career I have been very mindful of the folks that are marginalized whether it’s like in the workplace or as consumers of certain products that you know sometimes things are not designed with women in mind or people of color in mind or certain disabilities are not thought about which often leads to problems for again the people who are supposed to be building these products for but then also I think in the workplace like I have seen these things manifest and I think these things go hand in hand as people from different backgrounds or different walks of life. I think there has been so many studies now published on you know having a more diverse team actually leads to more creativity and better solutions because you’re able to see things from different angles and bringing all those perspectives together kind of creates a better outcome and so yeah that’s just been something that has been important to me throughout my career, that I have been mindful of with any of the teams that I have been on that we take that to heart and also design our research to reflect that.

Steve: So when you say design your research, that makes me think about sampling but that may be a very small view on what you’re talking about, how does it show up when designing a project?

Vanessa: Yeah, I definitely think sampling is a big part of it. I think where tech companies are located it’s easy to do convenience sampling and just say you know we will do something here in a lab. Luckily now remote research is more popular but back in the day where like a lot of physical labs were being used to you know test an app or show a new site that you just don’t want out there on the internet. That often meant you know you are sampling a geographic area where the income might be higher or there’s a certain distribution like whether it’s a gender ratio or a race ratio that might not reflect the full population. I think being very U.S. centric is often a thing with companies and research that we do and there’s a lot of additional hurdles sometimes to be inclusive. Like if you want to do a study that is conducted in people’s native language and you’re an international company there goes many dollars for recruiting and translation and kind of the scale that it now takes rather than doing something that’s local in your area. So that’s kind of what I mean by like the design, but then it can also mean you know all the way down to who is actually conducting the research. Again like if it has to be a native speaker or if it’s better with sensitive populations who might have mistrust when it comes to testing medical products or things like that like it all then requires a different level of planning and consideration in order to also make the participants feel comfortable and to collect that data in a way that’s respectful and yeah I mean I think I could go on and on. That’s kind of what comes to mind for me and what I have kind of pushed for and I think for the most part there hasn’t been any push back on whether it’s the right thing to do per se, but it really comes down to then the timing constraints and the financial constraints that push a lot of companies to say you know what? Not right now. So I have been a pretty strong advocate for when it makes sense to let’s take that time and kind of broaden our scope in order to make sure it works for the broader set of people, not just the convenient sample.

Steve: And you made a point about right diverse teams are more creative and then even thinking about who does the research.

Vanessa: Yeah, definitely. I think right now I’m in a context where we’re actually B2B and so I would say some of those factors have been a little bit less prominent for us and we tend to you know kind of go for the people who are using the product and try to just like expand our sample to that, but I’m sure that’s your experience too with B2B often you just have a way smaller population than in consumer where you have often hundreds of thousands of people you can contact like B2B might be. You have a list of 5,000 and by the time you add a few criteria now you have like 30 people that qualify. So that makes it a little tougher, but I think in the past when I’ve worked in SMB spaces or consumer type products then that has been more of a consideration and I think that’s also where vendors have sometimes come in for us where maybe we’re not the appropriate set of people to have this conversation or we might miss some of the nuance that exists in this conversation, so we’ve like when I was at Google we definitely would get outside help to kind of round out some of those conversations or kind of share like the interviews across a native Spanish speaker if we don’t have that on the team.

Steve: Does this come into how you approach hiring.

Vanessa: That’s a good question because I think with hiring I’ve not deliberately been in a situation where I’m like saying we need this race or this gender or this language because I think that can get very tricky and also potentially lead to discrimination in other ways, so hiring has largely been like merit based, but I think the part of hiring that I still find problematic is often people talk about pipeline issues and not being able to source candidates from different backgrounds and I often find like that there is actually something we can do about that and I think that’s where I focused more efforts is if I’m seeing candidates come through that are a little bit more narrow in kind of let’s say education. They’re all coming from a set of Ivy League schools or they’re kind of like located in a specific part of the country then I have kind of like worked with recruiting to say hey, can we kind of reach different hiring pools? Do you have access to like different communities where you can plug these jobs to kind of diversify the pipeline so we can make sure that at least candidates that have different levels of education or different background can at least have their resumes reviewed, be interviewed, things like that.

Steve: You know, in the time that you spent in your career so far. Have we made progress on these issues like how, how do you contrast what you see 2024. Like you said there’s still discrimination is not gone away.

Vanessa: Yeah. It’s hard to say. I’m not sure if we’ve made progress or not. I think during the pandemic when George Floyd was murdered there was definitely a heightened interest from companies to think about a lot of these topics. So one way that worked out is a lot of companies including Google and I was actually participating in like some of the hiring initiatives was to think about product inclusion specific roles and like really carve out space to say like we have people in the company who are kind of like trained and qualified to deal with maybe like more sensitive populations and then who are also actively putting programs together to like teach the broader company like how do you make sure that your product especially again when it comes to like language or anything AI related or kind of like anything that’s like visual is not discriminating against like oh the photos can’t pick up on darker skin or they can’t understand people with a certain dialect. Like how can we get that education out there? So I have seen a push in that. There was more consulting firms, more job postings that were specifically around product inclusion and bringing that academic knowledge into the tech field, but as far as like broader hiring trends I don’t know the data well enough across the full population, but I don’t know if I have seen like a uptick per se.

Steve: So maybe one last area to loop back to something else that we were talking about it and you described, you know, going from personal trader to attempt job that gave you exposure to, you know, things that led you into you accidentally research. In some ways it seems, I don’t know sort of circumstantial or opportunistic or something like it’s hard, it’s hard to plan for creating the conditions for that. And that’s, you know, what, is there any lesson or, you know, advice to people that that are listening or that you come across in your in your travels anyway about, you know, going from to me they seem like very different worlds to go to going from one bridging from one world into the other, you know, for you is, it was, I think a certain amount of happenstance and having the, you know, having the right lens on it. But I know does this lead to any guidance or advice for other people from your experience.

Vanessa: I think for many years, I had anxiety and regret around not starting my career in the field that I’m in sooner because I felt very very lost stumbling through all of the different fields and roles, and only in hindsight do the dots connect. As you were saying, they seem very, very different, but I think I’m better at what I do now because I learned the lessons in all of the different jobs. Even something like, again, being a personal trainer or executive assistant, being an executive assistant, I was able to sit in on more senior leadership meetings, and I really early picked up on short attention span, How do you get your point across concisely, What do they care about? And I think that made me a better researcher right away, even as I was still learning the practice because it taught me something about communication.

Then even reaching back to personal training, I think that made me a better manager because I like to think of myself very much as a peer to my team, where we are thought partners. They come to me with things, I come to them with things, and because they are all so driven, we don’t really have a lot of issues where I just have to enforce things or tell anyone to do anything.

But circling back to the personal training piece, I think that really put me in a headspace of okay, someone has a goal and they’re struggling with this, or they need support through this. How do you tap into the human psychology of – they want to get from A to B, we want to make it as sustainable as possible, do something that hopefully they enjoy enough that they can do it on their own.

And so really just getting into that mindset of how do I collaborate with another person and find common ways to address a problem and align with them so it doesn’t feel like I’m pushing you or forcing you, we’re kind of going towards this goal together. I think all of those little pieces along the way just shaped how I interact with people and I think has made me better at what I do today. Maybe just know that it’s all connected somehow.

Steve: Yeah. Yeah, I want to normalize part of what you said at least that, yeah, I also had regrets for not starting my career earlier I felt like I was late. And, you know, I’m, but that was long enough ago that many people I know didn’t know me then or wouldn’t know that about me but, you know, I had a complexly directed path to do what I do now. And, yeah, I think you’re right that we’re sort of all products of all of our experiences is one of the things I like about research but I think you’re right to extend it towards management and everything that there are a lot of different paths and then there are a lot of different ways of being. And that, yeah, if I wasn’t who I was I wouldn’t have gone on those paths and I wouldn’t be able to be all the weird mixes of things that makes me whatever kind of researcher or leader or, you know, all the things that you’re kind of bringing up. I like that about, I’ve always enjoyed that about our field that we’re just, we all come from different places. Even over the generations I think there’s still an interesting mix of that and then means like oh the next researcher that you meet, you know, done something that you didn’t know about like, and you get to work with them and like, you know, there’s something about personal training that shows up in an analysis or in a planning meeting or something and you get all these great stories from people. So I enjoy that. But yeah, I just, I started off by talking about myself here that I have, I know what that regret for me is like that I didn’t come into it out the gate early on and had to play catch up in a lot of ways too.

And maybe that’s a great place to kind of wrap up our conversation. Thank you so much for taking the time, sharing your own experiences and your own perspectives on the work that you’ve been doing all the way along and yeah it’s great to get to chat with you. Thank you.

Vanessa: Thank you so much for having me. it’s really an honor. And even when I was telling folks that Dollars to Donuts was coming back on my team everyone was really excited because a lot of us have your books and have listened to you and learned from you so I really appreciate being here.

Steve: All right, well, great. I hope they enjoy listening to you talk about all the great work that you’re doing.

Vanessa: Thanks.

Steve: Thank you. All right, that’s it for today. I really appreciate you listening to this episode. If you like Dollars to Donuts, recommend it to a friend or colleague or post about it on the social medias. You can find Dollars to Donuts in most of the places that you find podcasts. Review the show on Apple podcasts and go to portigal.com/podcast to find all the episodes including show notes and transcripts. our theme music is by Bruce Todd.