Portigal | 39. Mani Pande of Cisco Meraki

This episode of Dollars to Donuts features my interview with Mani Pande, Director and Head of Research at Cisco Meraki.

We used to do these immersion events where we would bring everybody who worked on, who was our stakeholder, to come and listen and talk to our customers. And we would do these focus groups, they were like a whole day event. There were folks from marketing and ops team who ran some of these focus groups. And when we got feedback about the immersion, it was very clear that everybody realized that when researchers are not doing the moderation, the kind of data that you get is not good. And the conversations were not that interesting. They didn’t feel that it was a good use of their time. So I think you can have your stakeholders experience it, that it’s not that easy to do moderation. – Mani Pande

Show Links

Help other people find Dollars to Donuts by leaving a review on Apple Podcasts.

Transcript

Steve Portigal: Welcome to Dollars to Donuts, the podcast where I talk with the people who lead user research in their organization. In case you don’t already know, I recently released a second edition of my classic book, Interviewing Users. This new edition is the product of 10 more years of me working as a researcher and teaching other people. It’s bigger and better. It’s got two new chapters, a lot of updated content, new examples, new guest essays, and more.

As part of releasing this new edition, I’ve been on a number of podcasts myself, including a conversation with Jane Portman that was part of her podcast, UI Breakfast. Here’s a quick excerpt from that conversation.

Jane Portman: As you’re training other researchers, you’re training experienced researchers, I’m thinking, what do you feel is common knowledge that they’re mastering well and that we’re all like good at and what things are surprisingly difficult?

Steve: You know, I think especially for people that are, they have a little bit of experience, but they’re kind of starting to blossom a little bit. One of the things they often need the most is confidence. And so people will often describe a scenario that they were in. People are messy and people are unpredictable and so all these things happen and so they go in with the best of intentions and plans and then things change a little bit. Somebody mentions their divorce, they don’t know what to do. And so I feel like my job isn’t to tell people that they’ve screwed up and that’s not how you do it. I think my job is to tell people that the thing that you encountered is very common. It’s a thing that a lot of researchers struggle with. I try to handle it this way, but there are situations when I handle it this way. Like I think I have a lot of like specific guidance and best practices, but all of those come with a lot of subjectivity and that it’s sort of the nature of the work to be a little confused or uncertain and to have to try things.

And by the way, there’s no right choice. When someone mentions their divorce, you know, not even me versus you, like me versus me. The next time I would do that interview, I would handle it differently. There’s a moment and if the divorce was brought up in minute three versus minute thirty, like it would play out differently. We’re not algorithms, we are, I think improvisation is a big part of it. So I think I want to help junior researchers feel okay about that there’s no one right way to handle this and that their way of, the fact that they felt confused and uncertain in a situation and they made this kind of, here’s how they addressed it, it’s, I rarely tell them like, well that’s the worst thing you could have done. It’s usually, they usually are doing their best. That confidence to make a different choice is kind of what I want to help somebody with.

You know, more experienced interviewers, I think I like, I like working with them because I think we can have a better, richer conversation about what are all these choices and what are the differences between them and you know, I love being in a workshop where I’ve got people with different experience levels because then I might give some guidance and then we can have different people suggest, well you know, here’s what I’ve done. Everyone can learn from each other and sometimes we can debate and I don’t mean that in like a right and wrong way but I think someone with experience, we can have a really interesting conversation where we look at one scenario and four or five different ways to handle it and we might disagree on what is sort of the optimal way.

I think what that surfaces is that as individuals, as interviewers, we’re all wired differently and we all have different instincts and different personalities and you can get away with things that I can’t because of my age or my gender or my energy and I can get away with things that you can’t and getting away is maybe there’s the wrong framing but there’s just so many interesting choices and it’s I love hearing about other people’s things and thinking like, oh could I have that amount of friendliness or that amount of stillness or that amount of curiosity or that amount of empathy, you know, could I present those things in different amounts? You know, that’s part of being an expert interviewer is you have your own personality and your own strengths and you can exercise different facets of that as the situation requires. I think, you know, a more experienced interviewer has more adaptability to that, has their core, you know, but also can put on different authentic, true to themselves faces and energies to kind of support different kinds of situations that are going to come up in these interviews.

Again, that was me speaking with Jane Portman on UI Breakfast. You can check out the whole episode and you should totally buy a copy of this new edition of Interviewing Users. You can also check out portigal.com/services to read more about how I work with teams and companies.

But now, let’s get to my conversation with Mani Pande. She’s the Director and Head of Research at Cisco Meraki.

Well Mani, thanks so much for coming on the podcast. It’s great to get the chance to chat with you today.

Mani Pande: Thank you, Steve, for having me on the podcast. I’ve been listening to your podcast for several years, so it’s great to be a guest on it.

Steve: Excellent. Do you want to start us off with kind of a little introduction to you and then we can build a conversation from there?

Mani: Sure. So my name is Mani Pande, and currently I lead the UXR team for Cisco Meraki. And, you know, Cisco is a really big company, and I am part of the networking division. And within the networking division, like, there are two types of primary products, I would say. Meraki, which is their SaaS offering, and then enterprise networking, which has primarily been their on-prem offering. And my team works across both Meraki and enterprise networking. So it’s a pretty big team, and it’s a big part of Cisco’s business because for Cisco, networking is still bread and butter. So there’s a lot of interesting work that the team does, and a lot of the work that they do does impact the products that we ship and also has a — hopefully has a lot of positive impact on Cisco’s bottom line.

Steve: Do you have any examples you can share about situations where research impacted something that the product was doing?

Mani: Yeah, so one of the — my manager is a big believer in the double diamond approach, you know, starting from doing foundational work and then moving on to doing — you know, once you have the design, testing it, doing concept testing, doing usability testing, and once you have shipped it, you are also — you also have some kind of metrics that you have used to define success and trying to measure those. So there are several examples that — where the team worked throughout the double diamond process and was able to make an impact not only just in defining, you know, what kind of product do we want to ship, but once we had some concepts, they helped define what are some of the hypotheses that we want to test, did a lot of concept validation, did a lot of usability testing because we had a lot of designs that we wanted to test, so pressure tested them in front of our customers.

And then, you know, once we had shipped the product, like we wanted to see that, you know, our customer’s happy with it. Do they really like it? Was it worth the effort? So, you know, doing some kind of like customer satisfaction surveys as part of getting continuous feedback from customers. And also, you know, even when we did those customer CSAP surveys, like we got a lot of open-ended comments, and we got some feedback from customers such, “Oh, you know, this product, like this is not working or that is not working.” So those were good early signals of what we needed to change before it escalated into a support issue.

So there are several projects that we have worked on. At Meraki, we always call our projects by planet names. So there’s a project called Jupiter. There’s a project called Aurora, which is all around, you know, providing more visibility for on-prem devices to show up on the Meraki dashboard so you can see them as well as manage them through the SaaS product.

Steve: And so is it the same, roughly the same, group of researchers that are working through the stages of the double diamond like you described?

Mani: Yes, I would say like it’s like, for example, for one of the projects that I mentioned, Aurora, it was a same researcher who worked through the whole process and obviously in very, very close collaboration with the designer. So they were a tight-knit team that worked throughout the double diamond process. So, you know, yes, they are.

Steve: Is that an example of a researcher being embedded? I know that’s kind of a buzzword. Is the researcher embedded with that team in that case?

Mani: And I know there are a lot of people have a lot of, especially research leaders have a lot of opinion about whether you should have an embedded model versus not an embedded model. In our case, I think just because of the complexity of the domain, having an embedded model is extremely important. I, you know, I have worked across, like I’ve worked at Wikipedia, I’ve worked with that Lyft, I have some B2B experience, I used to work at Success Factor where we made HR software, I worked at Samsung, I worked as a consultant where, you know, you just need to know a little bit to be effective.

But I have never, ever worked in such a technical domain, which is networking. Like every day at work, I feel like I, sometimes I feel like, okay, I know a little bit, but then there are days where I feel like I know my thing. So for us, I think the embedded model is extremely important because you need to have a little bit of domain expertise to be able to do research a little more intelligently and more meaningfully.

And another thing like I feel, this is my perspective that an embedded model works better. And, you know, even when I worked at Lyft, which I would say in terms of complexity is nothing compared to enterprise networking, we still had an embedded model because what an embedded model enables is relationships, which are harder to form if you’re not in an embedded model. Like for researchers, one of the things that, you know, we do is that we lead or we bring about change without authority. So for that, I feel like having relationships is extremely important.

In fact, you know, like there’s this article that I came across recently from Harvard Business Review, which, you know, they had listed like the three things that you need to do to lead without authority. One of them was relationships. Like having relationships with your PM partners, with your design counterpart, engineering, data science, it’s extremely important to be able to bring about change, to be able to show that, you know, what you are hearing from your customers matter, to be able to change hearts and minds. Because I sometimes feel that we are in the business of changing hearts and minds. You know, a lot of people, like I have worked with a lot of PM partners, they have very, you know, some of them have very strong opinions. So to be able to bring about change, I feel that having strong relationships is extremely important. So I am a big believer in the embedded model.

Steve: Are there other things that they have to do or that you encourage them to do to build the relationships in the way that you’re talking about?

Mani: There are various things like I have done all my career and then I also encourage my team to do. So one of the things is I feel like to have a good relationship, you need to bring, and this is more important if you’re in IT and you’re doing the research yourself, is to bring your design, to bring your stakeholders along with the right when you are conducting research. So that’s one thing that I always encourage my teams is invite people to come to your research sessions. Make sure that they are involved in helping you come up with the insights. Like obviously the researchers are going to do the heavy lifting. Like you don’t expect the PMs or the engineers or the data science to do the heavy lifting. But do a workshop with them and ask them, like what did you hear from some of the interviews that you attended? Like what resonated with you?

And also the other thing that comes out with that is that you have less resistance towards the end. People are less likely to challenge you. So it ensures that everyone is kind of on the same page from the beginning. So I feel that it’s also good for relationship building. So that’s one thing that I always tell the ICs to do.

And myself as a research leader, like I obviously try and build relationship with whoever are my counterparts. And I also try, I’ve always done is like build relationship, especially with, you know, PMs like who are like the head of the product that your team leads. So for example, when I was at Lyft, like I worked on the Driver app. So I used to meet with our head of product management for Driver at least one supporter to make sure that I also had a good relationship with them.

Another thing that I learned through that was working with them, it was easier to figure out what we should do long term because a lot of product managers are only thinking about, you know, what they have to deliver within the quarter or maximum the six months. Like if you build relationships with the leadership, like it enables your team to work on more long term projects. So that was just a learning that I had. And I always try and do that is like have a good relationship with the head of product, like people, you know, two or three levels above me. I mean, they have a lot of ideas.

Steve: How does the relationship support the longer term conversation?

Mani: Like they are thinking more about, you know, like where the business needs to be. They’re not so focused on the product roadmap. They are not so much thinking about, you know, this is the feature that I need to ship tomorrow. And also, you know, you can get them to say yes to something that you feel that the team should be working on, but that they might be a little bit of pushback from the product team. So if you get their blessings, you know, you can be working on projects.

Like, you know, as researchers, we have a lot of opinion. And I always tell people like you have to have a point of view. You are spending so much of time with the customers. You’re talking to them. Like if you don’t have a point of view on what research we need to do or what matters to our customers, then you’re probably not doing a good job. So like let’s say if you have a point of view of some research that needs to be done, but it’s much more long term. You know, you would probably not going to see the impact or nobody is thinking about in terms of their roadmap for the next quarter or the half. Like it’s easier to get a buy-in from the executive. And to be able to get that buy-in, like you have to be able to get that buy-in like you have to have a relationship with them. That’s what I have experienced and it has worked for me in the past. That’s what I do, but I also encourage like my ICs to do it.

Steve: So it’s you as the leader are having the relationship and the buy-in for the longer term pieces. This is not things that your ICs and researchers are focused on. This is your role as the leader.

Mani: But you know, a lot of people feel intimidated to go and meet the VP. So I encourage them to do it. And in fact, you know, like one of the other things when I used to do this was I would take like if I did this conversation, I would invite the ICs who were relevant for that conversation to be part of that conversation. So they didn’t feel intimidated to be having that conversation and they could also participate in that conversation and hopefully meaning going forward can do it themselves without me being there.

Steve: So when you talk about relationships, you’re creating them yourself. You’re encouraging others to do that. And then you’re, I guess, enabling or facilitating relationships between other people. You’re talking about a number of different fronts. for building these relationships– the workshops, inviting people to come to sessions, having these sort of, I guess, planning meetings or discussion meetings.

Mani: Yeah, ultimately my role as a research leader is to help my, enable my team. That’s how I think about it. That is one of my important goals. I would say not the only goal. So whatever I can do to enable that, I always try and facilitate that.

Steve: One thing that I’ve heard a lot and that I experienced myself is that the kind of relationship building you’re talking about, whether that’s workshops, participating in interviews, or just kind of meeting, is harder or it’s at least different when work is remote. And I wonder, have you seen changes in how you’re doing this relationship building or how you’re helping others to do it over the past few years?

Mani: Yeah, I mean you can think of it as a glass half full or half empty. That’s how I think about it. I think it’s still possible. In today’s world, like we have so many more tools. Like I would say this would have been so much harder 10 years back. Like doing a remote workshop is so much easier. Like we are all like you can use FigJam, you can use Miro. Like even our stakeholders, like everybody knows how to use those tools. And there are so many templates that you can leverage. Actually, you know, in some ways it’s easier to do it remotely versus do it in person.

But I agree like, you know, in person there’s a lot more energy to it. There is something about being in person at the same time. Like the wife is very different. But with the tools that we have, I think it’s just so much easier to do that. The last two jobs I have started, I’ve started them remotely.

Steve: That’s a good reframe on my question. I think glass half full, glass half empty is a lovely way to look at it.

Mani: And honestly, like if I compare them to the last two jobs that I have, like did I feel any difference? I would say it’s a little harder. It takes a little longer, right? But it’s not impossible. And you can get to that same level of relationship over time. So the one thing that I have done, the one thing that I have done is I have done a lot of work. And the one thing that I have done though is like I’ve also had this privilege like, you know, when I worked at Lyft or at Meraki, I live in the Bay Area. The offices are in the Bay Area. So when I go to the office, like I don’t, I think of those days as days of relationship building. So I go and I meet a lot of people. And at home it’s obviously a little more focused work or you could be in bigger meetings. And I think about work on how I spend my time in the office has also changed a little bit in the last four years.

Steve: Right, we might not have done explicitly relationship building, maybe not even having that as an intention, going into the office to build relationships.

Mani: Yeah. So not only just with stakeholders, I think it’s also important for the team. Like when I was at Lyft, our team, you know, like when we went to the office, we all went to the office. So it became more like a team day. Not very productive, I would say, but it was good for, you know, team bonding and it was good for us in terms of, you know, coming to know each other and just building our relationship and figuring out what kind of a team we are together.

Steve: You mentioned this article about change without authority and you said that they listed three things and one of them was relationships. Putting you on the spot, do you remember what the other two were?

Mani: I do actually because I did a blog about it. So I’ve given it a lot of thought. The other one was expertise. I think that’s also an important one for researchers because in the last, you know, few years, there’s all this, everybody talks about DIY research, you know, you can farm out research, everybody can do research, and there’s also this perception, you know, how hard is it to do research? Like, you only have to talk to people, right? Like, you and I are talking. We could be doing research right now. So I think it’s important for us as researchers to show our stakeholders that, you know, research is not easy as it appears. It’s very easy for us to do bad research and get bad insights from that.

So one of the things that I did at one of my previous jobs was we used to do these immersion events where we would bring, you know, everybody who worked on, who was our stakeholder, to come and listen and talk to our customers. And we would do these focus groups if they were like a whole day event. And in the beginning when we started that, there were folks from marketing and ops team who ran some of these focus groups. And when we got feedback about the immersion, it was very clear that everybody realized that when researchers are not doing the moderation, the kind of data that you get is not good. And the conversations were not that interesting. They didn’t feel that it was a good use of their time. So I think you can have your stakeholders experience it, that it’s not that easy to do moderation.

In fact, I feel like when I was in IC and if I times, like I would do four to five interviews in a day, I would be mentally exhausted. You know, doing moderation is one of the hardest things to do because you’re multitasking at another level. Like you’re trying to, I mean, I’m a little old school, I would take notes, I would try and listen to what the person is saying and figure out like, do I follow my interview script? Do I change it? So just having people experience that is important. So I think show and tell could be one way of showing that, you know, research is hard.

And then when it comes to quantitative research, like, you know, writing surveys, I think that’s a pretty specialized skill. I have seen people, you know, think that they can write surveys, but there’s a lot that goes into it. You can get absolutely wrong responses if your question is not well designed, if your scale is not well designed. So for quant research, like I have a little bit of a strong opinion on that, that I don’t think it should ever be DIY. It shouldn’t ever be unskilled stakeholders to write a survey. So that’s one thing, like, you know, just showing your expertise is one thing that you can do to leave without authority.

The other one they mentioned was about business. Organizational understanding is the third one. That is very important. That, you know, one of the things I will, as I said, you know, I have a long career, I worked for 20 years after grad school. I have seen often like researchers resist or don’t want to have a good understanding of how the company makes revenue, earns profit. They’re a little wary of that part of the business. I think it’s also important for us to understand, like, how does the company make profit? Like, for example, if you work for a B2B company, like, I would say that you should have a relationship with the sales team to understand how do they sell, what do they sell, like, what is some of the feedback that they get from customers.

So I think having a little bit of understanding of revenue and profitability is important. Like, for example, like, if you work for a company that’s in the gig economy, which has, you know, which has a marketplace, like, it could be Lyft, Uber or DoorDash, when that is the crux of the business marketplace. That’s how the company makes money. So understanding the dynamics of the marketplace is an important thing that a researcher should do. Let’s say if you are a researcher at Uber and you’re working on the Rider app, you should still have an understanding of the marketplace. Usually, you know, it’s the marketplace is kind of on its own, but irrespective of whether you work on the Driver app or the Rider app, you should have an understanding of how, you know, Riders and Drivers are matched, because ultimately that’s where the secret sauce happens and that’s where the company makes money.

I think a lot of us come from academic background and maybe a little bit of purist background. So maybe it could, this is just a hypothesis and that is where it comes from. I mean, I would say to a certain extent, even I had it, like, early in my career. It took a while for me to realize that understanding revenue, profitability is important. One thing that I would say, like, most researchers do agree on is understanding business priorities and, you know, doing research that aligns with business priority. I don’t see, I have never seen that resistance on that, but when it comes to revenue and profitability, I have seen researchers, like, have a little bit of resistance to that. Like, for example, I have some friends who work at Facebook and Google, you know, how they make money is ads, but many of them did not want to work on the ads team.

Steve: That seems different to me and just thinking about myself. That seems different to me than understanding like, how does the system match riders and drivers? When you say that, that kind of sparks my researcher curiosity. Like I think we would want to know that because what’s behind that secret wall and how do the gears all mesh? But again, my own bias here when you say working on ads is not, I kind of get that feeling too. Like I just think like, ugh, ads. And so to me that seems different. And I don’t know, I’m not trying to pin you down on something because you’re talking kind of subjectively what we each have our impressions. But I think this idea that, to your point about changing without authority, that there are these important things to understand. I guess there’s a difference between wanting to work on Facebook ads as your project and having enough of an understanding of the revenue model, which impacts everything you would ever do research on at Facebook I think.

Mani: So I think both. What I’m trying to get at is understanding the money part of the business is important. Like, and the money part of the business for different companies is different.

Steve: Yeah.

Mani: For a gig economy, it’s the marketplace. For Google and Facebook, it’s ads. So having some understanding of how your company makes money is important for researchers.

Steve: And not to flog a dead horse here, but I feel like companies like Google and Meta or Facebook, it seems like their culture is such that how the company makes money is sort of kept in a separate box and we’re going to come here and work on whatever the latest amazing thing that’s going to change the world Internet through balloons, you know, some amazing project. And so speaking out of my hat here, but I have some empathy for people that don’t want to think about the money because they’re not being sold that as an employee. I don’t know if that’s true. I’m hypothesizing that the company culture kind of keeps those things in separate buckets. But if you work at Lyft or Uber or DoorDash or Meraki, there’s a product and a service that’s much more essential to the conversation they’re having. Again, this is not my direct experience. I’m just kind of — I’m just giving you my biased interpretation of what you’re describing.

Mani: I think just the basic level is probably, I’m sure, like, you know, they keep it under wraps to a certain extent, but just like having a basic understanding and not having an aversion to it is important.

Steve: So when you talk about these factors, right, the understanding the business, so there’s some specificity, the expertise and the relationships, and you talked about Cisco and Meraki being just the level of complexity, the sort of technological and I guess industry specific stuff. Does that complexity become a compounding factor or something in trying to achieve those levels of those three factors that you’ve brought up?

Mani: I think especially for something this technical, having some domain expertise matters. And that goes back to having some, that goes back to the first one that I was talking about, which is expertise. Like having some domain expertise becomes important because if you want to have a meaningful conversation, you know, if you’re doing an interview, like you need to have some basic level of understanding to have a meaningful conversation. I know in research we say that, you know, no question is stupid, but if you have zero understanding, you’ll only have stupid questions, at least in this kind of a domain. So I think having some understanding is important, and that’s what I tell all my researchers.

Like after I joined Meraki, I have this cheat sheet I call “Mani’s Cheat Sheet” about networking, and every time I hear something that I don’t know, I get a version from Google and then I get a chat GPT version, like please explain it to a middle schooler version, which actually is the version that works for me. And that cheat sheet, as I’m like, “Huh, like 20, 30 pages long?” It just keeps on increasing. I’m not going to be an expert on networking. I don’t want to be, but I just want to know a little bit to be able to have meaningful conversations and to be also able to provide, you know, feedback to my team. Like sometimes, you know, when we have an important presentation, like I work with them to figure out like what are the insights, like what are some of our action items, but if I have no understanding of the domain, I can’t do that.

Steve: Can you talk about from the period of time that you came to Meraki, like what that progress has been and what research has gone from to where it is now?

Mani: So Meraki has seen incredible growth in design as well as research in, I would say, in the last one, two years, which is the opposite of where, you know, design as a field and UXR as a field is going at other companies. So we have, it’s also because, you know, as I was saying, like the company has really adopted the double diamond approach. So they see research as being an integral part of how you build products. So that’s one of the reasons why we have seen this big, massive growth in the last one year. And the reason why I came here was, you know, as I was telling you earlier, you know, like Cisco had these two products, like the SaaS product, the on-prem, and the strategy now is to, you know, convert, which is, you know, have like some of the on-products be able to be managed in the cloud.

So they brought these two teams together, the design team for the on-prem side and the design team from Meraki. So that’s how I got hired as the head of UXR because then, you know, the team grew because you had two different teams that merged and the complexity of the business and the complexity of the problems that the team was expected to answer grew because now you had two different businesses that you had to support. So that’s how I got hired and, right, and, you know, our team has grown quite a lot even in the last one year. Like I’ve hired, like just in March, three people have joined the team. And it’s also because, you know, everyone agrees that, you know, you need to do research if you want to build better products. So there is this hunger for research, so that’s the other reason why we’ve been able to grow despite, you know, the market going in the other direction.

Steve: So you came into this newly formed, newly merged organization that already had an appetite for a belief and a commitment to research.

Mani: Yeah, and that appetite has been growing. I would say it’s been a steady state because there are more and more teams that we are working with because Cisco networking is huge. As I was saying earlier, it’s like it’s their bread and butter. It’s the major chunk of their revenue. So there are more and more teams that we are working with and that’s one of the reasons why I’ve been able to hire researchers to support teams that did not do research previously but want to do research now.

Steve: Are there things that you are doing in your role that are behind that? In addition to there being these other teams, but do you think you’re responsible for the increasing the demand in any way?

Mani: I would say it’s a team effort. I would not put — I won’t say that it’s just me by myself. But obviously, you know, as a research leader — or I would say as a research professional, we have to evangelize research as much as we can all the time.

I can talk about my time at SuccessFactors. You know, when I joined SuccessFactors, they already had an IPO. Then they were bought by SAP in 2012 for like $3.5 billion. So the company was pretty successful, but they never had research at that time. And one of my — like what I had to do in the beginning was, you know, just evangelize research and make sure that it became part of how we build products. So in the beginning, I did a lot of like evaluated research to show impact and then, you know, slowly as our team grew, we — I mean, continued to do obviously evaluated research, but we did a lot of generated research as part of new product launches.

Steve: So you talk about that situation, they were bought into research enough that they hired you, but it doesn’t sound like the hunger that you’re talking about now wasn’t there at that time.

Mani: And also it’s like, you know, we’re talking 2013. I know the field has changed quite a lot in that time. But yes, the hunger wasn’t there. So my job was to create a hunger for research. And it was like very different tactics. Like I did a lot of usability research because that gets you — it’s a quick hit. That’s how I would describe it at times, you know, usability research. If you are looking for quick impact, you can get it pretty fast.

And you can get — you can find people who are on your side who become evangelists of research pretty easily. So that’s what I did a lot when, you know, when I had to build a team there because, you know, there was no research team. I wouldn’t say that — I mean, there was no research majority.

Steve: I hear relationships and expertise at least in that story.

Mani: Yes, there was because I remember when I joined, there was some usability testing that had been done. But whoever did it, they did not know how to define usability tasks. So when it was an unmoderated test and user testing, people were very confused about what was expected from them. And it was very clear that you needed somebody who knew the basics of usability testing to have — who should have set up the test.

Steve: Just switching topics slightly, you’ve kind of brought us back in time to some different roles that you’ve had and I wonder if we could go further back and maybe could you talk about how you entered the field of UX research.

Mani: I did not think I was going to become a UX researcher, honestly. Like I have a PhD in sociology. I thought I was going to become a college professor. That was my goal. But I’m — my husband is an engineer. So we moved to the Bay Area. And I looked for teaching jobs in the Bay Area. There were none. So I moved — I went — I got a job at Institute for the Future. I don’t know if you know about them, but they are a forecasting research company. So they do a lot of tech research. And I started working there for five years. And I did not even think the kind of research that I was doing was, you know, user experience research, like product strategy research. But that’s the kind of work that I did there.

One of the biggest clients that we had was Nokia. And I did a lot of ethnographic research, you know, going across — traveling to people’s home in India, U.S., and Brazil, and, you know, trying to understand, you know, how do they use mobile phones? Because this was like 2006, 2008. It was still a new phenomenon. And that helped — that was to help define Nokia’s strategy, especially for these markets for the next one to three years. And from that, I kind of transitioned into UXR. And in fact, you know, when I finished my Ph.D., I did not even know that there was an HCI field I completed in 2004. So it was still pretty early days of — at least from my perspective for the field.

Steve: When you found yourself doing global ethnographic fieldwork for Nokia, did you see any difference between how you were working in that context and the skills and approach that you had developed in your PhD?

Mani: There are cultural differences that you have to be aware of. One of the biggest things is you should be ready for anything. There are a lot of unknowns. I remember, like, we were doing this project, but we were working for Nokia. Like, we had — I was working with our clients, and we were supposed to go and interview someone, and the person didn’t show up. So there’s a lot of those things that you got to be ready for, like, you did not prepare for. Like, very rarely it’s happened to me that, you know, I did ethnographic research in the U.S., and we had no shows. But it’s happened in India so many times. It’s happened in Brazil. So you got to have these contingency plans and go with the flow. And I think those are some of the things that you have to be aware of.

And also, you know, especially, like, when I worked in Brazil, like, I obviously don’t speak the language. Like, I do not — like, for me, obviously, India is easier because I have lived in India. I know, you know, what the culture is, like, what are the things that you do. Like, for example, if you go to India — to anyone’s house in India, they are going to offer you tea, coffee, something to eat, and it’s polite to eat it. And it’s impolite if you don’t eat it. And when I went to Brazil, like, I had to work with translators, which I had never worked with before. So one of my big learnings was that it takes you twice as much to do the same interview because they are translating it for you.

So I think there are a lot of cultural nuances that you have to be familiar with when you go — especially when you do research outside of, you know, U.S. and maybe to a certain extent even Europe.

Steve: And did you see differences at that point in your career where you’re coming from an academic environment to a commercial environment? Were there points of transition for you as you moved from one to the other?

Mani: Yes. In terms of, like, I remember when I wrote my PhD dissertation, I took a year to do the analysis after I had finished my qualitative interviews. So I — and, you know, when I — this Nokia one that I was talking about, like, we were sharing research inside from every interview every day. So I think that is the big transition that you have to make. Like, if you come from academia is the pace. And also thinking about, you know, how — like, how does this impact the business? Like, how does this impact the product? And also working with so many stakeholders. Like, those are some of the big changes. Like, when I was in academia, the only time, like, I worked with others is my advisor provided me feedback for my dissertation. And when I published, I had these three reviewers who gave me feedback for some of the articles that I published. But, yeah, you know, the way you work with stakeholders is so different. And also you got to be, you — okay, we’re doing a lot of quick and dirty work, which obviously, you know, you don’t do it in academia. Especially if you do that, it won’t get published.

Steve: So after this Institute for the Future experience, was user research that was your career path at that point?

Mani: Yeah, and then I joined Wikipedia, which was actually a very amazing experience in terms of what I was able to do because — I mean, Wikipedia is like I have never met anybody who doesn’t like Wikipedia. I, you know, I’ve worked in different — as I said, you know, I have worked in different industries. Often we more than often, like, meet customers who say, like, I don’t like this about your product. I don’t like that about your product. But for Wikipedia, it was very different.

So one of the things that I did when I joined Wikipedia was I did the first ever survey of Wikipedia editors. So Wikipedia is, as you can probably guess, from their mission, unlike most companies, they barely do any tracking. Like, they don’t use, for example, cookies. So we used to rely on ComScore to even get the numbers for active readers for Wikipedia because they were not using cookies. So they did not know at that time, like, what percentage of editors are women because Wikipedia, even today, has a gender problem. So I did the first ever surveys of Wikipedia editors, and the answer at that time was 9%. The only 9% of people editing Wikipedia were women. And as a result of that, a lot of — there are fewer articles about women on Wikipedia versus men. So there’s a big gender bias in Wikipedia. So that is some of the work that I did when I joined Wikipedia, which I thought — and I also worked on the redesign of the mobile app. So I again did ethnographic research in the U.S., India, and Brazil for the redesign of the mobile app for Wikipedia. But all that work was very fulfilling just because of the amount of impact that you could have on users. Because I don’t remember the numbers now, but at that time there were 250 million active readers on Wikipedia every month, and there used to be like 7 to 8 billion page views.

Steve: As a user researcher over your career and maybe as a leader now, I don’t know, how do you think about different kinds of methods or different kinds of approaches to getting the data that you want to bring your team?

Mani: I mean, I’m trained as a mixed methods researcher, and when you think about mixed methods, I like to believe it’s on a continuum. There are some people who do small surveys and think that they are mixed methods researcher only doing descriptive statistics. There are some people who do regression analysis, who will do, I don’t know, structural equation modeling, latent class analysis, who are also mixed methods researchers. So it’s a little bit on a continuum, even for qualitative, right? Like you can be an ethnographer. You can be somebody who really believes in participant observation, or you can just do qualitative one-on-one interviews. With that said, like I do both because I just happen to be trained. When I did my PhD, I specialized in research methods, which meant that I had to prove that I could do both quant and work. So I did a lot of statistics. I did a lot, you know, I took a lot of qualitative courses. And in fact, you know, my dissertation was a qualitative dissertation, despite, you know, having a very deep background in stats. So I do both.

But with that said, I would say you should be methodologically agnostic. Like it should not, depending on what’s the problem that you’re trying to solve, you should figure out what’s the research, right research method. So I think that is what is more important. And that’s why I feel like having a mixed methods background is good, because then you just have more methods in your toolkit. So you can figure out like, you know, for this problem, probably a survey is a good solution. Or for if I want to find the answer to this problem, maybe doing ethnographic research, qualitative interviews is the right thing to do. And, you know, you can use all these methods throughout the process. Like you can do a survey, you know, along with some foundational research.

Like I can give you an example. When I worked at Wikipedia, I’m very proud of the feature that we released. I’ll talk about a little bit about that. So when I worked at Wikipedia, I interviewed this person in Salvador, which is, you know, in north of Brazil, in the Bahia district. And he was a Wikipedia editor. And obviously, like you’re a Wikipedia editor, you’re a big reader too. And at that time, he told me that, you know, he wanted to read Portuguese Wikipedia, but he felt that it was not mature enough. Some of the articles did not have the kind of detail that he was looking for. So he would often flip to English Wikipedia. So I thought that was an interesting thing that he mentioned. So after we did the ethnographic research, we followed it up with the survey and we asked people like, you know, do you, how many languages Wikipedia do you read? And, you know, we got data. Like if you read English, what else do you read? So obviously it was clear that, you know, English is the primary one. But more often than not, people read other language Wikipedia’s too. Especially outside of, you know, US I would say, even in Europe, like German Wikipedia is very big, but the German readers are also reading English Wikipedia. So, and that was also very clear in the survey data that we got. So we introduced, it’s even there today in the app. We introduced what we call the inter-language Wikilinks, which made it really easy to toggle between different language Wikipedia. So, for example, like if you search for an article, it tells you that, you know, this article is also available in this other language. You can set your primary languages, you can set your secondary languages. And all this came out through this, you know, ethnographic research that was done, followed it up with a survey, gave us more confidence that, you know, this would be a useful feature. And it still exists like 10, 12 years after we did that research.

Steve: That’s a great story and one detail that excites me is that small data point from one method. You find new questions in research. You found something and then you did some more research to kind of understand that phenomenon that you didn’t know.

Mani: Yeah.

Steve: That’s such a great example of that. That the ethnography was not sufficient to make something new, but it did point to a whole new effort to understand something, which then led to that feature.

Mani: Yeah, but ethnographic is that insight, which I don’t think I would have got from a survey, right? So it was, the survey just gave us more confidence that, yes, I think that’s worth building.

Steve: Yeah, that’s a good clarification. We’re going to talk about mixed methods and about different kinds of tools kind of feeding together. You talked early on about relationships, so it just makes me think about some of the different roles that we have around data science as a big part of what so many companies are doing. How do you see kind of the relationship between UXRs, mixed methods and otherwise folks and data science?

Mani: I think there’s a lot of synergy between what UX cards do and what data scientists do, because ultimately we are both researchers. It’s just that we are looking at different type of data. But if we bring it together, I strongly feel that we can have a much more holistic understanding of our users. And, you know, in my career, like I’ve had, especially at Wikipedia and Lyft, I was lucky to work very closely and actually even at Meraki, work with data scientists. So there are various ways we can work with data scientists. Like, for example, like if you’re doing segmentation, you know, often we do segmentation of our users just based on attitudinal data. But you can get the behavioral data from data scientists and bring that together to have, to just have a better segmentation model of your users. Various companies, they have dashboards to, you know, look at how their customers are doing and tracking. Many of them, especially in B2C, tend to be more behavioral based versus attitudinal. So I think there’s an opportunity there to have dashboards that not only just tell you what are the users doing, but also how are they feeling. I think that’s an important story to tell. So there’s that one can do.

Then there’s also an opportunity to work with data science during A/B experimentations. Like in my past companies, I have worked with data scientists when they have an A/B experiment to, you know, do a survey along with the experiment to see like if there are any differences between the control and the experimental group on not only what they’re doing, but how they are feeling. And the one thing that I learned is that there’s a little bit of lag. Like if people are unhappy, it takes a little while before they stop doing the thing. So like, you know, doing that survey gives you an early pulse that, okay, maybe this experiment is not working as well as it should. Maybe we should tweak something so that the users are happier. So I think that is one thing. So those are some of the opportunities that I see working with data scientists.

Steve: Is there anything else today in our conversation that I should have asked you about or you want to make sure that we talk about?

Mani: I can’t think of anything else right now. I’m sure I will think of something later.

Steve: Well, Mani, it’s been really lovely to speak with you and learn from you. I want to thank you again for taking the time to chat and share all your stories and experiences. It’s been great.

Mani: And thank you, Steve, for having me on the podcast.

Steve: There we go. Thanks for listening. Tell your friends. Tell your enemies about Dollars to Donuts. Give us a review on Apple Podcasts or any place that reviews podcasts.

Find Dollars to Donuts in all the places that have all the things or visit portigal.com/podcast for all of the episodes complete with show notes and transcripts. Our theme music is by Bruce Todd.

39. Mani Pande of Cisco Meraki

Show Links

Transcript

@DollarsToDonuts

Episodes

Praise for Dollars to Donuts