“Prepping for Execution: Metrics Interviews for Product Managers”: Tanvi Shah, Principal Product Manager at Upwork (Video + Transcript)

In this session, Tanvi Shah discusses the importance of metrics in product management and focuses on the concept of North Star metrics. North Star metrics are particularly important for prioritizing features, aligning with stakeholders, and measuring personal and company success. Shah outlines a four-step process for finding the North Star metric, which involves thinking about the business, identifying audience segments, brainstorming metrics, and narrowing down based on the stage of the business.

Transcript:

Tanvi Shah: Hi, everyone. I’m going to be talking about metrics and we’ll talk about interviews and we’ll dive into one specific topic, but let me quickly thank you, Amanda, for the introduction, but I just wanted to do a quick, better introduction here where I’ve basically worked as an engineer starting off at NetApp and then I transitioned into product management into the B2C world. Worked at a number of tech companies, both small and big, but being in B2C has helped me being a lot more metrics focused. I’ve learned a lot on the job as otherwise do. Personally, I’m also a mom. I have a six-year-old son. I also am a trained Indian classical dancer and I love reading books on the side. You might see me binging on two, three books at the same time. That’s a little bit about me.

Before we dive deep, I want you all to keep this in mind. Whatever I share today is like a toolbox. Use whatever you need, tweak it as you need it. It’s not gospel truth, you can change it. Don’t forget the big picture. When we talk about metrics, we get so deep into it, but we forget what the big picture is, so don’t forget that. Do a lot of mock practices when you’re thinking about interviews and prepping for interviews, and then run through more examples of interviews and compare it against the real world and quarterly statements of big public companies to understand what metrics they’re following. At the end of it, it’s really fun to understand metrics and to track them, so have fun while you’re doing this. It’s really important to keep the fun part of it here too.

All right, why do we need metrics? Let’s start there and then as we keep going through, I’ll ask a few more questions. Please interact in the comments. It helps me understand if some of this content makes sense or if not, we can diverge a little bit here. So why do we need metrics, quickly? We want to make decisions. We want to make projections. We want to have quarterly reports. We want to understand how an AB test works out. We want to understand what is the opportunity analysis for any feature that we’re trying to do, and that’s where metrics comes into play. There are three types of metrics, interviews generally. We talk about North Star metrics, there is a trade-off metric conversation or a diagnosis question. Diagnosis questions are not as much used these days, but I’ve still seen a few. North Star and trade-off, really big topics. North Star, there is another variation called dashboard. It’s kind of treated with the North Star here, so that’s why I’ve clubbed it together.

Today we’ll talk only about North Star because we just have a very limited amount of time. Why is a North Star metric important? Why do you think we should all care about it? Basically for three things. One, it helps us prioritize as PMs which feature is more important in the roadmap. It also helps align with stakeholders who might be talking about different metrics, and then you align metrics against company metrics and it ultimately helps succeed as a person because your performance review goals are definitely tied to this, and as a company it definitely helps them succeed.

Now, how do we find the North Star metric? It’s a four-step process. The first step is going to be thinking about the business. We’ll run through a mock question and we’ll go through the answers, but the first half, first piece here is thinking about the business. The second half of it is thinking about audience segments. These two then go into the third part, which is thinking about broadly brainstorming the metrics for the business and the audience, and then the last one, the fourth point, is to narrow it down based on where the business is at, in which stage is it, and we’ll go through all of them now.

Let’s run through this. What is the North Star metric for Airbnb? I think an example explains it better than simply giving ideas and how to do it in framework. For Airbnb, let’s start with what kind of business is Airbnb? In this case, the first thing that we want to start thinking about is the different types of services Airbnb offers or different business lines that it has, and the second one is it a B2B company or a B2C service? Again, they have business lines and what are the services here? With that, if you can take a stab at it, can you answer in the comments, what kind of services does Airbnb provide and what type of business models are these? This will help us dive into the next part. Add in the comments if you can answer the different types of services, types of business models.

Perfect. I see B2C. Yes, travel. Great. I’m seeing amazing things. One more thing to remember here is Airbnb has two major business lines. They do have the whole rental side of it. They also have experiences and that’s something that came up new. Now for each of these business lines, Airbnb travel is a B2C as some of you mentioned here, and Airbnb experiences both B2B and B2C because they do work with small businesses that are providing services. Now with that in mind, let’s talk about audience segments. How do we think about audience segments? Let’s run through a quick example.

For an audience segment example, let’s think about Amazon. For Amazon, there are three major types of audiences. We have the end consumers, there is the shopkeeper of B2B of business, and then there’s also the delivery person that’s involved in the Amazon side of things. [inaudible 00:06:22] is just an example to showcase the difference audience segment. What we are trying to do here is to then bring strong metrics. Who are the audience segments at this time for Airbnb? Can you answer in the comments based on the different business lines that we talked about? I’ll give a minute here. Who could be the audience segments for Airbnb Travel experience? Yes. Travelers host services. Yes, that’s correct. For experiences, who are the audience segments? Yes, consumers. Yes, yes, all of that is correct. I’m seeing good, but when we think about these different types of experiences, it’s necessary divided out.

For renters, there’s renters, there is the host, and for experiences, there are the experience seekers, the end users, there are hosts, and also there could be guides for physical tools or maybe there’s an in-between party that’s helped manage these services. This gives you the broad picture and now we go broad. Let’s try to find metrics for each business line and for each audience type. What we do is basically we use something like the Heart or the AARM metrics framework that’s out there to actually think about metrics for each of these audience segments. I’m going to pause here for a minute and ask again for the interaction in the comments. What are the metrics for each service type that we talked about? We have the Airbnb travel experience and the Airbnb experiences. What could the metrics be? If we go back to thinking about acquisition, engagement, monetization? What kind of metrics can we start thinking about for travel and for experiences?

Can some of you add it in the comments and then I can show what I came up with when I was doing it? Yes, number of bookings in a month, number of renters, hosts. Yes, very good. We also have, don’t forget, the Airbnb experiences. We want to ensure we are thinking of metrics for both service lines for the different audience segments. We have hosts on the platform, number of nights booked, and also visitors who are coming back. We have talked about metrics on all angles, also there’s revenue. Then on the experiences side, we are thinking about the number of experiences booked, revenue from these experiences, number of hosts, number of visitors. Again, all of the things that we did on the audience segment and the business side comes in here when we start brainstorming metrics.

Now, if we have a list of metrics, the next thing is to narrow it down to get to the North Star metric. What do you use to narrow down metrics now? We basically use stage of the product to define the company goals and to help us narrow down metrics. This is a rough framework where early stage companies are more about acquisition, product market fit goes into engagement. Growth is about user segment acquisition, engaging existing users, monetizing, and then expansion and maturity. Now, let’s think about the two business lines we talked about, Airbnb travel and Airbnb experiences. What kind of stage are they at? Can you answer in the comments again if you think Airbnb travel is growth or expansion or product market fit at this time? Yes, Airbnb travel is definitely mature. What about Airbnb experiences? Is it in the growth stage? Is it in the expansion stage? Is it in the product market stage? Yes. Airbnb experience is actually in the growth stage at this point. The comments are right on.

Mature phase, you think about monetization, retention as metrics. For growth phase for Airbnb experience, you’re thinking about engagement, monetization. Now this helps us narrow down from that bigger list of metrics to get to the North Star metric for Airbnb. Without giving away too much here, I wanted to basically take one beat here to really understand if we get the North Star. For Airbnb travel, who do you think is the North Star? We talked about a number of metrics here. Out of this, which is the North Star metric for Airbnb rentals or Airbnb experiences? Can anyone add it in the comments? Not sure would be one or two metrics at this point of time. Basically I know Priya asked what is the question? The question is trying to understand which is this one metric that really defines what should the company really aspire for?

Airbnb rentals, yes, nights booked is actually a good one. This is what it comes up with ultimately. Nights booked and revenue are actually the two metrics that they look at and they actually report this in all of the quarterly reports and Airbnb experiences is experiences booked revenue from experiences. Yes, the growth and maturity stage metrics look really similar, but I’m sure back in Airbnb they’re looking at a few deeper level metrics, but when we’re asked to report metrics, which is at a very high level, what is the North Star metric that I have to worry about as an Airbnb PM for renters or for Airbnb experiences or what is the CO looking at? These are the metrics they look at and they report on that. Basically this is a Q3 2022 readout of their quarterly report. Airbnb rentals and experiences, they reported 100 million nights booked and experiences booked 29% revenue year over year.

Now how are you going to use North Star metrics? You are going to use North Star metric as a PM most of the times to really prioritize your roadmap. But this exercise comes up a lot in the interviews and this is one of the frameworks of basically using it to get to your North Star metric in going broad and then narrowing down. I want to end it with saying that if you are interested in more metrics, if you’re interested in understanding more about trade-off and the others, well, definitely there is this … I think there’s a survey after this. Please express interest. But apart from that, these are the resources that you should look at. Lean analytics has been a Bible for all of my metrics things. I go back to it every time I interview.

I’ve heard about Gopractice.io. I have used a little bit of it. This is great for practicing and also don’t forget to mock interview with others, especially a lot of these execution interviews happen that way. That’s pretty much it. It was a quick rundown of metrics, generally takes longer, but if you have any other questions, please let me know. Amanda, I think we are right on time.

Amanda Beaty: Yes, you did great. Thank you. It’s a lot of interaction there. It looks like the audience enjoyed your talk. Thank you so much. Thanks to everybody for joining us and we’ll see you in the next session. Thank you so much.