Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• See open jobs at Indeed and check out open jobs at our trusted partner companies.
• Find more Indeed Girl Geek Dinner photos from the event – please tag yourselves!
• Does your company want to sponsor a Girl Geek Dinner in 2021? Talk to us!

Girl Geek X founder Angie Chang and Indeed Global Diversity & Inclusion Program Manager Allison Dingler from Austin, Texas kick off an Indeed Girl Geek Dinner in San Francisco, California.  Erica Kawamoto Hsu / Girl Geek X

Transcript of Indeed Girl Geek Dinner – Lightning Talks & Panel:

Angie Chang: Hello, and thank you all for coming out to Indeed Girl Geek dinner tonight. My name is Angie Chang and I’m the founder of Girl Geek X. We’ve been doing these Girl Geek Dinners at companies in the San Francisco Bay area now for a very long time, but I’m really glad that you’re here tonight for Indeed’s second Girl Geek Dinner.

Angie Chang: I’m really excited for all the talks that we’re going to hear tonight and please enjoy yourselves and meet someone new. At least one or two, maybe even three new people, get their LinkedIn, exchange LinkedIn information and maybe grab coffee later, ask about jobs, ask about jobs here. There’s always opportunities to level up and that’s why we keep doing this is because we like learning and hearing from other women in tech and other industries about things they learned on the way, and also what are the cool things they’re doing.

Angie Chang: So, please feel free to network afterwards. We also have things like a Girl Geek Podcast, in case you would like to find us on iTunes and all of the different podcasting services, we have a podcast. We also have a conference coming up. It’s a virtual conference we do every year for International Women’s Day. That’s going to be in March, so stay tuned. But I want to turn the microphone over to Alison, but say thank you so much for hosting us, Indeed.

Allison Dingler: Thank you. Thanks, Angie. All right, awesome. Thanks so much, everybody, for coming. This room is so packed, I love it. Yes, everybody excited? Yes. Ooh, energy. I’m about it. I’ve had a lot of caffeine today. I’m going to kick it off and start with our first tech talk of the evening.

Product Manager Lindsay Brothers gives a talk on “A/B Testing Pitfalls and Lessons Learned” at Indeed Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Lindsay Brothers: Hello. Beautiful. Hi, how are you doing? Good. It’s Tuesday. Okay, so I’m going to be talking about “A/B Testing Pitfalls and Lessons Learned”. Experimentation. It’s how we learn about the world around us. It’s something we started doing very early on as humans. It’s something we start as babies. How do we learn as babies? Well, we run the original A/B test, which is stick things into our mouths. And we learn, maybe we have a question, a hypothesis, is this edible? I can eat this.

Lindsay Brothers: Unfortunately, sometimes not so successful. It turns out to be dirt, and that test was not a success. Some tests fail. And other times when babies run this experiment, this A/B test, congratulations they’re tacos. Yay, it was a successful test. And we get to celebrate, we’ve learned something new and we ate tacos.

Lindsay Brothers: I’m Lindsay Brothers, I’m a product manager at Indeed. You can follow me on Twitter @LindsayBro. You may know Indeed, number one job search site worldwide. Who here has heard of Indeed? Everyone raise your hands. And who’s gotten a job through Indeed. Yes. That’s awesome. I love it. Yeah.

Lindsay Brothers: Okay, a little bit about Indeed so you have context for this talk. We help people get jobs. This is our mission. This is something we cared deeply about. And some context about just how big we are. We have 250 million unique visitors, 150 million resumes, 600 million salaries, and 25 million jobs. So, a lot of job seekers looking at a lot of jobs, millions of job seekers, millions of jobs.

Lindsay Brothers: We’re a very data driven company. We want to learn about job seekers and how do we do this? Well, A/B testing. What is A/B testing? A/B testing is a randomized experiment in which a new variant is tested against a control A to measure how they perform relative to each other. I just realized I don’t have a timer. Cool. So at any given time, we’re running hundreds of A/B tests to learn about job seekers, which leads to thousands of experiences. We’re running hundreds of A/B tests, which leads to thousands of experiences at any time.

Lindsay Brothers: Which means the person sitting next to you likely sees a very different Indeed than you do. This leads to many different lessons learned. We’re constantly running A/B tests and we’re constantly learning. Now this test, this talk is really about things that have gone not so well, pitfalls. We run a lot of A/B tests and we’ve made some very expensive, painful mistakes that I want to share with you.

Lindsay Brothers: The first pitfall, your metrics don’t matter. Second pitfall, big test, big failure. Pitfall three, most tests fail. And pitfall four, where does vision fit in? As we go through these pitfalls, I’ll share lessons learned. Let’s dive into the first pitfall.

Lindsay Brothers: Your metrics don’t matter. Or in this case my metrics didn’t matter and no one cared. This is a job alert, and you’ll hear more about Job Alerts in the next talk, as well. But this is the most common email we send out. It’s new jobs based on a query that a user has run. In this case, it’s a email developer jobs in San Francisco. Done a search on Indeed, signed up for this Job Alert, and they’re getting new jobs in their inbox, so they can apply to them as soon as they’re posted.

Lindsay Brothers: I was on this team as a product manager on this team and I was highly motivated to get more subscriptions, to get more people to sign up for these Job Alerts. This is an Indeed apply form. If a job is posted on Indeed, you go to apply for that job, you’ll see this form and you’ll apply directly on Indeed. What I wanted to test was adding a simple checkbox. Notify me when similar jobs are available. A job seeker will just check this check box, and the next day begin to receive new jobs in their email inbox. Pretty simple, right?

Lindsay Brothers: A versus B, no checkbox, that’s standard form, versus the test group with that checkbox. Now, it looks simple. It’s just a small change. But this test had a tremendous amount of data. Sure, it’s just one variable in a single test group, but look at all the things that are on this form. You can be logged in, you can be logged out, you can use your Indeed resume, you can attach your resume, you can attach a cover letter, you can add a phone number. So there’s all these different things you could do, just in a single form. So we had a lot of data to analyze.

Lindsay Brothers: And the first metric was really bad. We saw 0.1% decrease in applies. So just adding that single checkbox, less job seekers were going to fill out that form and apply to that job. And this was really scary. Not good. So there’s another team that exists, indeed.com is not just the Job Alerts team. There’s a team called the Indeed Apply team and they own this form, and their metric for success was the completion rate of this form. They wanted job seekers to apply to jobs on Indeed and to finish this form.

Lindsay Brothers: So, 0.1% may not sound like a lot, but we’re talking about millions of job seekers, millions of jobs. That actually can equal hundreds of thousands of applies lost by adding a single checkbox. So this team says to me, “What are you doing? Okay, we’re here to help people get jobs, not email subscriptions.” Yeah, I got some angry emails about that. This was our secondary metric. This was a metric I was keeping an eye on during the analysis of the tests, but it was their primary metric. It was what they cared most about, and they didn’t really care that I was getting more email subscriptions. I was getting a lot. Come on, I can put this on my eval. 50% increase in subscriptions. That looks really good, but they don’t care.

Lindsay Brothers: It felt like a massive win but we were losing applies. So, the question was, are we helping people get jobs or are we just helping them get email subscriptions? So, what is the impact of this test? Now keep in mind, they click that checkbox, they’re getting that job alert, they’re getting that new subscription, and there’s a lot of jobs in here. So, they got to be doing something with those jobs.

Lindsay Brothers: Well, let’s start looking at the standard email engagement metrics. They’re getting this new subscription, how are they engaging with it? Well, okay, they’re less likely to open it. That’s awesome. So, job seekers were checking this check box, maybe less likely to apply and less likely to open this email. And they were less likely to click through by 12%, so open rate was 12% lower, click through rate was 12% lower. Am I tricking people into this email? Is this spam? What am I doing?

Lindsay Brothers: But we had to look further down the funnel and what we found was the apply rate was actually higher. So, job seekers were less likely to open it, they’re less likely to click through, but they were more likely to apply to a job in that email by 0.25%, and this was very, very exciting. This meant that we were actually helping people get jobs. But we had to figure out the total impact. So had to do some math and we had to figure out the total downstream applies, more subscriptions and looking at that higher apply rate, how many additional applies were we getting? Was it making up for those lost applies? And it was, we were getting millions of additional applies from that email subscription. So, it was a success. Yeah, it looked real good on that eval.

Lindsay Brothers: Something to keep in mind is that looking at only short term metrics, that makes [inaudible]. Looking at only short term metrics can mean missing downstream impact. And they really didn’t care about getting additional email subscriptions, which of course they shouldn’t. But we were really aligned on the impact of applies and the power of applies. So, the first lesson I’d like to share is that downstream analysis is a powerful tool, but it does take time. It took time to figure out that we were having a higher apply rate in that email and we were getting those downstream applies. Cool.

Lindsay Brothers: So, second pitfall, big test, big failure. In 2017 this is what indeed.com looked like. And we were due for a facelift. The UX design team really wanted to update this and improve it. This was the vision. This is where we wanted to go. We wanted to modernize Indeed. Again, this is 2017.

Lindsay Brothers: Generally, when you’re doing an A/B test, you have a change and you have a result. You’re changing something on the product, a small variable change, then you see results. Well, okay, now keep in mind this is the entire search results page of Indeed. There’s a lot of things going on, and it was a really, really massive test. We were looking at, we were testing this old version versus this new version, and lots and lots of changes. Okay, lots and lots of changes going on on the site and then more changes we’re starting to see lots of metrics, and then there was lots of results, and more results, and more results and we couldn’t quite figure out what was going on and what changes were causing what metric going up or down. And it got really messy. It looked like that.

Lindsay Brothers: We had changed too many things at once. We wanted to run this massive A/B test where we were updating the old Indeed with this new, beautiful, massive, redesigned, gorgeous. Let’s skip ahead. Let’s go with this big vision. But we had changed too many things at once. And that meant our A/B tests were losing, metrics were going down, but we didn’t know why. And this is really expensive. Now, keep in mind, this is redoing the search on Indeed, that involves a lot of engineering effort.

Lindsay Brothers: We had to go back to the drawing board. 2017, we wanted to test this brand new, beautiful redesign, let’s modernize Indeed. So let’s start out where we started. We had understood when we were doing this test that too many changes at once meant we didn’t really understand what was happening. So we had to start from scratch. We had to really just redo this whole thing. What we had to do was test a single element at a time.

Lindsay Brothers: Now, this is a job card, so you do a search on Indeed, you’re going to see these job cards, which is job title, company location, maybe salary, some additional details. And this is an example of how we could test a single element at a time. You’re like, what’s changing here? It’s the spacing. So this is spacing as a single element. So a single variable in A/B test.

Lindsay Brothers: Another thing we had to do, like I had mentioned, we have this old older design that we wanted to update and originally it was just, let’s test all these elements at once. Something we also had to do was we had to switch to multi-variate tests. And so I say multivariate test, what do I mean? An A/B test in which all possible combinations of variations are tested at the same time.

Lindsay Brothers: Now let’s go back to that job card. We wanted to test a single element at a time. We want to break up all those elements so we can understand their impact. But we also want to A/B… the multivariate testing so we can go through all these different combinations. Now this is a job card. We got job title, company, location. There’s all these different elements to test. Well, let’s dive deep into a single element, which is salary. Let’s look at salary. Salary important. We like to make money at our job.

Lindsay Brothers: This is a single test where we’re just testing the element of salary, but what we’re going to do is a multivariate test to really dive deep into the UI of it. This is control, font size, 13.33 pixels. It’s not bolded and the color is gray. So, one element of the multivariate test is the font size variance, 12 pixels, 13.33, 14, and 16 pixels. Another part of the multivariate test is font weight, regular versus bolded, and finally color: gray, black, orange, and green.

Lindsay Brothers: That’s a lot. So we got four sizes, two weights, four colors. This is 32 groups. And were there spreadsheets? You know there were spreadsheets. I love spreadsheets. So, you’re like, whoa, that’s a lot going on there. Now, if we had not done a multivariate test, if we’ve just done A/B tests, so color as a single A/B test, font weight as a single A/B test, size as single A/B test, it would… like, here’s the control. Okay, that’s control. These would just be the groups.

Lindsay Brothers: So colors, one group, size, one group, you would only have eight different groups. But when you do multivariate testing, you get, you miss 24. So, 32 groups, well that’s a lot. But you get to explore how these elements play against each other. And we would have lost our winners. So if we had only done tests around those single UI elements one at a time, we would have totally missed these. And we would have not picked these. No UX designer would have picked these, because this one looks like Hulk. We call this one Hulk. It’s big. It’s green. It’s bold. No one was going to go with that. But we learned from this and we learned a lot.

Lindsay Brothers: Now, multivariate tests, of course, have some challenges. Well, you need sufficient traffic. I mentioned 32 different groups. Okay, you need enough traffic to learn anything from those groups. Also, significantly more complex analysis. So there’s a really good blog post. Robyn Rap is a data scientist on the Indeed engineering blog. You can look it up. And she actually talks about this specific test, and there’s literally equations in the blog posts about how to analyze it. It’s a lot of math. There’s some really strange combinations that you can get from multivariate tests, like the Hulk. No one was expecting that. We’re like, “All right, what do we do with that?” Okay.

Lindsay Brothers: But it really helps you optimize your UI at a very high velocity, which is really, really cool. We’re learning fast, we’re moving fast. And since August of 2018, we’ve run over 50 tests with over 500 groups, just on that search results page alone, and we’ve learned some really surprising things. Now remember when we started this test, it was old design, new design. All these changes and we couldn’t see what was going on. Metrics going up, metrics going down, not understanding the impact. But we saw really surprising things when we broke it up via by element, and then broke it up into multivariate tests so we could really learn quickly.

Lindsay Brothers: And something we totally missed was that changes were way more impactful on mobile. This data point lost, totally lost in that confusing analysis where everything was going up and down. But we had missed this, so when we broke up the elements, we learned more about the UI changes and their impact.

Lindsay Brothers: So, the lesson here, again, this was a very, very, very expensive mistake to make. We thought we could just skip ahead, come up with the new design. Let’s go there. Oh, we’re so modern. Wrong. Very, very expensive. So when we changed to testing small changes and testing them quickly, we learned a lot more, faster. Which goes directly into the next pitfall, which is most tests fail. And at Indeed, 70% of A/B tests fail. That means there’s not a clear winner. We know we put out this test group, it’s not amazing. It’s not going up. Whatever your metric is, it looks bad.

Lindsay Brothers: It’s really sad. It’s like, what’s the point of this? Why am I here? Why am I running all these tests if they all fail? Why did I make my engineers build this or build this test if it just… It’s just sad. Want to hide under a blanket. There are three different reasons, really, to run A/B tests. We can get faster wins, we can get a better design and we can get a better understanding of the product. So how exactly do we get to these things?

Lindsay Brothers: Well, let’s say we run a test and the test is positive. Your metric goes up. Well, awesome, you have a KPI win. Woo hoo. That’s great when that happens. Or okay, let’s say the test is neutral. It did not change. And this was actually quite common in that test I just discussed, where we were testing all these around elements, all many, many multivariate tests, spreadsheets galore, lots of neutral stuff. That actually means there’s a lot more design flexibility. So if it’s neutral, it means you can play with that element. You can maybe make it a little more prominent or you can do things with it. It’s not a fixed element. It’s more, you can be creative with it. And so that’s actually really exciting.

Lindsay Brothers: And, of course, if something’s negative, what we found is, okay, don’t touch it. Do not touch that element. Like I said, we broke up all these different elements. We’re testing many different variations of these elements, and some things came back consistently negative, and we’re like job seekers like that, we’re not going to do anything with it. And so you’re still winning. Even if a test is neutral, you learn that you can do things with that element. If it’s negative, don’t touch that element. Move on, try something new.

Lindsay Brothers: So, it is successful. Sometimes it’s disappointing when it doesn’t go the way you planned or your hypothesis is totally off, but we do get wins from this. And the lesson here is to change what winning means. So, even when we’re running all these different A/B tests, the metrics aren’t looking great. You’re still learning about your users, are still learning about, for us we’re learning about job seekers and how they interact with Indeed and how they look for jobs.

Lindsay Brothers: So that goes directly into pitfall four, where does vision fit into all of this? We have this big vision. Okay, so let’s go back to that test. This is where we were. So this is again that job card, you do a search on Indeed. You see all these jobs. This was the first iteration after testing. There’s a lot of stuff going on there. And this was the vision. We like to empower design at Indeed. We love to have visions. We love to think about where are we going with things. But when you’re doing a lot of A/B testing and, where does this all fit in? I’m running all these different, I’m testing all these elements. I’m trying to learn. I have these hypotheses. But where does the vision fit into this?

Lindsay Brothers: Well, even failures can inform design vision. Let’s go back to… Now which one is this? Okay, so this is the vision. This is the vision we originally started with, 2017. This is our vision. This is where we wanted to go in 2017. And we started to remember that first big test, not so good. Then we started to break up elements, did multivariate testing, lots of spreadsheets, and we started learning. We got lots of negative and neutral and positive, and we learned things like no blue or underlined needed on the job title. Doesn’t matter if you have that, job seekers are still going to click on the job.

Lindsay Brothers: We can add more spacing. Like I mentioned, spacing was a single A/B test where we’re looking at that element and playing with spacing. We can add more white space. We can do more there. Salary needs to be more prominent. Like I talked about that salary test, we could make it big and green and bold and people love it, they’re clicking through. Do not touch this. If you can apply on Indeed, that Indeed apply little tag, if we changed it, negative. Any change, negative. Do not touch it, keep it there. Font size. We played with font size, of course, the font size was way too small.

Lindsay Brothers: So again, this is where we started after that first iteration, that first iteration of testing, this was our vision, but then we ran all these tests, testing all these different elements, many variations, and we ended up here. Now, you can see salary’s a lot more prominent. There’s more spacing. We don’t need that underline. We played with underline, you don’t need that. So, as we learned about these different elements, tests were negative, tests were positive, tests were a lot were neutral, we were able to take these learnings and put it back into the vision. So we had a vision that more aligned with how job seekers were using our site.

Lindsay Brothers: Sometimes it can feel like a vision is just this beautiful thing you create and it doesn’t always align with how people use your products. And with A/B testing and with learning from many, in fact, failed tests, we were able to take that back and have a vision that aligned with how job seekers use Indeed. So it’s not just a design win, it’s also a business win. We had a lot of KPI wins, many tests not so great, but we did have ones that were delivering KPI wins and we’re able to implement that back into the vision.

Lindsay Brothers: So the lesson here is both, of course, the design vision guides your testing. We had UX designers who were thinking about where Indeed should be, where we could go, and that led us to define some of these tests. But also those A/B tests, as we’re running them, went back into the vision. So, to recap, your metrics don’t matter. Remember we ran this amazing checkbox test and it looked horrible. At first, applies were going down on that page. So, we had to align our metrics, we had to agree on those downstream applies. And downstream impact can take longer. It can take much longer to pan out, but it can prove really valuable.

Lindsay Brothers: Next up, big tests, big failures. That’s when we ran this massive redesign of the Indeed search, and we thought we could just skip ahead from this old design to this new beautiful design. We were wrong. That was big mistake. So, breaking up tests really helps understand impact. So breaking up these tests into different elements, multivariate tests, those multivariate tests really help us understand the different UI elements. And most tests fail. So indeed 70% of tests fail and that meant that we had to redefine winning.

Lindsay Brothers: So, failed test just means there’s lack of flexibility. That element is important to users. They care about it, maybe don’t change it. And a neutral test means that there is flexibility. So, if something’s neutral, maybe we can do more with it, maybe we can play with it. And then finally, where does vision fit in? So, that big redesign test, we had a vision of where we wanted to go. We’d tried testing it all by itself, didn’t work, but we were able to use it to guide those single element tests, the multivariate test, and it helped us define a test plan. But also as we ran those tests, it helped us inform the vision and we adapted our vision to what made more sense for our users and our job seekers.

Lindsay Brothers: So my question for you is, where will your testing failures take you? If you run A/B tests, if you run tests with your users, they will fail and things will go wrong, and you will run tests and will feel like an engineering waste of time. Oh boy. But there’s still learnings to be had, so where will those failures take you? Thank you so much for your time. Please clap. Here’s my information, so feel free to shoot me an email, LindsayB@indeed.com or Twitter, @LindsayBro. If you tweet, I get metrics. Awesome. Thank you so much. So Allie is going to give a little intro.

Allison Dingler: Thanks, Lindsey. Everyone give it up one more time for Lindsay, A/B testing. Yes, I’m here for it. Awesome. Everyone having a good time so far. Getting some good food in your belly, some good drinks, some good friends, yes? Awesome. So I’m going to kick it off. We have our next tech talk that’s about to get started. We have Janie and Rohan here, so I’ll have y’all come on up and get you going. Do you want another microphone?

Janie Clarke: Yeah.

Allison Dingler: Microphone.

Janie Clarke: Thank you.

Rohan Kapoor: [inaudible].

Janie Clarke: Can you get a timer?

Allison Dingler: Timer.

Senior Product Manager Janie Clarke gives a talk on “AMP for Email” at Indeed Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Janie Clarke: Hi everyone. My name is Janie Clark and I’m a product manager here at Indeed. I’m here today with Rohan Kapoor who’s a software engineer on my team, and we’re here to talk about our adventures with AMP for Email. Did you click? All right, there’s a little bit of content here that might be repeated from Lindsay’s talk. As you probably just learned, we are very data driven company here. We test a lot of things. And Rohan and I work on Job Alerts, which Lindsay also used to work on.

Janie Clarke: It’s an important product for Indeed. I assure you it’s not the only product, even though we’re talking about a lot tonight. Job Alerts is a big product. We have over 250 million active Job Alerts subscriptions. We send it to over 60 countries in 28 different languages. And we do a lot of A/B testing within this email. At any given time we have dozens of A/B tests running.

Janie Clarke: You’ve also seen this before. This is a Job Alert email. Basically it’s a way for job seekers to get the newest jobs emailed to them. We’re here today to talk about our adventures in AMP. So first, I’ll give a quick background about traditional email development and some of the challenges that we had faced with it, and we were hoping AMP would solve these challenges for us. And then Rohan will go into some technical considerations and obstacles that we faced and how we solved them. And finally, we’ll share some of the results of our testing so far.

Janie Clarke: So first up, traditional email development, or why email development sucks. Do we have any email developers in the room or anyone who has worked on it before? Woo! Email geeks. So you may have heard, email development can be difficult. Because emails are sent out as static HTML content, it has to be able to support whatever old arcane email client that your users may be using to read their email. Outlook 2007, being a good example of this.

Janie Clarke: Many different email clients have their own special rules about what kind of content and markup they require and can render. And if your inspect the source of an email, that in your inbox, you may notice that it’s a mess. Lots of nested tables, inline CSS with special rules for different Outlook versions. Again, some of you may have noticed this before. But Outlook aside, even modern email clients like Gmail have issues of their own.

Janie Clarke: So remember, at Indeed our mission is to help people get jobs, and on our team we do that by sending Job Alert emails to millions of job seekers every day, so that they can get the latest results for their search. We know from extensive experimentation on the site, as you know, and also within our email, that job seekers get a lot of value from personalized contextual information about the jobs that they’re looking at. Providing them with this information helps them make better decisions about which job they want to click on. It helps them apply to the right jobs and it helps people get jobs.

Janie Clarke: But whenever we do a test in the Job Alert email to add this information, we run into a problem. And this happens. Have you seen this before in Gmail? The email clips, since traditional email cannot load any external content, no JavaScript, no external CSS files, all the content has to be static and contained within the HTML. And along with all those nested tables that you need for Outlook and custom inline CSS, for the older email clients, it can be a real challenge to squeeze in all of the information that you want.

Janie Clarke: So it’s something we’re constantly trying to find the right balance on our team. I’m including as much information and as many jobs in the email as we can, while also not running into clipping. So there’s a certain size limit that Gmail hits, when the email will clip if the HTML is over that limit. So we’ve tested adding more jobs to the Job Alert, more content always leads to more clicks, more engagement, more applies. But the more jobs we add, the more it clips. Right now, about 5% of our emails clip.

Janie Clarke: Another problem we’ve run into with Job Alerts is how to include the most Up-to-date information. I’m going to go into a little bit about sponsored jobs. A sponsored job is one where the employer is paying to promote that job and get it in front of job seekers. When we’re sending Job Alerts, we send out both sponsored and organic jobs. We will not send the job alert if there are no new organic jobs, but we do allow the sponsor jobs to be a little bit older.

Janie Clarke: And the thinking there is that, if the employer is paying to get this job promoted to more job seekers, they’re actively trying to fill that position. So, we want to help them do that by including the older job. But it’s also a challenge because job seekers can open their email hours or even days after we send it. And if a job is older, it’s more likely to be closed by the time they see it. If you look closely at this screenshot, you might notice a difference between the job at the top, which is sponsored, and the organic job below it.

Janie Clarke: In our Job Alerts, we worked around this problem in a pretty clever way. When the user opens their Job Alert, a request will be sent to our server to fetch the latest sponsor job for that slot and that alert, and a screenshot would be taken and we would render it into an image. So we call this image ads. Image ads were really clever workaround for the problem of how to show the latest sponsored jobs, but they came with their very own problems. So, every email renders HTML a little bit differently. Many of them render poorly, and different clients have handled different images in very strange ways.

Janie Clarke: So some problems that we’ve faced with this are giant sponsor jobs or tiny sponsor jobs, grainy images, you name it, we’ve run into it and fixed it. So getting image ads right was something we had struggled with on our team for a long time. And because of this, when Google first announced AMP for Email, using it to replace image ads was the first thought that we had. We are also very excited about how AMP for Email wouldn’t necessarily need to include all of the older markup that’s required by older clients because it’s only supported by a certain newer clients. Now I’ll give a little bit of background about AMP.

Janie Clarke: AMP is a web component framework that is used to help create interactive websites, stories, emails and ads. AMP is designed to create a user first experience, which they define as being mobile first and loading fast. And it’s especially helpful for users on poor quality connections because they try to load the most important content first. AMP also does not allow for content to change positions once it’s loaded. Meaning that all the page elements have to have a fixed width and height.

Janie Clarke: The reason for this is they want to avoid the page jumping around as it loads, which you’ve probably noticed a lot on the internet is the thing that happens, especially on slower connections. And AMP achieves these goals by providing a set of predefined components that you can use to build web pages. AMP for Email is a way to offer email users an interactive experience within email, by allowing certain AMP components to be used in email. It brings a lot of modern app functionality directly into emails that has been impossible before.

Janie Clarke: If you use Google Docs, you may have seen this email, which allows you to reply to a comment directly from your inbox without having to leave. It’s pretty amazing. I use it every day. It’s a killer use of the AMP functionality. Some of the benefits of AMP for Email are, it can dynamically load content from a remote server, not just images, but also text. Users can also submit forms and information to a remote server. So users can submit feedback and content to you. And lastly, the layout can change as the user interacts with it.

Janie Clarke: So in some ways this allows email developers to build an entire web application inside of an email. AMP for Email allows for a level of customization that has literally never been possible before in email. It’s very pretty exciting if you’re an email person like me. So back to our use case. Google opened up the AMP for Email preview in April of 2018, and we jumped on the opportunity to participate in the developer preview. The timing was really good because we had an intern starting for the summer in May, and at the time Google expected their launch to be around September. So it would be perfect for an intern project. He’d be able to see his functionality go live, sound really great. So we assigned him the project of creating an AMP version of the Job Alert that loads the sponsor jobs using AMP.

Janie Clarke: Now the reason we assigned it to an intern, it was hard to justify putting full time resources from the team onto this, due to the large time frame before launch and the general unpredictability. And this has been consistently a challenge when working with AMP, which we’ll go into more later. So now Rohan is going to talk about what it was like to actually work with AMP.

Rohan Kapoor: Before jumping into AMP, I wanted to take a second to talk about emails traditional development. Traditionally emails have had two MIME types, the text MIME type and an HTML MIME type. So a modern email client, such as a Gmail or Outlook on the web, that can support HTML will read HTML, and text only clients that run into terminal, something like Mutt, will read only the text part.

Rohan Kapoor: So then came AMP, which was implemented as a new third MIME type. Clients that support the AMP MIME type will read the AMP part and display that, while other clients will fall back naturally to HTML and text. So, there’s full backwards compatibility, there’s no risk that an email client will display incorrect content or garbage just because you start sending AMP.

Rohan Kapoor: As Janie mentioned, one of the biggest considerations that Google had when they built AMP was mobile first experiences. In a mobile world, it’s quite common for the user’s device to lose their connection as they’re moving around from place to place. And on the web, AMP works around this by using AMP caching and caching some components on the user’s device. This reduces the network traffic required to load pages. But, we’re talking about email. And in email, developers have to send fallback content like above, which can be used if the network request fails to return data. So in this case, this email failed to load this data, and so this data that was preloaded as fallback content shows up so that there’s not a giant white space where it should be.

Rohan Kapoor: With AMP’s emphasis on user first design and development, there’re some imperatives that we found particularly tricky to work with. For example, AMP requires that all content has an integer valued width and height. For an email system, that means that at the time you’re sending it, the system needs to know what the width and height of all of your dynamically generated content will be, even though that content doesn’t exist yet.

Rohan Kapoor: So, in our case, we fetch text like the job snippet dynamically. And it can lead to situations where the content is too long and gets truncated. See the text sponsored on the image on the slide. Or too short with extra padding on the ends. And on this slide we can see that the difference in height is pretty remarkable between the job in the middle and the one above and below it.

Rohan Kapoor: Now AMP also suffers from the clipping problem that Janie was mentioning, but it behaves very differently than the traditional HTML email. If the AMP MIME type is larger than 100 kilobytes, the email client silently drops it and falls back to the HTML MIME type. There’s also additional limitations on the size of the entire MIME tree that can cause the whole email to just disappear into the void. It’s also important to keep in mind that this is still an email. Everything is still happening inside an email client, inside the user’s browser. And UI elements such as lightboxes may not work exactly the way you expect them to.

Rohan Kapoor: So we had created mock ups for replicating an Indeed view job page inside our Job Alert email. This was a little bit of a slimmed down version, just because you know it’s running inside an email. And the idea would be that the user clicks on the job and instead of leaving the email, inside the email themselves, inside the email itself, they can view the job description. However, it didn’t quite work the way that we expected it to.

Rohan Kapoor: So the video loads, so you can see, when you click on the job at the top, you get a normal job description. But as you scroll down and then click in, it never scrolled up. The job description is up there. And if you go all the way down to the bottom and then click in, the job description is gone. But actually it’s all the way up there. Because of the way AMP content is rendered, for security reasons it all runs inside an iframe. It has no idea what the viewport is and so lightboxes don’t quite work the way we would think they would.

Rohan Kapoor: So we emailed the team at Google, filed a ticket, and a little bit of back and forth happened, and then they closed the ticket, and said that they’re going to remove AMP lightbox from the list of email approved components because they couldn’t find a way to make it work. So, at Indeed, we’ve built our own email service provider or ESP. And so our journey with AMP begun by adding support for sending the AMP MIME type through that platform. As I said earlier, the AMP MIME type, oops. The AMP MIME type is backwards compatible, so any Indeed application that doesn’t support sending AMP would have no change in behavior. But any application that is sending AMP is received by a client that supports AMP, everything will work fine.

Rohan Kapoor: We also ran into a bunch of specific challenges while working with AMP, and there’s some interesting workarounds that we wanted to share. As Janie mentioned, we started working with AMP during a developer preview period, and at the time AMP for Email was considered bleeding edge. As many of you may know, when you’re working with bleeding edge software, sometimes you have to be ready to bleed. One of the biggest challenges that we ran into was that AMP specifications changed a lot during this time, and the documentation frequently lagged behind. And here’s one such story.

Rohan Kapoor: One day a QA in our team reached out to me and he was telling me that none of our AMP emails were working anymore. And keep in mind that these were emails that were in his inbox from yesterday and worked fine yesterday, but it’s dynamic. So what worked yesterday may not work today. What happened was that where the sponsor jobs were supposed to be, there was just a large white boxes. So we opened the developer tools, looked in the error console, and it’s full of incomprehensible red error text. And all of the error text is minified, so no idea what any of it means.

Rohan Kapoor: So we reached out again to our developer contacts at Google, and they told us that, “Oh yeah, they have now enforcing Gmail-specific CORS headers. The documentation doesn’t come out yet, but it’ll be there in like a week.” And they sent us a quick and dirty version, but basically we had to now add the Gmail-specific CORS headers. That’s an example that we use, and without those all the Ajax requests to fetch the content failed.

Rohan Kapoor: So AMP initially required that the domain that it sends requests to matches the email sender domain, and at Indeed we use alertatindeed.com when we’re sending Job Alerts. However, jobs can live on a variety of different domains depending on what country they’re for. So jobs in the UK live at www.indeed.co.uk, for example. You’ll notice that indeed.com, where the email is coming from, and indeed.co.UK where the job is hosted don’t match.

Rohan Kapoor: So, one possibility that we had was to change the sender email address to match the job domain, but this raised another potential issue for us. It could dilute the sender reputation of the indeed.com domain. So, for those of you that send a lot of email, you’re probably familiar with the concept of sender reputation. Basically, all of the incoming email providers like Gmail, Outlook, Yahoo, have a score that they assign to emails coming from your domain. And switching to a new domain like indeed.co.uk would have no sender reputation and we’d lose all of the reputation we had from indeed.com. So this was not really something we wanted to do.

Rohan Kapoor: What did we end up doing? Well, the most straightforward solution that we could come up with was creating a proxy web app. This web app lived at an indeed.com sub domain. In our case it was called ampxy.indeed.com, and it accepted a base 64 encoded URL that told it which domain to actually go fetch the job from. And it handled all of the AMP security validation, handled the CORS headers, and then did the request to the real domain to get the data and passed it back. We called this solution AMPXY for AMP Proxy, and it effectively allowed us to perform cross-site origin requests while masking them, so that as far as AMP and the Gmail client were concerned, we were still hitting indeed.com.

Rohan Kapoor: Ironically, a few weeks after we built this solution, the team at Google reached out to us and told us that based on our feedback, they were removing the same domain requirement. Again, bleeding edge software developer preview all of that. Well, fortunately there was some other benefits that AMPXY gave us, so it wasn’t all just wasted effort. AMP for Email doesn’t allow for any redirects, and if redirects are present, requests fail automatically with no error. AMPXY allows us to proxy the redirect on the server side and once again hide it from Google and make it look like everything’s fine.

Rohan Kapoor: Second, QA testing. So, since AMPXY exposes a single endpoint, we can put that through the QA firewall and have a very simple, straightforward way to test in QA rather than having to open up a bunch of different end points for various functionality. But there’s one significant downside: URL lengths. Because we’re using a base 64 encoding, your URLs ended up almost twice as long as before they started, which makes email clipping much, much worse. In the future we’re planning on building a URL shortening system, which we will integrate into AMPXY, which will hopefully allow us to have shortened URLs and use them in AMP and everything will hopefully work, but it hasn’t been built yet.

Rohan Kapoor: Unlike regular HTML, email AMP does have strict validation requirements. If AMP content validates, it shows up. If it doesn’t validate, you get HTML content instead. And this doesn’t actually show up in any way that you can see. So, one such scenario here is that the AMP spec doesn’t allow you to import functions and then not use them. We use AMP list components for sponsored jobs, and every time we have a sponsor job, it goes in its own unique AMP list.

Rohan Kapoor: So if we have no sponsor jobs at the time, we’re rendering an email, then we don’t use any of our AMP list components, but the import was still there. We learned the hard way that we had to remove the import, and so now it’s wrapped in a nice if-statement to make sure that we don’t fail validation that way.

Rohan Kapoor: As I mentioned, when emails fail validation, there’s no reporting back from the Gmail system as to why it failed or the quantity that failed or anything like that. There is however, this very convenient developer sandbox where you can copy and paste in your entire email, and it will tell you line by line what you did wrong. So, in this example, the tag image is disallowed, because AMP doesn’t allow you to use images, you have to use AMP images instead. Some dynamic caching magic stuff, I’m sure.

Rohan Kapoor: So as a result we are adding AMP validation to our internal ESP because as you’ve probably heard, Indeed is a data driven company. We love our metrics. We want to know how many of our emails that are leaving fail validation and why and hopefully correct them.

Rohan Kapoor: One last thing with AMP is that there’s additional security requirements than traditional email. It’s a trend here. You take traditional email and then add a bunch more requirements and then you get AMP. So to pass AMP validation, you must always pass DKIM, SPF, and DMARC, and email is must always be encrypted in transit using TLS. If not, they magically disappear into the ether. We learned that one the hard way, too.

Rohan Kapoor: So, rounding up the list of important considerations with AMP is that users probably aren’t reading your email exactly at the time you’re sending them. So Gmail supports rendering AMP up to 30 days after the email has been sent, at which point it will permanently switch over to the HTML content forever. So this means that as a developer, when you’re sending out your emails, all your URL endpoints, all of your paths, all of that has to be valid for 30 days, otherwise the email fails. Another use case for fallback content, which can be displayed if those requests do fail.

Rohan Kapoor: So, few technical takeaways. Working with bleeding edge software is hard. Specifications change frequently and you have to be willing to adapt at all times. You have to plan for fallback content with AMP. You are in a mobile world, network connections change all the time and you don’t want large holes in your email where the dynamic content was supposed to be. And you have to find a way to work around the fixed width and height limitations. Basically, you want to make sure that parts of your email don’t clip internally. There’s no giant white spaces, so find some sort of an easy medium. Make sure all of your content that’s coming dynamically is capped at that limit.

Rohan Kapoor: And now I’m going to hand it back to Janie to talk about some of our tests results and conclusion so far.

Janie Clarke: Thank you. How is our test going so far? You learned all about A/B tests before. We test everything, and AMP is no different. We are running AMP in an A/B test right now, that’s targeted at gmail.com users only. So the control group does not send the AMP MIME type at all and the test group does send the AMP MIME type. When we are also testing a few other little functionalities, but the main thing we’re testing right now is the sponsor jobs. So far when we look at our test at the aggregate level, we’re not seeing much change in our test group, and there’s a reason for that.

Janie Clarke: When we compare emails that were opened as AMP to emails that were not opened as AMP, they were just opened as regular HTML, we do see two times more clicks in the AMP opened emails, which is a really great promising early results. So, the reason for this is mobile. Google has not started to roll out AMP for mobile gmail.com yet–for the Gmail app. And in Job Alerts, 82% of our opens are on mobile. So that means most of the users that are getting that AMP MIME type aren’t seeing it. So since AMP is currently not supported on mobile, our test results are pretty limited.

Janie Clarke: It’s hard to spot many behavior changes when we look at the test at the aggregate level. And so we’re waiting right now for Google to start rolling out the AMP functionality for the Gmail app, so that we can really see how it does at full scale. We’re in a little bit of a holding pattern right now, just watching and waiting. We do have some future ideas for features that take advantage of AMP that we’re really looking forward to testing and we’re working on them right now. One of them is an interactive NPS survey as shown here, so we can show NPS right at the bottom of the email and users can answer the question, even type in some feedback for us without even leaving Gmail.

Janie Clarke: We also working on an interactive unsubscribed surveys, similar idea, someone can unsubscribe and tell us why they’re unsubscribing right there. So, it’s a great way to capture some user feedback from your email users. So here are a few things to consider about AMP from a product manager’s point of view. Firstly, be aware of the current limitations when you’re planning. As I mentioned before, there are some challenges with mobile support and with how certain components behave.

Janie Clarke: And secondly, you have to plan out how you’re going to measure your test. We didn’t go into much detail about that, but since AMP adds interactivity to your emails, you need to know what actions you might want to measure and make sure that you’re logging those, tracking them and whatever system they use, so that you can see what’s happening inside the email. And lastly, designing for AMP brings new challenges. So it’s different from designing for a traditional email and it’s also different from designing for the web or mobile. Like the lightbox case that we mentioned, the design that you have in mind when you first do it might not be how it actually works in real life. So you have to be aware of that and be ready to adapt.

Janie Clarke: So we’re very excited about AMP but cautiously. So as I mentioned, it totally changes email. It lets email do some things that have never been possible before. It brings the whole conversion funnel directly inside the email. In short, we think it’s awesome. If you’re excited about AMP as we are, we recommend giving it a try. You do need to have some development bandwidth to work on it. But there may be dragons. So there are some challenges when it comes to working with AMP, so just make sure you’re prepared. Thank you.

Allison Dingler: Everyone hyped? Everyone ready? Yeah? Can I get woo hoo? Y’all can do better than that. Here we go, awesome. I’m going to pass it over to Galina.

Indeed girl geeks: Galina Merzheritaskaya, Nitya Malhotra, Erin McGowan, and Alison Yu speaking on women in leadership at Indeed Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Galina Merzheritskaya: Hi, everyone. My name is Galina. I’m a QA Engineer on the data science platform at Indeed. I have a few question we got from panelists tonight, and once I finish them, I would like to move to the audience and hear your questions. Before I start with my questions I would like all panelists to introduce themselves.

Nitya Malhotra: Hi everyone. My name is Nitya. I’ve been at Indeed for about five years now. I’m an Engineering Manager. I transitioned from an IC to engineering manager about two years ago. Before that, I was a product manager with Merrill Lynch, Bank of America, and then Indeed. And yes, that’s where I’ve been ever since.

Erin McGowan: I’m Erin McGowan. I’ve been at Indeed for three and a half years. I’m our Associate Site Lead in Seattle. That’s part of our chief of staff organization, so we’re looking at overall site health and ensuring engineering growth across the site.

Alison Yu: Hi everyone. I am Alison. I am the Open Source Community Manager here at Indeed. I’m part of the Open Source Program office and I report directly into the Engineering Capabilities Organization. I’ve been here a year and a half. I think I said that. I have a little bit of a cold, so if I start to cough, I’ll exit to stage left.

Galina Merzheritskaya: Thank you. I think all of us know that tech is a male dominated industry statistically. You have this number. Does anyone know the ratio between men and women in tech for the past five years? Okay.

Audience Member: 10:1.

Galina Merzheritskaya: 10:1. Close. So, Catalyst is a nonprofit organization collects a lot of data to help women at workplace, and they provide the data that 30% of women in tech industry, only 20% in a leadership position, and Forbes also did their own research showing that you have some growth coming from 3% to 6%, from average 15 to 17. So there’s some work done, there are some changes, probably it’s why you’re here.

Galina Merzheritskaya: So I want to ask panelists, what do you think can be done to change the situation? To make it better and have more female, the women in a male dominated industry?

Erin McGowan: [inaudible].

Nitya Malhotra: Hang on for a second. I think something that in my mind would really make a difference is A, seeing more women in leadership positions. This is something that I personally find motivating, or demotivating sometimes, to not see. But how do we do that? How do we actually get to that place? Now, I know that from Indeed, I’m again speaking from Indeed’s perspective, I know that support, having either great mentors or great support within your organization has also been really helpful.

Nitya Malhotra: Another thing that Indeed has been doing is we’ve actually been partnering with a few programs. I’m actually more familiar with the program in the Seattle office where we’ve been partnering with the Ada Developers Academy. The Ada Developers Academy is a training program for women who have not necessarily been through the traditional CS program in school, and it’s basically a bootcamp after which they go through an internship and then end up joining a bunch of tech companies.

Nitya Malhotra: So the Ada program in Seattle has actually been pretty amazing. I actually ended up working with a lot of ADs as we call them in Seattle, and it’s been great to see so many women in the Seattle office, thanks to the Ada program, and they’ve all been doing such an amazing job. So that in itself has been pretty encouraging to see more women in general in a lot of the offices. So, I think just it’s a numbers game out there and seeing more women in tech is I think what is going to solve the problem in the end.

Galina Merzheritskaya: And [inaudible]. Is it a major program for anyone who works at Indeed or anyone from outside [inaudible].

Nitya Malhotra: It’s actually outside Indeed but they end up partnering with Indeed and a bunch of other tech companies as well, and do internships at Indeed and end up getting them full-time offers with tech companies.

Galina Merzheritskaya: Thank you.

Erin McGowan: And in San Francisco, Indeed is also partnering with Techtonica. They are hosting a cohort of 15 on site now. It’s the second cohort. The first cohort has four interns that are currently interning here at Indeed. The other from that cohort are in interning at other tech offices. So San Francisco is also helping to increase the funnel to help grow women in these engineering roles.

Alison Yu: And I can expand on Techtonica. I am based on the San Francisco office so I work closely with them. One of Techtonica’s missions is to make sure that women and non-binary individuals who have non-traditional tech backgrounds have a way to join the tech industry. So they put them through a bootcamp, essentially, and then help place them with jobs. At Indeed, we not only host them but we, let’s last quarter, also help get them to Grace Hopper. So we did sponsor travel and passes for Tectonicans who went and hosted a project at Grace Hopper’s open source day, which the Open Source Program office did sponsor.

Alison Yu: So, we’re trying to make sure that their name is getting out there, that there is more ways for people to get involved with tech. And then just being a part of the Open Source industry as well is, one thing that we like to stress is that technology isn’t all about codes. So there’s so many different jobs within tech that don’t require you to be a hard engineer or a coder, even. So there’s so many different facets of tech that have roles open. So marketing, legal, et cetera. So, I think there’s a lot of other ways that people can evolve in tech, which aren’t traditionally thought of.

Galina Merzheritskaya: That’s right. Thank you, Alison. You all [inaudible]. Great. Based on the value you see right now, have you, having mentioned of leadership roles, how did you land there? How do you get to this current destination where you’re right now?

Erin McGowan: Sure. I’ll take this one first. I didn’t always start in tech. I actually started in hard science as a lab tech, and it was awful. If you think tech has low men to women ratios, hard sciences has nothing on that. I was the only woman in both labs that I worked in. And looking at it and as we’re using computers, running instruments, it was interesting and fascinating way more than the hard science. So at that point I transitioned into tech largely in quality assurance. I learned how to become an [S debt 01:09:45] at Microsoft way back when. And I always said, “I don’t want to be a manager. I don’t want to be a manager. That’s awful.”

Erin McGowan: And then I started mentoring people and seeing them grow and I was like, “Oh, this is what management is about. I can help drive careers.” At one of my jobs prior to Indeed, I took the plunge and I accepted a leadership role. I had an awesome, awesome female advocate who was big on ensuring that women had access to those leadership roles, and I loved it. When I joined Indeed, I came in as a quality assurance manager, actually in Austin, and I grew my team from four, and before I transitioned I had a team, teams in Tokyo, Austin, and Seattle, and tried to help them grow.

Erin McGowan: And also growing the next generation of leaders really helped fill my bucket. So even if you think you don’t want to be a manager right now, you never know, don’t close that door, leave it open.

Galina Merzheritskaya: You’d like to add anything?

Nitya Malhotra: Yeah, I can go. I actually started as a product manager in, I was working with Merrill Lynch. So I ended up being a product manager. Now after school, despite good grades, et cetera, I had a computer science degree, I wasn’t really sure if I was maybe cut out for software engineering. And I know a lot of women who have gone through similar experiences, it’s amazing how many women I know who have basically had similar experiences. Anyways, so at that point, it was when I missed getting my hands dirty and actually coding. And that’s when I realized that, “Oh, this is maybe something that I do want to get into.” And at Indeed is when I would say I completely transitioned back to a software engineer.

Nitya Malhotra: And again, I think my journey, even at Indeed was, it took a lot of work, a lot of, I would say a lot of coaching and a lot of mentoring from really great managers, really good mentors, that really helped. Because there was always this, [inaudible] I did this, but is it really that great? Everyone could have done this. And it required a lot of my manager be like, hey, you know what, this is actually something good that you’ve accomplished, et cetera.

Nitya Malhotra: So, I would say it was a lot of that. It was a lot of help from my managers and mentors to actually get me to a state where I felt that I was confident enough to go ahead and then ultimately make a switch from IC to tech lead to then engineering manager. And I think that transition off of that has happened really smoothly.

Alison Yu: Yeah. I started out in clean tech in solar, right, if anyone remembers when [inaudible] happened. That was a fun time. I really got thrown into the tech bubble in a sink or swim situation. From there I transitioned from clean tech into tech. I was really lucky. I had a great manager who I actually followed from one company to the next, and she really encouraged growth. And here’s some different ways you can help expand what you’re doing.

Alison Yu: Manage some vendors, manage different contractors, figure out how your style is, and that’s how I’ve gotten to where I am. I don’t currently manage any people, but I do manage many different relationships cross departmentally and within my own program. So, external and internal. That’s how I’ve navigated the waters.

Galina Merzheritskaya: It sounds like you can come from individual contribution to a manager. Maybe you can advise something, what steps look like to come from IC to manager.

Nitya Malhotra: I can take this one one first. When I was thinking about moving, becoming an engineering manager, I think my biggest question was, am I going to lose technical focus? My goal was to be an engineering manager who was also really technically strong. And I was really worried about either plunging into management a little too early and losing the technical focus.

Nitya Malhotra: Fortunately that has not happened. What I have learned over time is that being an engineering manager has given me, I might not be directly making a lot of technical contributions, but it’s given me the chance to make those contributions by influencing others. And that has been the big switch in my thinking. So I can still be involved in really big, really technical changes.

Nitya Malhotra: The only difference is I’m influencing those changes rather than actually executing on them. And that has also been pretty, I still feel technically involved, at the same point in time, I’ve still felt that I could influence others careers. Coaching has also been really, really rewarding and I have no regrets.

Erin McGowan: I think what I would tell someone who is interested in going from being an IC into management, is to try mentoring. See if there’s an opportunity to mentor an intern or a new hire, and see how you like that. As that is successful, working with your manager to say, you know what, I want to do this full time. Where do you see my skill gaps? What do you see that I need? Is there any training that you can attend, books that you would suggest? And making sure that your manager is aware.

Erin McGowan: If your manager is not receptive, finding someone who has seen some of that leadership, your mentorship to get as an ally. Unfortunately, sometimes we need to have that external ally who’s not our direct manager. But the biggest thing is doing it, showing leadership, stepping up in meetings, stepping up to volunteer to take on some of that extra unofficial leadership. As people see you in that role, it is a lot easier for them to see you in a full time management role.

Alison Yu: Yeah, and I would expand on that as well. Not only just finding one-on-one mentorship, but if you can try to step into a leadership position in what we would call here a ERG, like Lena is leading the women in tech group at in the San Francisco office for Indeed. If you can find different programs where you can help lead and bring people together, it makes a big impact. People see you. It’s a highly visible role and it’s really easy way to also find other people who can help mentor you.

Alison Yu: Because otherwise, unless you’re asking for mentorship or asking for help, if you don’t raise your hand and look for those opportunities, people don’t know. So I think the first step is really asking and looking for those opportunities. Prior to joining Indeed, for example, I led the philanthropy efforts at my last company for about three and a half years prior to joining Indeed, but I didn’t have a formal role there. But that gave me the tools to actually be able to step into more of a leadership role where I had the experience behind me.

Alison Yu: And it was something that, even though it wasn’t official, I could put on my resume, I could talk about my experience. So, I think as Erin said, getting your hands dirty and actually doing it is really 80% of the work.

Galina Merzheritskaya: Yeah, I can probably [inaudible]. You try and you like it but we have [inaudible]. You probably had a lot of advice in your career path. Can you guys discuss for some areas, some industries that at least have [inaudible] of sharing? can you share maybe one of the advice with us?

Alison Yu: Sure. I’ll start. One of the things, one of the pieces of advice that I’m still grappling with is, don’t focus so much on perfection and don’t burn yourself out. I am probably a Type A type of person. If you know me, I very much focus on perfection. But one thing I’ve learned over the years is, if you’re burned out, you’re not putting out your best work. You might be running for perfection but you’re not going to get there if you’re completely burned out. So, take time for yourself and rest. I know it seems counterintuitive, but recharging is really one of the most important things that you can do.

Galina Merzheritskaya: That’s great

Erin McGowan: Sure. I think the piece of advice that has stuck with me the most is, if you’re not uncomfortable, you’re not growing. It can be easy to get into a role or into a position where you’re comfortable, you know what you’re doing, you know you have this. But when you’re in that steady state, you’re not pushing yourself and you’re not growing. So it’s okay to get comfortable for a little bit, but then bump yourself to that next level and look for that harder challenge, because that is when you’re going to grow and take your career to the next step.

Nitya Malhotra: Completely second that, by the way, completely. My end, I would say the biggest career advice that I have received is–something that I’ve also really followed, is I’ve really looked out for mentors outside my manager as well. Whether it’s peers or managers on another team, basically looked around the office for people that I know I can go to with questions about various different things. I have a mentor that I go to for technical questions. I have a mentor that I go to if I have questions around being a manager.

Nitya Malhotra: There’re so many different ways. There’s so many different people that I know to reach out to if I have questions on so many different things, and having that these are often the people who will end up supporting you and end up giving you that additional visibility as well within the office. That’s something.

Nitya Malhotra: Another thing that this applies specifically to mentoring, and this advice that I received while mentoring which I’ve always moved–paid forward is, when mentoring, always think about the why rather than, focus on the why rather than the what. No matter what it is that you’re teaching someone, focusing on why something is important rather than what, actually really helps a concept stick. I think this is just in general, good advice that I’ve always passed on.

Galina Merzheritskaya: Okay. So really great advice. I don’t [inaudible]. This is a good time to take questions. If you have a question, they have a mic over there [inaudible] so we can [inaudible].

Vanessa: Hi [inaudible]. Is this working?

Galina Merzheritskaya: No.

Vanessa: Hi, can you hear me?

Galina Merzheritskaya: Yeah.

Vanessa: Cool. My name is Vanessa and I was wondering if you could give me some advice for a situation I find myself in pretty often, which is I oftentimes find that when giving my opinion on a situation, it’s either implied or explicit that I back my decision up with data. Whereas I find a lot of male colleagues can just give their opinion and not back it up with data, and be considered credible. When I back up my decision with data, sometimes it’s met amicably and sometimes it’s met with skepticism, even if it’s based in reality. So I’m wondering if you have any advice for those kinds of situations.

Galina Merzheritskaya: Tough one [inaudible].

Erin McGowan: That’s a tough one. I am really, really happy that Indeed does not have a culture like that. I think the biggest thing is to be consistent. And I would also, with those men being questioned, ask. Be the one to stand up and ask them, what are you basing this on? What did you see that led you to this result? And be consistent with that and they can come to expect the same level of questioning as yourself.

Nitya Malhotra: I would ask the other questions, are there others in the room who probably feel similarly? Are you the only one in the room who is feeling that way or there may be others who are feeling similarly. Maybe this is a wider problem as well and needs to be addressed in a wider way, but that’s the other avenue that I would go down, and the me–

Alison Yu: Yeah. I would also say, if you feel like this is the only… you’re the only one on the team for example, bring it up to your manager, talk about it. If they don’t realize that it’s an un-bias–or an unconscious bias, then how can they address it? So, I think that’s something that we should be talking about more in the workplace, anyways, I am very happy that we don’t have that culture here at Indeed. But I think that until you raise that issue with others and even talking to your teammates one-on-one, pull them aside, say, “Hey, why are you questioning me about this?” Until you have those open conversations with them, I don’t think the situation can change. But I think that’s your first step.

Nitya Malhotra: And I think it’s fair to ask for data, but I think it should be applied consistently. I think it’s absolutely fair to ask for data when you’re making a statement. But the inconsistency is the issue. And the other thing that I would say is don’t let that stop you from actually providing your input. Even if you have to back it by data, go ahead, keep going strong with providing your opinions, even if you have to back it with data. But don’t stop doing that.

Vanessa: Thanks.

Galina Merzheritskaya: Anyone else? Everyone seem to ask big questions, tough questions. Okay it’s 8 o’clock already.

Audience Member: I’m not sure if anyone, any of you, y’all said this already, but can you tell me how long it took you to get into leadership and are there any other steps that could take to be there?

Erin McGowan: Sure. I’ll go first. From the time that I really decided I wanted to move into leadership, was probably about 12 months, and that included doing an extensive six month leadership program that had some very intensive training sessions, talking with other leaders in the organization. This was prior to Indeed. It’s really going to vary, depending on your organization. I’ve had people on my team that I was able to get from an IC into a management, that was anywhere between say six months and 18 months, depending on where they were at on their career when they started addressing interest and wanting to go to management.

Nitya Malhotra: And I was actually similar situation once I figured out that I wanted to be in a leadership position, was about asking, “Hey, what are the next steps that I can take, move into a tech lead role?” And tested that out for a while, then moved on to engineering management. It was a similar 12 month period for me. The advice would be if that’s something that you’re interested in, I would say ask for it and test it out, see how it goes.

Galina Merzheritskaya: One time I got advice that if you’re solving problems, you lead something. It can be project, a team, a team has issues and so you have to handle them, overcome them, failure. Then it’s just next step if you want to make them official or if you want to just [inaudible]. If you want to make official you speak with the manager and [inaudible] talk with what you want and how fast you want it.

Nitya Malhotra: Yes.

Galina Merzheritskaya: What should I do to make manager? Steps just what can [inaudible] and what can make a [inaudible]. If it’s two months, three months [inaudible] you solve the basic problem [inaudible]. For [inaudible] company, tomorrow you [inaudible] [inaudible]. Yeah.

Alison Yu: Yeah. It seems like it really does vary depending on where you start in your career path. If you start very early on, knowing that you want to manage someone, it’ll take you much longer than if you’ve been in your career for five, 10 years. So I think it’s just knowing what you want and going after it. When I manage people, I actually did a job switch and looked for particular roles that had managing positions and where I would hire on a team. I know that’s not always the most ideal way to do it, but sometimes the company that you’re currently at lacks resources. Here at Indeed, we thankfully don’t lack resources. I just decided that that wasn’t the path that I wanted to be on at the moment. So, many different ways.

Galina Merzheritskaya: [inaudible]. Anyone else?

Audience Member: Hi. I just wanted to ask, what advice do you have for somebody who is starting off in their career but wants to have influence within the team? What kind of strategies or what did you do when you were in that position to influence the decisions that your team makes or just to have some influence, because as a leader, obviously you have to make decisions and have influence over your team. So do you have any advice for people and they’re starting their careers?

Nitya Malhotra: So, actually this is a good question back to you. I would ask yourself what is the thing–What is the… there’s always, no matter what the team, what the company, there is always some room and some scope for improvement. And finding things that either you are passionate about getting… about the team getting better about team’s process improving, about how the development processes. Or if you want to let’s say, if you’re really passionate about this particular, it could even be at this particular class in your service that you think is not tested for example. Or you see a bunch of errors and no one’s caring, but there’s so many.

Nitya Malhotra: I would say pick something that, let’s say, that you are passionate about that, let’s say, bothers you and go ahead and fix it. Things like these, the tiny, tiny things that you spot and as you keep improving these, people are going to notice that you’re taking the initiative to go ahead, find something that you don’t like, and improve it. I think that is a great leadership quality in itself. And once you start doing that in the small level, you will start doing that on a larger scale, as well.

Erin McGowan: I would say one of the mistakes I made early in my career, is I was afraid to speak up. I would be sitting in a meeting room and I would be afraid to actually voice my opinion. I would have the thoughts and I would just be afraid to actually put them out there. Put yourself out there. Don’t be afraid to say, what about this? Have we considered this? You know what, I tried this and it really didn’t work. They’re not going to bite. At least if you’re in a good workplace, they’re not going to bite. So I would say just go ahead and speak up. They want to hear your voice. You’re in that room for a reason, so don’t be afraid to use it.

Nitya Malhotra: Also to be fair, that never gets easy. Today there have been times where I am like, “Oh, should I see this? Everyone else seems to know.” But that never gets easy and that’s always a struggle. That’ll continue to be a struggle, times you just have to push past it.

Alison Yu: I would say try to find gaps and try to become a subject matter expert in one of those areas. I was on a marketing and communications team across multiple different companies. I specialized in social media for a while, then I became more broad. And because of my expertise in one area, people came to me from many different departments, from different business units. Even though I was in the marketing team and I sat in marketing, people from different engineering teams would come to me and ask, “How do I market my product better?”

Alison Yu: So once you get your name out there and you’re proven that you’ve done the research, you’re doing a good job, people will seek you out anyway. So focused on something that you’re passionate about, that really fires you up because that’s something that will be recognized that you’re doing a good job and then they will naturally follow.

Galina Merzheritskaya: I can add the two pieces. One as jealous [inaudible]. If you think I don’t know team, I was like, “Oh my God, she loves this team, loves the product, what can I improve? Like everything’s great.” And then, “Oh is it [inaudible] feels great.” And he’s [inaudible] today [inaudible] every day since 5:00 PM, do you guys use [inaudible] and then he’s trying to give [inaudible] aspect. Not [inaudible] but [inaudible]. Whereas where I can help to improve this. And as it little by little you have this guy like, “Hey, this is actually not a big problem.” And [inaudible] somewhere you can show the need to do such.

Galina Merzheritskaya: You’re also independently and [inaudible] are like this is my research and definitely how can you grow. And I’ve learned [inaudible] goals, that sometimes will follow when you lose everyone else, you just have your own goals, but for [inaudible] before breaking them, bring on some rules that exist. And knowing how everything works. So like okay then you can figure out different ways that basically can prosper in that area.

Alison Yu: I’m just going to add to that I think different perspectives and the way that different people will look at a problem or a situation, can only make a product or a team stronger. So even if you think, “Hey, my opinion, what is it really valid?” Your opinion is valid so never question that. But also know that the different ways that you look at a problem is a different way that someone might have never seen it, and you can be revealing something about a weakness that maybe no one else on the team had thought about and that can be a really great thing for you and for the team.

Galina Merzheritskaya: That’s true. Okay.

Audience Member: Hi there. I have a question about how do you grow on the job? I know that you mentioned that there is a women in leadership program that you went to to get a lot of training before you became a manager. What if there’s no training like that or you’ve thought about becoming a manager maybe down the road, how do… what kind of things have you done, maybe in the past, to prepare yourself to get to this road? Do you read books? Do you listen to podcast or do you go to meet up to meet mentors? Yeah.

Erin McGowan: Yes. All of the above. I think the biggest thing, if you’re looking for some leadership training and your organization doesn’t offer it, look external and ask. Often time your organization will pay for external training. Most of the time there are budgets for that and people don’t even know to ask. So ask for it. Perhaps they can bring some in. You’re probably not the only one. There are a number of books that you can help read. I [inaudble] in the management before podcasts were a big thing, but I know that a lot of people now are using podcasts to help learn and having a mentor. Having a manager mentor that is not within your organization can be huge.

Erin McGowan: You’re able to really have good conversations with them without any fear of bias or… I asked this question, is my manager going to think I’m not ready? So having someone who’s outside, whether they’re outside just your group or your company, is invaluable to really help understand where you as a person need to go.

Nitya Malhotra: I’m not great with podcasts or it reading outside work. For some reason after work, I just switch off and I… it’s really hard. So a lot of my growing I feel happens, majority of it for me has happened at work. Something that I found useful is putting myself in, let’s say my manager’s shoes, and thinking about A, how would I have done this differently? How would I have done this better, or do I like what they’re doing? As time has gone by, this has created a mental model of how it is that I want to be as a manager, and that has really helped me model my experiences with my reports as well.

Alison Yu: Yeah, I am also the same way. I do not want to read or listen to a podcast about work after work. I work enough, so I think one of the things that I try to do, is at work or when I travel for work, so speaking at conferences, et cetera, one of the things that I try to do is make sure I go to the networking events at those things. Talking to my peers in the same industry, seeing where they are in their careers, how they’ve been able to progress, and then finding other people who are already in other higher management roles and seeing and talking to them about what was their career path, if they’d be open to mentoring.

Alison Yu: I think that’s really important that you want to see what other people in your same industry are doing, and not just only stick to what’s the norm in your company. Because sometimes when you’re so isolated and so siloed in your own company, you might not notice that the path that your management runs, it’s not the same as the rest of the industry. So I think that’s also very important to keep just a tab on.

Galina Merzheritskaya: Yeah. Sometime it’s called shadowing, if you think you would like to try some role and be like a measurable, you speak with your and you’re like, “Hey, I want to see what you do so I can know what you do from seeing what you’re doing.” And then you will have questions and then you will have a pass while you actually [inaudible]. And then you always say, “Oh, I did [inaudible].” And if you have an opportunity to speak with the manager be clear what you want, or how will the manager help me.

Galina Merzheritskaya: Or like if it’s not in your company, there are so many programs outside. We have so many mentorship right now. There are many [inaudible] to step up higher and you learn. It’s a great time to do it now because, if you can do more then you can do tomorrow [inaudible]. It makes it so that managers who are [inaudible] can draw a time just to help you to grow as well. Yeah. I think that’s fine. And we have time for a few more questions if you have any. Yes, okay.

Audience Member: Hi. Just wanted to firstly thank you guys for sharing your experience. My question was around conflict and some strategies that you use when you experience conflict in the workplace. I know sometimes women can be more of the supportive or the accommodating role, where men can be more dominant in that conflict situation, so just wondering what tips and tricks you have for dealing conflict.

Erin McGowan: I think the first tip is to make sure that both parties are leaving emotion at the door. If either party starts to escalate with emotions, asking for a timeout and table, and have that conversation at a later time. You’re not going to have any kind of a productive conversation if someone is angry, yelling, or on the flip side, if someone is upset, perhaps crying. You’re not going to be able to have a good healthy conversation, table it and schedule time. If you need to involve a neutral third party, either your manager or that person’s manager, or both, to help, if you are afraid of emotions escalating again.

Erin McGowan: I think the other thing is that some conflict is healthy. It’s okay to disagree, but you want to make sure that you are disagreeing respectfully, and making sure that you’re not crossing any of those professional lines disagreeing. Having data can really help. This is where I am on this because one, two, three, can you help me understand how you came to this?

Erin McGowan: So, really understanding their point of view can help bridge. You might be closer than you thought, but you’re talking sideways. You’re just in a different place, but you might still actually be really close to each other. So taking the time to understand where they’re coming from and how they got there. And helping them to do the same on your side.

Nitya Malhotra: I might have slightly different opinion, especially around the emotions. I know emotions in general are turned on as this bad term. I actually don’t think they’re bad. I think that a lot of times when there are strong emotions associated with how someone’s feeling, there’s generally a reason behind it, and it’s always, I think it’s important to figure out what that reason is. So even if there is emotion around it, obviously we don’t want things to escalate, but that emotion is stemming from something and it’s really important to figure out where that’s coming from.

Nitya Malhotra: So stating, let’s say in a conflict, stating exactly why you’re saying something, you’re stating where you’re coming from, what your intentions are. My intentions are not to disagree with you, but I really think that this is the right thing that we should bring. I’m just giving a stupid example. But stating your intentions, where you’re coming from, I think that really helps clarify and make someone understand that you’re not actually attacking them and that you’re coming from a place of logical reason.

Nitya Malhotra: Crucial conversations was this class that actually I took at, it was one of the trainings that was offered at Indeed. I thought it to be super helpful, not just in my work life, but also in my personal life. I would recommend, I would look into it as well.

Alison Yu: I think for me it’s making sure that everyone’s on a neutral playing field. Making sure that someone doesn’t feel ganged up on. For example, taking any of those external situational feelings that could happen if you were to say, “Hey, I want to address this.” But you’re doing it, for example, in the cafeteria or in a meeting room full of other people. I think there’s a time and place for everything, so if you are feeling those emotions, I don’t think they’re necessarily bad, but I do think that you do need that time to have everyone simmer down a little bit, so you have the root of the issue that you’re actually trying to get to, versus the feelings that are really getting there.

Alison Yu: I do agree that when you have those charged feelings, there is something that is sparking that and you need to get to the issue and resolve that, but you need to also do it in a cool, calm, collected manner, because you going in there guns blazing is not going to help with anything, neither you or the other party. But making sure to have it on a neutral ground, I would say before you really escalate it to managers, try to see if, hey, can you work this out with your peer? Is there a way you can take this out? Try going for a coffee and just going and talking it out. Usually that, I think, helps it.

Alison Yu: It’s not necessarily in the office or in front of others, but it gives you a neutral ground and it’s time away where other people can’t really hear what’s going on, and I think that’s very important when you’re trying to hash out the details when there’s a serious conflict.

Galina Merzheritskaya: And I’ll just maybe say super [inaudible] personal. If you personally got offended or someone’s [inaudible]. All my [inaudible] and game matters. I’ve been thinking about it and [inaudible] up there. You guys [inaudible]. And I [inaudible] solve problems, do it once I think. I’ve been told if you have to do [inaudible] five times breathe in, and breathe out five times [inaudible], you will get more oxygen to your brain and you will be feeling a little more rational. You cool down and then you can move on to the next step. You can also agree to [inaudible] partner [inaudible]. Thank you.

Audience Member: Thank you.

Galina Merzheritskaya: And I think one more question then we’ll close. Okay. One, two, three. Okay, I [inaudible]. Thank you so much for your time.

Allison Dingler: Awesome. I won’t take up any more time. Another round of applause for our amazing panelists up here, yes.

Thank you to Indeed’s Alison Yu, Erin McGowan, Lindsay Brothers, Janie Clarke, Rohan Kapoor and Galina Merzheritskaya for speaking at Indeed Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Thanks to all the girl geeks who came out to Indeed for dinner, networking, talks, panel discussion and more networking!  Erica Kawamoto Hsu / Girl Geek X

Our mission-aligned Girl Geek X partners are hiring!

• See open jobs at Indeed and check out open jobs at our trusted partner companies.
• Find more Indeed Girl Geek Dinner photos from the event – please tag yourselves!
• Does your company want to sponsor a Girl Geek Dinner in 2021? Talk to us!