Speakers:
Katie Jansen / CMO / AppLovin
Alice Guillaume / Director, Marketing / AppLovin
Helen Wu / Director, Growth Partnerships / AppLovin
Swetha Anbarasan / Network Engineer / AppLovin
Laura Pfister / Software Engineer / AppLovin
Anusha Ramesh / Principal Software Engineer / AppLovin
Sonal Gupta / Principal Software Engineer / AppLovin
Transcript of AppLovin Girl Geek Dinner – Panels:
Katie Jansen: I’m Katie, I’m the CMO here at AppLovin, and we just wanted to thanks to Girl Geek X. This is awesome. We haven’t really done anything like this before, so this is our first time, and they’ll probably ask you for feedback at some point so feel free to give it to us. This is a new thing for us. I would also like to thank Brianna. Brianna, where are you? Brianna is the one, and Marissa who is probably not here right now, she’s out front, they really put all this together, they got all the food, which was outstanding, and the color coded name tags, and they were down here putting the seats here today all on their own, so I just wanna give a big shout out for all your hard work, thank you so much.
Katie Jansen: If you don’t know a lot about AppLovin or even if you do, real quickly, we’re a comprehensive platform that connects app developers of all sizes to about two billion global users each month. We have a growth monetization and publishing services that really focus on giving game developers the expertise and insights they need to grow their business. Whoops. We’re only about six years old, and we, I think we have about 160 employees, 165, and up until recently we were self-funded. So we’re actually pretty small compared to a lot of the companies around here, but we’ve really experienced hyper growth in the past few months, even just this year, you could say. We have about seven offices worldwide and our valuation is a little north of two billion. So we have a very few amount of people doing a lot of work and producing a lot of great things, well beyond just the revenue we’re bringing in.
Katie Jansen: Today we thought we could partner with Girl Geek to give you a real unique perspective on how to adapt when you’re in a fast growing company, because we have a very strong perspective on that, and we grow our own roles kind of within that company. Today I have Helen and Alice, and they’re gonna kinda start things off with me. We’re gonna get the business perspective on that, and then next we’re gonna move over to engineering and ops and really get a more technical perspective on how we scale and how we’re adaptable. And oh, I’m sorry, there was an agenda as well. Panel one and panel two, and then we can eat and drink some more. Alright, so let’s start things off with you guys just telling me a little bit about what you do here at AppLovin and about yourself.
Alice Guillaume: Hi everyone. This is really great turnout, I’m super excited to be here. A little bit about myself, my name is Alice. I’m the Director of Marketing here at AppLovin. I’ve been here for four years. My journey here started out on growth and as the business has scaled I’ve taken on creative services and most recently, I built out that team to be over 30 people, and it’s currently one of the largest teams in the business.
Helen Wu speaking at AppLovin Girl Geek Dinner.
Helen Wu: I’m Helen. I’m the Director of Growth Partnerships at AppLovin. I joined in 2013, and since then, have grown and trained our global teams who are responsible for helping our customers achieve their performance goals. And a little fun fact about me, I used to attend these Geek Girl Dinners like six years ago, so in 2012, I was in the audience like you guys learning the stories and experiences of other geek girls, so I’m very excited to be here today.
Katie Jansen: I didn’t know that. You saved that for this. So you guys both come from a different background. It’s not like you were in this industry when you first came to AppLovin, so what drew you to AppLovin, and what do you enjoy most about being here?
Alice Guillaume: Why don’t you…?
Helen Wu: Okay cool. So I joined because I wanted to be around the smart and hardworking people in this company. I wanted to learn how to build a business, and before I joined I was at a large bank. I did internet … I did research on internet trends, and I learned that mobile advertising was gonna be growing at a tremendous rate and we were just at the very beginning of this long term growth trajectory. So in 2013, AppLovin was, had already been operating for about a year and a half. They had invested heavily into building the platform and product, and was starting to make their first business hires.
Katie Jansen: I remember interviewing with a couple people at the company and just being so inspired and in awe of how smart and passionate they were. I was like, I just wanna be around these people all the time and work with them and learn from them, and you know, because I knew where mobile advertising was headed, I really believed in our CEO Adam’s business vision of us being this growth engine for app developers. So I knew there was gonna be endless opportunities to learn and to grow with the business. So I joined and I haven’t looked back since.
Katie Jansen: Alice?
Alice Guillaume: My journey’s similar but a little bit different in a sense that I had also come from a very structured background. I had grown up in management consulting, banking, sales and training, that’s what I thought I wanted to do with my life, and until I reached a point where I realized I needed something more. I wanted to work in a place where I could make an impact, where I could bring ideas and promote change. I needed that in my career.
Alice Guillaume: I did not know very much about mobile advertising to be totally honest, and I took a chance and I took a risk to interview at AppLovin, and I was nervous because it was a deviation from a very structured route, and actually taking that risk was probably one of the best decisions I’ve made.
Alice Guillaume: And two things drew me to AppLovin — one, and I think this echoes what Helen said is the people, and two is the learning opportunity. So for me when I was interviewing, I realized that everyone was so smart, and so much smarter than who, than I am, but they were down to earth enough to wanna teach me. So I thought that was really cool. Two was that leadership really cared about my growth, and they were invested in helping me get to where I needed to be. That was also something that, for me, made a big impact on my career here.
Alice Guillaume: And three was the mentality about work. People here are really passionate about making things better and not doing things just for the sake of doing them. If something is menial, or something is low value, let’s find a way to like automate that. And that really spoke to me because it demonstrated the company’s value in innovation and change. So all of those things really inspired me. I wanted to be around those people, and I wanted to be a part of that race.
Katie Jansen: And the fourth reason is Helen, right? Didn’t you guys go to school together?
Alice Guillaume: Yes. Fun fact about me and Helen. Helen and I went to college together and she was my big sister there, and that’s how I learned about AppLovin. And she was gonna kill it when she joined.
Katie Jansen: Right. And is there anything surprising you guys have learned since being here?
Helen Wu: Yeah absolutely. So I mean I think it’s just learning that I would be constantly stretched and challenged all the time. I mean before I joined, again, I was at a large bank where the work environment and culture was very different. Even though the financial markets were changing every single day, I felt like the mindset and the processes were very routine and very stagnant. So I wanted to be in a place where I felt like I could add value, where I could learn something new every single day.
Helen Wu: The most important thing I learned over the last five years is the value of being fast and flexible, which is staying in tune with changing market opportunities, being ready to action, adapt, and re-adapt. And I think the speed and mindset with which we operate is really the key to our tremendous business growth that we’ve had since 2012.
Helen Wu: Also, you know I think everyone in this company has the ability to wear a lot of hats to help out in ways that are beyond what’s just being asked of them. So I think really people, I think people here are very, very keen to like help the business but they’re also very invested in helping each other. So you know I think hiring people who can add to this culture of continuing pushing each other, you know, challenging each other to ask intelligent questions, to take risks, and to do better than their last is really important for continuing to grow our business.
Katie Jansen: So you guys have grown from being individual contributors to being managers now. You both manage teams. Alice as you mentioned, Alice is actually on my team, and then you have one of the biggest teams at the company right now, and you have a big team too. What do you guys typically look for when you’re hiring, because lately it feels like we’re always hiring. So what do you look for? How do you kind of figure that out in an interview? What are some tips you can give to these guys about that?
Alice Guillaume: Definitely. So I’m very passionate about hiring and recruiting. The resume is important, but for me it’s really the human and the psychological aspect of who you are. When I interview candidates there are two main things that I care about.
Alice Guillaume: The first one is ability to learn. So you will hear throughout the theme of our panel that the only constant is change, and that’s, I think, a core thing that has drive our company to be so successful today is constantly evolving.
Alice Guillaume: To be able to move that fast, we need to hire people who are open to learning, who are open to self improving, and who are receptive to knowledge and feedback. For example, when our team ramped up from 15 to over 30, that doesn’t happen over night. That’s a collaborative effort of everybody on the team from the individual contributors to the leads, and that really requires that openness and heart to be flexible.
Alice Guillaume: The second thing I care about is grit and passion. So I think that speaks to the first one is be receptive, be open to learning, and two is apply that in your day to day and be able to put in the amount of work that it takes, and you need to have the passion to be able to wanna do that. Those are things for me. You wanna add anything, Helen?
Helen Wu: Yeah definitely. I totally agree with everything Alice said. I think we spend a lot of time thinking about hiring and who we hire and why we hire. I also wanna add that I look for candidates who have really deep comfort with data, simply because we have a lot of it, which is readily available and waiting for someone to take, extract meaning out of it. So our growth analysts do the really important work of analyzing the data, building models that can inform us of, you know, how we can drive better performance for our customers.
Helen Wu: The second thing I look for is some love of solving puzzles of any kind. So even if a candidate doesn’t have quantitative or technical background, you know, I look for any indication that they might be, for example like a chess master, or an avid player of strategy based board games, anything that suggests that they’re a strategic problem solver who can look multiple steps ahead and formulate a plan. And these kinds of things demonstrate to me that someone possesses deep problem solving skills, which we need in our business to continue to grow the way we have.
Katie Jansen: And so what about, okay let’s someone’s gotten hired, they’re on your team, so let’s fast forward a little bit. How do you guys as managers coach and manage, you know, keep your staff motivated? Again we’re small, we’re scrappy. How are you able to kinda get in there and work with them in way that’s not micromanaging but also helping them move forward? Especially when we change goals all the time, right, or we’re flexible or we fail fast, right?
Alice Guillaume: Yeah, go ahead.
Helen Wu: I think it’s a really good point, and I think in addressing staff motivation, I think it’s important to also discuss the culture and the space in which we work. So we have a very open office concept where analysts, managers, and VPs all sit next to each other, and they’re sitting next to other functional teams. We encourage a lot of questions to be asked openly, a lot of ideas to be raised, and so problem solving, learning, and teaching happens organically at our desks and out in the open, and it happens in every sort of of like direction you can imagine between levels of the organization and across different functions. I think those are really great opportunities for people to learn and to grow all the time.
Helen Wu: And then, on my team we do regular check ins between managers and the employees where I like to take our employees out to coffee, and it usually starts with asking questions like “How are you doing,” “What’s going well for you,” “What do you need help with,” and these are important opportunities for an employee to share their ideas and questions that they have, or goals, and get support from their manager on how get there. And I really like that these are outside of the office, like over coffee, because it’s a more casual environment where people can connect on a more personal level and dive deeper into the discussion.
Alice Guillaume: This is a really good question. I think there are a lot of things a manager can do in the day to day to help foster this, but for me the fundamental value, and I don’t take any credit for this ’cause someone else taught me this, is empathy. So for me it’s about recognizing that every single member of your team is human. So for me it’s spend time with them.
Alice Guillaume: I do walk around on my team and other teams, probably ’cause I just like talking to people a lot, but I like getting to know them, getting to know then in a work environment, but also outside of work. As a manager that also helps me connect the dots when I see opportunities for people who can help each other out or would just be a good fit to talk to one another, whether it’s a mentorship relationship or not. So that’s one thing.
Alice Guillaume: Another thing is with my direct reports, don’t be afraid to talk about things that are not work related. I think building that relationship is really important to having tougher conversations too. That openness is super key. That last thing I would say is have your EQ feelers out. So how is the team feeling?
Alice Guillaume: I have a team of 30 people, it can be hard sometimes to know how everybody is feeling, but there are ways to kind of learn that, whether it’s through your own direct managers or just through day to day talking to people and being sensitive to that.
Alice Guillaume: Like if something’s going or the team’s feeling stressed out, can we take a break, can we go for a coffee, can we catch up, can we lighten the mood a bit. I’ll crack a joke, I’ll play some songs or something like that. Or if you have a life event going on in the team, like celebrate that. Their win is the team’s win, but most importantly it’s winning everyone over as people.
Katie Jansen: Yeah, I mean I think what I heard both of you guys say is it’s about the individual, and it’s about connecting in different environments. I will say having worked at other companies before, AppLovin is very unique in that we actually have a very low amount of meetings here.
Katie Jansen: As the CMO, I can easily go sometimes with only one or two meetings in a day. I’m not in back to back meetings all day, so that actually gives us the time to connect with individuals outside of your team and on your team, we really take the open office concept, like why have it if you’re not gonna actually leverage it, so we’ll just do impromptu meetings or conversations that’ll just start up right at the desks versus going into conference rooms and having meetings.
Katie Jansen: For me, having been at so many other, not a ton I guess, but at a few other companies, the concept of not being in back to back to back meetings and actually to connect with your team and other teams in a real environment, and also leveraging things like Slack and you know the different tools that we have is what makes a big difference too, which is what you said.
Katie Jansen: And what about mentors? How do you guys leverage mentors on the team? Do you leverage mentors on the team? Do you think it’s valuable?
Alice Guillaume: Absolutely, it’s valuable. So my team it depends, I focus on what the purpose of the mentorship is. So an example could be if I have a new manager, I want them to have a resource where they can talk to that person more candidly than they probably could with me to help them grow. In that case what we’ve done on my team is we’ve sourced externally through networks of our leadership team, to provide that person with a mentor, and we’ve seen that has been really effective.
Alice Guillaume: An alternative example could be just pairing a junior person and a senior person together on the team. That use case is more internally driven, so you know, making sure this person has a support on the team that they can learn from them about their day to day jobs as well. So absolutely mentorships are important. I had mentors as well. They’re your greatest advocate, they can help honestly point out areas where you can improve on more candidly, and it’s just nice to have someone to talk to sometimes.
Helen Wu: Yeah, definitely everything Alice said I agree with too. In thinking about how I think about mentorship, I think it’s also worth discussing like how I define it. I think mentorship is also a two way partnership where both parties can get a lot out of that interaction. Not only is it a way for the mentee to share their ideas and goals and get support and guidance from an experienced person, but it’s also a great opportunity for the mentor to teach, to pass on knowledge, and to get perspective on what’s happening at other levels of the organization.
Helen Wu: I think this is actually one of the very critical interactions between an employee and their own manager, so we invest a lot into this, definitely. And then as far as more long term mentorship that we have in the team, it’s most commonly gonna be when a new hire joins, we try to pair them with a more experienced analyst who can teach them about the role, about the business, and that’s a really great opportunity for the experienced analyst to practice their leaderships skills and to help develop the team.
Katie Jansen: So this is probably a harder question, but we have small teams here, we’re a small company, and so outside of this like, okay this is a new hire that’s coming in and they’re getting paired with someone senior, how do you help people find mentors? What are some practical ways? ’Cause I feel like quite honestly I go to these events and everyone’s like “Let’s find a mentor, it’s really important, and that all is organic. Thank you.” And I don’t think that … I mean I think that’s great, and I guess it should be organic, but there’s gotta be ways to kinda facilitate that process. So have you employed that at all, or what have you guys done?
Alice Guillaume: So I’ve tried … That’s a really good question. It can be tricky. I’ve employed both assigning on my team, so I look at kind of like what the needs are and I have paired people together, but I also allow them the flexibility to kind of figure out if it works for them. I tend to think if it’s more natural it’ll work a little bit better, but at least I’m providing an opportunity to that person to try it. The other thing I’ve tried is just observing. So there are natural kind of relationships that form through time just kind of, that’s just how humans are, and if I see that happening then I can have that conversations with both parties and see if that would make sense as a match. So I’ve tried both ways. I will say the one where I match it has probably has less chances of working than the organic one, but because there’s not always an organic fit, I still have to go try.
Katie Jansen: Do you have anything to add, Helen?
Helen Wu: Yeah, I think it’s definitely tricky to you know try to pair employees with new mentors. So I think the check-ins with people to understand where do they need help, how things are going for them is really helpful for understanding what they need. On the business growth side we … The nature of our work is very collaborative too, so I think there’s a lot of opportunities to build partnerships that are cross function. So for instance if two people, you know, are both familiar with a specific account or specific type of game we can pair them together to share their own unique experience and perspective, and so we build these partnerships based on business opportunities that are coming up too. And then, oftentimes, this is when we can pair an employee with a more experienced member of the team, and this person can serve as a mentor.
Katie Jansen: I wasn’t gonna say this, but I’m gonna, probably like a year and a half ago Adam, who’s our CEO, told me “Go mentor Helen,” which you might have suspected since I was like “Hello, Helen. You’re not on my team. Would you like to get coffee with me” randomly once a month at this scheduled time? But I’ll be honest. It probably took like about a year, ’cause for a while it felt maybe a little awkward and forced, but I noticed in the past probably five to six months, I’m like okay like this feels like a real, and we’re both getting something out of it to your point mentor-mentee relationship, it took a while sometimes I will say with the pairings, but you can hit your stride.
Katie Jansen: And then I have a mentor that I met, it wasn’t at this group. It was at Women in Wireless, that’s not their name now and I don’t remember their name to be honest with you. They rebranded. But it was a Women in Wireless event, and I was on a panel about mentors, and the EVP of revenue over there was just so impressive to me, and so I afterwards sent her a thank you note for being on the panel, and then two weeks later followed up with coffee, and now she’s definitely my mentor, and it’s been probably three years. And she’s not in the same vertical as me. She’s in finance and rev ops, which is very different, but I still learn quite a bit from her, but it was almost like, I guess, I was courting her or something. But she was open to it so it worked out okay.
Alice Guillaume: Sometimes you have to go get it.
Katie Jansen: Yeah I know. I think at that point I realized I don’t have a mentor, maybe I need one. So that’s all we had today, and I think we wanna open it up if there’s any questions from the audience. Yes, go.
Audience Member: I have a question for Alice.
Katie Jansen: Am I supposed to give her the mic?
Audience Member: I can talk loudly.
Katie Jansen: Alright. Do it.
Audience Member: I was just curious so your title is director of marketing, which is fairly straightforward, of creative services. What does that include like more on a day to day basis?
Alice Guillaume: Definitely. And what’s your name?
Audience Member: Jessica.
Alice Guillaume: Jessica, okay. Good question. So yes I’m director of marketing, but I lead the creative services team. So what we do is we make video graphic and playable ads for our customers, and that is marketing because we are helping them market their apps and trying to help them sell their product. My day to day is, I manage a couple of amazing teams of designers and game developers, and we focus on market research, story boarding, producing, QA-ing, and looking up the results.
Katie Jansen: Yeah and that team rolls into my team as the chief marketing officer.
Audience Member: Alright hi everyone, my name is Regina and thank you for this panel. I’m actually in marketing, and that’s actually one of the reasons I came. I was like “Oh they’re talking about marketing, yes!” But my question is this, because there’s been a lot of call about diversity and inclusion, and I’m like this is great, but especially because you were talking about hiring, what I find is is that when I’m interviewing a lot of companies aren’t really looking at transferable skills. Like if you’re trying to get more diverse candidates then you can’t really expect that everybody is cut from the same cloth, and love that you know yeah you came from banking and …
Katie Jansen: I didn’t come from banking. No banking here.
Audience Member: Yeah yeah yeah, insurance and teaching, and now I’m in tech. So I’m just wondering as people who have done that pivot successfully and have stayed, because I’ve been at a few companies, just like what is your advice to that because there’s a lot of talk about diversity but then I’m like “What do you mean I’m not a fit?” So there’s a little bit of frustration there.
Katie Jansen: I actually I have, we are hiring a director of content right now, and I have … Maybe yes, we’re still in the interview process. But there was a candidate who I would say, and you know Lewis here has been part of the interview process is not what was described up here. She is not from tech, she’s actually from nonprofit, and so I really had to challenge myself and really focus on what are the skills that this job needs, does it really need a tech background, and I don’t think it does, actually. I think that if she’s really good at the brand, and she’s smart, he or she, they can learn the content. So this person is midway through, and I’m not sure if this individual is gonna work out one way or another, but I move …
Katie Jansen: Okay yes, you. You’re here. But I’ve moved her through the process and she’s done quite well because she does understand like she can run a persona project, she understands how to go through and do a customer journey project, and those are the skills that I need. She can learn the content, essentially, and so that’s what I’m trying … And Alice is reporting to me, I have had clear conversations before where I will say I am not talking about skin color. I am not talking about gender.I am talking about this person is different than we normally hire, and I don’t wanna talk about team fit. Do they fit the company? ’Cause when you say team fit that is actually just who people in the team wanna hang out with in my opinion, and that’s uncomfortable to say, but it’s kinda the truth. So how can we …
Katie Jansen: And so now we have more people on the team that are, you know, commuting from an hour and a half away, or they have kids that are in high school, and that isn’t really what the make-up of our team used to be. We have someone who just started who is a former professor. Like it is different and I’m really trying to push the team to do that ’cause I think we will be better and we will push the envelope more if we can start to do that. Did you guys have anything you wanted to add? Nope. Okay. Right there.
Audience Member: My name is Aurora, I also work in gaming and I’m curious about what games you guys are playing.
Katie Jansen: I don’t play games. That’s the truth. Honest truth. But they do, so go for it. I have two kids so I’m busy. I’m super busy.
Alice Guillaume: I just have a dog, so I can play games. Video games? Console games? Mobile games?
Audience Member: Mobile.
Alice Guillaume: I spent $500 on Cooking Fever. I finished the game. Task management, resource management, things where I feel like I’m accomplishing things.
Aurora: What was the name of the game, sorry?
Alice Guillaume: Cooking Fever. Look it up.
Katie Jansen: She’s like, I’m steering clear of that game. I don’t wanna spend $500.
Helen Wu: I do play games a lot, so I usually will download like the top free games, like the top 10 to 20 just to understand what’s out there in the marketplace. We do work with 90% of the top gaming developers out there in the ecosystem. So that’s part of the market research and intelligence that we do. And then beyond that I’m really addicted to word games right now, just like can you find all the words in this, like, jumble, and like improve your vocabulary and train your brain, and beyond that, I love playing board games. We actually do board game night like every two weeks or so at this office, which our creative team and game developers started, and it’s, we do it like on a Friday, we order pizza, we drink some beers and it’s just like everyone having a great time.
Katie Jansen: I play Wordscapes. I do play a few games, but I’m nothing like you guys. And I do insist we play board games and do puzzles with the kids, it’s really fun. There was a question back there.
Audience Member: I’m the back, so I’ll stand up. I’m Jessica and I work at Lime, and to Helen’s point I belong to a private organization of Rubik’s cube solvers. So we’re kind of going through hyper growth as well, so as a small company, what is something that you value that you’ve seen large companies lose along the way that you think is not worth sacrificing?
Helen Wu: That’s a really great question. I think as a still-small organization we really value, just the people that we spend time with. You know my greatest moments of joy are just my colleagues like challenging me to figure out, like, you know, how can we deep dive into this operational change together and how can we figure it out, and like how can we find a better way to get this solution, or let’s just geek out together about the data and about the product, and so I think it’s, you know, people coming together to solve the difficult challenges of the business and investing and spending time to help each other grow and learn along the way. I think it’s so important to hire people who can add to this culture because you know that’s, I think that’s what’s gotten us to where we are over the last five years, and I think at a larger company, I hope we never lose that part of our culture, is just like people geeking out together about like data and product and how to do it best.
Alice Guillaume: That’s a really good question, Jessica, so adding onto that I would say for me it’s remembering to keep things nimble. So our team has grown really large, and that means we need to change too. So some of the processes that we had in place don’t work when you’re at a larger scale, and you know, for me, I don’t wanna get in the way of really talented people, too, who can run with things. So taking a moment, even if things are going at 500 miles per hour, just stop and think about what still needs to exist in the structure, what can we break, and what we can recreate. I think that’s really important because it doesn’t make sense to keep things the same way if the team is no longer the same way anymore.
Katie Jansen: And we just did that yesterday. Alice and I sat down and we said some of these things aren’t working and some of these things are, and we haven’t rolled it out to those of you who are on analysis teams, so I won’t announce it here. But we are moving and changing, and I would just add, I think, which really you hit on in some ways, is being okay with failure. Fail, and this whole, fail fast thing, but I’ve been at this company for six years now, so what they have done so great is be able to say this isn’t working, now let’s move. And maybe not even, it’s not maybe even a pivot, but like let’s change, let’s scrap this, let’s go forward, like this isn’t working, we’re not gonna hang out here and try to make it work forever.
Katie Jansen: Now, admittedly, being self-funded that’s a lot easier to do when you don’t have a board. So I do wanna say that with some context because I think that we benefited from that quite a bit, but being able to fail and just move quickly has been pretty big. And not having a lot of meetings. I will reinforce that. Every time when you start to grow, all these meetings get going, and we’ve done a good job of keeping them to a minimum, and that’s one of the things Alice and I just did. We looked at the meetings she had, and we’re gonna start cutting some of the meetings ’cause we started noticing there was people in too many meetings and did they really need to be. I think ,ready, do you wanna go to the next panel? Yes you do.
Katie Jansen: Okay in the interest of time we’re gonna go ahead and get started, but feel free to get up and get a drink and move around as you want to. So now we’re gonna move more into focusing on tech, so this is engineering and ops here that you see represented here. This team works out of our Palo Alto office. So our business team mostly is here in San Francisco, and then Palo Alto is finance, ops, engineering, HR, some of HR is here, we split. You know how HR is. So this is gonna be about embracing change through actually building systems, so that actual adaptability we talked about on more of like a, you know, broad base and this is actually on systems itself. So this is Sonal, Anusha, Laura, and Swetha, and why don’t you guys just tell the audience a little bit about yourself. Take it away.
Sonal Gupta: Hi, I’m Sonal. Can you guys hear me? I work as a principal software engineer at AppLovin, and I’ve been working here for a little more than five years now. Anusha and I actually started a week apart from each other. So the infrastructure team, our main goal is to ensure high performance and scalability in our system. Essentially we wanna make sure that our ads are served in the most efficient and sort of fast way possible such that we are not only, the response times are as minimum as they could be, but we’re also not using as many resources and servers. So yeah that’s what my team does, and I work as part of the team and then some other things that I work on are revenue tracking infrastructure, among other things.
Katie Jansen: Anusha?
Anusha Ramesh: Hi, I’m Anusha, I’m also a principal software engineer at AppLovin. Like Sonal said, we’ve both been here about five years and two months now. So it’s been a while. And I actually work on the platform team. Funny enough all four of us actually work on different teams, so you’re gonna get a slightly different aspect on our engineering side, and the platform team, what we do is we make sure that all the data that we get from internal sources, external sources, get into the right databases, the right locations in the right time so that everybody in engineering, business side, everybody can actually use it. So some of the systems I work on are things like Pub/Sub system and custom counting frameworks.
Laura Pfister: Hi, I’m Laura, and I’m a software engineer on the ad server optimizer team. I’ve worked for AppLovin for almost three years now, and on the optimizer team, we strive to answer, essentially, two questions related to ad serving. So whenever we get an ad request we’re trying to determine, one, whether or not we wanna show an ad and spend the time and resources that it takes to do that, and then two, what ad do we wanna show and what valuation is that ad placed at. And so we’re actually making use of the data that Anusha’s team and the platform team give us to answer these questions.
Swetha Anbarasa: And hi my name is Swetha, and I work as a network engineer as part of the operations team at AppLovin. So we provide the backbone for all the stuff engineering does. So primarily my responsibilities include designing and configuring and maintaining the production and development networks at AppLovin, and also help with like maintaining our data centers as well as optimizing our network routes. I’m also responsible for capacity planning as our businesses grow.
Katie Jansen: Yeah so obviously we are processing a lot of data and doing a lot as you’ve heard here, and as I mentioned early on, we do that with just 160 people right now worldwide.
Katie Jansen: Let’s get real specific and talk about how and when do we decide to automate. You heard the panel before this talking about how that’s been a real advantage for us.
Katie Jansen: And then really what’s the best way to automate. If you guys could give some examples around tools, I think that would really help the audience here too to, like, get real specific. So you wanna take it away, Swetha?
Swetha Anbarasa: Yeah.
Katie Jansen: Okay.
Swetha Anbarasa: When did we decided to automate was a point where we needed to really scale. When the business started, we were able to you know manage everything with a few servers in a rack, managed or something, and it was all working fine. But as our businesses grew, we had to scale very quickly and we had increasing demands.
Swetha Anbarasa: From a networking perspective, our main concern was to find a vendor that actually has a development driven product. With the major players in the industry at that point, it was kind of a challenge to find a vendor that was actually development driven for our networking, so we had to do a little bit of research, and after some evaluation, we ended up using Cumulus Linux.
Swetha Anbarasa: Cumulus Linux is an open box solution. So you could actually install … It’s a switching software, so you could actually install the software on a bare model Linux machine and start working with it. So some of the advantages for going with that sort of a vendor is you can use the software to automate very easily, and we chose Chef as our automation tool. Chef is based on Ruby, so it was easy for us to write code in, as well as our server infrastructure also uses Chef.
Swetha Anbarasa: So networking and our servers together, everything is in one place in Chef, and we have like one repository where all of us contribute and change stuff, so it was easy for us to manage. And another advantage is, they were able to provide us with virtual images, so we were actually able to simulate a data center before we actually invest money and like buy stuff and actually deploy. A lot of it was being tested before we could deploy, so that was giving us a big advantage.
Swetha Anbarasa: Because we had limited resources, automation was actually playing a very big role, and we had our reasons to choose Chef and a few of the open networking platforms. And we also use Juniper routers which have been automated using Python scripts, and we integrated with RunDec to run a job in case we wanna do any sort of configuration changes or upgrades. For a small company, I think we’ve done a lot of automation, and we’re able to scale quickly with limited resources because of that.
Katie Jansen: Okay. And then how about on the engineering side? Maybe Sonal and Laura, what was it like, can you give some examples of automating some processes?
Sonal Gupta: The ops team automated the process of bringing up servers, and now engineering team had to make sure our services could also be deployed on all the servers that were being brought up, without a lot of intervention from us, and one of the big things we pride on in engineering is how agile our development is.
Sonal Gupta talks about automating processes at AppLovin Girl Geek Dinner.
Sonal Gupta: We push to prod every single day, and when I originally joined and even two years after I had been working, we used to push multiple times a day. So a single, any code change that you make could affect all the 60 billion requests that we are serving. For me my first year I was always very nervous when I was pushing that button to sort of deploy to production, ’cause there was no-
Laura Pfister: So nervous.
Sonal Gupta: Yeah. But it’s, I mean, ’cause we didn’t have a lot of structure around how to push and how to make sure that the changes we were making were very limited. So we tried to build an A/B testing sort of test control framework that would ensure that any new changes we push had very minimal impact and we could ensure that once it was verified, the changes were assessed, only then was it pushed to production. And all of these changes, we couldn’t like hack them, do Eiffel statements, or you know basic sequel queries, ’cause they were all prone to errors. So we had to like come together sort of build a structure that was away from human error, we could validate all the changes we were making, and we could triage and sort of back track the issues, and how we push something, when we push something, and how do we fix it.
Laura Pfister: Right and to add to Sonal’s point, so the A/B testing framework that we put into place in order to kind of combat all these issues is really really important to the productivity of the optimizer team. So without this, we were kind of able to A/B test maybe one or two features in like a week or something. The process was slow and we might have to wait for one feature to be clear just because we just didn’t have the bandwidth to do a lot of different things and once. And then it was hard for us to analyze the how one feature might impact the entire system, and it might be hard to pinpoint exactly which feature might be causing an issue, and we might need to even have reverted the entire all of production to kind of backtrack that problem. So with this new framework in place, we’ve been able to really really increase productivity and then just the overall safeness of our system so we don’t have as much of that fear when we’re pushing, we can really control exactly how much of our traffic is diverted into one feature, and really pinpoint and analyze the impact at each and every feature. So we can get a lot of features out and do it really quickly, and that’s really made us much more productive, much faster, and it’s just really a lot safer.
Laura Pfister: So I think that’s one of the biggest things that have allowed us to scale a lot of optimizer features and also infrastructure features really, really quickly as we’ve grown.
Katie Jansen: So switching gears just a little bit then, so we talked about how we automate processes, but what about how we grow and scale with those, have you had to change processes over time? Have you had to scrap them completely and start over, and what does that look like and how easy is that to do?
Anusha Ramesh: Like I mentioned when I did my introduction, I work on a custom counting framework as part of my daily jobs to say, but way back in 2014, if you go all the way back then we were a pretty small company still, we were starting to grow pretty rapidly at that time, and all of our data, all of the answers that we were trying to get from our data, we would use SQL queries. That’s what everybody does, you query your database and you get your answers and you look at ’em and you analyze. And as we grew and grew as the year went by, our data got bigger and bigger and bigger, and suddenly these SQL queries took an hour to come back with data. An hour and a half. They’d hit database connection issues, and networking issues, and so at the end of it, we were like, okay, well, this doesn’t quite work. It’s not gonna scale as fast as we are growing.
Anusha Ramesh: We actually took that entire system, redesigned it, re-implemented it, and pushed out a brand new custom counting framework that provides the answers and the same results that we wanted out of those SQL queries in ten minutes. We can do a query on that database, get results in ten minutes. And this entire thing because of how fast we move as a engineering department, we had it in from design to production in three to four months, which is pretty quick in my opinion, and then, of course, years go by. We keep growing, growing, growing, and with all the alerting and monitoring that we have in place, we realized that we’re soon gonna hit a limit there too. So what do we do then? Alright, scrap the whole thing. And we literally went down back to the square one, ’cause I had to redesign the whole system. Redesigned it, re-implemented it, re-pushed it to production three months later in 2016, and knock on wood, that is the system we have in place today and it will continue to go. And I bug ops about it all the time, but we have nice alerting and monitoring systems that ops has set up for us that keep our systems in place.
Laura Pfister: Another thing to go with the scalability, so as data has increased, the business has increased, and logic would follow that campaigns and ad inventory have also increased. With this increase, the ad selection process has also kind of gotten bogged down at times. There’s more ads to go through, so every single thing that you do in that part of the system is gonna start to be slower. This is a gradual increase, and, you know, it didn’t really impact things, but sometimes you wanna scale things, or sometimes scalability is about kind of dealing with making things larger, but in this case we wanted to actually add campaign breadth, a variety of additional campaigns to serve on a different component of our system called real time bidding, and in this system we are bidding on other ads, ad request from other systems, and in order to do this, we need to meet a certain millisecond threshold. When we tried to add more campaigns in this system, we found that we were no longer meeting this millisecond threshold. We were looking at the data, kinda seeing like okay nothing’s happening here so we’ve gotta do some analysis. We used a tool called a flame graph to actually analyze the CPU usage of our ad server and see where things were taking the longest amount of time.
Laura Pfister: We looked, and sure enough it was in our ad selection process. We were able to use this to see exactly where we were spending the most time, and then we were able to analyze this and go through and figure out where we could make cuts, where we could figure out how to do the exact same thing, get the same result and produce the same ad that we would in the previous system, but do it faster. Through this process, we were actually able to increase speed as much as three times faster in some cases, and drop CPU load a lot. It was really a great success and just another example of how we were able to scale things quickly, and I mean the entire process was like a few weeks, maybe.
Katie Jansen: How do you decide that? How do you decide when to just scrap it all together like you were talking about, or keep working on it and kinda push through? ’Cause it sounds like, especially in your example, Anusha, it was almost like broke and we were in trouble. Do we have something that’s in place that you know lets us know about that ahead of time, or is there some kind of meeting or team situation that occurs for you guys to decide, hey this is looking like it might need to be changed, we should invest more? What does that look like?
Anusha Ramesh: Specifically for this, well not even specifically, all of our systems are actually, we have a common tools package which platform keeps up to date, and most of our services plug into this system and use our tools to send metrics, send stats up to Graphite, which is one of our monitoring tools. And Graphite is amazing. If you ever see our office in Palo Alto you’ll see just TVs full of graphs everywhere, monitoring the most, like every little thing you could think of.
Katie Jansen: I like that you think that’s amazing.
Anusha Ramesh: They’re really cool-looking graphs, though. We have a ton of TVs up everywhere.
Sonal Gupta: I think it’s interesting when people walk by they see all these graphs, they don’t know anything what they’re about.
Anusha Ramesh: [crosstalk] … tiny little spikes somewhere of one thing going wrong, and suddenly you’re like oh wait, that’s wrong, we need to figure out what went wrong there.
Katie Jansen: To be fair, we use those in our videos and all sorts of things for marketing.
Anusha Ramesh: [inaudible] promotional videos. That’s one of the big things that tells us when a system is coming to a limit. We have stats on our databases, are we hitting some size limit where we can’t grow any further, are we hitting a CPU limit, can we not process things a little faster, are we hitting a backup somewhere where files are just sitting around just waiting to be processed and we don’t know. And if you don’t know about that, then that’s a problem, because we are gonna be behind. Our reporting pages aren’t gonna show the right thing, business side won’t see the right data until maybe hours ahead and then they’ll come to us and be like “Where’s the data?” And we have to be like “Uh, let’s go find it.” But all of these tools in place, Graphite, Zabbix, Chef, all of these things help you automate and figure out these alerts that might happen, and for the counting framework, that was the biggest thing that actually told us this is not gonna work. We hit a threshold that we had that wasn’t full but not there, like an 80% threshold, decided to do some things to minimize it for the short term, and then decide a long term solution.
Anusha Ramesh: I think that’s one thing in engineering that we do a lot, is we do a short term solution that is like for five, one week, two weeks, let’s fix the problem immediately so we continue to run, and then figure out a long term process, a long term solution that we can push out in a month, month and a half that will actually fix the entire solution and we can go for another year, or two, or three, or however long it may be, with the system.
Sonal Gupta: One of things I’d like to add to that is so one of the very first and sort of the most drastic scrapping of things we did was our ad server used to be in PHP, and a lot of our other components were in PHP as well. While we were debugging an issue with real time bidding and sort of why aren’t we bidding fast enough, why aren’t we winning as much, we realized it was actually the choice of language that was the bottleneck for our system. We decided, I mean we could’ve continued to patch and sort of improve the PHP system, make it faster, sort of find some wins here and there, but we realized that it was sort of avoiding the unavoidable, ’cause at some point we would have to move away from that system. We took a step back and we decided, okay what can we do, and we decided to come up with a C++ ad server solution. I mean it was a sort of a big undertaking ’cause most people on the team didn’t even know C++, but knew how powerful the language was, so within a few months we all decided to learn C++ and had a working ad server solution that not only supported our in-network ad serving but also our real time bidding solution.
Sonal Gupta: And over the year, then we migrated all our other services away from PHP to Java, or C++, and this is like something that’s very fun. One of our first intern projects was to move a service away from PHP to Java, and the intern finished it in two months. That’s sort of the trend at AppLovin. We try to get, if there’s a need and we realize there is a problem, we don’t shy away from completely scrapping the system, and more importantly we try to sort of learn quickly, everybody grows together, and try to get a solution out as soon as possible, and not a hacky solution but something that will work long time. I mean we’re currently still using our C++ ad server, and everybody on the team, we are always looking to make changes even if they’re like small changes to the two string function so we can get like little wins here and there and continue to optimize our system.
Katie Jansen: Yeah. That actually brings me to our final question, which is what about looking forward? You’re building this stuff now, how do you future-proof it? Can you future-proof anything? Maybe you could each give me your thoughts on that and then we’ll open it up for Q&A.
Swetha Anbarasa: Sure. think yes, we can future-proof things to an extent. So from the ops perspective, the way we try to plan is we don’t just have 2X or 3X capacity. We actually wanna have at least 10 times the current load. That’s the capacity that we look at, ’cause we’ve actually run into instances where that really helped us and we really needed the bandwidth and the servers to help. I know Anusha agrees, ’cause we’ve run into a lot of issues where we had 40 gig of bandwidth and it was actually, we never used it that much, but suddenly one day we needed it, it was during the holidays.
Anusha Ramesh: It’s always [crosstalk]-
Swetha Anbarasa: Yeah it’s always during the holidays that we need it, and we actually had a 40 gig bandwidth between U.S. east and west, and we were able to do the data transfer quickly without anyone even noticing or waking up anybody or like disturbing them from their holidays. So that’s how we plan, and I think one of the important or like key concepts about being future-proof is to have excellent monitoring, and for the size of our company, I feel like the monitoring we have is really good.
Swetha Anbarasa: We use Graphite. So Graphite is beautiful, colorful, so it’s numbers in graphs. So you’re not looking actually at tables, you’re not looking at boring data, you’re actually looking at graphs, and spikes, and dips, and it’s all like time and data, so it’s easy for us to understand, and you could actually add any component to Graphite and start tracking it even before you understand why we are tracking it. You know maybe a couple of months later you run into an issue, and you go back to the one thing that you were tracking, and you look at it, and you are like “Okay, thank god I actually have these stats.” So Graphite keeps collecting, and it works with Zabbix and also holds SNMP data from networking devices. So it’s sort of like one place where we can look at a lot of our stats and attributes and data to monitor.
Swetha Anbarasa: Apart from that we also work with, we also have like Google Cloud Compute, so we have systems where we can just spin up several servers if we need it and then tear them down once we are done. So that also helps us to be future-proof to some extent, and also maybe like from the networking side we also look for vulnerabilities that keep coming, we proactively patch our systems, do upgrades, be on the lookout for anything that’s going on, and kind of have everything up to date so we don’t have to worry about when something actually hits us. Yeah.
Katie Jansen: Laura, you wanna … Final thoughts?
Laura Pfister: Yeah, so when I’m working on something it’d say it’s probably a little bit more granular, I would say for one thing the monitoring that we have in our system definitely makes it really easy to see any small changes. So if there is an issues we’re gonna catch it faster, and that’s really, really important. From more a designing things for future proof perspective, I’d say, since I’m adding a lot of new features, a lot of new functionality, whenever I’m looking to add new things I try to look not just for, will this last like a week, but designing it in such a way that I’m trying to think what else might I use this for. So that’s just kind of an important way that I approach pretty much all the code that I write.
Katie Jansen: Anusha?
Anusha Ramesh: So like Swetha said, we definitely have the ops, like if they have 10X capacity, platform will use it. At some point in time we will 100% use it. If it’s just an emergency data from one side of the country to the other because something went wrong, we will use it.
Anusha Ramesh: I know I bug ops every single day for random stuff, and I’m glad that they allow me to do that. But as for future-proofing, a lot of our systems they’re all repeating the same thing, monitoring, monitoring, monitoring, because that is the biggest thing you can do. And our Java services that we use, we all have one base set of tools that we plug into every single Java service, and it makes every single developer, they don’t have to know how this tool is written in Java, they just use it and they know that as soon as they use it, five minutes later it’s gonna end up in Graphite, it’s gonna, you can pull up a dashboard on it, you can see a graph of it, and it’s done, and it’s there for you to see forever. Until you stop writing to it and it goes away.
Anusha Ramesh: But you at least have the ability to do that, and that’s one of the things that is really nice is it just exists and people use it. Funny enough, this monitoring tool, we actually were trying to figure out some issue, like we were trying to test a new database, something was going wrong, we realized that it’s actually our tool that writes to Graphite that was not working. We took it around, we redid it, and a day later we actually got 200X improvement out of it just by rewriting the whole thing, and we went from about 10,000 writes a minute to two and a half million theoretically writes a minute.
Anusha Ramesh: And it was like, of course, that’s all theoretical numbers and it’s more like a million writes a minute, but in the end it’s, like it went out to production, like it went out into our code system. Nobody knows that it actually got changed on the back end, but they’ll use it. They’ll see an improvement, and it’s just kinda there. And that’s the cool thing about having these systems that make it future-proof. They exist, you use it, things show up in graphs, it works. And you have to think about if you’re gonna do those million writes a minute, can your system handle it? And that’s future-proofing. And that’s kinda what we try to do at least a little bit.
Katie Jansen: Any final thoughts?
Sonal Gupta: I’m gonna add something that Anusha told us earlier in the day, that even with the issue the monitoring system had we still only needed two servers for getting all the metrics from all our different Java services. So the revenue tracking infrastructure that I work on, ’cause it’s such a tricky component, and that is where we track our revenue, any changes I make could potentially make us lose events, make us lose our revenue. So when I started, I integrated with the monitoring service, ’cause the platform team had done just such an amazing job giving it to us.
Sonal Gupta: One of the best things is when you look at our data, it follows a very certain path, and if there’s any variance in it, you will be able to tell there’s an issue. So that’s one of the reasons why monitoring becomes so important ’cause we can see every little detail of our system and how it changes. So for specifically, for example, for the revenue tracking infrastructure I can see how many revenues events I’m tracking every single hour, minute, day, and if there’s a change, I can, we can determine sooner than later that there was a change, and it was usually correlate to a certain push or some sort of event that happened and we can either go ahead and revert the change or sort of push a fix out quickly. So, yeah.
Katie Jansen: Cool. Alright in the interest of time here I’m gonna open it up to Q&A. Maybe take one or two, and then ’cause I know I believe there’s dessert. Yes, there’s dessert and drinks. I wouldn’t wanna promise dessert if it wasn’t there. Any questions? Yes, right there in the middle.
Audience Member: Hi, I’m Kayce.
Katie Jansen: Hi, Kayce.
Audience Member: First of all it’s been so fun to listen to you guys talk. You’re so passionate about what you do. This has been really engaging to hear all of you talk. I guess my one question is what’s the one thing that you wish you could do faster ’cause you’re going at such a big speed? How do you communicate together? I mean you’re all on four different teams so how do you make sure you aren’t messing up each others’ stuff and what’s the one thing you just wish you could do faster?
Katie Jansen: That’s a good question.
Anusha Ramesh: Thank you for your question. One of the things about not messing each others’ stuff up, is we are on four different teams so we do kind of work on separate projects, and we do, at the very beginning back in like 2013, when we started, we were using GitHub, Algin, all that good stuff for code version control and things, but we didn’t have a good code review process in place, and over the time as we grow from just maybe two or one engineer in a team, to four, five, eight engineers in a team, we’ve put a code review process in place, we’ve put in a structure so that we actually don’t step on each others’ toes. We have branches, we have to push the master, and then push from master to production, and do all these things, basically red tape if you wanna call it, before we can actually push it out. But even at that point, with all of these processes, we still push code every single day to production. We still push out our stuff. It’s because we have a good, I guess we are all used to it now, and it’s such a good process that we have in place that it doesn’t seem, it’s streamlined. That’s the word I was looking for. It’s a streamlined process that actually works for us.
Sonal Gupta: And I think we are still a very small team, so really we all have a lot of different responsibilities and roles we play every day, and we are small enough that they’re very separate from each other even if you’re part of the same team. So there’s a lot of communication and as you had mentioned, because you know facilitating sort of the open office structure, whenever we have questions or we wanna discuss something, we just go to the person’s desk, it doesn’t have to be like an elaborate meeting, and so really, like, if there’s something you’re working on and you think somebody else might have done it before you can just go ask them. It’s as easy as that.
Katie Jansen: Yeah I mean I’m obviously not on your team, but I do talk to, you know, the VP of engineering quite often when I have, which is rare by the way.A lot of times VPs of engineering don’t wanna talk to the CMO and they are annoyed by us, and that’s not the case here [at AppLovin]. So I would say – or you know to the engineers themselves – I would say communication is really key, and just different types of communications, whether it’s taking the time to talk to someone, or we really leverage Slack. I can’t say enough how much we leverage Slack. We have multiple channels, we will do direct messages, however we want to use it, and it’s been pretty awesome.
Katie Jansen: And then we all use Asana, I would say too. They use Asana, and then they made my whole team use Asana, and they didn’t like it but they like it now. It took a while. I actually try to sometimes mirror some of the processes they’re using, because I think that it helps. Obviously I’m not gonna do Graphite and those things, but like the communication processes, trying to do similar things helps us communicate better as cross functional teams across the company too. A question back there, yeah?
Audience Member: Hi my name is [inaudible]. So great you guys are so flexible, when the systems just not working you change it. My question is when you change the database, you know scrap it, how is decision made? How do you be able to push that idea?
Swetha Anbarasa: I think the decisions are mostly a combined discussion. Like Katie said we don’t have that many meetings, so in case something like this comes up, we would do a discussion where we pull in like the VP of ops, engineering and then the concerned people in the team teams that are involved, and sort of like white board it, get everybody’s opinion, write down some stuff that, you know our top technologies that we wanna use, and then kinda get everybody’s opinion. And again we would like discuss over Slack in a particular channel and kinda like arrive at a conclusion. So it’ll be more of a discussion with the team’s concerned.
Sonal Gupta: And I think just when everybody wants the best for the product and the company, it’s very easy to come to a quick conclusion.
Katie Jansen: Yeah, we’re all on the same page.
Sonal Gupta: Yeah.
Katie Jansen: Alright, one more question. Go for it.
Audience Member: So as the SDK provider, you deal with a lot of customers and face a lot of issues. How do you track the issues that are reported in and do you have analytics on them to figure out like where to fix things?
Sonal Gupta: Actually with have a separate product team, Jimmy’s team, that sort of does that for us. So one of the amazing things about engineering is that because of our product team they take care of sort of triaging and determining the issues before they even get to us. So by the time an issue comes to us, it’s probably something that is important. And you know you work with the SDK team a lot.
Laura Pfister: Yeah, I’ve done a lot of those kind of ad server, sever side settings, and yeah I only get the ones that we’re certain are a problem immediately. So having that kind of filter above us makes it so that we’re able to get those out really, really quickly and prioritize them as much as possible.
Katie Jansen: Yeah, and the product team, while they are business people, have some engineering and like background, so they can triage pretty well too. So I think that kind of having a team that’s in the middle there makes a big difference. Alright, I’m gonna wrap it. We are available to answer more questions, but I’m gonna let everyone go get drinks and dessert. Thank you very much for coming.
