Shea Kelly: Is that working? Am I on? Welcome, everybody. Happy Monday, or as we say in my household, and everyone knows this here, happy day. How many people watch Sponge Bob? Then, you know what I’m doing. Listen. I really, I want to reiterate what Angie said. First and foremost, we are actually just thrilled and delighted that you’re here, and I mean that. We have been talking about doing an event like this for a while, but it’s always so backed up, the schedule like when can we get and do one?

Shea Kelly: We’re thrilled to do it for a couple of reasons. One is probably the obvious — we’re a growing business. We’re very deliberate and intent about wanting to continue to expand and diversify our company with rock star talent, and so, at a minimum, we want to expose and have people get greater awareness of Sumo, who we are, what we do, but obviously, the biggest thing, I’ll be honest, I think we have joy today in having some of our fabulous team members, who are all smiling at me now, share some of what we do.

Shea Kelly speaking at Sumo Logic Girl Geek Dinner.

Shea Kelly: You can get some insight into, aside from Sumo, I think it’s an industry that we’re in that’s pretty darn exciting and interesting, and we hope you get some good things from it. It’s a big week for us because this actually is the week of our second annual user conference. We have some 600 people registered to be at a hotel in Burlingame later this week. We don’t do anything small, so we thought in the same week, let’s have a board meeting, and let’s do a Girl Geek Dinner and just six other things. It’s like, but we like to go big or go home as they say, so again, welcome. I do want to also tell you, in case you didn’t notice, when you leave later, there’s a gift bag for you up front with a little bit of what we would call Sumo schwag. You cannot leave without a Sumo squishy. In the bag is a Sumo squishy for those … So many people are like, “Are we going to get a squishy?” You’re getting a squishy. They are so popular. Go on a campus, and it’s like they look at you, go, “Hi. What are your jobs? Can I have a squishy?” It’s like, “You don’t want to work for me. You just want a squishy.”

Shea Kelly: Okay. I am only going to take a few minutes here before the main speakers. I am not the main speaker. Take a few minutes. Just tee up a little bit about Sumo Logic, who it is that we are, what we do. I don’t know how many folks are aware of us, so I’m just going to do a quick welcome and intro there. The welcome, I think, is done. Then, we’re going to shift to our presenter. Bret is going to focus as we’ve talked about the whole, how we enable customers to be super successful in using our product is, we’re coming at it obviously from different ways — we come at in terms of the user experience and the design and how do we think about what customers need, and how they use the product, and how do we build that into the product, which then leads to Riya, who is going to talk to us about from an engineering and development perspective, how do we do that? How do we continue through the process? It is literally non-stop, and you’ll see that and hear that in what we talk about today. How do we infuse that into our development of the product?

Shea Kelly: The third piece is learning. Stacy is going to talk about the tools and all the things that customers need, and whether it’s docs and it’s training and it’s certification. They need to be enabled, so they’ll use the product, right? That’s pretty obvious. Oh, lights are out. Okay, so we’re into a groovy mood here. Okay, this is good. No, I like this. Okay, thanks. We do it right here. Then, the last piece, Jen’s going to speak to down here at the end is security and privacy. I mean, it is obviously, you can’t pick up a newspaper or see anything or think about your own data, personal data, but companies are migrating, as we’re going to talk a lot about, their data to the cloud. That has with it inherent, oh my gosh, how do I make sure that it is secure, it’s private, and that we have all the compliance and regulatory things in place? Jen’s going to talk a bit about that. Then, at the end, we’re going to open it up for discussion. I’ve already had a few questions that have been raised that will feed into the discussion later. Into that mix, we’re going to add the lovely Mary Ann O’Brien who’s sitting here who leads a big team for us in sales, and she can also bring a bit of perspective in terms of how we go out and speak to customers, okay? You can always throw a question out in between. Otherwise, I think what we’ll do is save a lot of it for the end. Sound good? Happy Monday? We’re still good? All right, you’ve had food. I hope it was good.

Shea Kelly: Again, I’m just going to go through this quickly, but if you think about and say, “Well, what does Sumo Logic do?” what we would say our tagline is, is Sumo Logic is a machine data analytics service that helps companies to build, run, and secure their modern applications. You say, “Okay, what does that mean? What does that really translate into?” If you think about the path over the years, the transformation from, I guess what I call the old school with data centers and on-prem solutions where data actually is stored and housed, and the transformation, as you know, this digital transformation we’re living in and will continue is to the public cloud.

Shea Kelly: What the natural outcome of that is that we’re building a different kind of software, and it is a software that is always on. It’s a software that, in many cases, or these modern applications that result from that are customer-facing. They are revenue-generating. Those are new aspects of something. It’s a 24-hour if you think about it in terms of availability. Take out your phone. Think of the things that you take for granted now, and all of that data has to somehow, in the private cloud, in the process, be built, run, and secured in that manner.

Shea Kelly: To us, that leads to three things, and I hope this translates. If not, sometimes, the … I don’t know if the white on white is looking good here, so the very first thing is obviously the cloud. Data smog rating, you know this. It’s going to continue too. It’s going in a rapid phase, and it’s going across industries and verticals into some of the sectors that before, would have been hesitant to, into banking and in insurance. You’ve got that migration happening pretty quickly and pretty voluminously, if that is a word.

Shea Kelly: What that second part of that is it leads to this notion of continuous delivery. It’s always on, and that’s where it translates to the humans, to the people who are building and running and securing these apps because they’re not doing it the way they used to do it before. Now, if you think about it, and you think about the old way of developing software, the old … What do they call? Waterfall, so you would have an idea. You’d put it in development. It would go through a whole bunch of steps, and at some point later, something would spit out on a disk, and you’d take it somewhere. That’s not what happens anymore, right?

Shea Kelly: here’s a different way of architecting these modern apps, and so, what has to happen and has to follow that is we have to have a supporting mechanism for these teams who do it. For us, we say, okay, if it’s all going in the cloud, and it’s continuous delivery, it’s always on, that’s where Sumo Logic comes in. What we do is we provide a product that’s called continuous intelligence. What that does is it gives these teams who are having to architect those modern apps in a different way and who have to manage and do things in a different way as teams, it gives them an analytics product that helps them to again, build, run, and secure those apps in a way that meets these modern needs.

Shea Kelly: Just quickly, we are about now eight years in business. I think our founder might still be over here, so I’m going to have him give a wave. Christian Beedgen. Should I tell the story of how Sumo came to be? I won’t do that, but anyway, it’s a dog. It’s not Sumo. Catch me for the story. Anyway, so, Christian is our founder. Bruno’s still here. He’s our founding VP of product and strategy, and so, eight years in business, we are now some 1,600 customers strong… 50,000 users I think… so it is proliferating pretty quickly. We have, I think as a lot of SaaS models do what you’d call a land and expand. We get in on a use case, and then over time, they say, “Well, it works so well here. We need to expand this,” and it easily can expand across the organization.

Shea Kelly: We are serving companies, like I say, in every vertical, in every industry. If they have a modern application, they have got to have a way to analyze and think about that data as they build, run, and secure it. We have customers from Anheuser-Busch to Pinterest to … Who are some of the others? As I’m thinking about it, JetBlue or Betfair, Hearst, and so, again, these are also global organizations, in many cases. For us, it’s, like I say, eight years strong. We’re some $250 million in investment, so we were no longer a startup. We’re a late-stage private company, I would call us, funded by some of the best and brightest in the Valley, and so we’ve got a ways to go on what my last point to be here is this journey. We’re just getting started in some ways. I mean, this proliferation of data, if you think about it, I am going to step in here just third from the right. If you think about it, that’s 2018.

Shea Kelly: Machine data by 2020 is predicted to account for 40% of all data created. If you think about it, this machine data, which is what we focus on with our application service, is growing at such an exponential rate. Customers and companies have got to find a way to analyze that data. This is the background. Again, if you just think about it, it’s digital transformation, software as a service, companies are going digital, modern applications always on, and through that process, which is what leads to our discussion today, is this question of, how do we make sure, because customers may have different needs and how they do these things, that we are listening to them and looking at all of their needs and figuring out how we build that into the product ultimately through how do we do security and compliance around it, okay? With that, I am going to turn it off to the real presenters. I’m going to start here with Bret. Here’s our group. Here you go.

Bret Scofield speaking at Sumo Logic Girl Geek Dinner.

Bret Scofield: Hello. Good? Okay, perfect. I’m Brett Scofield. I lead UX research at Sumo Logic. UX research is a relatively new discipline, so I wanted to talk a little bit about what that actually means. Design’s been around for a while, and I think product development has also been around for a while. UX research is really a combination of … or it works very closely with those two, so a lot of times, what we’re doing is getting feedback from our customers and from our non-customers on ideas. Sometimes, these ideas are really big ones. Should we build this feature? Is there a place for this new product? That type of thing. Sometimes, it’s more granular. We’ve decided, yes, we’re going ahead with building a metrics feature, and now, we need to get into the nitty-gritty of what exactly that looks like and how our customers are going to use that type of thing. UX research, I converse with our customers pretty much all the time. I advocate for them heavily, so I’m always telling their stories to our engineering, product management, designers, just really emphasizing like, “Hey, our customers are running into so much pain with this. Can we please, please change something?” et cetera.

Bret Scofield: Then, on a personal level, obviously, I enjoy Boomerangs. This is with my work wife. Hi, Rebecca. Then, I run a lot. I’m a marathoner, so I’d love to chat about all those things. I’m on the team, I’m the food contrarian for all things, so people are always like, “Hey, have you tried this awesome new like hybrid taco thing?” I’m always like, “That sounds like it sucks.” That’s usually my role. Then, I wanted to get into a little bit about why I chose Sumo, why I’m working here.

Bret Scofield: First off, we have a lot of really interesting gnarly problems to solve, and that’s great. You start off Monday morning, like today, you really dive into those problems. You’re working with engineers. You’re working with security people. You’re working with a bunch of people who are really, really smart and have a lot of things to say. That’s amazing, and that’s great to work on those problems, but then, our team does a really good job of balancing that with a little bit of levity. We go to lunch with the UX team. We talk about tacos. We talk about horoscopes. We talk about whatever else. That gives you enough breathing room and enough space to go back in in the afternoon and to like really dig into those problems again.

Bret Scofield: I also really appreciate that there is a huge customer focus throughout the organization. That’s why we’re giving this talk. We’re talking today about how the customer influences everything that we do at Sumo Logic, and so, it wasn’t hard to pull that together with all these other women because everyone in the organization really thinks about the customer and keeps them at the forefront of the decisions that they’re making.

Bret Scofield: Okay, so today, I’m going to be talking through the product development process, and I’m going to be talking through the two halves of it, which here are running concurrently. Instead of sequentially, instead of having all of this discovery happen before delivery happens, we have both of these processes going all the time at Sumo Logic. The first part of this, the discovery bit is really answering those like super-high level questions. Should we build this thing? Is there a market for this thing? That type of, it’s exploratory, it’s really like pie in the sky sort of stuff.

Bret Scofield: Then, after we’ve figured something out from there, we’ve determined, yeah, there is space for this, this is something that our customers would actually want and would use, then those findings are given to delivery focused teams. These teams have an idea. Now, they’re like, “OK, we need to build a metrics functionality or we need to build this type of functionality,” and so they need to dive into more of the specifics of that. They’re figuring out, “OK, with our customers, how exactly are they going to do what we want them to do in this? Is it a dropdown? Is it a visualization?” They get into the meat and bones of that.

Bret Scofield: What I’m going to talk about is how our customers drive both of those processes. With the discovery processes mentioned, these are the really big high-level questions, and the major question that we’re trying to answer here is, are we building the right thing? Is there space for this thing that we want to do? One of the recent … Well, not recent. One of the things that I’ve been working on for a long time at Sumo logic has been our personas. Many of you are familiar with personas. These personas are not real, but they are an amalgamation of our customers and their mental models.

Bret Scofield: The reason that we went through the exercise of creating personas is really because we want to give the entire organization a common vocabulary. Melinda has a certain mindset and approach to the product, so when she goes into Sumo Logic, she’s likely trying to accomplish a specific set of things. She has a certain familiarity with a product. She goes in in a certain frequency. Andre, completely different. He uses Sumo Logic for different things, goes in with a little bit more uncertainty, all these types of things.

Bret Scofield: When we refer to our personas, I can say, “Hey. Stacy in documentation, I’m dealing with an Andre.” She immediately knows what that means, and she knows what sort of pains he’s likely to be feeling, where he’s coming from, all that type of stuff. The process of creating our personas, we attacked this in, first, a qualitative way. We met with a bunch of our customers on site, watched them do work. We talked to them about their educational background, their career trajectory, what are the things that are painful for them, et cetera.

Bret Scofield: Then, after that, we aggregated all the data, came out with three personas. Then, during a hackathon, we went through the process of deriving quantitatively what our personas look like. Sumo Logic, as Shea mentioned, has a ton of data on what our customers are doing in the product. We analyzed it during this hackathon. We ran a k-means clustering algorithm on all that data, and we derived the key use cases. Those key use cases actually mapped really well to the personas that we had derived qualitatively. There’s both a quantitative and a qualitative backing for these. Then, they’ve permeated the organization. I think almost everyone at Sumo Logic knows who Kathy and Andre and Melinda are, so.

Bret Scofield: Then, as mentioned, so this was a discovery project. As mentioned, the discovery stuff often influences the delivery things, so now, in a lot of the delivery projects that are going on, we define at the very beginning who is this specific feature for? How do we expect an Andre to approach this? How do we expect a Melinda to approach it? How would it be different? Those types of things.

Bret Scofield: Then, I want to talk a little bit more about the delivery process. As mentioned, the delivery process is when that sort of overarching thing is defined. We know we need to build this specific thing, but the question that we’re seeking the answer here is, are we actually building that thing right? Are we doing the things that need to be done so that a customer can actually do the thing that they want to do in here? With delivery specific things, we start at the very broadest level, and then we narrow in.

Bret Scofield: One of the things that we commonly do here is participatory design. We have quite a good relationship with internal users of Sumo Logic, so our customer success team uses Sumo Logic heavily. They also interface with our customers. Sales engineering, similar case. We bring them in. We also bring in our own engineers. They love dogfooding Sumo Logic, and they generally like working with us, so we bring them in. We do design exercises where it’s pretty much, we start with a blank canvas, and we have them draw, what is your 10-star experience? What would this future look like in an ideal world?

Bret Scofield: We’re actually, as Shea mentioned, we’re holding our user conference on Wednesday and Thursday of this week, and we’re doing a large-scale participatory design with a bunch of our customers. We validated that there’s a specific feature that we want to build, and so we’ve made kind of a Lego kit for all of our customers to build their ideal version of this. It’s a super fun thing, and it’s really great because it gets buy-in from a lot of people. They feel like they are a part of this thing.

Bret Scofield: After that, the researchers and the designers will aggregate a lot of the feedback, and they’ll put it together, and they’ll start actually working with the pixels. This looks a little washed out, but they’ll start building designs and prototypes. Once those are in a solid enough state, we’ll put those in front of our customers. This is actually one of our customers in Australia. He’s expressing disappointment, which is … that sometimes happens with research. He’s a little bit bummed about one of the things that we had sort of neglected to redesign. We were launching this redesign, and there were some areas where we’re like, “Oh, well de-prioritize those. It won’t be a big deal.” Then, we found out from a ton of our customers, “No, it actually is a really big deal, and they are really upset, they want us to redesign this.”

Bret Scofield: This changed the trajectory of the project. We stopped, and we said, “Okay. We actually do need to redesign these few screens and make this a cohesive polished experience for our customers.” Then, after something has been designed and released to all of our customers, we measure. We work very closely with the product management team and then with our engineers to instrument the log data so that we can see what people are doing with this new feature. Are they exiting Sumo Logic right after? Are they continuing with their workflow? What exactly is being done with this new thing?

Bret Scofield: This is great because it allows us to measure success, so if we can see a lot of engagement with this, we’re pretty happy. It also sort of allows us to jump off with more qualitative research. If we see, for instance, that everyone’s exiting Sumo Logic after using this new thing, then maybe we should delve into that. We should hear some stories. We should figure out what’s going on and make some adjustments. We’re using the Google HEART framework actually to drive which sort of metrics we’re tracking for this.

Bret Scofield: Yeah, to sum everything up, I just talked through how the customer feedback drives these two halves of the product development process at Sumo Logic, so we have the delivery process and the discovery process. Both of those are heavily influenced by the thoughts and feedback that come from our customers. Then, I’m going to hand this over to Riya for her perspective from engineering.

Riya Singh: Thank you. Do I use this too? Okay, cool. Lights are bright. Hi. Hi, everyone. How’s everyone doing? My name is Riya. My name is Riya. I’m in the engineering development team. I’m a team lead of the data engine team. Quick show of hands, how many engineers out here? Software engineering. That’s one-third of the room maybe. Okay, cool. Feel free to connect with me on LinkedIn if you have more questions after my talk, and I’m happy to answer more questions.

Riya Singh: The theme of today’s talk is, how does customer feedback impact different product areas? I’m going to give you the engineering perspective or how we are listening to our customers. A little bit of background about myself, I joined Sumo about four years ago. I started as a software engineer. Today, I’m the team lead of data engine team. My team is about five people, and we mostly work on streaming systems at Sumo, so anything that requires real-time, low latency applications like live tail, live dashboards, alerting, and, so on.

Riya Singh: I’m also a dance fitness instructor, so my mornings are all work, work, work. Nights are all dance, dance, dance, so it’s been a lot of fun. Sometimes, I’d work. I love Sumo because I can combine both my passions. I do all kind of flash mobs at work and all those fun stuff at work, but the main reason I’ve been with Sumo for years is because I’ve seen many companies in which engineering is given a project and said, “Okay, these are your requirements. Go and build it.” Very few times, we try to understand why we are building it or if you’re building the right thing or why is this important? Sometimes, I feel engineering does not get that perspective. In Sumo, it’s ground up, right? As soon as you’re trying to decide what to build next, we’ll ask the right questions. The advantage that we have is that we are our customer zero, so we are using our product as much as anybody else. You kind of understand why this is important, and that perspective really makes you make good products.

Riya Singh: Let’s talk about customer focus development. Some things are very strategic, right? Sometimes, you want to build products ahead of the market, right? We want to build things that nobody else is building. We want to be innovative. We want to be strategic. There are a lot of projects, which come through deciding to be innovative, so that aside, there are other things that we do when we get customer feedback from various channels. We work very closely with Bret and her team, so as soon as she’s going to a customer and having her discussions, we are reading back the reports come back in, and we try to figure out, “Okay, where are the different things we’re developing incoming?” and build something to help that customer.

Riya Singh: Two very nice things that we have in Sumo, I’ll just describe them a little bit of detail. This is my kind of favorite place to hang out. It’s the ideas portal. We have a public ideas portal wherein customers can go and mark the ideas or features that they want to see in our product. It’s very nice because you can see where it was created, how many words are there, and what’s the progress. That’s a very nice way to find out what is important to our customers, and we can build the right thing. That gives us a lot of perspective.

Riya Singh: The other place, which I really, really like as well is we have our own public Sumo Dojo, a Slack channel where all of our customers or most of our customers are there. That is nice in a way because it’s very real-time, right? They’re asking a question, and sometimes, they’re helping themselves or we are answering some questions. It adds that human connection to the conversation when it’s real time, right? That is one very nice place where we get our ideas from. Yeah, so as I said, we talked to field. We have all these channels, and we try to get feedback and build the right things.

Riya Singh: Cool. Shifting gears a little bit, one thing in engineering for running a SaaS service is very important is that we can’t just build and forget about it. The system has to run. It has to run 24/7. It has to run at 100% availability, which is easier said than done. A lot of elbow grease, which goes on in trying to make sure that the service scales, and it works under different load conditions. Engineering realizes that it is very important that the current service work, right, so we spend a lot of time in trying to make sure that our service are reliable. Some ways how we do it, we use our own product to monitor our own services, so this is an example of an outage dashboard. We’re trying to monitor different lag latencies. There is a group called the IRC, incident response coordinator, so in case our metric here does not look good, we declare it as an outage. All the teams are involved in trying to resolve it as soon as possible.

Riya Singh: When I joined Sumo, we were about 80 people. We had maybe 10 micro services. Today, we have 500+ and I think more than 50+ micro services. Being able to monitor and run the service at scale is a challenge, and we have learned so much through just trying to run this well. We have learned that things that work when you’re a small company does not scale when you’re a bigger company. You have to automate as much as possible. Anytime you’re adding more humans to it, there is more chances of errors coming through, so the human side of scaling does not translate to the skill. We spend a lot of time in trying to automate things so that if it in case something goes down, we can recover from it as soon as possible without human interaction.

Riya Singh: Despite all the works, sometimes outages happen, and you move on, but one thing very nice about Sumo and its culture is that our focus is very clear. We are one with our customers, so in case an outage happens, we all gather together in an outage war room and try to help each other to resolve it as soon as possible. You’ll see people leaving the meetings, leaving any presentation and coming and helping. It was very nice for me to see when I joined Sumo that everybody’s so helpful here. If an outage happens, we do post mortems within a day. At this point, any new feature development stops, and we try to work through the action items that have come through the outage post mortem.

Riya Singh: Our policy is no repeat offenders, so in case we find the root cause and it has happened once, we will find the ways to ask the five whys and fix it, but you want to make sure it doesn’t happen again, so no repeat offenders is our metric to see how well we are doing.

Riya Singh: Okay. Talking about outages kind of makes my head hurt too because I’ve been to, well, more than a couple of them, but we’re getting better.

Riya Singh: Let me just shift things a little bit and talk about this newest initiative that I’ve been leading at Sumo. I’m leading a new team at Sumo called the Quick Wins team. As you know, we have grown so fast that today, we have 50,000 users and 1,600 customers, right? They’re very engaged customers, so they’re constantly asking us for new things that they want to see in the product. Sometimes, a team is not tasked to do those small things because they’re already working on the next big initiative, right? These smaller things or UX feedback somehow doesn’t get prioritized. I’m sure you guys have noticed that too in your companies that the smaller things are sometimes skipping through.

Riya Singh: We started a new team called the Quick Wins team, and a secret sauce is the sriracha because a little goes a long way, and so we’re trying to make all these small, small changes, which makes our product better over time. It’s a cross-functional team, so we have … It’s a small team, one UI, one UX, one backend, one docs, one PM. One nice thing about this is we are growth hackers.

Riya Singh: We are not tied to any particular section of the product, so it’s not like somebody who knows search cannot do dashboards or somebody who knows dashboards cannot look at collection. We are training people and empowering them to be able to make progress in any part of the product, removing any kind of silos that exist.

Riya Singh: We’re trying to make the product easier to use for Andre, for Melinda, and all the personas that we have. The aim is to improve the NPS, which is net promoter score, which is a metric showing how much our customers like our product. That’s it for mine. Stacy shall talk about learning.

Stacy Kornluebke: Hi. I’m Stacy Kornluebke in case any of you here looked out at the agenda and wondered how to pronounce that. It’s a frequent question. I am the manager of training and documentation at Sumo Logic, which if it’s a lot for you to say, we just kind of reduced it down to the learn team. I am also the mother of three wonderful boys. I have replaced my couch three times in case anyone’s wondering. People say to me, “Wow, that must be a lot of work.” I say, “No, it’s a lot of fun, but I do miss my furniture,” so I am an avid fan of the arts. I am kind of infamous for dragging my friends two and a half miles down Manhattan to see Gustav Klimt’s The Woman in Gold. If you don’t feel like making that trek, you can just watch a very wonderful movie with Ryan Reynolds, and you’ll see the painting a lot.

Stacy Kornluebke: I’m also a voracious reader. I have 80 out of my goal for 100 books this year. If you find me later for networking opportunities, do not ask me what I’ve read. Once you reach this level of quantity, there’s very little quality. I like a lot of steampunk and vampire and werewolf books. It gets me through the night, but in all honesty, what I really like to do is make users confident. My whole goal is to give them the materials they need so that they feel that they are able to use the product the way they want to.

Stacy Kornluebke: People often ask me what’s special about Sumo. I did leave Cisco to come here, and I did it for a number of wonderful reasons. One of the top reasons is that we have very technical and sophisticated users. These people do a lot with their day, and they’re pretty knowledgeable to begin with, so they’re fun to write for. They’ve got a good background. They’re also, interestingly enough, highly collaborative.

Stacy Kornluebke: One of the things that I don’t think we’d addressed enough is how cool Sumo users really are. I watched them in a training class helping each other, and I don’t mean people from their own company. They were answering questions for people for other companies. They’re very proud to know the product, and they share it well.

Stacy Kornluebke: I happen to be a member of the customer service team, and it’s a really nice place to be, but everyone at the company has a huge drive towards making customers successful. I can go to Bret frequently and ask how things would look or what’s going to happen or what’s the next design. I can often come to Riya and say, “Hey, what’s the Quick Wins team doing this week?” so I never feel shut out of what we’re doing, and I feel that it’s always highly based on customer feedback. They work very closely with them. Jen’s giving me the face, but I can always come to her for compliance and security reasons. She’s really a very great resource if you need to understand GDPR.

Stacy Kornluebke: Sumo Logic also has a great sense of fun. What you’re looking at, in case you’re worried, is a suitcase full of 70 Sumos. The squishies that you’re going to receive tonight are extremely popular with our students. We didn’t get them shipped in time, so in addition to other interesting things I have done with my life, I took 70 Sumos in a bag through JFK, and TSA did not ask me one question.

Stacy Kornluebke: Think the last thing that I want to talk a little bit more about is that when I came here, we had this great phrase about how Sumo Logic was going to be democratizing data. At first, I was trying to imagine, what did that really look like? The more I learned about the product, the more people shared information with me, the more I understood the importance of having information at your fingertips. A lot of information usually gets siloed off, and in my world, when I get siloed off, I don’t really know as much about the customer as I think I do. My whole goal is to make sure that they learn about the product.

Stacy Kornluebke: All right, so if everything’s perfect, and you came here, and you loved it, what’s the big challenge with Sumo? What have you been doing for the last 18 months? Christian’s giving me the face, so I think he’s wondering too. To be perfectly honest, there’s a lot of challenge in the world of learning. The world has kind of gone from a directed approach. “Hey, I’m going to tell you what to do. These are the five steps to use this toaster,” to an open approach.

Stacy Kornluebke: There’s Stack Overflow, there’s GitHub, there’s just Googling it, right? Everywhere you go, you try and find the answer to your question. If you want people to use your product correctly, you need to make sure that you’re answering their questions because if you’re just telling them how you think it’s being used, they’re going to stop coming to you, and they’re going to go someplace where they can find the answer. That may not be a good answer, and it may not be a professional answer, but it’s giving them what they need. Users absolutely have the right to expect that you’re going to help them accomplish the task they want with your product.

Stacy Kornluebke: What do you know about your users? The first thing to kind of try and answer their questions involves understanding your users, and so, we’ve talked to you through two presentations about Melindas and Andres. We’ve talked to you from a design perspective and from a development perspective, but from my perspective, they need to know about the product. Melinda’s on it. She comes to Illuminate. I have a million Melindas. They tell me that the docs are like the Bible. They love it. They go into it every day. That’s because they use the product every day. Our product has a query language, and for her, it’s a second language. She knows what to type, what to find, where to go.

Stacy Kornluebke: Andre is a less frequent user of our product. I don’t know if you’ve ever tried to speak a foreign language that you only use once a month, but as you can imagine, trying to look things up, hoping that you got the right translation, the right inflection, that you understand the subtleties of the language, it’s really not going to happen if you pull out one of these dictionaries. It’s also not going to work quite so well with Google Translate, so you want to be able to get advice and understand what do these terms actually mean and how can I accomplish my task.

Stacy Kornluebke: If you’re trying to understand how to grow your knowledge in any part of your product, use the data you have already. Usually, Google Analytics is quite popular. Everybody uses it. No one knows what to do with the data. I suggest that you get the data, and understand a few things about what you want people to do with that information. For us, we were extremely excited to have people coming in, but then, we had to spend a lot of time figuring out, are they spending enough time on page views, do we want them to leave this page and find other pages, or do we want them to exit? Google will tell you what people are doing. You need to understand whether those behaviors are helping or hurting them.

Stacy Kornluebke: We also were very fortunate to have marketing reach out and give us information on SEO because a high number of our users Google. We needed to know whether or not we were obeying search algorithm rules. We also use net promoter scores. I don’t know … How many people are using NPS? Does anyone want to … Okay, so I see we’re talking to maybe 10 people. All right, so, the way NPS works is that it’s a ten … Excellent. No. You’re using my argument against me. I like it. I frequently do that to other people in the audience, so I deserved it.

Stacy Kornluebke: Net promoter score. On a basis of one to 10, a nine and a 10, those people are going to recommend you. Anyone below a nine isn’t as thrilled about you as you think they are. While they’re using you, they’re not excited about you. Hopefully, when they give you that rating, they’re going to tell you something. If they say, “Great product, but the docs suck,” I am the one who will find you and say, “Hey, what can we do to make it better?” but in these scores, that information does give you a sense of whether or not they’re truly satisfied. If only a nine or 10 matters, then you need to step it up and make sure you’re not making a seven or an eight.

Stacy Kornluebke: We also have this cool direct feedback mechanism in the docs, so down here where you see, was this article helpful? Yes, no. Leave feedback. I know that doesn’t seem that revolutionary, but we get feedback twice a week to our information based on this from customers. That’s a high level of participation. People come in. They feel very comfortable. They can say yes or no, but frequently, they can say, “Hey, I was expecting it to do this.” So long as you’re responsive to that, people will come back to you.

Stacy Kornluebke: We were also fortunate because we’d spend so much time making data visualizations. We could use our own tools, which was pretty awesome and a great kind of group project, and then again, I have to thank the UX team for giving us their customer journey because we could see the pain points people had with the product and try and align them to whether or not they were having a problem reaching information, training, documentation. Was it just an education issue, or was it something that was more of a product problem, and we were not going to get doc around it? It was going to have to be fixed.

Stacy Kornluebke: I speak a lot to search because 60% of our users and specifically, our struggling users, are coming to us for this reason, but if you are trying to figure out how to reach more users, find how they find you. If that’s through your Twitter feed, if that’s through your Slack channel, make sure that you know how your most struggling users are coming to you because if you aren’t speaking their language and you’re attempting to force them to use another tool, good luck. I don’t think anybody wants to learn one more thing. They want to find it the way they normally find it.

Stacy Kornluebke: Also, I recommend using your own tools. I know I’ve said it. I know you’ve seen all of us pull a dashboard showing how we use our product with our data, but what this allowed us to do was to take our certifications, take our data, and show it to sales. We pulled in customer success to make a meaningful dashboard. It became a group project. By involving more of the company, you’re going to get more people starting to understand what customers are struggling with and how they can help them. Frequently, our salespeople come in and they see who is certified and who isn’t and whether that makes a difference in whether people buy or renew. A certification is a great way to learn about our product, and it’s one indicator of whether or not people understand and feel confident with it.

Stacy Kornluebke: Okay, so how do I get answers to them? Well, everyone’s going to have to come up with a different way of doing this. For us, we found that a number of users desperately wanted some way to interact with a human being. They wanted immediate answers, and whether that was a Slack channel or that was an in-person training class, they were happy. Just bring me somebody who can answer my questions. The first thing we did with certification and training was we started holding what we called Cert Jams. We took the user classes that we hold at Illuminate. We came to cities where there were at least two, 300 Sumo Logic users. We said, “Hey, there’s a free class if you come today. Come in. Learn. Get a t-shirt.” Jane and Kerry were helping me out today, so somewhere around here, they’re wearing their t-shirts. Yes, so that, believe it or not, was a big thing for people.

Stacy Kornluebke: Quick answers to questions online as I’ve mentioned before. I know SEO is kind of this dirty word in the engineering world. It’s what marketing people do, and we shouldn’t touch it, but if you don’t, you don’t get the results you want. Then, just quick answers in real time, so if you can’t talk to a person, if you’re out on Anchorage and you’re using our product, you can still get on Slack. Anyone can find and ask a question.

Stacy Kornluebke: Here’s a picture of our Cert Jams. This is kind of cool for people. They’re really happy. We call this the t-shirt shot at the end. When they get their certification, they get their shirt. For every city that we go to, we try and take at least one of these. We’re doing at least 25 this year, and we hope to do 30 next year, but this is really a great networking opportunity for anybody who uses our product, and it’s a chance for people in our company to kind of get to know our users in a non-sales context.

Stacy Kornluebke: These are all the mistakes that we made with SEO. Please don’t make our mistakes. Please make new ones.

Stacy Kornluebke: We, first of all, did not know how search engines worked, so we had to use the SEM rush report to understand things like the algorithm will punish you for dashes and for underscores instead of dashes. Good to know. Our tool was automatically generating underscores all the time for us to take out those spaces, so we had to change the default setting, and magically, our documentation was popping to the top for our search results. Good to know.

Stacy Kornluebke: Use their terms. One of the things that I also find with learning materials, with documentation is we frequently try and force people into an exact definition of exactly what we’re doing, which is great, but it’s not the term people are searching for. A classic example of this is we talked very specifically to two-factor authentication because there were two factors when a lot of people were looking for MFA. We do not have the right to try and teach them the difference between multi-factor and two-factor. We can take them to the two-factor page, and then, they can see the difference between that and multi-factor, but please just get them to the information that they want. Use the term they use even if it’s a little hacky. They want to find their stuff.

Stacy Kornluebke: Then, the last bit is please avoid stubs. I’m just going to give you this bit of an SEO advice. If it’s under 200 words, your search engine hates you. I know there’s this great trend to try and bring down documentation to short little stubs, easy-to-read pages. Please bind up all those little stubs into a single page. You user will thank you because they don’t want to click through 60 pages, and your search engine will thank you because then, it will think it’s a real document. You can also make your own SEO mistakes. I recommend it. Keep having fun, but get people what they need.

Stacy Kornluebke: We also created an in-product learn tab. What we discovered was that people just want to go to the product, find what they need to do to learn, and go through stuff. Well, that’s great, but I have a 30-video library, and there’s no easy way to stick that all in the product. What we did was we had people from the UX team help us out with a design, and engineering helped us implement just kind of a short five-video series. We used APIs from our knowledge base to pull in the tutorials, and then, we provided quick links to other parts of the product. Now, this is really useful to a lot of people. They come in. They watch one quick video. They review the tutorials, and they’re done. It’s not a long-term thing. It’s a help in onboarding, but it made a big difference.

Stacy Kornluebke: You also sometimes have to accept change. As wedded as I am to SEO and I like forums, and I want information to be found, and we’re going to be the next Stack Overflow, a community where you have a timed response where people have SLAs and they’ll get back to you even within a day is not as cool as it used to be. People like Slack. Slack offers real-time response to questions. It makes people happy. They’re not really concerned if the next person can Google it. They’re getting the answer they need. Being where your users are is kind of what you need to provide.

Stacy Kornluebke: Also, just remember this is a process, so use any data that you have, CSAT, NPS, Google Analytics. Whatever you’ve got, start trying to understand your users. Please work with any group that you have that has similar customer based mindsets, so if you’ve got a UX team, if you’ve got a customer success team, if you’ve got a field full of salespeople that desperately want to help customers, reach out to them. Don’t hesitate to ask a couple of questions to do your job better.

Stacy Kornluebke: I also recommend that you join some grassroots movements. I like Write the Docs. It’s a bunch of people that take a more hacker-based approach to writing documentation, but it’s also just a user conference where people come who work with problems every day. We can talk about our struggles with trying to use analytics data. Talk to your users directly. I don’t know, depending upon your corporate culture, how open people are to that, but Sumo’s really never barred us from reaching out and saying, “Hey, how could we make things better?” if it’s just sitting silently on a call with someone who is working with the customer, that’s good too. Understand what people are really saying about what you’re offering and what you’re teaching about the product.

Stacy Kornluebke:  I want to say thank you. I know I gave you a to-do list, but that’s kind of what I do. Now, we’re going to hand you over to Jen who’s going to talk to you about security and privacy.

Jen Brown: I’m Jen Brown. I am DPO here at Sumo Logic. As you see, I control everything security and privacy as far as GRC goes. That means I get to work with all of the external auditors and do all that fun internal audits policies. I’m putting you to sleep already, so yes. I’m also a contributor to Dark Reading. If any of you aren’t aware of what Dark Reading is, it’s a really great resource for technology news, so check it out if you haven’t and don’t know about it. I’ve been in this space for over 20 years. I’m a grandmother. You can see. I’m blessed to be booed. That’s what they call me, grandma to Lucy and Will, and then Max is in the middle there. He’s over here sleeping. He’s our security mascot. Happy to be at Sumo. I’ve been here for about two years now, was here for about six months as a consultant, liked it, so I came on. We kind of did a try before you buy. They made sure they liked me. I made sure I liked them. Luckily, it all worked out.

Jen Brown: Our group is broken into three different groups. We’ve got the Security Operations Center, so we’ve got a manager for that who’s building that out. We actually are just beginning that journey. As I said, I do compliance, all our external audits. You can see we have PCI, SOC 2, ISO, CSA Star, HIPAA, Privacy, and then we also do risk management. We’re also looking or going towards FedRAMP certification, so a lot of fun to be had, and then DevSecOps. We’ve got an engineer that we poached from finance, which sounds really strange, but our last two hires have been from the finance groups. I have to be really careful when I’m walking around that group. I have to kind of skirt it, so nothing hits me, but he’s helping us to automate everything possible. I’m going to show you one of the things he built for us to save a lot of time and make our customers much, much more happy.

Jen Brown: What we’ve got here is we’ve got a self-service portal. When I first came onto Sumo, what would happen is if a customer needed something like our PCI AOC, which is an attestation of compliance, the salesperson would go in. They’d enter a JIRA ticket. It would come to us. We’d have to make sure there’s an NDA in place. There was this back and forth. Depending on how busy we were, if I had audits, whatever, sometimes, this could take three to four weeks to get a document. Doesn’t equal happy customers at all.

Jen Brown: What Mike built for us … Let me just pull this up real quick. Hey, Brian. Is Brian here? I was going to take you through the portal, but this will be good enough. There’s an NDA on the front of it, which really helps us reduce that time of making … We don’t have to go and make sure there’s an NDA with a customer before we let them see this. They agree to it, and this is what they’re going to see. They’re going to see that they can pick from any of these documents here to get sent to them. If they’re our customer, so if they’re in Salesforce or they’re a customer or prospect in Salesforce, they’re going to get those documents right away. I mean, instant. Just no more two to three weeks. They get it right away.

Jen Brown: If they’re not in Salesforce, there’s going to be a little bit of investigation that goes on. Jane who works with us back there, she’s going to go to the sales team, and she’s going to find out like, should this person who’s requesting this actually get these documents? This has made our customers really, really happy. It has cut down a lot of work for us too, which of course, makes us really happy. The other thing we’ve done that I was going to show you is we put our DSR portal on here. Anybody working with GDPR? We put our data subject request portal on this as well, so it’s really automated a lot of what we’re doing.

Jen Brown: All right, so, as you can see, just in a quarter alone, we had 546 from customers and 616 requests from prospects. That’s a lot of JIRA tickets we didn’t have to deal with, and again, a lot of happy customers. It’s decreased the time, as I’ve said. It enables us to be more transparent with our customers because we’re able to put more documentation out there, and we’re always trying to find new things that we can put out there. At first, it was just the attestations and certifications, and now, we’ve got our pen test results out there. We’re always trying to provide more.

Jen Brown: The other thing on that portal that I wasn’t able to show you because it got cut off is we’ve got a place where customers can come to us and say, “You’re not compliant with fill-in-the-blank.” I mean, there’s so many laws and regulations out there. They’re able to tell us what it is they need us to be compliant to. There’s the German privacy law. There’s the New York CFR, so on and so forth, so we’re able to get that from them, which I think I’ve got a minute to teach.

Jen Brown: Yes, so we’re able to, instead of just throwing a dart and trying to figure out what maybe customers want, we’re able to hear from them. We’re hearing from sales all day long. We love sales, but it’s better to hear it from the customers and really see what it is that they want. Just based again on a quarter’s worth of data, we found out that 11 customers really need it. I’m not even going to try to … Maybe you can pronounce it, Christian. It’s a German privacy law. They need us to go get compliant with that. New York has got a new financial services law on the book. We’ve got 11 customers who need us to look at that.

Jen Brown: Australia has got our IRAP. We’ve got another 11 customers who are asking for that. Then, GLBA, we’ve got 10. There’s some others that people are asking for, but we’re able to really see what people are looking for us to be able to show and demonstrate compliance against. That’s really helped us to build that roadmap instead of just guessing because there’s hundreds you can choose from.

Jen Brown: All right, so that was mine. Mine was quick, sweet, and short, but we’re always trying to automate what we’re doing in security, so that way, when we build our roadmap, we’re building it correctly and adding more and more and more every time that we do. Our customer input influences all our decisions at Sumo Logic. I mean, it’s just a really, really big point. Customers are so important. That includes how we design, engineer, like end-to-end. It’s amazing. We share a common goal and partner closely with our customers, impact cross-functionally. I mean, as the women said, it is really end-to-end. This reflects the core value of what we … We’re in it with our customers. All right.

Shea Kelly: All right. Thank you, Jen. Everybody still with us? Happy Monday still? Okay, okay, okay. Good, good, good. We wanted to set some time aside. Is that too loud? I feel like I’m talking really loud. Set some time aside at the end for some questions, so I thought we’re going to scooch … How about if we scoot our chairs over? Does that work? Is this not working anyway? While we’re doing that, I actually want to shout some thank yous out, so first and foremost to Angie, and to our team Stacy and Bronwyn for the recording and all the help to get this recorded for us, so we’ve got it in perpetuity.

Shea Kelly: Second, if she’s back there, I want to shout out to Tori Lee. Is Tori here? All this food, beverage, everything, Tori is magnificent. She does this for us every day, and I was thrilled she was able to do it for all of us here today. Obviously, I want to thank our panel, our fabulous panel. I want to thank all of you for coming. We’re going to shift to a few questions, and we’ll see. We have mics we can hand out and around, so if anyone has the questions, just put a hand up, and we can bring a microphone to you, or you can shout it and we can probably just repeat it. Okay. Max is out. Yes, we are very dog-friendly. At any one point, what do we have? Like 50 here a day? Yes, that’d be great. Yes. Thank you. Is he up? We’re just going to adjust lights a little bit here. Did somebody have a question?

Audience Member: Yes. Right here.

Shea Kelly: Oh, I’m so sorry. Okay.

Audience Member: First of all, thank you. The set of the presentation is one of the best I’ve seen because you touched on real everyday life in a company versus how you, being a woman, makes a difference through your job. No. Thank you. I mean, I got from each of the presentation something that I could relate to but that brings the question to Bret and Riya. You both … and actually, to the learning lady.

Shea Kelly: She had to leave, just so you know.

Audience Member: Yeah. All of you, your presentation talked about the product management but there’s no product management talking, and each of you … Obviously, I’m in product management, so I’m concerned, but you are doing things that almost, throughout my career, part, all of it would fall under my responsibility. Can you talk about the product management dynamics in the company?

Shea Kelly: You want to repeat over there so …

Jen Brown: Louder.

Shea Kelly: Do you guys have mic? Can you turn it on?

Jen Brown: Product management dynamics. He’s Bruno. He’s one of our co-founders.

Bruno Kurtic: Hi. Could somebody-

Riya Singh: Hello.

Bruno Kurtic: Hi. Could you repeat the question because nobody behind here could hear it?

Shea Kelly: This is Bruno Kurtic.

Bruno Kurtic: Here. I’m going to give you the mic. In fact, we’d throw with this thing around.

Audience Member: I’ll shorten the question. First of all, the compliments still hold. One of the best set of presentation in Geek Girl Dinner ever because it touched on real life topics. Instead of what being a woman in tech means, you talked about what doing my job as a professional means, which I can … and happily more to relate to, but back to you. Many of the responsibilities that I as a product manager or head of product view as my responsibility were amazingly well-described by people that their title is not product management. Can you describe the dynamics of product management in the company interacting with UX, with engineering, with security?

Bruno Kurtic: Sure. We have a really easy job in product management. We just sit around and all these guys do everything else. That’s how it works here. No. Just to be serious, we don’t actually talk about ourselves as product management or engineering. We actually call ourselves product development, and all of these people here are in product development, bar one, who has a big input into product development. We run very integrated teams, right? Everybody has their primary responsibility, product, for strategy and requirements, user experience for design, engineering for architecture, things like that.

Bruno Kurtic: Ultimately, we basically break teams up into small units that have cross-functional team members who do the things that are necessary for that unit to succeed. Usually, that’s rallied around specific customer outcomes, so when we build things at Sumo, we don’t build features. We build outcomes. We focus ourselves around whose life are we going to make easier if we do X, Y, and Z. That’s why when you talk to everybody across the company, you’ll hear a lot of things that are relevant to other cross-functional topics. We usually don’t have very stringent decision-making that you get to decide this, you get to decide that. The team decides. They do the best that they can. That’s why.

Audience Member: Thanks. Great. Thanks.

Riya Singh answers a question from the audience at Sumo Logic Girl Geek Dinner.

Riya Singh: I’ll just add a little bit more to it. Firstly, we’re missing one of our PMs — She is on a maternity leave, so that’s why there’s one missing function here. I work very closely with Lavinia and all the other product management as well. I think when we said that engineering takes in customer input, obviously, the vision comes from the product management, right? It’s not me you or it’s just her or one person. It’s all together, right? Customer experience is important to us. Especially with things-

Audience Member: It was a compliment because you said product management, so I think I was wondering you have no job for the product manager. It was a compliment because there was no dedicated speaker as a product manager, but you spoke product management. It was a compliment. That’s why I was asking.

Riya Singh: Yeah. I hope Lavinia will be back soon, and we can talk more. Yeah. More questions, guys?

Audience Member: Yeah, I got it. First, thank you for the wonderful presentation and the hospitality. You talked about that collaborative environment, for example, when there is a problem, you go to the war room, you drop everything, and then you solve the problem. My question is, how does this culture scale when the number of projects, applications, and customers grow?

Riya Singh: More war rooms. Every room is a war room. The concept of war room was particular to outages. The definition of an outage is that it shouldn’t happen too often. It’s not that you’re spending all the time in the war room, right? We are working very hard to make sure that these outages don’t happen over time. As I said before, no repeat offenders, right? It may sound stressful when you are in this war room, an outage situation, but it’s not frequent. The frequency is just going down with time. Does that answer your question?

Audience Member: I was stressing the example, the war room, but on a larger scale, the idea that there’s cross-functionality, collaboration, I think that’s a recent change in culture. How does that scale?

Riya Singh: If the team is larger, how do we make sure this cross collaboration happens?

Shea Kelly: Maybe Jen, you …

Jen Brown: Yeah, so communication. We’re always working together. He thinks it’s a laser. Sorry, he’s very excited. Sorry. We work really closely together. We make sure that there are people who are on point for when things like outages happen, items like that. I mean, even though we’re growing, I don’t know if it’s just unique to Sumo, but we just haven’t seem to have that problem yet. I mean, Christian?

Christian Beedgen: We’re always… It’s a divide and conquer sort of strategy. We make room for more teams. We make more teams, so we keep them cross-functional. Then, over time, as we need to grow, we needed to grow more leaders as well, and then there’s sometimes an additional layer of cross-functional leadership discussion, so to tie it all up, it’s a tried matrix, I think.

Jen Brown: A good example of that is when I first started in security, we had a security engineer, we had me. Now, we have DevSecOps. We’ve got the SOC engineer, so we are growing and adding more teams and functionality like Christian said. Does that answer better? You look like … Okay.

Audience Member: Who are your direct competitors, and then how you differentiate yourselves from them?

MaryAnn O’Brien: All right, so good question. When I came to Sumo, I actually have a lot of friends that work at one of our direct competitors. As I started to research … Actually, to be quite honest, I had never heard of Sumo Logic when a recruiter had reached out, so I had an opportunity to research and do my own level of understanding in terms of the company itself. I’ll just tell you one of the primary competitors that we’ll see especially on the enterprise side is Splunk, if you’re familiar with Splunk. Another … I lead a mid-market sales team, and one of the actually big competitors that we see a lot is open source, so if you’re familiar with ELK, they’re actually … We run into them quite a bit more actually, just as equally, probably about 50/50. In my particular segment is ELK, as well as Splunk are the two primary.

MaryAnn O’Brien: Overall, that’s mainly on our log side, but we also have a unified logs and metric strategy, which includes metrics, so every so often, we will also see some monitoring and metrics type of competitors like Datadog as an example. Does that help?

Shea Kelly: Anybody else?

Audience Member: It’s a really big microphone. Yeah. All right, it’s awesome. I always want to pause a microphone. Thank you for the presentation in [inaudible]. It’s really good. My question is that, listen to your customers, and since you have so many customers, I’m pretty sure there’s a lot of feature, there’s a lot of requests, a lot of I wonder if some want that. Based on experience, what do you find the best way to prioritize and how do you choose what feature or what new things to work on first?

Riya Singh: That’s very in the PM to answer that question about prioritization. Bruno, you want to take that?

Bruno Kurtic: It’s a loaded question, but we start with the overall strategy. What is the true north? What are we trying to achieve as a company? I oftentimes tell people that strategy’s knowing what not to do. It’s not knowing what to do. We start with the strategy. We align those strategies with customer outcomes. We focus on certain set of things as few as we can. It’s not always easy, right? You always try to kind of do less, but you end up doing a lot more than you probably can, and you should. We try to really tightly scope what problems are we trying to solve.

Bruno Kurtic: We try to sort of align on what is the minimum amount of work we need to put in to produce something that actually changes the outcomes for the customer. We really like to work agile and deliver products out to customers. We’ve built a very sophisticated way to surface new capabilities in production to individual customers even though we’re a multi-tenant SaaS service, so we have very fine-grained controls over what we deliver to customers. We did that because we wanted to be able to give customers things early, get feedback, iterate, iterate, iterate, improve, and that’s how we develop. It has to be aligned with strategy. It has to be aligned with positive customer outcomes. It has to be in scope of certain time horizon, usually six to nine months, and then, you just fill the backlog, and you burn down.

Audience Member: Thank you.

Shea Kelly: Any other question?

Audience Member: I heard the mention of customer success. I was just wondering if you also have a customer support team. Is it one thing for you guys? How do you deal with like 24 by seven support or service?

Riya Singh: Our learning lady is missing. We do have customer success and customer support. The way I understand it, so we have our support portal where customers can put in support requests. Customer support is mostly trying to solve their immediate use cases or reporting bugs or something is not working, helping them troubleshoot. That’s customer support. Customer success is a little more broader. They’re trying to make solutions to help our customers. They’re actively trying to find out why this customer is not giving us a good score or how is he using the product, and how can we increase usage adoption within their accounts? They’re effectively trying to make them more powerful in using of the product.

Speakers:
Katie Jansen / CMO / AppLovin
Alice Guillaume / Director, Marketing / AppLovin
Helen Wu / Director, Growth Partnerships / AppLovin
Swetha Anbarasan / Network Engineer / AppLovin
Laura Pfister / Software Engineer / AppLovin
Anusha Ramesh / Principal Software Engineer / AppLovin
Sonal Gupta / Principal Software Engineer / AppLovin

Transcript of AppLovin Girl Geek Dinner – Panels:

Katie Jansen: I’m Katie, I’m the CMO here at AppLovin, and we just wanted to thanks to Girl Geek X. This is awesome. We haven’t really done anything like this before, so this is our first time, and they’ll probably ask you for feedback at some point so feel free to give it to us. This is a new thing for us. I would also like to thank Brianna. Brianna, where are you? Brianna is the one, and Marissa who is probably not here right now, she’s out front, they really put all this together, they got all the food, which was outstanding, and the color coded name tags, and they were down here putting the seats here today all on their own, so I just wanna give a big shout out for all your hard work, thank you so much.

Katie Jansen: If you don’t know a lot about AppLovin or even if you do, real quickly, we’re a comprehensive platform that connects app developers of all sizes to about two billion global users each month. We have a growth monetization and publishing services that really focus on giving game developers the expertise and insights they need to grow their business. Whoops. We’re only about six years old, and we, I think we have about 160 employees, 165, and up until recently we were self-funded. So we’re actually pretty small compared to a lot of the companies around here, but we’ve really experienced hyper growth in the past few months, even just this year, you could say. We have about seven offices worldwide and our valuation is a little north of two billion. So we have a very few amount of people doing a lot of work and producing a lot of great things, well beyond just the revenue we’re bringing in.

Katie Jansen: Today we thought we could partner with Girl Geek to give you a real unique perspective on how to adapt when you’re in a fast growing company, because we have a very strong perspective on that, and we grow our own roles kind of within that company. Today I have Helen and Alice, and they’re gonna kinda start things off with me. We’re gonna get the business perspective on that, and then next we’re gonna move over to engineering and ops and really get a more technical perspective on how we scale and how we’re adaptable. And oh, I’m sorry, there was an agenda as well. Panel one and panel two, and then we can eat and drink some more. Alright, so let’s start things off with you guys just telling me a little bit about what you do here at AppLovin and about yourself.

Alice Guillaume: Hi everyone. This is really great turnout, I’m super excited to be here. A little bit about myself, my name is Alice. I’m the Director of Marketing here at AppLovin. I’ve been here for four years. My journey here started out on growth and as the business has scaled I’ve taken on creative services and most recently, I built out that team to be over 30 people, and it’s currently one of the largest teams in the business.

Helen Wu: I’m Helen. I’m the Director of Growth Partnerships at AppLovin. I joined in 2013, and since then, have grown and trained our global teams who are responsible for helping our customers achieve their performance goals. And a little fun fact about me, I used to attend these Geek Girl Dinners like six years ago, so in 2012, I was in the audience like you guys learning the stories and experiences of other geek girls, so I’m very excited to be here today.

Katie Jansen: I didn’t know that. You saved that for this. So you guys both come from a different background. It’s not like you were in this industry when you first came to AppLovin, so what drew you to AppLovin, and what do you enjoy most about being here?

Alice Guillaume: Why don’t you…?

Helen Wu: Okay cool. So I joined because I wanted to be around the smart and hardworking people in this company. I wanted to learn how to build a business, and before I joined I was at a large bank. I did internet … I did research on internet trends, and I learned that mobile advertising was gonna be growing at a tremendous rate and we were just at the very beginning of this long term growth trajectory. So in 2013, AppLovin was, had already been operating for about a year and a half. They had invested heavily into building the platform and product, and was starting to make their first business hires.

Katie Jansen: I remember interviewing with a couple people at the company and just being so inspired and in awe of how smart and passionate they were. I was like, I just wanna be around these people all the time and work with them and learn from them, and you know, because I knew where mobile advertising was headed, I really believed in our CEO Adam’s business vision of us being this growth engine for app developers. So I knew there was gonna be endless opportunities to learn and to grow with the business. So I joined and I haven’t looked back since.

Katie Jansen: Alice?

Alice Guillaume: My journey’s similar but a little bit different in a sense that I had also come from a very structured background. I had grown up in management consulting, banking, sales and training, that’s what I thought I wanted to do with my life, and until I reached a point where I realized I needed something more. I wanted to work in a place where I could make an impact, where I could bring ideas and promote change. I needed that in my career.

Alice Guillaume: I did not know very much about mobile advertising to be totally honest, and I took a chance and I took a risk to interview at AppLovin, and I was nervous because it was a deviation from a very structured route, and actually taking that risk was probably one of the best decisions I’ve made.

Alice Guillaume: And two things drew me to AppLovin — one, and I think this echoes what Helen said is the people, and two is the learning opportunity. So for me when I was interviewing, I realized that everyone was so smart, and so much smarter than who, than I am, but they were down to earth enough to wanna teach me. So I thought that was really cool. Two was that leadership really cared about my growth, and they were invested in helping me get to where I needed to be. That was also something that, for me, made a big impact on my career here.

Alice Guillaume: And three was the mentality about work. People here are really passionate about making things better and not doing things just for the sake of doing them. If something is menial, or something is low value, let’s find a way to like automate that. And that really spoke to me because it demonstrated the company’s value in innovation and change. So all of those things really inspired me. I wanted to be around those people, and I wanted to be a part of that race.

Katie Jansen: And the fourth reason is Helen, right? Didn’t you guys go to school together?

Alice Guillaume: Yes. Fun fact about me and Helen. Helen and I went to college together and she was my big sister there, and that’s how I learned about AppLovin. And she was gonna kill it when she joined.

Katie Jansen: Right. And is there anything surprising you guys have learned since being here?

Helen Wu: Yeah absolutely. So I mean I think it’s just learning that I would be constantly stretched and challenged all the time. I mean before I joined, again, I was at a large bank where the work environment and culture was very different. Even though the financial markets were changing every single day, I felt like the mindset and the processes were very routine and very stagnant. So I wanted to be in a place where I felt like I could add value, where I could learn something new every single day.

Helen Wu: The most important thing I learned over the last five years is the value of being fast and flexible, which is staying in tune with changing market opportunities, being ready to action, adapt, and re-adapt. And I think the speed and mindset with which we operate is really the key to our tremendous business growth that we’ve had since 2012.

Helen Wu: Also, you know I think everyone in this company has the ability to wear a lot of hats to help out in ways that are beyond what’s just being asked of them. So I think really people, I think people here are very, very keen to like help the business but they’re also very invested in helping each other. So you know I think hiring people who can add to this culture of continuing pushing each other, you know, challenging each other to ask intelligent questions, to take risks, and to do better than their last is really important for continuing to grow our business.

Katie Jansen: So you guys have grown from being individual contributors to being managers now. You both manage teams. Alice as you mentioned, Alice is actually on my team, and then you have one of the biggest teams at the company right now, and you have a big team too. What do you guys typically look for when you’re hiring, because lately it feels like we’re always hiring. So what do you look for? How do you kind of figure that out in an interview? What are some tips you can give to these guys about that?

Alice Guillaume: Definitely. So I’m very passionate about hiring and recruiting. The resume is important, but for me it’s really the human and the psychological aspect of who you are. When I interview candidates there are two main things that I care about.

Alice Guillaume: The first one is ability to learn. So you will hear throughout the theme of our panel that the only constant is change, and that’s, I think, a core thing that has drive our company to be so successful today is constantly evolving.

Alice Guillaume: To be able to move that fast, we need to hire people who are open to learning, who are open to self improving, and who are receptive to knowledge and feedback. For example, when our team ramped up from 15 to over 30, that doesn’t happen over night. That’s a collaborative effort of everybody on the team from the individual contributors to the leads, and that really requires that openness and heart to be flexible.

Alice Guillaume: The second thing I care about is grit and passion. So I think that speaks to the first one is be receptive, be open to learning, and two is apply that in your day to day and be able to put in the amount of work that it takes, and you need to have the passion to be able to wanna do that. Those are things for me. You wanna add anything, Helen?

Helen Wu: Yeah definitely. I totally agree with everything Alice said. I think we spend a lot of time thinking about hiring and who we hire and why we hire. I also wanna add that I look for candidates who have really deep comfort with data, simply because we have a lot of it, which is readily available and waiting for someone to take, extract meaning out of it. So our growth analysts do the really important work of analyzing the data, building models that can inform us of, you know, how we can drive better performance for our customers.

Helen Wu: The second thing I look for is some love of solving puzzles of any kind. So even if a candidate doesn’t have quantitative or technical background, you know, I look for any indication that they might be, for example like a chess master, or an avid player of strategy based board games, anything that suggests that they’re a strategic problem solver who can look multiple steps ahead and formulate a plan. And these kinds of things demonstrate to me that someone possesses deep problem solving skills, which we need in our business to continue to grow the way we have.

Katie Jansen: And so what about, okay let’s someone’s gotten hired, they’re on your team, so let’s fast forward a little bit. How do you guys as managers coach and manage, you know, keep your staff motivated? Again we’re small, we’re scrappy. How are you able to kinda get in there and work with them in way that’s not micromanaging but also helping them move forward? Especially when we change goals all the time, right, or we’re flexible or we fail fast, right?

Alice Guillaume: Yeah, go ahead.

Helen Wu: I think it’s a really good point, and I think in addressing staff motivation, I think it’s important to also discuss the culture and the space in which we work. So we have a very open office concept where analysts, managers, and VPs all sit next to each other, and they’re sitting next to other functional teams. We encourage a lot of questions to be asked openly, a lot of ideas to be raised, and so problem solving, learning, and teaching happens organically at our desks and out in the open, and it happens in every sort of of like direction you can imagine between levels of the organization and across different functions. I think those are really great opportunities for people to learn and to grow all the time.

Helen Wu: And then, on my team we do regular check ins between managers and the employees where I like to take our employees out to coffee, and it usually starts with asking questions like “How are you doing,” “What’s going well for you,” “What do you need help with,” and these are important opportunities for an employee to share their ideas and questions that they have, or goals, and get support from their manager on how get there. And I really like that these are outside of the office, like over coffee, because it’s a more casual environment where people can connect on a more personal level and dive deeper into the discussion.

Alice Guillaume: This is a really good question. I think there are a lot of things a manager can do in the day to day to help foster this, but for me the fundamental value, and I don’t take any credit for this ’cause someone else taught me this, is empathy. So for me it’s about recognizing that every single member of your team is human. So for me it’s spend time with them.

Alice Guillaume:  I do walk around on my team and other teams, probably ’cause I just like talking to people a lot, but I like getting to know them, getting to know then in a work environment, but also outside of work. As a manager that also helps me connect the dots when I see opportunities for people who can help each other out or would just be a good fit to talk to one another, whether it’s a mentorship relationship or not. So that’s one thing.

Alice Guillaume: Another thing is with my direct reports, don’t be afraid to talk about things that are not work related. I think building that relationship is really important to having tougher conversations too. That openness is super key. That last thing I would say is have your EQ feelers out. So how is the team feeling?

Alice Guillaume: I have a team of 30 people, it can be hard sometimes to know how everybody is feeling, but there are ways to kind of learn that, whether it’s through your own direct managers or just through day to day talking to people and being sensitive to that.

Alice Guillaume: Like if something’s going or the team’s feeling stressed out, can we take a break, can we go for a coffee, can we catch up, can we lighten the mood a bit. I’ll crack a joke, I’ll play some songs or something like that. Or if you have a life event going on in the team, like celebrate that. Their win is the team’s win, but most importantly it’s winning everyone over as people.

Katie Jansen: Yeah, I mean I think what I heard both of you guys say is it’s about the individual, and it’s about connecting in different environments. I will say having worked at other companies before, AppLovin is very unique in that we actually have a very low amount of meetings here.

Katie Jansen: As the CMO, I can easily go sometimes with only one or two meetings in a day. I’m not in back to back meetings all day, so that actually gives us the time to connect with individuals outside of your team and on your team, we really take the open office concept, like why have it if you’re not gonna actually leverage it, so we’ll just do impromptu meetings or conversations that’ll just start up right at the desks versus going into conference rooms and having meetings.

Katie Jansen: For me, having been at so many other, not a ton I guess, but at a few other companies, the concept of not being in back to back to back meetings and actually to connect with your team and other teams in a real environment, and also leveraging things like Slack and you know the different tools that we have is what makes a big difference too, which is what you said.

Katie Jansen: And what about mentors? How do you guys leverage mentors on the team? Do you leverage mentors on the team? Do you think it’s valuable?

Alice Guillaume: Absolutely, it’s valuable. So my team it depends, I focus on what the purpose of the mentorship is. So an example could be if I have a new manager, I want them to have a resource where they can talk to that person more candidly than they probably could with me to help them grow. In that case what we’ve done on my team is we’ve sourced externally through networks of our leadership team, to provide that person with a mentor, and we’ve seen that has been really effective.

Alice Guillaume: An alternative example could be just pairing a junior person and a senior person together on the team. That use case is more internally driven, so you know, making sure this person has a support on the team that they can learn from them about their day to day jobs as well. So absolutely mentorships are important. I had mentors as well. They’re your greatest advocate, they can help honestly point out areas where you can improve on more candidly, and it’s just nice to have someone to talk to sometimes.

Helen Wu: Yeah, definitely everything Alice said I agree with too. In thinking about how I think about mentorship, I think it’s also worth discussing like how I define it. I think mentorship is also a two way partnership where both parties can get a lot out of that interaction. Not only is it a way for the mentee to share their ideas and goals and get support and guidance from an experienced person, but it’s also a great opportunity for the mentor to teach, to pass on knowledge, and to get perspective on what’s happening at other levels of the organization.

Helen Wu: I think this is actually one of the very critical interactions between an employee and their own manager, so we invest a lot into this, definitely. And then as far as more long term mentorship that we have in the team, it’s most commonly gonna be when a new hire joins, we try to pair them with a more experienced analyst who can teach them about the role, about the business, and that’s a really great opportunity for the experienced analyst to practice their leaderships skills and to help develop the team.

Katie Jansen: So this is probably a harder question, but we have small teams here, we’re a small company, and so outside of this like, okay this is a new hire that’s coming in and they’re getting paired with someone senior, how do you help people find mentors? What are some practical ways? ’Cause I feel like quite honestly I go to these events and everyone’s like “Let’s find a mentor, it’s really important, and that all is organic. Thank you.” And I don’t think that … I mean I think that’s great, and I guess it should be organic, but there’s gotta be ways to kinda facilitate that process. So have you employed that at all, or what have you guys done?

Alice Guillaume: So I’ve tried … That’s a really good question. It can be tricky. I’ve employed both assigning on my team, so I look at kind of like what the needs are and I have paired people together, but I also allow them the flexibility to kind of figure out if it works for them. I tend to think if it’s more natural it’ll work a little bit better, but at least I’m providing an opportunity to that person to try it. The other thing I’ve tried is just observing. So there are natural kind of relationships that form through time just kind of, that’s just how humans are, and if I see that happening then I can have that conversations with both parties and see if that would make sense as a match. So I’ve tried both ways. I will say the one where I match it has probably has less chances of working than the organic one, but because there’s not always an organic fit, I still have to go try.

Katie Jansen: Do you have anything to add, Helen?

Helen Wu: Yeah, I think it’s definitely tricky to you know try to pair employees with new mentors. So I think the check-ins with people to understand where do they need help, how things are going for them is really helpful for understanding what they need. On the business growth side we … The nature of our work is very collaborative too, so I think there’s a lot of opportunities to build partnerships that are cross function. So for instance if two people, you know, are both familiar with a specific account or specific type of game we can pair them together to share their own unique experience and perspective, and so we build these partnerships based on business opportunities that are coming up too. And then, oftentimes, this is when we can pair an employee with a more experienced member of the team, and this person can serve as a mentor.

Katie Jansen: I wasn’t gonna say this, but I’m gonna, probably like a year and a half ago Adam, who’s our CEO, told me “Go mentor Helen,” which you might have suspected since I was like “Hello, Helen. You’re not on my team. Would you like to get coffee with me” randomly once a month at this scheduled time? But I’ll be honest. It probably took like about a year, ’cause for a while it felt maybe a little awkward and forced, but I noticed in the past probably five to six months, I’m like okay like this feels like a real, and we’re both getting something out of it to your point mentor-mentee relationship, it took a while sometimes I will say with the pairings, but you can hit your stride.

Katie Jansen: And then I have a mentor that I met, it wasn’t at this group. It was at Women in Wireless, that’s not their name now and I don’t remember their name to be honest with you. They rebranded. But it was a Women in Wireless event, and I was on a panel about mentors, and the EVP of revenue over there was just so impressive to me, and so I afterwards sent her a thank you note for being on the panel, and then two weeks later followed up with coffee, and now she’s definitely my mentor, and it’s been probably three years. And she’s not in the same vertical as me. She’s in finance and rev ops, which is very different, but I still learn quite a bit from her, but it was almost like, I guess, I was courting her or something. But she was open to it so it worked out okay.

Alice Guillaume: Sometimes you have to go get it.

Katie Jansen: Yeah I know. I think at that point I realized I don’t have a mentor, maybe I need one. So that’s all we had today, and I think we wanna open it up if there’s any questions from the audience. Yes, go.

Audience Member: I have a question for Alice.

Katie Jansen: Am I supposed to give her the mic?

Audience Member:  I can talk loudly.

Katie Jansen: Alright. Do it.

Audience Member: I was just curious so your title is director of marketing, which is fairly straightforward, of creative services. What does that include like more on a day to day basis?

Alice Guillaume: Definitely. And what’s your name?

Audience Member: Jessica.

Alice Guillaume: Jessica, okay. Good question. So yes I’m director of marketing, but I lead the creative services team. So what we do is we make video graphic and playable ads for our customers, and that is marketing because we are helping them market their apps and trying to help them sell their product. My day to day is, I manage a couple of amazing teams of designers and game developers, and we focus on market research, story boarding, producing, QA-ing, and looking up the results.

Katie Jansen: Yeah and that team rolls into my team as the chief marketing officer.

Audience Member: Alright hi everyone, my name is Regina and thank you for this panel. I’m actually in marketing, and that’s actually one of the reasons I came. I was like “Oh they’re talking about marketing, yes!” But my question is this, because there’s been a lot of call about diversity and inclusion, and I’m like this is great, but especially because you were talking about hiring, what I find is is that when I’m interviewing a lot of companies aren’t really looking at transferable skills. Like if you’re trying to get more diverse candidates then you can’t really expect that everybody is cut from the same cloth, and love that you know yeah you came from banking and …

Katie Jansen: I didn’t come from banking. No banking here.

Audience Member: Yeah yeah yeah, insurance and teaching, and now I’m in tech. So I’m just wondering as people who have done that pivot successfully and have stayed, because I’ve been at a few companies, just like what is your advice to that because there’s a lot of talk about diversity but then I’m like “What do you mean I’m not a fit?” So there’s a little bit of frustration there.

Katie Jansen: I actually I have, we are hiring a director of content right now, and I have … Maybe yes, we’re still in the interview process. But there was a candidate who I would say, and you know Lewis here has been part of the interview process is not what was described up here. She is not from tech, she’s actually from nonprofit, and so I really had to challenge myself and really focus on what are the skills that this job needs, does it really need a tech background, and I don’t think it does, actually. I think that if she’s really good at the brand, and she’s smart, he or she, they can learn the content. So this person is midway through, and I’m not sure if this individual is gonna work out one way or another, but I move …

Katie Jansen: Okay yes, you. You’re here. But I’ve moved her through the process and she’s done quite well because she does understand like she can run a persona project, she understands how to go through and do a customer journey project, and those are the skills that I need. She can learn the content, essentially, and so that’s what I’m trying … And Alice is reporting to me, I have had clear conversations before where I will say I am not talking about skin color. I am not talking about gender.I am talking about this person is different than we normally hire, and I don’t wanna talk about team fit. Do they fit the company? ’Cause when you say team fit that is actually just who people in the team wanna hang out with in my opinion, and that’s uncomfortable to say, but it’s kinda the truth. So how can we …

Katie Jansen: And so now we have more people on the team that are, you know, commuting from an hour and a half away, or they have kids that are in high school, and that isn’t really what the make-up of our team used to be. We have someone who just started who is a former professor. Like it is different and I’m really trying to push the team to do that ’cause I think we will be better and we will push the envelope more if we can start to do that. Did you guys have anything you wanted to add? Nope. Okay. Right there.

Audience Member: My name is Aurora, I also work in gaming and I’m curious about what games you guys are playing.

Katie Jansen: I don’t play games. That’s the truth. Honest truth. But they do, so go for it. I have two kids so I’m busy. I’m super busy.

Alice Guillaume: I just have a dog, so I can play games. Video games? Console games? Mobile games?

Audience Member: Mobile.

Alice Guillaume: I spent $500 on Cooking Fever. I finished the game. Task management, resource management, things where I feel like I’m accomplishing things.

Aurora: What was the name of the game, sorry?

Alice Guillaume: Cooking Fever. Look it up.

Katie Jansen: She’s like, I’m steering clear of that game. I don’t wanna spend $500.

Helen Wu: I do play games a lot, so I usually will download like the top free games, like the top 10 to 20 just to understand what’s out there in the marketplace. We do work with 90% of the top gaming developers out there in the ecosystem. So that’s part of the market research and intelligence that we do. And then beyond that I’m really addicted to word games right now, just like can you find all the words in this, like, jumble, and like improve your vocabulary and train your brain, and beyond that, I love playing board games. We actually do board game night like every two weeks or so at this office, which our creative team and game developers started, and it’s, we do it like on a Friday, we order pizza, we drink some beers and it’s just like everyone having a great time.

Katie Jansen: I play Wordscapes. I do play a few games, but I’m nothing like you guys. And I do insist we play board games and do puzzles with the kids, it’s really fun. There was a question back there.

Audience Member: I’m the back, so I’ll stand up. I’m Jessica and I work at Lime, and to Helen’s point I belong to a private organization of Rubik’s cube solvers. So we’re kind of going through hyper growth as well, so as a small company, what is something that you value that you’ve seen large companies lose along the way that you think is not worth sacrificing?

Helen Wu: That’s a really great question. I think as a still-small organization we really value, just the people that we spend time with. You know my greatest moments of joy are just my colleagues like challenging me to figure out, like, you know, how can we deep dive into this operational change together and how can we figure it out, and like how can we find a better way to get this solution, or let’s just geek out together about the data and about the product, and so I think it’s, you know, people coming together to solve the difficult challenges of the business and investing and spending time to help each other grow and learn along the way. I think it’s so important to hire people who can add to this culture because you know that’s, I think that’s what’s gotten us to where we are over the last five years, and I think at a larger company, I hope we never lose that part of our culture, is just like people geeking out together about like data and product and how to do it best.

Alice Guillaume: That’s a really good question, Jessica, so adding onto that I would say for me it’s remembering to keep things nimble. So our team has grown really large, and that means we need to change too. So some of the processes that we had in place don’t work when you’re at a larger scale, and you know, for me, I don’t wanna get in the way of really talented people, too, who can run with things. So taking a moment, even if things are going at 500 miles per hour, just stop and think about what still needs to exist in the structure, what can we break, and what we can recreate. I think that’s really important because it doesn’t make sense to keep things the same way if the team is no longer the same way anymore.

Katie Jansen: And we just did that yesterday. Alice and I sat down and we said some of these things aren’t working and some of these things are, and we haven’t rolled it out to those of you who are on analysis teams, so I won’t announce it here. But we are moving and changing, and I would just add, I think, which really you hit on in some ways, is being okay with failure. Fail, and this whole, fail fast thing, but I’ve been at this company for six years now, so what they have done so great is be able to say this isn’t working, now let’s move. And maybe not even, it’s not maybe even a pivot, but like let’s change, let’s scrap this, let’s go forward, like this isn’t working, we’re not gonna hang out here and try to make it work forever.

Katie Jansen: Now, admittedly, being self-funded that’s a lot easier to do when you don’t have a board. So I do wanna say that with some context because I think that we benefited from that quite a bit, but being able to fail and just move quickly has been pretty big. And not having a lot of meetings. I will reinforce that. Every time when you start to grow, all these meetings get going, and we’ve done a good job of keeping them to a minimum, and that’s one of the things Alice and I just did. We looked at the meetings she had, and we’re gonna start cutting some of the meetings ’cause we started noticing there was people in too many meetings and did they really need to be. I think ,ready, do you wanna go to the next panel? Yes you do.

Katie Jansen: Okay in the interest of time we’re gonna go ahead and get started, but feel free to get up and get a drink and move around as you want to. So now we’re gonna move more into focusing on tech, so this is engineering and ops here that you see represented here. This team works out of our Palo Alto office. So our business team mostly is here in San Francisco, and then Palo Alto is finance, ops, engineering, HR, some of HR is here, we split. You know how HR is. So this is gonna be about embracing change through actually building systems, so that actual adaptability we talked about on more of like a, you know, broad base and this is actually on systems itself. So this is Sonal, Anusha, Laura, and Swetha, and why don’t you guys just tell the audience a little bit about yourself. Take it away.

Sonal Gupta: Hi, I’m Sonal. Can you guys hear me? I work as a principal software engineer at AppLovin, and I’ve been working here for a little more than five years now. Anusha and I actually started a week apart from each other. So the infrastructure team, our main goal is to ensure high performance and scalability in our system. Essentially we wanna make sure that our ads are served in the most efficient and sort of fast way possible such that we are not only, the response times are as minimum as they could be, but we’re also not using as many resources and servers. So yeah that’s what my team does, and I work as part of the team and then some other things that I work on are revenue tracking infrastructure, among other things.

Katie Jansen: Anusha?

Anusha Ramesh: Hi, I’m Anusha, I’m also a principal software engineer at AppLovin. Like Sonal said, we’ve both been here about five years and two months now. So it’s been a while. And I actually work on the platform team. Funny enough all four of us actually work on different teams, so you’re gonna get a slightly different aspect on our engineering side, and the platform team, what we do is we make sure that all the data that we get from internal sources, external sources, get into the right databases, the right locations in the right time so that everybody in engineering, business side, everybody can actually use it. So some of the systems I work on are things like Pub/Sub system and custom counting frameworks.

Laura Pfister: Hi, I’m Laura, and I’m a software engineer on the ad server optimizer team. I’ve worked for AppLovin for almost three years now, and on the optimizer team, we strive to answer, essentially, two questions related to ad serving. So whenever we get an ad request we’re trying to determine, one, whether or not we wanna show an ad and spend the time and resources that it takes to do that, and then two, what ad do we wanna show and what valuation is that ad placed at. And so we’re actually making use of the data that Anusha’s team and the platform team give us to answer these questions.

Swetha Anbarasa: And hi my name is Swetha, and I work as a network engineer as part of the operations team at AppLovin. So we provide the backbone for all the stuff engineering does. So primarily my responsibilities include designing and configuring and maintaining the production and development networks at AppLovin, and also help with like maintaining our data centers as well as optimizing our network routes. I’m also responsible for capacity planning as our businesses grow.

Katie Jansen: Yeah so obviously we are processing a lot of data and doing a lot as you’ve heard here, and as I mentioned early on, we do that with just 160 people right now worldwide.

Katie Jansen: Let’s get real specific and talk about how and when do we decide to automate. You heard the panel before this talking about how that’s been a real advantage for us.

Katie Jansen: And then really what’s the best way to automate. If you guys could give some examples around tools, I think that would really help the audience here too to, like, get real specific. So you wanna take it away, Swetha?

Swetha Anbarasa: Yeah.

Katie Jansen: Okay.

Swetha Anbarasa: When did we decided to automate was a point where we needed to really scale. When the business started, we were able to you know manage everything with a few servers in a rack, managed or something, and it was all working fine. But as our businesses grew, we had to scale very quickly and we had increasing demands.

Swetha Anbarasa: From a networking perspective, our main concern was to find a vendor that actually has a development driven product. With the major players in the industry at that point, it was kind of a challenge to find a vendor that was actually development driven for our networking, so we had to do a little bit of research, and after some evaluation, we ended up using Cumulus Linux.

Swetha Anbarasa: Cumulus Linux is an open box solution. So you could actually install … It’s a switching software, so you could actually install the software on a bare model Linux machine and start working with it. So some of the advantages for going with that sort of a vendor is you can use the software to automate very easily, and we chose Chef as our automation tool. Chef is based on Ruby, so it was easy for us to write code in, as well as our server infrastructure also uses Chef.

Swetha Anbarasa: So networking and our servers together, everything is in one place in Chef, and we have like one repository where all of us contribute and change stuff, so it was easy for us to manage. And another advantage is, they were able to provide us with virtual images, so we were actually able to simulate a data center before we actually invest money and like buy stuff and actually deploy. A lot of it was being tested before we could deploy, so that was giving us a big advantage.

Swetha Anbarasa: Because we had limited resources, automation was actually playing a very big role, and we had our reasons to choose Chef and a few of the open networking platforms. And we also use Juniper routers which have been automated using Python scripts, and we integrated with RunDec to run a job in case we wanna do any sort of configuration changes or upgrades. For a small company, I think we’ve done a lot of automation, and we’re able to scale quickly with limited resources because of that.

Katie Jansen: Okay. And then how about on the engineering side? Maybe Sonal and Laura, what was it like, can you give some examples of automating some processes?

Sonal Gupta: The ops team automated the process of bringing up servers, and now engineering team had to make sure our services could also be deployed on all the servers that were being brought up, without a lot of intervention from us, and one of the big things we pride on in engineering is how agile our development is.

Sonal Gupta talks about automating processes at AppLovin Girl Geek Dinner.

Sonal Gupta: We push to prod every single day, and when I originally joined and even two years after I had been working, we used to push multiple times a day. So a single, any code change that you make could affect all the 60 billion requests that we are serving. For me my first year I was always very nervous when I was pushing that button to sort of deploy to production, ’cause there was no-

Laura Pfister: So nervous.

Sonal Gupta: Yeah. But it’s, I mean, ’cause we didn’t have a lot of structure around how to push and how to make sure that the changes we were making were very limited. So we tried to build an A/B testing sort of test control framework that would ensure that any new changes we push had very minimal impact and we could ensure that once it was verified, the changes were assessed, only then was it pushed to production. And all of these changes, we couldn’t like hack them, do Eiffel statements, or you know basic sequel queries, ’cause they were all prone to errors. So we had to like come together sort of build a structure that was away from human error, we could validate all the changes we were making, and we could triage and sort of back track the issues, and how we push something, when we push something, and how do we fix it.

Laura Pfister: Right and to add to Sonal’s point, so the A/B testing framework that we put into place in order to kind of combat all these issues is really really important to the productivity of the optimizer team. So without this, we were kind of able to A/B test maybe one or two features in like a week or something. The process was slow and we might have to wait for one feature to be clear just because we just didn’t have the bandwidth to do a lot of different things and once. And then it was hard for us to analyze the how one feature might impact the entire system, and it might be hard to pinpoint exactly which feature might be causing an issue, and we might need to even have reverted the entire all of production to kind of backtrack that problem. So with this new framework in place, we’ve been able to really really increase productivity and then just the overall safeness of our system so we don’t have as much of that fear when we’re pushing, we can really control exactly how much of our traffic is diverted into one feature, and really pinpoint and analyze the impact at each and every feature. So we can get a lot of features out and do it really quickly, and that’s really made us much more productive, much faster, and it’s just really a lot safer.

Laura Pfister: So I think that’s one of the biggest things that have allowed us to scale a lot of optimizer features and also infrastructure features really, really quickly as we’ve grown.

Katie Jansen: So switching gears just a little bit then, so we talked about how we automate processes, but what about how we grow and scale with those, have you had to change processes over time? Have you had to scrap them completely and start over, and what does that look like and how easy is that to do?

Anusha Ramesh: Like I mentioned when I did my introduction, I work on a custom counting framework as part of my daily jobs to say, but way back in 2014, if you go all the way back then we were a pretty small company still, we were starting to grow pretty rapidly at that time, and all of our data, all of the answers that we were trying to get from our data, we would use SQL queries. That’s what everybody does, you query your database and you get your answers and you look at ’em and you analyze. And as we grew and grew as the year went by, our data got bigger and bigger and bigger, and suddenly these SQL queries took an hour to come back with data. An hour and a half. They’d hit database connection issues, and networking issues, and so at the end of it, we were like, okay, well, this doesn’t quite work. It’s not gonna scale as fast as we are growing.

Anusha Ramesh: We actually took that entire system, redesigned it, re-implemented it, and pushed out a brand new custom counting framework that provides the answers and the same results that we wanted out of those SQL queries in ten minutes. We can do a query on that database, get results in ten minutes. And this entire thing because of how fast we move as a engineering department, we had it in from design to production in three to four months, which is pretty quick in my opinion, and then, of course, years go by. We keep growing, growing, growing, and with all the alerting and monitoring that we have in place, we realized that we’re soon gonna hit a limit there too. So what do we do then? Alright, scrap the whole thing. And we literally went down back to the square one, ’cause I had to redesign the whole system. Redesigned it, re-implemented it, re-pushed it to production three months later in 2016, and knock on wood, that is the system we have in place today and it will continue to go. And I bug ops about it all the time, but we have nice alerting and monitoring systems that ops has set up for us that keep our systems in place.

Laura Pfister: Another thing to go with the scalability, so as data has increased, the business has increased, and logic would follow that campaigns and ad inventory have also increased. With this increase, the ad selection process has also kind of gotten bogged down at times. There’s more ads to go through, so every single thing that you do in that part of the system is gonna start to be slower. This is a gradual increase, and, you know, it didn’t really impact things, but sometimes you wanna scale things, or sometimes scalability is about kind of dealing with making things larger, but in this case we wanted to actually add campaign breadth, a variety of additional campaigns to serve on a different component of our system called real time bidding, and in this system we are bidding on other ads, ad request from other systems, and in order to do this, we need to meet a certain millisecond threshold. When we tried to add more campaigns in this system, we found that we were no longer meeting this millisecond threshold. We were looking at the data, kinda seeing like okay nothing’s happening here so we’ve gotta do some analysis. We used a tool called a flame graph to actually analyze the CPU usage of our ad server and see where things were taking the longest amount of time.

Laura Pfister: We looked, and sure enough it was in our ad selection process. We were able to use this to see exactly where we were spending the most time, and then we were able to analyze this and go through and figure out where we could make cuts, where we could figure out how to do the exact same thing, get the same result and produce the same ad that we would in the previous system, but do it faster. Through this process, we were actually able to increase speed as much as three times faster in some cases, and drop CPU load a lot. It was really a great success and just another example of how we were able to scale things quickly, and I mean the entire process was like a few weeks, maybe.

Katie Jansen: How do you decide that? How do you decide when to just scrap it all together like you were talking about, or keep working on it and kinda push through? ’Cause it sounds like, especially in your example, Anusha, it was almost like broke and we were in trouble. Do we have something that’s in place that you know lets us know about that ahead of time, or is there some kind of meeting or team situation that occurs for you guys to decide, hey this is looking like it might need to be changed, we should invest more? What does that look like?

Anusha Ramesh: Specifically for this, well not even specifically, all of our systems are actually, we have a common tools package which platform keeps up to date, and most of our services plug into this system and use our tools to send metrics, send stats up to Graphite, which is one of our monitoring tools. And Graphite is amazing. If you ever see our office in Palo Alto you’ll see just TVs full of graphs everywhere, monitoring the most, like every little thing you could think of.

Katie Jansen: I like that you think that’s amazing.

Anusha Ramesh: They’re really cool-looking graphs, though. We have a ton of TVs up everywhere.

Sonal Gupta: I think it’s interesting when people walk by they see all these graphs, they don’t know anything what they’re about.

Anusha Ramesh: [crosstalk] … tiny little spikes somewhere of one thing going wrong, and suddenly you’re like oh wait, that’s wrong, we need to figure out what went wrong there.

Katie Jansen: To be fair, we use those in our videos and all sorts of things for marketing.

Anusha Ramesh: [inaudible] promotional videos. That’s one of the big things that tells us when a system is coming to a limit. We have stats on our databases, are we hitting some size limit where we can’t grow any further, are we hitting a CPU limit, can we not process things a little faster, are we hitting a backup somewhere where files are just sitting around just waiting to be processed and we don’t know. And if you don’t know about that, then that’s a problem, because we are gonna be behind. Our reporting pages aren’t gonna show the right thing, business side won’t see the right data until maybe hours ahead and then they’ll come to us and be like “Where’s the data?” And we have to be like “Uh, let’s go find it.” But all of these tools in place, Graphite, Zabbix, Chef, all of these things help you automate and figure out these alerts that might happen, and for the counting framework, that was the biggest thing that actually told us this is not gonna work. We hit a threshold that we had that wasn’t full but not there, like an 80% threshold, decided to do some things to minimize it for the short term, and then decide a long term solution.

Anusha Ramesh: I think that’s one thing in engineering that we do a lot, is we do a short term solution that is like for five, one week, two weeks, let’s fix the problem immediately so we continue to run, and then figure out a long term process, a long term solution that we can push out in a month, month and a half that will actually fix the entire solution and we can go for another year, or two, or three, or however long it may be, with the system.

Sonal Gupta: One of things I’d like to add to that is so one of the very first and sort of the most drastic scrapping of things we did was our ad server used to be in PHP, and a lot of our other components were in PHP as well. While we were debugging an issue with real time bidding and sort of why aren’t we bidding fast enough, why aren’t we winning as much, we realized it was actually the choice of language that was the bottleneck for our system. We decided, I mean we could’ve continued to patch and sort of improve the PHP system, make it faster, sort of find some wins here and there, but we realized that it was sort of avoiding the unavoidable, ’cause at some point we would have to move away from that system. We took a step back and we decided, okay what can we do, and we decided to come up with a C++ ad server solution. I mean it was a sort of a big undertaking ’cause most people on the team didn’t even know C++, but knew how powerful the language was, so within a few months we all decided to learn C++ and had a working ad server solution that not only supported our in-network ad serving but also our real time bidding solution.

Sonal Gupta: And over the year, then we migrated all our other services away from PHP to Java, or C++, and this is like something that’s very fun. One of our first intern projects was to move a service away from PHP to Java, and the intern finished it in two months. That’s sort of the trend at AppLovin. We try to get, if there’s a need and we realize there is a problem, we don’t shy away from completely scrapping the system, and more importantly we try to sort of learn quickly, everybody grows together, and try to get a solution out as soon as possible, and not a hacky solution but something that will work long time. I mean we’re currently still using our C++ ad server, and everybody on the team, we are always looking to make changes even if they’re like small changes to the two string function so we can get like little wins here and there and continue to optimize our system.

Katie Jansen: Yeah. That actually brings me to our final question, which is what about looking forward? You’re building this stuff now, how do you future-proof it? Can you future-proof anything? Maybe you could each give me your thoughts on that and then we’ll open it up for Q&A.

Swetha Anbarasa: Sure. think yes, we can future-proof things to an extent. So from the ops perspective, the way we try to plan is we don’t just have 2X or 3X capacity. We actually wanna have at least 10 times the current load. That’s the capacity that we look at, ’cause we’ve actually run into instances where that really helped us and we really needed the bandwidth and the servers to help. I know Anusha agrees, ’cause we’ve run into a lot of issues where we had 40 gig of bandwidth and it was actually, we never used it that much, but suddenly one day we needed it, it was during the holidays.

Anusha Ramesh: It’s always [crosstalk]-

Swetha Anbarasa: Yeah it’s always during the holidays that we need it, and we actually had a 40 gig bandwidth between U.S. east and west, and we were able to do the data transfer quickly without anyone even noticing or waking up anybody or like disturbing them from their holidays. So that’s how we plan, and I think one of the important or like key concepts about being future-proof is to have excellent monitoring, and for the size of our company, I feel like the monitoring we have is really good.

Swetha Anbarasa: We use Graphite. So Graphite is beautiful, colorful, so it’s numbers in graphs. So you’re not looking actually at tables, you’re not looking at boring data, you’re actually looking at graphs, and spikes, and dips, and it’s all like time and data, so it’s easy for us to understand, and you could actually add any component to Graphite and start tracking it even before you understand why we are tracking it. You know maybe a couple of months later you run into an issue, and you go back to the one thing that you were tracking, and you look at it, and you are like “Okay, thank god I actually have these stats.” So Graphite keeps collecting, and it works with Zabbix and also holds SNMP data from networking devices. So it’s sort of like one place where we can look at a lot of our stats and attributes and data to monitor.

Swetha Anbarasa: Apart from that we also work with, we also have like Google Cloud Compute, so we have systems where we can just spin up several servers if we need it and then tear them down once we are done. So that also helps us to be future-proof to some extent, and also maybe like from the networking side we also look for vulnerabilities that keep coming, we proactively patch our systems, do upgrades, be on the lookout for anything that’s going on, and kind of have everything up to date so we don’t have to worry about when something actually hits us. Yeah.

Katie Jansen: Laura, you wanna … Final thoughts?

Laura Pfister: Yeah, so when I’m working on something it’d say it’s probably a little bit more granular, I would say for one thing the monitoring that we have in our system definitely makes it really easy to see any small changes. So if there is an issues we’re gonna catch it faster, and that’s really, really important. From more a designing things for future proof perspective, I’d say, since I’m adding a lot of new features, a lot of new functionality, whenever I’m looking to add new things I try to look not just for, will this last like a week, but designing it in such a way that I’m trying to think what else might I use this for. So that’s just kind of an important way that I approach pretty much all the code that I write.

Katie Jansen: Anusha?

Anusha Ramesh: So like Swetha said, we definitely have the ops, like if they have 10X capacity, platform will use it. At some point in time we will 100% use it. If it’s just an emergency data from one side of the country to the other because something went wrong, we will use it.

Anusha Ramesh: I know I bug ops every single day for random stuff, and I’m glad that they allow me to do that. But as for future-proofing, a lot of our systems they’re all repeating the same thing, monitoring, monitoring, monitoring, because that is the biggest thing you can do. And our Java services that we use, we all have one base set of tools that we plug into every single Java service, and it makes every single developer, they don’t have to know how this tool is written in Java, they just use it and they know that as soon as they use it, five minutes later it’s gonna end up in Graphite, it’s gonna, you can pull up a dashboard on it, you can see a graph of it, and it’s done, and it’s there for you to see forever. Until you stop writing to it and it goes away.

Anusha Ramesh: But you at least have the ability to do that, and that’s one of the things that is really nice is it just exists and people use it. Funny enough, this monitoring tool, we actually were trying to figure out some issue, like we were trying to test a new database, something was going wrong, we realized that it’s actually our tool that writes to Graphite that was not working. We took it around, we redid it, and a day later we actually got 200X improvement out of it just by rewriting the whole thing, and we went from about 10,000 writes a minute to two and a half million theoretically writes a minute.

Anusha Ramesh: And it was like, of course, that’s all theoretical numbers and it’s more like a million writes a minute, but in the end it’s, like it went out to production, like it went out into our code system. Nobody knows that it actually got changed on the back end, but they’ll use it. They’ll see an improvement, and it’s just kinda there. And that’s the cool thing about having these systems that make it future-proof. They exist, you use it, things show up in graphs, it works. And you have to think about if you’re gonna do those million writes a minute, can your system handle it? And that’s future-proofing. And that’s kinda what we try to do at least a little bit.

Katie Jansen: Any final thoughts?

Sonal Gupta: I’m gonna add something that Anusha told us earlier in the day, that even with the issue the monitoring system had we still only needed two servers for getting all the metrics from all our different Java services. So the revenue tracking infrastructure that I work on, ’cause it’s such a tricky component, and that is where we track our revenue, any changes I make could potentially make us lose events, make us lose our revenue. So when I started, I integrated with the monitoring service, ’cause the platform team had done just such an amazing job giving it to us.

Sonal Gupta: One of the best things is when you look at our data, it follows a very certain path, and if there’s any variance in it, you will be able to tell there’s an issue. So that’s one of the reasons why monitoring becomes so important ’cause we can see every little detail of our system and how it changes. So for specifically, for example, for the revenue tracking infrastructure I can see how many revenues events I’m tracking every single hour, minute, day, and if there’s a change, I can, we can determine sooner than later that there was a change, and it was usually correlate to a certain push or some sort of event that happened and we can either go ahead and revert the change or sort of push a fix out quickly. So, yeah.

Katie Jansen: Cool. Alright in the interest of time here I’m gonna open it up to Q&A. Maybe take one or two, and then ’cause I know I believe there’s dessert. Yes, there’s dessert and drinks. I wouldn’t wanna promise dessert if it wasn’t there. Any questions? Yes, right there in the middle.

Audience Member: Hi, I’m Kayce.

Katie Jansen: Hi, Kayce.

Audience Member: First of all it’s been so fun to listen to you guys talk. You’re so passionate about what you do. This has been really engaging to hear all of you talk. I guess my one question is what’s the one thing that you wish you could do faster ’cause you’re going at such a big speed? How do you communicate together? I mean you’re all on four different teams so how do you make sure you aren’t messing up each others’ stuff and what’s the one thing you just wish you could do faster?

Katie Jansen: That’s a good question.

Anusha Ramesh: Thank you for your question. One of the things about not messing each others’ stuff up, is we are on four different teams so we do kind of work on separate projects, and we do, at the very beginning back in like 2013, when we started, we were using GitHub, Algin, all that good stuff for code version control and things, but we didn’t have a good code review process in place, and over the time as we grow from just maybe two or one engineer in a team, to four, five, eight engineers in a team, we’ve put a code review process in place, we’ve put in a structure so that we actually don’t step on each others’ toes. We have branches, we have to push the master, and then push from master to production, and do all these things, basically red tape if you wanna call it, before we can actually push it out. But even at that point, with all of these processes, we still push code every single day to production. We still push out our stuff. It’s because we have a good, I guess we are all used to it now, and it’s such a good process that we have in place that it doesn’t seem, it’s streamlined. That’s the word I was looking for. It’s a streamlined process that actually works for us.

Sonal Gupta: And I think we are still a very small team, so really we all have a lot of different responsibilities and roles we play every day, and we are small enough that they’re very separate from each other even if you’re part of the same team. So there’s a lot of communication and as you had mentioned, because you know facilitating sort of the open office structure, whenever we have questions or we wanna discuss something, we just go to the person’s desk, it doesn’t have to be like an elaborate meeting, and so really, like, if there’s something you’re working on and you think somebody else might have done it before you can just go ask them. It’s as easy as that.

Katie Jansen: Yeah I mean I’m obviously not on your team, but I do talk to, you know, the VP of engineering quite often when I have, which is rare by the way.A lot of times VPs of engineering don’t wanna talk to the CMO and they are annoyed by us, and that’s not the case here [at AppLovin].  So I would say – or you know to the engineers themselves – I would say communication is really key, and just different types of communications, whether it’s taking the time to talk to someone, or we really leverage Slack. I can’t say enough how much we leverage Slack. We have multiple channels, we will do direct messages, however we want to use it, and it’s been pretty awesome.

Katie Jansen: And then we all use Asana, I would say too. They use Asana, and then they made my whole team use Asana, and they didn’t like it but they like it now. It took a while. I actually try to sometimes mirror some of the processes they’re using, because I think that it helps. Obviously I’m not gonna do Graphite and those things, but like the communication processes, trying to do similar things helps us communicate better as cross functional teams across the company too. A question back there, yeah?

Audience Member: Hi my name is [inaudible]. So great you guys are so flexible, when the systems just not working you change it. My question is when you change the database, you know scrap it, how is decision made? How do you be able to push that idea?

Swetha Anbarasa: I think the decisions are mostly a combined discussion. Like Katie said we don’t have that many meetings, so in case something like this comes up, we would do a discussion where we pull in like the VP of ops, engineering and then the concerned people in the team teams that are involved, and sort of like white board it, get everybody’s opinion, write down some stuff that, you know our top technologies that we wanna use, and then kinda get everybody’s opinion. And again we would like discuss over Slack in a particular channel and kinda like arrive at a conclusion. So it’ll be more of a discussion with the team’s concerned.

Sonal Gupta: And I think just when everybody wants the best for the product and the company, it’s very easy to come to a quick conclusion.

Katie Jansen: Yeah, we’re all on the same page.

Sonal Gupta: Yeah.

Katie Jansen: Alright, one more question. Go for it.

Audience Member: So as the SDK provider, you deal with a lot of customers and face a lot of issues. How do you track the issues that are reported in and do you have analytics on them to figure out like where to fix things?

Sonal Gupta: Actually with have a separate product team, Jimmy’s team, that sort of does that for us. So one of the amazing things about engineering is that because of our product team they take care of sort of triaging and determining the issues before they even get to us. So by the time an issue comes to us, it’s probably something that is important. And you know you work with the SDK team a lot.

Laura Pfister: Yeah, I’ve done a lot of those kind of ad server, sever side settings, and yeah I only get the ones that we’re certain are a problem immediately. So having that kind of filter above us makes it so that we’re able to get those out really, really quickly and prioritize them as much as possible.

Katie Jansen: Yeah, and the product team, while they are business people, have some engineering and like background, so they can triage pretty well too. So I think that kind of having a team that’s in the middle there makes a big difference. Alright, I’m gonna wrap it. We are available to answer more questions, but I’m gonna let everyone go get drinks and dessert. Thank you very much for coming.