Shea Kelly: Is that working? Am I on? Welcome, everybody. Happy Monday, or as we say in my household, and everyone knows this here, happy day. How many people watch Sponge Bob? Then, you know what I’m doing. Listen. I really, I want to reiterate what Angie said. First and foremost, we are actually just thrilled and delighted that you’re here, and I mean that. We have been talking about doing an event like this for a while, but it’s always so backed up, the schedule like when can we get and do one?
Shea Kelly: We’re thrilled to do it for a couple of reasons. One is probably the obvious — we’re a growing business. We’re very deliberate and intent about wanting to continue to expand and diversify our company with rock star talent, and so, at a minimum, we want to expose and have people get greater awareness of Sumo, who we are, what we do, but obviously, the biggest thing, I’ll be honest, I think we have joy today in having some of our fabulous team members, who are all smiling at me now, share some of what we do.
Shea Kelly speaking at Sumo Logic Girl Geek Dinner.
Shea Kelly: You can get some insight into, aside from Sumo, I think it’s an industry that we’re in that’s pretty darn exciting and interesting, and we hope you get some good things from it. It’s a big week for us because this actually is the week of our second annual user conference. We have some 600 people registered to be at a hotel in Burlingame later this week. We don’t do anything small, so we thought in the same week, let’s have a board meeting, and let’s do a Girl Geek Dinner and just six other things. It’s like, but we like to go big or go home as they say, so again, welcome. I do want to also tell you, in case you didn’t notice, when you leave later, there’s a gift bag for you up front with a little bit of what we would call Sumo schwag. You cannot leave without a Sumo squishy. In the bag is a Sumo squishy for those … So many people are like, “Are we going to get a squishy?” You’re getting a squishy. They are so popular. Go on a campus, and it’s like they look at you, go, “Hi. What are your jobs? Can I have a squishy?” It’s like, “You don’t want to work for me. You just want a squishy.”
Shea Kelly: Okay. I am only going to take a few minutes here before the main speakers. I am not the main speaker. Take a few minutes. Just tee up a little bit about Sumo Logic, who it is that we are, what we do. I don’t know how many folks are aware of us, so I’m just going to do a quick welcome and intro there. The welcome, I think, is done. Then, we’re going to shift to our presenter. Bret is going to focus as we’ve talked about the whole, how we enable customers to be super successful in using our product is, we’re coming at it obviously from different ways — we come at in terms of the user experience and the design and how do we think about what customers need, and how they use the product, and how do we build that into the product, which then leads to Riya, who is going to talk to us about from an engineering and development perspective, how do we do that? How do we continue through the process? It is literally non-stop, and you’ll see that and hear that in what we talk about today. How do we infuse that into our development of the product?
Shea Kelly: The third piece is learning. Stacy is going to talk about the tools and all the things that customers need, and whether it’s docs and it’s training and it’s certification. They need to be enabled, so they’ll use the product, right? That’s pretty obvious. Oh, lights are out. Okay, so we’re into a groovy mood here. Okay, this is good. No, I like this. Okay, thanks. We do it right here. Then, the last piece, Jen’s going to speak to down here at the end is security and privacy. I mean, it is obviously, you can’t pick up a newspaper or see anything or think about your own data, personal data, but companies are migrating, as we’re going to talk a lot about, their data to the cloud. That has with it inherent, oh my gosh, how do I make sure that it is secure, it’s private, and that we have all the compliance and regulatory things in place? Jen’s going to talk a bit about that. Then, at the end, we’re going to open it up for discussion. I’ve already had a few questions that have been raised that will feed into the discussion later. Into that mix, we’re going to add the lovely Mary Ann O’Brien who’s sitting here who leads a big team for us in sales, and she can also bring a bit of perspective in terms of how we go out and speak to customers, okay? You can always throw a question out in between. Otherwise, I think what we’ll do is save a lot of it for the end. Sound good? Happy Monday? We’re still good? All right, you’ve had food. I hope it was good.
Shea Kelly: Again, I’m just going to go through this quickly, but if you think about and say, “Well, what does Sumo Logic do?” what we would say our tagline is, is Sumo Logic is a machine data analytics service that helps companies to build, run, and secure their modern applications. You say, “Okay, what does that mean? What does that really translate into?” If you think about the path over the years, the transformation from, I guess what I call the old school with data centers and on-prem solutions where data actually is stored and housed, and the transformation, as you know, this digital transformation we’re living in and will continue is to the public cloud.
Shea Kelly: What the natural outcome of that is that we’re building a different kind of software, and it is a software that is always on. It’s a software that, in many cases, or these modern applications that result from that are customer-facing. They are revenue-generating. Those are new aspects of something. It’s a 24-hour if you think about it in terms of availability. Take out your phone. Think of the things that you take for granted now, and all of that data has to somehow, in the private cloud, in the process, be built, run, and secured in that manner.
Shea Kelly: To us, that leads to three things, and I hope this translates. If not, sometimes, the … I don’t know if the white on white is looking good here, so the very first thing is obviously the cloud. Data smog rating, you know this. It’s going to continue too. It’s going in a rapid phase, and it’s going across industries and verticals into some of the sectors that before, would have been hesitant to, into banking and in insurance. You’ve got that migration happening pretty quickly and pretty voluminously, if that is a word.
Shea Kelly: What that second part of that is it leads to this notion of continuous delivery. It’s always on, and that’s where it translates to the humans, to the people who are building and running and securing these apps because they’re not doing it the way they used to do it before. Now, if you think about it, and you think about the old way of developing software, the old … What do they call? Waterfall, so you would have an idea. You’d put it in development. It would go through a whole bunch of steps, and at some point later, something would spit out on a disk, and you’d take it somewhere. That’s not what happens anymore, right?
Shea Kelly: here’s a different way of architecting these modern apps, and so, what has to happen and has to follow that is we have to have a supporting mechanism for these teams who do it. For us, we say, okay, if it’s all going in the cloud, and it’s continuous delivery, it’s always on, that’s where Sumo Logic comes in. What we do is we provide a product that’s called continuous intelligence. What that does is it gives these teams who are having to architect those modern apps in a different way and who have to manage and do things in a different way as teams, it gives them an analytics product that helps them to again, build, run, and secure those apps in a way that meets these modern needs.
Shea Kelly: Just quickly, we are about now eight years in business. I think our founder might still be over here, so I’m going to have him give a wave. Christian Beedgen. Should I tell the story of how Sumo came to be? I won’t do that, but anyway, it’s a dog. It’s not Sumo. Catch me for the story. Anyway, so, Christian is our founder. Bruno’s still here. He’s our founding VP of product and strategy, and so, eight years in business, we are now some 1,600 customers strong… 50,000 users I think… so it is proliferating pretty quickly. We have, I think as a lot of SaaS models do what you’d call a land and expand. We get in on a use case, and then over time, they say, “Well, it works so well here. We need to expand this,” and it easily can expand across the organization.
Shea Kelly: We are serving companies, like I say, in every vertical, in every industry. If they have a modern application, they have got to have a way to analyze and think about that data as they build, run, and secure it. We have customers from Anheuser-Busch to Pinterest to … Who are some of the others? As I’m thinking about it, JetBlue or Betfair, Hearst, and so, again, these are also global organizations, in many cases. For us, it’s, like I say, eight years strong. We’re some $250 million in investment, so we were no longer a startup. We’re a late-stage private company, I would call us, funded by some of the best and brightest in the Valley, and so we’ve got a ways to go on what my last point to be here is this journey. We’re just getting started in some ways. I mean, this proliferation of data, if you think about it, I am going to step in here just third from the right. If you think about it, that’s 2018.
Shea Kelly: Machine data by 2020 is predicted to account for 40% of all data created. If you think about it, this machine data, which is what we focus on with our application service, is growing at such an exponential rate. Customers and companies have got to find a way to analyze that data. This is the background. Again, if you just think about it, it’s digital transformation, software as a service, companies are going digital, modern applications always on, and through that process, which is what leads to our discussion today, is this question of, how do we make sure, because customers may have different needs and how they do these things, that we are listening to them and looking at all of their needs and figuring out how we build that into the product ultimately through how do we do security and compliance around it, okay? With that, I am going to turn it off to the real presenters. I’m going to start here with Bret. Here’s our group. Here you go.
Bret Scofield speaking at Sumo Logic Girl Geek Dinner.
Bret Scofield: Hello. Good? Okay, perfect. I’m Brett Scofield. I lead UX research at Sumo Logic. UX research is a relatively new discipline, so I wanted to talk a little bit about what that actually means. Design’s been around for a while, and I think product development has also been around for a while. UX research is really a combination of … or it works very closely with those two, so a lot of times, what we’re doing is getting feedback from our customers and from our non-customers on ideas. Sometimes, these ideas are really big ones. Should we build this feature? Is there a place for this new product? That type of thing. Sometimes, it’s more granular. We’ve decided, yes, we’re going ahead with building a metrics feature, and now, we need to get into the nitty-gritty of what exactly that looks like and how our customers are going to use that type of thing. UX research, I converse with our customers pretty much all the time. I advocate for them heavily, so I’m always telling their stories to our engineering, product management, designers, just really emphasizing like, “Hey, our customers are running into so much pain with this. Can we please, please change something?” et cetera.
Bret Scofield: Then, on a personal level, obviously, I enjoy Boomerangs. This is with my work wife. Hi, Rebecca. Then, I run a lot. I’m a marathoner, so I’d love to chat about all those things. I’m on the team, I’m the food contrarian for all things, so people are always like, “Hey, have you tried this awesome new like hybrid taco thing?” I’m always like, “That sounds like it sucks.” That’s usually my role. Then, I wanted to get into a little bit about why I chose Sumo, why I’m working here.
Bret Scofield: First off, we have a lot of really interesting gnarly problems to solve, and that’s great. You start off Monday morning, like today, you really dive into those problems. You’re working with engineers. You’re working with security people. You’re working with a bunch of people who are really, really smart and have a lot of things to say. That’s amazing, and that’s great to work on those problems, but then, our team does a really good job of balancing that with a little bit of levity. We go to lunch with the UX team. We talk about tacos. We talk about horoscopes. We talk about whatever else. That gives you enough breathing room and enough space to go back in in the afternoon and to like really dig into those problems again.
Bret Scofield: I also really appreciate that there is a huge customer focus throughout the organization. That’s why we’re giving this talk. We’re talking today about how the customer influences everything that we do at Sumo Logic, and so, it wasn’t hard to pull that together with all these other women because everyone in the organization really thinks about the customer and keeps them at the forefront of the decisions that they’re making.
Bret Scofield: Okay, so today, I’m going to be talking through the product development process, and I’m going to be talking through the two halves of it, which here are running concurrently. Instead of sequentially, instead of having all of this discovery happen before delivery happens, we have both of these processes going all the time at Sumo Logic. The first part of this, the discovery bit is really answering those like super-high level questions. Should we build this thing? Is there a market for this thing? That type of, it’s exploratory, it’s really like pie in the sky sort of stuff.
Bret Scofield: Then, after we’ve figured something out from there, we’ve determined, yeah, there is space for this, this is something that our customers would actually want and would use, then those findings are given to delivery focused teams. These teams have an idea. Now, they’re like, “OK, we need to build a metrics functionality or we need to build this type of functionality,” and so they need to dive into more of the specifics of that. They’re figuring out, “OK, with our customers, how exactly are they going to do what we want them to do in this? Is it a dropdown? Is it a visualization?” They get into the meat and bones of that.
Bret Scofield: What I’m going to talk about is how our customers drive both of those processes. With the discovery processes mentioned, these are the really big high-level questions, and the major question that we’re trying to answer here is, are we building the right thing? Is there space for this thing that we want to do? One of the recent … Well, not recent. One of the things that I’ve been working on for a long time at Sumo logic has been our personas. Many of you are familiar with personas. These personas are not real, but they are an amalgamation of our customers and their mental models.
Bret Scofield: The reason that we went through the exercise of creating personas is really because we want to give the entire organization a common vocabulary. Melinda has a certain mindset and approach to the product, so when she goes into Sumo Logic, she’s likely trying to accomplish a specific set of things. She has a certain familiarity with a product. She goes in in a certain frequency. Andre, completely different. He uses Sumo Logic for different things, goes in with a little bit more uncertainty, all these types of things.
Bret Scofield: When we refer to our personas, I can say, “Hey. Stacy in documentation, I’m dealing with an Andre.” She immediately knows what that means, and she knows what sort of pains he’s likely to be feeling, where he’s coming from, all that type of stuff. The process of creating our personas, we attacked this in, first, a qualitative way. We met with a bunch of our customers on site, watched them do work. We talked to them about their educational background, their career trajectory, what are the things that are painful for them, et cetera.
Bret Scofield: Then, after that, we aggregated all the data, came out with three personas. Then, during a hackathon, we went through the process of deriving quantitatively what our personas look like. Sumo Logic, as Shea mentioned, has a ton of data on what our customers are doing in the product. We analyzed it during this hackathon. We ran a k-means clustering algorithm on all that data, and we derived the key use cases. Those key use cases actually mapped really well to the personas that we had derived qualitatively. There’s both a quantitative and a qualitative backing for these. Then, they’ve permeated the organization. I think almost everyone at Sumo Logic knows who Kathy and Andre and Melinda are, so.
Bret Scofield: Then, as mentioned, so this was a discovery project. As mentioned, the discovery stuff often influences the delivery things, so now, in a lot of the delivery projects that are going on, we define at the very beginning who is this specific feature for? How do we expect an Andre to approach this? How do we expect a Melinda to approach it? How would it be different? Those types of things.
Bret Scofield: Then, I want to talk a little bit more about the delivery process. As mentioned, the delivery process is when that sort of overarching thing is defined. We know we need to build this specific thing, but the question that we’re seeking the answer here is, are we actually building that thing right? Are we doing the things that need to be done so that a customer can actually do the thing that they want to do in here? With delivery specific things, we start at the very broadest level, and then we narrow in.
Bret Scofield: One of the things that we commonly do here is participatory design. We have quite a good relationship with internal users of Sumo Logic, so our customer success team uses Sumo Logic heavily. They also interface with our customers. Sales engineering, similar case. We bring them in. We also bring in our own engineers. They love dogfooding Sumo Logic, and they generally like working with us, so we bring them in. We do design exercises where it’s pretty much, we start with a blank canvas, and we have them draw, what is your 10-star experience? What would this future look like in an ideal world?
Bret Scofield: We’re actually, as Shea mentioned, we’re holding our user conference on Wednesday and Thursday of this week, and we’re doing a large-scale participatory design with a bunch of our customers. We validated that there’s a specific feature that we want to build, and so we’ve made kind of a Lego kit for all of our customers to build their ideal version of this. It’s a super fun thing, and it’s really great because it gets buy-in from a lot of people. They feel like they are a part of this thing.
Bret Scofield: After that, the researchers and the designers will aggregate a lot of the feedback, and they’ll put it together, and they’ll start actually working with the pixels. This looks a little washed out, but they’ll start building designs and prototypes. Once those are in a solid enough state, we’ll put those in front of our customers. This is actually one of our customers in Australia. He’s expressing disappointment, which is … that sometimes happens with research. He’s a little bit bummed about one of the things that we had sort of neglected to redesign. We were launching this redesign, and there were some areas where we’re like, “Oh, well de-prioritize those. It won’t be a big deal.” Then, we found out from a ton of our customers, “No, it actually is a really big deal, and they are really upset, they want us to redesign this.”
Bret Scofield: This changed the trajectory of the project. We stopped, and we said, “Okay. We actually do need to redesign these few screens and make this a cohesive polished experience for our customers.” Then, after something has been designed and released to all of our customers, we measure. We work very closely with the product management team and then with our engineers to instrument the log data so that we can see what people are doing with this new feature. Are they exiting Sumo Logic right after? Are they continuing with their workflow? What exactly is being done with this new thing?
Bret Scofield: This is great because it allows us to measure success, so if we can see a lot of engagement with this, we’re pretty happy. It also sort of allows us to jump off with more qualitative research. If we see, for instance, that everyone’s exiting Sumo Logic after using this new thing, then maybe we should delve into that. We should hear some stories. We should figure out what’s going on and make some adjustments. We’re using the Google HEART framework actually to drive which sort of metrics we’re tracking for this.
Bret Scofield: Yeah, to sum everything up, I just talked through how the customer feedback drives these two halves of the product development process at Sumo Logic, so we have the delivery process and the discovery process. Both of those are heavily influenced by the thoughts and feedback that come from our customers. Then, I’m going to hand this over to Riya for her perspective from engineering.
Riya Singh speaking at Sumo Logic Girl Geek Dinner.
Riya Singh: Thank you. Do I use this too? Okay, cool. Lights are bright. Hi. Hi, everyone. How’s everyone doing? My name is Riya. My name is Riya. I’m in the engineering development team. I’m a team lead of the data engine team. Quick show of hands, how many engineers out here? Software engineering. That’s one-third of the room maybe. Okay, cool. Feel free to connect with me on LinkedIn if you have more questions after my talk, and I’m happy to answer more questions.
Riya Singh: The theme of today’s talk is, how does customer feedback impact different product areas? I’m going to give you the engineering perspective or how we are listening to our customers. A little bit of background about myself, I joined Sumo about four years ago. I started as a software engineer. Today, I’m the team lead of data engine team. My team is about five people, and we mostly work on streaming systems at Sumo, so anything that requires real-time, low latency applications like live tail, live dashboards, alerting, and, so on.
Riya Singh: I’m also a dance fitness instructor, so my mornings are all work, work, work. Nights are all dance, dance, dance, so it’s been a lot of fun. Sometimes, I’d work. I love Sumo because I can combine both my passions. I do all kind of flash mobs at work and all those fun stuff at work, but the main reason I’ve been with Sumo for years is because I’ve seen many companies in which engineering is given a project and said, “Okay, these are your requirements. Go and build it.” Very few times, we try to understand why we are building it or if you’re building the right thing or why is this important? Sometimes, I feel engineering does not get that perspective. In Sumo, it’s ground up, right? As soon as you’re trying to decide what to build next, we’ll ask the right questions. The advantage that we have is that we are our customer zero, so we are using our product as much as anybody else. You kind of understand why this is important, and that perspective really makes you make good products.
Riya Singh: Let’s talk about customer focus development. Some things are very strategic, right? Sometimes, you want to build products ahead of the market, right? We want to build things that nobody else is building. We want to be innovative. We want to be strategic. There are a lot of projects, which come through deciding to be innovative, so that aside, there are other things that we do when we get customer feedback from various channels. We work very closely with Bret and her team, so as soon as she’s going to a customer and having her discussions, we are reading back the reports come back in, and we try to figure out, “Okay, where are the different things we’re developing incoming?” and build something to help that customer.
Riya Singh: Two very nice things that we have in Sumo, I’ll just describe them a little bit of detail. This is my kind of favorite place to hang out. It’s the ideas portal. We have a public ideas portal wherein customers can go and mark the ideas or features that they want to see in our product. It’s very nice because you can see where it was created, how many words are there, and what’s the progress. That’s a very nice way to find out what is important to our customers, and we can build the right thing. That gives us a lot of perspective.
Riya Singh: The other place, which I really, really like as well is we have our own public Sumo Dojo, a Slack channel where all of our customers or most of our customers are there. That is nice in a way because it’s very real-time, right? They’re asking a question, and sometimes, they’re helping themselves or we are answering some questions. It adds that human connection to the conversation when it’s real time, right? That is one very nice place where we get our ideas from. Yeah, so as I said, we talked to field. We have all these channels, and we try to get feedback and build the right things.
Riya Singh: Cool. Shifting gears a little bit, one thing in engineering for running a SaaS service is very important is that we can’t just build and forget about it. The system has to run. It has to run 24/7. It has to run at 100% availability, which is easier said than done. A lot of elbow grease, which goes on in trying to make sure that the service scales, and it works under different load conditions. Engineering realizes that it is very important that the current service work, right, so we spend a lot of time in trying to make sure that our service are reliable. Some ways how we do it, we use our own product to monitor our own services, so this is an example of an outage dashboard. We’re trying to monitor different lag latencies. There is a group called the IRC, incident response coordinator, so in case our metric here does not look good, we declare it as an outage. All the teams are involved in trying to resolve it as soon as possible.
Riya Singh: When I joined Sumo, we were about 80 people. We had maybe 10 micro services. Today, we have 500+ and I think more than 50+ micro services. Being able to monitor and run the service at scale is a challenge, and we have learned so much through just trying to run this well. We have learned that things that work when you’re a small company does not scale when you’re a bigger company. You have to automate as much as possible. Anytime you’re adding more humans to it, there is more chances of errors coming through, so the human side of scaling does not translate to the skill. We spend a lot of time in trying to automate things so that if it in case something goes down, we can recover from it as soon as possible without human interaction.
Riya Singh: Despite all the works, sometimes outages happen, and you move on, but one thing very nice about Sumo and its culture is that our focus is very clear. We are one with our customers, so in case an outage happens, we all gather together in an outage war room and try to help each other to resolve it as soon as possible. You’ll see people leaving the meetings, leaving any presentation and coming and helping. It was very nice for me to see when I joined Sumo that everybody’s so helpful here. If an outage happens, we do post mortems within a day. At this point, any new feature development stops, and we try to work through the action items that have come through the outage post mortem.
Riya Singh: Our policy is no repeat offenders, so in case we find the root cause and it has happened once, we will find the ways to ask the five whys and fix it, but you want to make sure it doesn’t happen again, so no repeat offenders is our metric to see how well we are doing.
Riya Singh: Okay. Talking about outages kind of makes my head hurt too because I’ve been to, well, more than a couple of them, but we’re getting better.
Riya Singh: Let me just shift things a little bit and talk about this newest initiative that I’ve been leading at Sumo. I’m leading a new team at Sumo called the Quick Wins team. As you know, we have grown so fast that today, we have 50,000 users and 1,600 customers, right? They’re very engaged customers, so they’re constantly asking us for new things that they want to see in the product. Sometimes, a team is not tasked to do those small things because they’re already working on the next big initiative, right? These smaller things or UX feedback somehow doesn’t get prioritized. I’m sure you guys have noticed that too in your companies that the smaller things are sometimes skipping through.
Riya Singh: We started a new team called the Quick Wins team, and a secret sauce is the sriracha because a little goes a long way, and so we’re trying to make all these small, small changes, which makes our product better over time. It’s a cross-functional team, so we have … It’s a small team, one UI, one UX, one backend, one docs, one PM. One nice thing about this is we are growth hackers.
Riya Singh: We are not tied to any particular section of the product, so it’s not like somebody who knows search cannot do dashboards or somebody who knows dashboards cannot look at collection. We are training people and empowering them to be able to make progress in any part of the product, removing any kind of silos that exist.
Riya Singh: We’re trying to make the product easier to use for Andre, for Melinda, and all the personas that we have. The aim is to improve the NPS, which is net promoter score, which is a metric showing how much our customers like our product. That’s it for mine. Stacy shall talk about learning.
Stacy Kornluebke speaking at Sumo Logic Girl Geek Dinner.
Stacy Kornluebke: Hi. I’m Stacy Kornluebke in case any of you here looked out at the agenda and wondered how to pronounce that. It’s a frequent question. I am the manager of training and documentation at Sumo Logic, which if it’s a lot for you to say, we just kind of reduced it down to the learn team. I am also the mother of three wonderful boys. I have replaced my couch three times in case anyone’s wondering. People say to me, “Wow, that must be a lot of work.” I say, “No, it’s a lot of fun, but I do miss my furniture,” so I am an avid fan of the arts. I am kind of infamous for dragging my friends two and a half miles down Manhattan to see Gustav Klimt’s The Woman in Gold. If you don’t feel like making that trek, you can just watch a very wonderful movie with Ryan Reynolds, and you’ll see the painting a lot.
Stacy Kornluebke: I’m also a voracious reader. I have 80 out of my goal for 100 books this year. If you find me later for networking opportunities, do not ask me what I’ve read. Once you reach this level of quantity, there’s very little quality. I like a lot of steampunk and vampire and werewolf books. It gets me through the night, but in all honesty, what I really like to do is make users confident. My whole goal is to give them the materials they need so that they feel that they are able to use the product the way they want to.
Stacy Kornluebke: People often ask me what’s special about Sumo. I did leave Cisco to come here, and I did it for a number of wonderful reasons. One of the top reasons is that we have very technical and sophisticated users. These people do a lot with their day, and they’re pretty knowledgeable to begin with, so they’re fun to write for. They’ve got a good background. They’re also, interestingly enough, highly collaborative.
Stacy Kornluebke: One of the things that I don’t think we’d addressed enough is how cool Sumo users really are. I watched them in a training class helping each other, and I don’t mean people from their own company. They were answering questions for people for other companies. They’re very proud to know the product, and they share it well.
Stacy Kornluebke: I happen to be a member of the customer service team, and it’s a really nice place to be, but everyone at the company has a huge drive towards making customers successful. I can go to Bret frequently and ask how things would look or what’s going to happen or what’s the next design. I can often come to Riya and say, “Hey, what’s the Quick Wins team doing this week?” so I never feel shut out of what we’re doing, and I feel that it’s always highly based on customer feedback. They work very closely with them. Jen’s giving me the face, but I can always come to her for compliance and security reasons. She’s really a very great resource if you need to understand GDPR.
Stacy Kornluebke: Sumo Logic also has a great sense of fun. What you’re looking at, in case you’re worried, is a suitcase full of 70 Sumos. The squishies that you’re going to receive tonight are extremely popular with our students. We didn’t get them shipped in time, so in addition to other interesting things I have done with my life, I took 70 Sumos in a bag through JFK, and TSA did not ask me one question.
Stacy Kornluebke: Think the last thing that I want to talk a little bit more about is that when I came here, we had this great phrase about how Sumo Logic was going to be democratizing data. At first, I was trying to imagine, what did that really look like? The more I learned about the product, the more people shared information with me, the more I understood the importance of having information at your fingertips. A lot of information usually gets siloed off, and in my world, when I get siloed off, I don’t really know as much about the customer as I think I do. My whole goal is to make sure that they learn about the product.
Stacy Kornluebke: All right, so if everything’s perfect, and you came here, and you loved it, what’s the big challenge with Sumo? What have you been doing for the last 18 months? Christian’s giving me the face, so I think he’s wondering too. To be perfectly honest, there’s a lot of challenge in the world of learning. The world has kind of gone from a directed approach. “Hey, I’m going to tell you what to do. These are the five steps to use this toaster,” to an open approach.
Stacy Kornluebke: There’s Stack Overflow, there’s GitHub, there’s just Googling it, right? Everywhere you go, you try and find the answer to your question. If you want people to use your product correctly, you need to make sure that you’re answering their questions because if you’re just telling them how you think it’s being used, they’re going to stop coming to you, and they’re going to go someplace where they can find the answer. That may not be a good answer, and it may not be a professional answer, but it’s giving them what they need. Users absolutely have the right to expect that you’re going to help them accomplish the task they want with your product.
Stacy Kornluebke: What do you know about your users? The first thing to kind of try and answer their questions involves understanding your users, and so, we’ve talked to you through two presentations about Melindas and Andres. We’ve talked to you from a design perspective and from a development perspective, but from my perspective, they need to know about the product. Melinda’s on it. She comes to Illuminate. I have a million Melindas. They tell me that the docs are like the Bible. They love it. They go into it every day. That’s because they use the product every day. Our product has a query language, and for her, it’s a second language. She knows what to type, what to find, where to go.
Stacy Kornluebke: Andre is a less frequent user of our product. I don’t know if you’ve ever tried to speak a foreign language that you only use once a month, but as you can imagine, trying to look things up, hoping that you got the right translation, the right inflection, that you understand the subtleties of the language, it’s really not going to happen if you pull out one of these dictionaries. It’s also not going to work quite so well with Google Translate, so you want to be able to get advice and understand what do these terms actually mean and how can I accomplish my task.
Stacy Kornluebke: If you’re trying to understand how to grow your knowledge in any part of your product, use the data you have already. Usually, Google Analytics is quite popular. Everybody uses it. No one knows what to do with the data. I suggest that you get the data, and understand a few things about what you want people to do with that information. For us, we were extremely excited to have people coming in, but then, we had to spend a lot of time figuring out, are they spending enough time on page views, do we want them to leave this page and find other pages, or do we want them to exit? Google will tell you what people are doing. You need to understand whether those behaviors are helping or hurting them.
Stacy Kornluebke: We also were very fortunate to have marketing reach out and give us information on SEO because a high number of our users Google. We needed to know whether or not we were obeying search algorithm rules. We also use net promoter scores. I don’t know … How many people are using NPS? Does anyone want to … Okay, so I see we’re talking to maybe 10 people. All right, so, the way NPS works is that it’s a ten … Excellent. No. You’re using my argument against me. I like it. I frequently do that to other people in the audience, so I deserved it.
Stacy Kornluebke: Net promoter score. On a basis of one to 10, a nine and a 10, those people are going to recommend you. Anyone below a nine isn’t as thrilled about you as you think they are. While they’re using you, they’re not excited about you. Hopefully, when they give you that rating, they’re going to tell you something. If they say, “Great product, but the docs suck,” I am the one who will find you and say, “Hey, what can we do to make it better?” but in these scores, that information does give you a sense of whether or not they’re truly satisfied. If only a nine or 10 matters, then you need to step it up and make sure you’re not making a seven or an eight.
Stacy Kornluebke: We also have this cool direct feedback mechanism in the docs, so down here where you see, was this article helpful? Yes, no. Leave feedback. I know that doesn’t seem that revolutionary, but we get feedback twice a week to our information based on this from customers. That’s a high level of participation. People come in. They feel very comfortable. They can say yes or no, but frequently, they can say, “Hey, I was expecting it to do this.” So long as you’re responsive to that, people will come back to you.
Stacy Kornluebke: We were also fortunate because we’d spend so much time making data visualizations. We could use our own tools, which was pretty awesome and a great kind of group project, and then again, I have to thank the UX team for giving us their customer journey because we could see the pain points people had with the product and try and align them to whether or not they were having a problem reaching information, training, documentation. Was it just an education issue, or was it something that was more of a product problem, and we were not going to get doc around it? It was going to have to be fixed.
Stacy Kornluebke: I speak a lot to search because 60% of our users and specifically, our struggling users, are coming to us for this reason, but if you are trying to figure out how to reach more users, find how they find you. If that’s through your Twitter feed, if that’s through your Slack channel, make sure that you know how your most struggling users are coming to you because if you aren’t speaking their language and you’re attempting to force them to use another tool, good luck. I don’t think anybody wants to learn one more thing. They want to find it the way they normally find it.
Stacy Kornluebke: Also, I recommend using your own tools. I know I’ve said it. I know you’ve seen all of us pull a dashboard showing how we use our product with our data, but what this allowed us to do was to take our certifications, take our data, and show it to sales. We pulled in customer success to make a meaningful dashboard. It became a group project. By involving more of the company, you’re going to get more people starting to understand what customers are struggling with and how they can help them. Frequently, our salespeople come in and they see who is certified and who isn’t and whether that makes a difference in whether people buy or renew. A certification is a great way to learn about our product, and it’s one indicator of whether or not people understand and feel confident with it.
Stacy Kornluebke: Okay, so how do I get answers to them? Well, everyone’s going to have to come up with a different way of doing this. For us, we found that a number of users desperately wanted some way to interact with a human being. They wanted immediate answers, and whether that was a Slack channel or that was an in-person training class, they were happy. Just bring me somebody who can answer my questions. The first thing we did with certification and training was we started holding what we called Cert Jams. We took the user classes that we hold at Illuminate. We came to cities where there were at least two, 300 Sumo Logic users. We said, “Hey, there’s a free class if you come today. Come in. Learn. Get a t-shirt.” Jane and Kerry were helping me out today, so somewhere around here, they’re wearing their t-shirts. Yes, so that, believe it or not, was a big thing for people.
Stacy Kornluebke: Quick answers to questions online as I’ve mentioned before. I know SEO is kind of this dirty word in the engineering world. It’s what marketing people do, and we shouldn’t touch it, but if you don’t, you don’t get the results you want. Then, just quick answers in real time, so if you can’t talk to a person, if you’re out on Anchorage and you’re using our product, you can still get on Slack. Anyone can find and ask a question.
Stacy Kornluebke: Here’s a picture of our Cert Jams. This is kind of cool for people. They’re really happy. We call this the t-shirt shot at the end. When they get their certification, they get their shirt. For every city that we go to, we try and take at least one of these. We’re doing at least 25 this year, and we hope to do 30 next year, but this is really a great networking opportunity for anybody who uses our product, and it’s a chance for people in our company to kind of get to know our users in a non-sales context.
Stacy Kornluebke: These are all the mistakes that we made with SEO. Please don’t make our mistakes. Please make new ones.
Stacy Kornluebke: We, first of all, did not know how search engines worked, so we had to use the SEM rush report to understand things like the algorithm will punish you for dashes and for underscores instead of dashes. Good to know. Our tool was automatically generating underscores all the time for us to take out those spaces, so we had to change the default setting, and magically, our documentation was popping to the top for our search results. Good to know.
Stacy Kornluebke: Use their terms. One of the things that I also find with learning materials, with documentation is we frequently try and force people into an exact definition of exactly what we’re doing, which is great, but it’s not the term people are searching for. A classic example of this is we talked very specifically to two-factor authentication because there were two factors when a lot of people were looking for MFA. We do not have the right to try and teach them the difference between multi-factor and two-factor. We can take them to the two-factor page, and then, they can see the difference between that and multi-factor, but please just get them to the information that they want. Use the term they use even if it’s a little hacky. They want to find their stuff.
Stacy Kornluebke: Then, the last bit is please avoid stubs. I’m just going to give you this bit of an SEO advice. If it’s under 200 words, your search engine hates you. I know there’s this great trend to try and bring down documentation to short little stubs, easy-to-read pages. Please bind up all those little stubs into a single page. You user will thank you because they don’t want to click through 60 pages, and your search engine will thank you because then, it will think it’s a real document. You can also make your own SEO mistakes. I recommend it. Keep having fun, but get people what they need.
Stacy Kornluebke: We also created an in-product learn tab. What we discovered was that people just want to go to the product, find what they need to do to learn, and go through stuff. Well, that’s great, but I have a 30-video library, and there’s no easy way to stick that all in the product. What we did was we had people from the UX team help us out with a design, and engineering helped us implement just kind of a short five-video series. We used APIs from our knowledge base to pull in the tutorials, and then, we provided quick links to other parts of the product. Now, this is really useful to a lot of people. They come in. They watch one quick video. They review the tutorials, and they’re done. It’s not a long-term thing. It’s a help in onboarding, but it made a big difference.
Stacy Kornluebke: You also sometimes have to accept change. As wedded as I am to SEO and I like forums, and I want information to be found, and we’re going to be the next Stack Overflow, a community where you have a timed response where people have SLAs and they’ll get back to you even within a day is not as cool as it used to be. People like Slack. Slack offers real-time response to questions. It makes people happy. They’re not really concerned if the next person can Google it. They’re getting the answer they need. Being where your users are is kind of what you need to provide.
Stacy Kornluebke: Also, just remember this is a process, so use any data that you have, CSAT, NPS, Google Analytics. Whatever you’ve got, start trying to understand your users. Please work with any group that you have that has similar customer based mindsets, so if you’ve got a UX team, if you’ve got a customer success team, if you’ve got a field full of salespeople that desperately want to help customers, reach out to them. Don’t hesitate to ask a couple of questions to do your job better.
Stacy Kornluebke: I also recommend that you join some grassroots movements. I like Write the Docs. It’s a bunch of people that take a more hacker-based approach to writing documentation, but it’s also just a user conference where people come who work with problems every day. We can talk about our struggles with trying to use analytics data. Talk to your users directly. I don’t know, depending upon your corporate culture, how open people are to that, but Sumo’s really never barred us from reaching out and saying, “Hey, how could we make things better?” if it’s just sitting silently on a call with someone who is working with the customer, that’s good too. Understand what people are really saying about what you’re offering and what you’re teaching about the product.
Stacy Kornluebke: I want to say thank you. I know I gave you a to-do list, but that’s kind of what I do. Now, we’re going to hand you over to Jen who’s going to talk to you about security and privacy.
