VIDEO
More than 100 Girl Geeks gathered at Zendesk HQ in San Francisco to hear Engineering and Product leaders share how they build technology that champions the customer service experience.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
- See open jobs at Zendesk and check out open jobs at our trusted partner companies.
- Does your company want to sponsor a Girl Geek Dinner in 2021? Talk to us!
Girl Geek X team: Gretchen DeKnikker and Sukrutha Bhadouria welcome the crowd to Zendesk Girl Geek Dinner in San Francisco, California.
Speakers:
Shawna Wolverton / SVP, Product Management / Zendesk
Swati Krishnan / Software Engineer / Zendesk
Erin McKeown / Director, Engineering Risk Management / Zendesk
Alethea Power / Staff Software Engineer, Site Reliability / Zendesk
Eleanor Stribling / Group Product Manager / Zendesk
Sukrutha Bhadouria / CTO & Co-Founder / Girl Geek X
Gretchen DeKnikker / COO / Girl Geek X
Transcript of Zendesk Girl Geek Dinner – Lightning Talks & Panel:
Sukrutha Bhadouria: Welcome to the Zendesk sponsored Girl Geek dinner tonight. I’m Sukrutha. This is Gretchen. Thanks for joining us. I love all the color around. I love your hair, lovely lady. Anyway, a little bit … you also, all of you. I quickly want to recap what Girl Geek X is. So why you see that up there, Girl Geek X is an organization with Angie, Gretchen, and I working to make it easier for women and people who identify as women or anything you want to identify yourself as, anyone, to come and network outside of work, find out more about other companies that have a great culture and have really, really innovative products, such as Zendesk. At dinners like these, you have the first and the third hour reserved for networking, so I hope you’ve been chatting away and making connections so when you actually want to work at the company or apply there, it makes it easier. It’s like you have inside information.
Sukrutha Bhadouria: Zendesk has sponsored a few times before so they’ve been such a great ally and with Shawna working here now. We used to work together before. Not directly but in my head we did work directly. So when she reached out to us we were super excited that we’d have another Zendesk dinner coming up. Today, we do not just dinners. Once we hit the 10 year mark with Girl Geek X, we started doing virtual conferences, which we’ve had two so far. We also have a podcast so search for Girl Geek X and we’re looking for more ideas on topics so listen to what we have and suggest topics. Sign up for our mailing list through our website, girlgeek.io. We also launched our swag store today so–
Gretchen DeKnikker: Did you guys see it?
Sukrutha Bhadouria: You did?
Gretchen DeKnikker: That’s so cute. It’s one of these little guys.
Sukrutha Bhadouria: Yeah, so Gretchen’s nicknamed those characters pixies because they’re pixelated.
Gretchen DeKnikker: It’s a great name so it’s not just that I nicknamed [inaudible].
Sukrutha Bhadouria: So please share on social media tonight everything that you see here, eat, listen to, learn. The hashtag for tonight is Girl Geek X Zendesk and that’s enough from me. This is Gretchen, like I said.
Gretchen DeKnikker: Okay, Sukrutha said everything we always say except for, please join me in welcoming the amazing Shawna Wolverton.
Shawna Wolverton: Thank you. I am just so incredibly impressed with what these women have built over 10 years. I was looking at their site today. Over 200 dinners. This is an amazing organization and we’re incredibly honored to host tonight. A little food, a little networking and hopefully, maybe, you’ll even learn a few things. We have an agenda because this is what we do at Zendesk. Everything starts with an agenda. We have checked in. That’s good. We ate. We all successfully avoided the caution tape, so it’s a classy establishment here at Zendesk. And we’re going to do some lightning talks and then stick around, we’re going to do a big group picture and then, there’s a whole other hour after we talk at you for a while with dessert and some more chatting.
SVP of Product Management Shawna Wolverton emcees Zendesk Girl Geek Dinner, stating “it’s amazing to get a seat at the table and to look around and see people who look like you.”
Shawna Wolverton: I am Shawna Wolverton. I am the SVP of Product here at Zendesk. I joined about six months ago and it has been an amazing six months. I feel sort of corny a lot. People ask me all the time, how’s the new job? How’s the new job? I feel like a little bit of a Hallmark card. Like it’s so great. But what’s been amazing and I really just sort of figured this out is that I was trusted right out of the gate, right? I was able to go out and be competent. At week 2, I was on stage. In month 3, I’m in front of investors and in front of the press. And it’s just been so amazing to be given that trust. And I was incredibly lucky to join Zendesk in a cohort of women executives. We hired a CIO, as well as our chief customer officer, onto the executive board when I joined and then I looked up and we already had women in our CFO seats as well as in the head of people.
Shawna Wolverton: And it’s amazing to get a seat at the table and to look around and see people who look like you. So you did not, though, come here to listen to me jabber around on about how much I love my job. But we have four incredibly accomplished speakers tonight and we’re going to start with Swati who’s going to talk to us about metaprogramming. At the end, we’ll have some time for Q&A so definitely stick around for that. Swati.
Software Engineer Swati Krishnan gives a talk on “Code that writes code: Metaprogramming at Zendesk” at Zendesk Girl Geek Dinner.
Swati Krishnan: Hi, everyone. Hi, everyone. I’m Swati. And today I’m going to be talking about code that writes code or metaprogramming here at Zendesk. So first, a little bit about me. I’ve been a software engineer in the code services organization at Zendesk for around two and a half years now. In this time, I have learned and contributed to many projects. But of the cooler and fun things that I got to learn here was metaprogramming and I hope that I can share it with all of you out here.
Swati Krishnan: So first of all, what do you mean by metaprogramming? Well, most programs are built on language constructs. These language constructs could be classes, methods, objects, et cera. Metaprogramming, basically, allows you to manipulate these language constructs at run-time. So why is Ruby as a language particularly suited to metaprogramming? Well, that’s because Ruby’s a dynamically typed language. What this means is that it allows you to access and manipulate these language constructs at run-time. This is a difference from what statically typed languages would let you do normally.
Swati Krishnan: So how do we leverage metaprogramming here at Zendesk? So Zendesk is like a Rails shop. So this basically means that we have a lot of products and apps that are built on Rails. For those of you that don’t know, Rails is a Ruby based web framework. Web frameworks have to be pretty flexible so this means that a lot of modules and libraries in Rails such as Active Support, Active Record, et cera, heavily leverage metaprogramming. So by using Rails, Zendesk by proxy, uses a lot of heavy lifting that comes from metaprogramming.
Swati Krishnan: This talk is not going to be about rails. This talk is about account feature flags at Zendesk and how we use a bit of metaprogramming magic to add some more fun and color to them. So before I launch into that, what exactly do you mean by account feature flags? Here at Zendesk, when a developer ship new code, we do so behind … I don’t know why this is so … okay. So whenever developers … that’s better. I’m just going to leave it. So here at Zendesk, whenever developers ship new code, we do that behind something called feature flags. So whenever a feature flag is down to 0%, that basically means that that feature is not available on any accounts. Is this fine? Are you sure? Okay. When it’s done to 100%, that means that it’s available on all accounts.
Swati Krishnan: So this basically gives you mechanism to roll out a feature slowly so it can go from 0 to 100% and you could also roll it back quickly so that if things go wrong, if there’s a bug in the code or if customers aren’t really appreciative of the features–which doesn’t happen. It doesn’t happen, but there’s a slight possibility so you should always [inaudible]. So basically this feature flag framework allows developers to ship code in a more reliable way.
Swati Krishnan: So the way that this is built, it basically means that developers now have a method called has feature name question mark available on the account object. So that whenever they’re trying to ship this new feature, they can just basically go if your account object has … they can basically just go that if and say that if your account object has this part of the feature turned on, that means that we can now execute the new feature’s specific code. If it doesn’t have the feature turned on, that means that we can just fall back to our old non feature specific code or just execute old code.
Swati Krishnan: So how can we simplify the existing account class structure so that we can basically add this feature so that developers can by proxy enable new feature specific code? So one way to do that would be basically to just open the account class to add your has whatever your feature name is, question mark, method inside that. All that this method would be doing would just be checking the database to see if the feature flag is turned on for the specific account or not.
Swati Krishnan: So when a developer has a new feature to add, what they’ll basically do is just go into this account class, add a method called def has feature XYZ question mark and it’ll do the exact same thing, which means that it’ll basically call the database and check if the feature flag is turned on for the account.
Swati Krishnan: But there are also several problems with this sort of an approach and you should not be doing this. And that’s because it encourages repetition a lot so whenever a developer wants to add their own feature, they’d basically be like going to this class, adding their method, making that database call to check if the feature flag is turned on. In coding and in Ruby in general, we try to discourage repetition, because if there’s a way to get something done with as few lines of code and concisely as possible, then it should definitely be trying to use that.
Swati Krishnan: The other kind of obvious disadvantages that means that every developer, whenever they want to write this particular has their own feature method on an account object. But how could I get [inaudible] implementation off fetching from the database so this is just encouraging reinventing the wheel, which is something that we don’t want developers to do because that’ll just add potential for more bugs. Because if everyone gets to write their own implementation, then you can have more bugs pop up from that.
Swati Krishnan: And last but not least, why should we do it in this brute force driven way when metaprogramming gives you more cleaner, elegant ways to solve the same problem? So the metaprogramming solution to this is basically just adding list of features. So over here I added a couple of features, but you can increase this with how many ever features you want. And then in these six magical lines, we’ll just be iterating over this features list. And we’ll be calling the Ruby metaprogramming magical method … actually the Ruby magical dynamic generation spell which is basically just going to define a new method based on that item that it’s picked from the list. It’ll just interpolate that in the method name and then, voila. I don’t know if I said that correctly. And then you basically get a method which will then make that database call with what it’s picked up from the features list.
Swati Krishnan: So this basically means that all that a developer now has to do to get the has feature available method is to just add their feature name to this features list and then whenever the Ruby app will boot up and start, it’ll automatically create the has their feature name available method on the account object so they don’t have to write their own implementation. They don’t have to repeat themselves. They don’t have to do anything much.
Swati Krishnan: So this was just one of the benefits and applications of metaprogramming. There are several others, such as the open class implementation, which will basically let you add your own functionality over any method in the class. So you basically even go and open up like the [inaudible] method in the ink class which is a code Ruby library class. And you can add your own functionality, like logging or benchmarking, to it. Another kind of interesting one would be the Active Record library in Rails. So Active Record for those of that don’t know is object relation and mapping. So basically if you have something like user dot name in your code or user dot name equal to Swati in your code, Active Record will magically figure out that this call response to the user’s table in your database. If that user’s table has a column called name, then it’ll automatically create the [inaudible] methods for you so user dot name and user dot name equal to will already be created for you so you don’t have to define it yourself.
Swati Krishnan: So yeah. These [inaudible] the applications of metaprogramming. This is just a glimpse of all that it can do. But I hope that you found this informative and will probably use it in your own work. Thank you.
Director of Engineering Risk Management Erin McKeown gives a talk on “Staying Cool Under Pressure – Lessons from Incident Management” at Zendesk Girl Geek Dinner.
Erin McKeown: Hello, everybody. My name is Erin McKeown. I just want to first say welcome. I’m so excited that Zendesk is hosting this event. I’m even more excited to share with you guys a couple of lessons I’ve learned through managing incidents throughout my career. To quickly introduce myself, like I said, my name is Erin McKeown. I’m the director of engineering risk management here at Zendesk. I have the great pleasure of leading a team of–a group of teams, actually, that I like to think of in three different categories, which is really our first line of defense, threat prevention, and recovery. When I say the first line of defense, we have what we call our Zendesk Network Operation Center. We actually have Kim Smith here with us who leads the ZNOC. Hello, Kim. Everybody say hi. She’s visiting from Madison. So Kim has the pleasure of running a very, very awesome team that takes–monitors and takes care of our systems 24/7 365 a year. Like I said, they do all kinds of monitoring. They put out fires. They escalate to different engineering teams when there’s something that is a little bit larger that they need help with.
Erin McKeown: In addition to our ZNOC, we also have our incident management team. They partner very closely with our ZNOC, and they’re responsible for running all of our response and coordination of any service incident that we have here at Zendesk. On the other side of that, we have our business continuity and disaster recovery. These are really the areas of which we focus on planning for training employees and testing on how we can recover from business disruptions. So that can be anything from a natural disaster that impacts one of our office facilities to a natural disaster that may actually take out an entire AWS region. Everyone cross your fingers that that does not happen.
Erin McKeown: So this is one of my favorite quotes. It’s a little nerdy, but Franklin Roosevelt said, “A smooth sea never made a skilled sailor.” Disclaimer, I’m not a sailor, but stick with me here. I think what Frank is trying to get at here is that no matter what, there’s always going to be challenges that come up and we are going to have to deal with adversity and we can plan and we can do all kinds of things to get prepared for events to take place but at the same time, we need to take these as an opportunity to continue to learn and to grow. And so, I’m just going to share with you guys two events that I’ve actually been a part of and two important lessons I’ve learned from them. I really wanted to dig in and give you guys a real technical incident type of conversation but I didn’t want you to fall asleep.
Erin McKeown: So the first event that I’m going to talk about is from 2011. Back in 2011, there was a 9.1 earthquake off the coast of Japan and it actually was a mega underwater earthquake that took place. As a result of that, there was a tsunami that then hit a nuclear power plant and caused a meltdown of the Fukushima power plant. This is considered the second biggest radioactive event accident to have happened to Chernobyl. I don’t know if you guys are watching the HBO series, but kind of along those lines.
Erin McKeown: So a very, very devastating event. We actually had an office in Tokyo with 250 employees on the 50th floor of a high rise building when the earthquake happened. You can imagine how scary that would really be. That was the first wave of it. And then the tsunami hit and there was devastation across the entire the eastern side of Japan. And then this huge threat of radioactivity that was potentially threatening Tokyo. These employees went through the ringer. I mean it took us about a week and a half to confirm where all of our employees were, make sure that they were safe, make sure that their families were safe, that they had what they needed. All the work that was going on in Tokyo completely stopped. It’s fine. There was other people that picked it up and things to do.
Erin McKeown: I think that what we learn from this type of an event is no matter what, people are our most important asset. As a company, you consider it family and I think one of the challenges that companies do have is really understanding that line between responsibility and just doing the right thing for their employees. In this particular event, we actually considered chartering planes to get our employees out. We didn’t have to do that because it turned out everything was going to be okay, but, yeah. So bottom line from this experience, to highlight that despite the fact that the office was unoperationable for weeks at that point, everything was fine business wise. All we cared about was the employees being safe and their families being safe.
Erin McKeown: So this is another event that took place in 2012. Hurricane Sandy. It actually impacted the eastern seaboard, caused over an estimated $70,000,000,000 dollars worth of damage. Another very, very human wise devastating event. I’m not going to talk about that one. Part of this in this one, a startup that became … well, I wasn’t working there but partnered with them on some things. I wasn’t responsible for their DR. They actually had their data center in downtown Manhattan. I don’t know if you guys know that there’s data centers in downtown Manhattan but from a risk standpoint, I would not be having my data center downtown Manhattan.
Erin McKeown: Anyway, they completely lost power. They lost backup generator power. They didn’t have a disaster recovery plan. They didn’t have their data backed up. So they were pretty much dead in the water. They had to sit there and wait and see if everything would come back or if it wouldn’t. So the big lesson from them here is luckily, the services came back. They were able to continue their operations, but they quickly implemented a disaster recovery and backup data … sorry. Data backup policy.
Erin McKeown: So I think one of the things from this experience is really understanding again, first and foremost, the people aspect is the most important, but when you start thinking on the business side of things. Especially for a company like Zendesk, that our data is our bread and butter, that’s where you want to be putting some focus and making sure that you’re considering that and making plans for it. So yep. Just kind of the takeaway from that is we do consider people. Again, I think about it from a Zendesk standpoint because like Shawna, I absolutely love it here. I’ve been here for four years. They’re going to have to drag me out kicking and screaming. Again, I do believe that we’re a company that … you know, people first. We do believe that also our data’s pretty important too. So thank you so much.
Staff Software Engineer, Site Reliability, Alethea Power gives a talk on “Computer, Heal Thyself: Automating Oncall, So You Can Sleep Through It” at Zendesk Girl Geek Dinner.
Alethea Power: Hi. This is my talk. Computer heal thyself: automating oncall. So you can sleep through it. My name is Alethea Power. I’ve worked in auto-remediation, which is what I’m going to cover, and I’ll explain that term in a minute. I’ve worked in auto-remediation for about 10 years. I built one of the world’s first and largest auto-remediation services. And now I’m building an auto-remediation service in conjunction with Kim and the ZNOC team here at Zendesk.
Alethea Power: So what is the purpose of auto-remediation? Well, tech companies have been finding through the dev ops revolution, not revelation. I mean I guess it’s kind of a revelation. Over the past number of years, that they can get better product quality, faster product development velocity, and higher service reliability if they give product engineering teams both the responsibility and the authority to manage the full life cycle of the software that they’re writing. So that means not just writing code but the engineers who write the code also push the code out to production. They operate the code in production. And they respond when there are outages in production.
Alethea Power: So this causes a virtuous tight loop. The engineers who are writing the code are best equipped to solve problems when they occur and when those problems occur, it gives those engineers a lot of extremely useful information about how to change that code or repair it. So quality goes up, speed goes up, et cera, et cera. But this introduces a whole new set of responsibilities for software engineers that they have not traditionally had to take care of which means we have to provide them with tools to make these jobs easier so that they can focus on the part they understand and not have to worry about lots of things that distract them from the focus of the code.
Alethea Power: So auto-remediation is meant to be a tool to help address with your mediation of outages. And I’m not talking about the Fukishimas of the world. I’m talking about much more frequent outages. The kind that happen 20 times a day. The kind that happen at 4 A.M. and at 4:45 and at 6 and at 5:30 A.M. So what does this look like in practice? Traditionally, you have a monitoring system that detects when you have outages in your infrastructure with your services. That monitoring system gives alerts to engineers. Now this could be in the form of engineers sitting in front of a dashboard of alerts 24/7 watching it. It can be in the form of alerts paging engineers in the middle of the night and waking them up. Yeah, et cera, et cera.
Alethea Power: And then engineers take their own knowledge and documentation recorded in what’s frequently called runbooks to execute various commands in the production environment to try and solve these problems. So these commands can be things like, if you have an application that’s wedged, maybe you’ll restart it. If you have a hard drive that’s full and maybe it’s full because there’s a bunch of errors spewing into a log. Then maybe you truncate that log. If you’re in the middle of being attacked. If you’re in the middle of a DDoS attack. Maybe you changed some routing rules to black hole incoming requests.
Alethea Power: So these are the kinds of things I’m talking about. So in auto-remediation service replaces these two components. The engineer gets replaced with a service and the runbooks get replaced with remediation code. So instead of having human readable documentation about what to do, you have blocks of code. And the auto-remediation service goes and executes this code in response to alerts in the monitoring system. And then engineers can sleep through the night. Their talents are better used for, for instance, instead of waking up to restart a service that has a memory leak, they can be well rested in the morning and figure out why it has a memory leak and fix that.
Alethea Power: And in general, we can take better advantage of the knowledge that we have across all of our engineers. The engineers that are being woken up and the engineers that are watching these dashboards. We’ve got a lot of really knowledgeable, talented, intelligent people. And we want them to be able to use their skills in the most sophisticated and interesting ways possible. So we’re trying to automate as much as we can.
Alethea Power: So I’m going to look at an example here. This is a configuration file for the auto-remediation service that we’re building. I tried to design the configuration language to be as simple as possible while also being flexible enough for what we’re trying to accomplish. So let’s walk through it. This file says if you have this issue on these hosts, then it should run this job in response but don’t run it more often than that. So specifically if the osquery agent is busted, web servers in us-west-1, then you want to run this block of remediation code but don’t do it more than five times per hour per cluster. Make sense?
Alethea Power: So let’s go look at this thing right here so we can understand how that looks. So we’re also building an SDK, mostly built by engineers on Kim’s team. And this SDK includes a lot of convenience objects and convenience methods so that the people writing remediations can focus just on the logic that they care about and they don’t have to worry about things like SSH authentication and properly rotating keys and how do they get authentication into AWS so they can reboot EC2 hosts and stuff like that. We abstract all that away for them and we do it in ways that make our security compliance team happy. Every remediation uses a different SSH key magically.
Alethea Power: So this remediation you can see in four lines of code. It could fix this problem. So let’s walk through these lines. First, you import our SDK so you get all of these convenience objects and methods. Then you subclass our remediation class and override the run method and inside of that, you get this convenience object. If it’s an alert on a host, you get self.host. The remediation doesn’t even have to know what host it’s working on. It can if it wants self.host.name. We’ll tell you a host name but you don’t have to. And you get this method, self.host.run, which magically does lots of SSH things in the background and can run this command to go restart that service.
Alethea Power: So it’s that straightforward. We’re trying to make it as simple as possible for our engineers. It’s pretty complicated on the backside. Here’s a pretty simplified picture of what the backside looks like. So, Swati, you did a magic thing with a dot. I don’t know how to do it so I’m just going to go point. So that thing, the alert mapper, pulls in alerts from PagerDuty. That’s who we use for monitoring or where we consolidate our alerts. And it runs those alerts through the configuration like the configuration files we were just seeing and calculates what remediation jobs to run, inserts those jobs into the database, and then that thing, the job launcher, pulls the jobs from the database, hands them as config files to Kubernetes and Kubernetes executes them inside of containers. We’re running them in containers because I’ve built this before and engineers make jobs that take 100 gigs of ram and all the CPU you can use so we don’t want any job to choke out the others. And lastly, since we have this nice infrastructure in place already with a beautiful SDK, we’re giving people the ability to launch proactive jobs using a CLI to do things like kernel upgrades and other stuff that’s not necessarily responding to alerts. All right. Thank you.
Group Product Manager Eleanor Stribling gives a talk on “ML in Support: Infusing a flagship product with innovative new features” at Zendesk Girl Geek Dinner.
Eleanor Stribling: Hi, everyone. My name’s Eleanor Stribling. I’m a group product manager here at Zendesk. What that means is I manage other product managers. And what I wanted to tell you about today was how we’re using machine learning in Support, our largest, oldest product. A little bit about me. I would also say that Zendesk is really the best place I’ve worked in in tech. I’ve been here a year. Before that, I’ve been in all kinds of companies ranging from a company that’s like a 100,000 people all the way to a teeny, tiny social impact startup and this experience overall has been just amazing. I work with obviously lots of really smart people, so definitely encourage you to explore this if it’s of interest to you.
Eleanor Stribling: One of the reasons I really like Zendesk and I like working on this product is … well, it’s not evil. But also, it really helps people do their jobs and do them well and that’s why I’m so excited about this particular project. Putting machine learning in Support, because like I said, this is the product that a lot of our customers use. Use it a lot. They’re in it everyday. And we want to help them do their jobs better and more efficiently. So machine learning is a great way to do that.
Eleanor Stribling: I want to do a little bit of clarification of terms. So when I say Support, I might mean something different than what you imagine it to mean. So most people when they say, most normal people who don’t work here, when they say support, they mean calling support like I need to call support because I’ve got a question. That kind of usage. What I’m going to talk about is the product Support. So Support, as I mentioned, is our oldest product. Until recently, it was Zendesk. And basically it’s a system for creating tickets or issues, moving them through a system, making sure the right people see them at the right time and then resolving them. And that’s kind of the core of what we offer. So it’s a really cool place to work because we have huge impact on a lot of users.
Eleanor Stribling: So a pause here and then zoom up a little bit. When you think of machine learning as part of customer service or customer support, what do you think of? What do you sort of imagine? Chances are, you imagine something like this. So this is one of our products. This is AnswerBot. And it is exactly what the name connotes. It is a bot that answers questions for people in chat. So in this example, you’re connecting to a chat. You’re asking some basic questions. AnswerBot looks at the text and predicts a response and then serves it to you. If the prediction is strong enough and if it doesn’t, as you can see right here, it’s going to escalate it to an agent. That went by really fast but trust me, that’s what it did.
Eleanor Stribling: So that’s usually how we think about customer support with ML, right? Bot answers your questions. I think this is a great product and it does lots of great things. Among them, it means that customers don’t always have to talk to a person. So I definitely have my moments. I think we all do when we really don’t want to talk to a person and in these circumstances, it’s great. But the problem with answer bots, generally, not just ours, is that people do want human connection. So it’s great for deflecting some issues but sometimes when you call support, you just want to talk to a person. How do I get to a person, you might scream into the void.
Eleanor Stribling: So really the question that we have now as a very customer centric company building a product that’s supposed to help you build relationships, is how do we help people inject that humanity that customers want, they want to experience. How do we help them do more of that? How can we help them be more efficient? And I think we started looking at machine learning as a way to do that in Support. This is also, I think, important kind of context. We do this really cool report every year about customer experience trends. So if you’re interested in customer experience, generally, if you’re a data person, definitely check this out because I think it gives you good perspective or if you want to apply for a job, just saying, it will give you really good perspective into the landscape.
Eleanor Stribling: So there’s a lot going on here but basically people expect answers fast. They want it on every channel that you have. They expect you to be on every channel. They really want you to be proactive but you’re probably not doing that so there’s a lot of pressure right now on these customer support organizations. So in this environment of I just want a person but I also want a person with all this other stuff, how do you manage that? So when we first looked at taking this approach of we got this giant product people know and love. It’s like where they spend their whole day in a lot of cases at work. We first started with the question, how can we use machine learning to help customers manage complexity. Because we are going up market. We’ve got more and more customers who have huge agent teams. Like about 40% of our annual revenue comes from customers that have over 100 agents. So these are not small companies. There’s a lot of complexities.
Eleanor Stribling: So we kind of started there, but then realized pretty quickly as a customer centric company that really, what we were asking is how can we use machine learning to make our customers even better at their jobs? And really even beyond that, how can we help them make their jobs less stressful? If you imagine being an agent or a manager of support agents or even an administrator of a system like this, there’s a lot riding on you. There’s a ton of stress. People are calling you stressed out, saying I’ve been trying to talk to a person for however long. It’s often not pleasant and so, I think, to make jobs for these folks easier is one of the reasons I joined Zendesk, because again, it’s something that’s actually improving people’s lives and it’s definitely not evil.
Eleanor Stribling: So what we wanted to do was figure out, how do we add little things to this so that it won’t blow you away, like the machines aren’t taking your job, but we’re giving you little tools to do everything that much better, that much faster. So again, being a customer centric company, we looked at the main groups of customers that use our product, which you see across the top there. Agents, managers, administrators. And then we thought about, for each of them, as you can see down the side there, what their goals are and then we thought about what we could use ML to do for them. How could we help them do their jobs with this rich set of data that we have for each customer?
Eleanor Stribling: So first of all, agents. So they really need to get happy customers. Like if you’re finally getting that touch of humanity in your support experience, you want your customer to leave happy, right? It satisfies them. It satisfies the customer. Everyone’s incentives are aligned. So the plan here is because agents are often working in complex environments, they can be very high turnover environments, we wanted to figure out a plan to–and what we’re working now, actually–is essentially crowdsourcing agent responses. So we can start suggesting next steps for people as they’re working on a ticket. And that’s really huge. Again, in somewhere that’s really fast paced, maybe you’re working on something you’re not familiar with, we’re kind of there to lend them a helping hand and help them be a little bit faster and more efficient and give people more relevant answers.
Eleanor Stribling: For managers, so managers are leading a team of agents and they really need these agents to be efficient and make people happy and they care about CSAP. Part of that is making sure you got the right number of people, the right people and the right number, in the right place to answer questions. So here we’re looking at grouping relevant data together. So for example, if you have a ticket that comes in and it’s one of a hundred tickets about the same topic, we want to surface that in a really clear and simple way for managers so they can respond effectively. Either by getting agents with the right skills. Maybe they figured out a response they want to communicate to their team. That kind of thing so that they can get on top of it. Another thing that we’re working on managers that I think will really help is predicting surges. So we can look at the agent staffing that they’ve had at any given time. Maybe it’s a time of it’s really busy like around Christmas for example or maybe it’s just every Wednesday. What do I need? The other thing we’re working on here is figuring out how to surface that intelligence so managers can do their job better so we’re giving them a little boost.
Eleanor Stribling: And then finally, administrators. So these are the folks that set up Zendesk and maintain Zendesk. And so their main thing is that no ticket, no issue kind of gets undealt with. And I think that there’s kind of a constant stress that they have that something will not be answered because they somehow messed up the settings. So the great thing about administrators from a data science perspective is they kindly label a lot of data for us. We don’t want them to stop doing that but what we can do is learn from how they label data for us. And what that means is we can help make sure that no ticket goes unanswered. That if they don’t assign something that makes sense, we can provide suggestions, updates for them, but also for managers in real time so that they can change the routing. So there’s a lot of really cool things we can do that would really have real time impact in small ways on our customers to, again, make their job better, make it easier and less stressful. And really, that’s one of the reasons I work in tech. Because I want people’s lives to be made better through it.
Eleanor Stribling: And finally, if you follow me on Medium or Twitter, you know I’ve got kind of this weird thing about Harry Potter and I had studied language in Harry Potter. But to me, this project is kind of like that. It’s like we’re taking something that’s everyday that people are used to staring at for hours on end and we’re adding little things that are unexpected and kind of cool. And so that’s why I think that this is such a great space to be in. Because we’re having like huge impact by making little and also extremely cool changes to the experience. We are also hiring in that team. Shameless plug. We’re hiring in that team for a data science engineer and a data scientist and I’m also hiring for a product manager, so if you’re interested in any of those, definitely come see me after. Thank you.
Shawna Wolverton: All right. Thank you to our amazing speakers. Why don’t you guys actually all come back up and we can do a little Q&A. I think there’s going to be some folks out with mics wandering around. Maybe. There you go. We don’t need all the mics. So we got about ten minutes for Q&A if anyone has questions about the talks or Zendesk or you know, we know a lot of things. Trust us. It’s fun. No? Careful, we’ll ask you … oh, great. Right … oh, you’re close but then we got one up here.
Zendesk girl geeks: Shawna Wolverton, Swati Krishnan, Erin McKeown, Alethea Power and Eleanor Stribling answer audience questions at Zendesk Girl Geek Dinner.
Audience Member: Hi. Alethea, I really enjoyed yours as someone who’s been woken up so many times from PagerDuty. Like God bless you. Can you talk more about the code behind what makes all that wonderful magic run?
Alethea Power: Yes, but there’s so much of it. Maybe it’s better to go into details after the Q&A?
Audience Member: I will find you. Thank you.
Shawna Wolverton: I have a feeling–
Alethea Power: I guess I could give you like a 30 second. It’s all written in Python. We use Aurora on the backside for the database. Like I said, we put containers into Kubernetes. I don’t know. That’s a very quick, quick, quick. It looked like you frowned when I said Python.
Audience Member: Oh no.
Alethea Power: Okay, so don’t find me afterwards. No, no, seriously. Totally come ask.
Audience Member Thank you.
Shawna Wolverton: Heard one up here.
Audience Member Hi. This is a question for Swati. You mentioned metaprogramming and I’m actually really interested in dynamic programming languages, such as Python, but you mentioned you mostly work with Ruby. So I was just curious if you ever worked with other languages, such as Python, for instance?
Shawna Wolverton: Lovers and haters of Python.
Swati Krishnan: Thanks for the question. My internship project here was in Python. So yes, I’ve worked with Python before. That was dealing with, I don’t know if you’ve heard about [inaudible], but that’s like a graph database implementation in Python. So I worked in that quite a bit and yeah, Ruby and Python are very similar, interchangeable somewhat. Yeah. Any more questions about the?
Audience Member: Talk to me more about it.
Swati Krishnan: Sure, catch me and then I can talk to you about my Python work. Sure.
Shawna Wolverton: Question.
Audience Member: Hi. I have a question for Alethea. So no doubt that it’s great that you’re not woken up at 4 A.M. or on call. Agreed with that. But I’m curious, one of the philosophies of DevOps is that when engineers feel the pain of the alerts, they’re more motivated to fix it. And so do you find that maybe the engineers aren’t as motivated to fix it and if so, is that actually a problem?
Alethea Power: That is such a good question. So this service is in the process of being built right now, but like I said, I built this in the past and had years of experience running it in the past. That’s why we surface very public metrics from it. So rather than feel the pain in a way that makes them bleary eyed and less capable of doing their jobs, they feel the pain in the sense of error budgets and visible metrics and things like this. So, yeah.
Shawna Wolverton: For the record, blameless accountability.
Alethea Power: This is true. I’m actually a big fan of blameless accountability.
Audience Member: I’m also just curious as to how many engineers helped you to build this and how long it typically takes?
Alethea Power: So it’s me and two engineers on Kim’s team. We spent a while designing because there were some security compliance constraints we had to hit and also, we’ve purchased a number of companies, so we have to be able to work with a wide variety of infrastructural decisions. So it took us a few months to figure out high level, how to design the system so that it would do all of that. And once we knew roughly what we were doing, I don’t know, what would you say? We’ve got about 80% of the code written in two months. Something like that.
Audience Member: [inaudible].
Alethea Power: Yeah. We’re cranking right now.
Audience Member: Hi. I have a question for Eleanor. So, I don’t know anything about your product, Support, but I’m assuming there’s a dashboard so when the customers come to open a ticket, is there a knowledge base? I was going to ask you, are you using machine learning to help the customer before they open a ticket.
Eleanor Stribling: Yes. So we’ve got a product called Guide, which is basically a help center. It’s a really easy use, out of the box kind of help center. So yeah, we’ve got that product. We also have AnswerBot, which I mentioned, which helps people before they even reach out to a person to try and resolve their issue before that. And we also have a bunch of tools for people who administer help centers to help them figure out what to write articles about so from those three dimensions, we try to take care of them before they need to reach out.
Audience Member: Got it. Thank you.
Shawna Wolverton: Going once. Oh, one more. [inaudible].
Audience Member: I have a question for Erin McKeown. She and Kim Smith and I actually started Zendesk on the same day, a little more than four years ago. But Erin, when you started, you were the first person to work in business continuity and disaster recovery here, and now you’ve built out quite a practice. I’m just wondering if you have any sort of quick tidbits, lessons learned, insights on that experience over the last four years?
Erin McKeown: Yeah. Well, that’s a really good question. Yeah, I started out as business continuity disaster recovery program manager and that kind of scope grew quite a bit. We had a lot of activity on our intimate management so we built out an entire team that is really churning now and doing amazing work. And so, been switching focus a little bit to prioritize different things and build out different teams. I’m actually, right now, hiring a disaster recovery manager who then will hire three analysts under them so I’m really excited about the progress that’s being made there. But I think what I tried to do was focus on what I could actually manage and actually what I could take on and be honest with myself about that. Because I think I started out of the gate being like oh, I’m going to do all of these things and quickly was like, oh gosh. Got to pace it back a little bit.
Erin McKeown: Again, having very supportive upper management and with that whole perspective has really helped us get progressively down the line, but, yeah, it’s been a fun journey over the four years for sure.
Shawna Wolverton: One in the back.
Audience Member: Hi. This question’s for Eleanor. I was just curious, and it seems that the product you’re thinking about might not be as mature. How do you deal with customer questions around validation of the algorithm or you mentioned you’re going to forecast demand search. How do you deal with where they’re like, well, how is this true or how do I know you’re giving me the right guidance because I don’t trust the machine or the model?
Eleanor Stribling: Yeah, that’s a great question. I actually saw a really … this influenced me a lot. A talk by someone from PagerDuty at a conference a little while ago. And I talked to him about it after because we were thinking about doing some similar things and he was saying that really the biggest challenge was getting people to adopt the ML because they didn’t trust it. And so I think the approach that we’re taking is very much opt in, we’re going to validate all of these algorithms we’re writing. We’re going to validate them all with customers before we start and make those early validation customers EAP customers, we hope, to sign them up so they can sort of see it in action and be part of making sure it works the way they need it to. So I think that that’s one tack.
Eleanor Stribling: But I think it’s also the reason behind the strategy that we’re not going to suddenly say oh, we’re going to use ML to route all of your tickets. Like trust us, it works. We’re not going to do that. We’re going to very gradually introduce little things that help people a little bit. And they don’t even have to take the suggestion if they don’t want to. But the hope is that over time, they begin to trust it. It doesn’t replace them. It doesn’t replace necessarily even huge amounts of their workflow. It just makes it a little bit better for them and I think that that’s definitely going to have to be the first phase of how we approach this. And then we’ll see.
Shawna Wolverton: I think one more question. Yeah? But we’ll all be here afterwards. Feel free to find us.
Audience Member: Hi. I have a question for Eleanor, too.
Eleanor Stribling: Sure.
Audience Member: It’s a continuation to what she asked. So with every customer that opt ins with you, do you retrain your model and then how do you know, how good is your model?
Eleanor Stribling: Yeah, so, great question. So we are doing individual customer models. I think that that’s really because each customer’s quite different and we definitely have customers with a ton of data and we want to make sure that we customize the solution to them. I think that’s how we’re going to get the best result. In terms of validating it, I think that, again, we’re going to need to do a couple of steps. I think with some of our biggest customers, we have some customers who are already really interested in this. So I think that there’s an opportunity there to get them on board. Have them help us test it effectively. I think we will be gating some of these things, so we’ll give them options to roll it out to portions of their organization. We have a lot of customers who deploy it in multiple areas in the organization. So do that gradually. Make sure they’ve got some training around it. But I think, again, really the strategy needs to be we’re going to get some customers who we know it works for them and they can help us evangelize it, because otherwise, I don’t think people won’t necessarily trust it. [inaudible] own data. Does that answer your question?
Audience Member: Yeah, yeah, yeah.
Eleanor Stribling: Great.
Shawna Wolverton: All right. Thank you lovely speakers. We fed and watered you. We educated you a little bit. And in exchange, you get to learn why it would be so amazing and awesome to work here. I want to introduce Lauren from our recruiting team.
Lauren Taft: Hi, everyone. Thanks so much for coming. I’m Lauren Taft, manager of recruiting for technical and university recruiting and Stephanie, who’s over there, who’s our senior tech recruiter. Just wanted to tell you a little bit more about Zendesk. We have 145,000 customers, 2,600 employees. Our headquarters is here in San Francisco. We have 16 global offices. Our product is in 160 countries. It touches 60 languages. And we have 1.4 billion yearly interactions processed.
Lauren Taft: So a little bit about Zendesk recruiting. We’re growing at scale. There’s tons of opportunity and with opportunity comes impact. And then a little bit more about what our values are here. We practice empathy, focus on relationships, and be humbledent, which is humble and confident together that we made as one word. Kind of a fun little spin. We thought it’d be great to show you a video. Oops. I should pause this for a second. We made this for International Women’s Day and it’s a little bit of what it feels like to be a female here at Zendesk.
Video Speaker: Oh okay, one word.
Video Speaker: One word to describe her? Badass.
Video Speaker: Oh, I would totally call her a badass.
Video Speaker: Badass.
Video Speaker: Is badass one word?
Video Speaker: Okay, two words. She’s amazing, but she is also a badass, which is pretty cool. She has a special way of like seeing things within you that you might still be trying to grasp or shore up and she’s like no, you’re there. You’re ready.
Video Speaker: Any time she gives me feedback, it’s often very direct, and sometimes a little shockingly direct, but it never upsets me because I know that it’s coming from a place in her heart where she wants to be my best self.
Video Speaker: She had a really genuine talk with me, which I really appreciated. It was kind of like a big sister talk and it was a talk that I’ve never gotten from anyone at work. She just did it in such a genuine, motherly way. The way that she approached the situation, I really respected, and I realized why she deserves to be in a leadership position.
Video Speaker: Wow. She said all that? Trying to put into words the emotions that are there around it. It’s wonderful to feel recognized. I feel like that’s something a lot of women don’t ask for or expect. I had women like that in my own life, and it is super meaningful to me in terms of just being a person in this world to be able to affect somebody like that, so.
Video Speaker: She’s really helped me to push myself outside my comfort zone. To own those aspects of being a woman that at times can appear or make us feel a little bit more limited. I think her favorite word was, use that emotion and passion for good. To help get things done. To help drive what’s important to your team and your organization and that’s the first time I’ve really looked at it that way. How do I take that crazy wild but super passionate part of me and put that in a place and use that in a way that can get good things done?
Video Speaker: I really love it when women have a conviction or a boldness to put themselves out there and say this is a thing that I want and then to go get it. And it’s been so cool to see her succeed and push herself and push others and grow Zendesk over the past couple of years.
Video Speaker: We would talk about what we’d like, what we didn’t like about our jobs and what we wanted and she took the steps to communicate, make it clear what her goals were, but she didn’t just wait for things to happen. And that’s what is mostly inspiring is that she took her destiny into her own hands. She went and took classes outside of work and was able to move her career in the direction that she wanted to.
Video Speaker: I would say that she’s helped me by demonstrating that you … it’s always easier to take responsibility for your current situation and how to get to where you want to be. She’s shown me that it’s good to not necessarily wait for opportunities to show up, but to go after them aggressively. Even if you’re not sure how they’re going to pan out and even if sometimes other people are telling you not to go after the thing, that if your gut is telling you to go after the thing, you should do it.
Video Speaker: I’m actually surprised at how many strong, powerful, motivated, intelligent women that I’ve met since I’ve been here. More than I’ve ever met in my life. It helps me to drive myself to be better, but it’s also just a really good support network.
Video Speaker: We’re hoping we can spread the joy.
Video Speaker: You are definitely spreading the joy. If there was like one moment this week that I needed this most of all, it was like right now, today.
Video Speaker: I’m so glad to hear that.
Video Speaker: So, thank you.
Lauren Taft: Uh oh. I don’t know what’s going on. There we go. So I hope you guys enjoyed that video. Just gives you a good sense of what it’s like to be here. If you’re interested, come chat with us. It was a pleasure hosting you all. We had a bit of swag snafu so check your inboxes for an Amazon gift card. We’re very appreciative that you’re here, and we are going to take a group picture.
Zendesk Girl Geek Dinner group picture – thanks for coming out and joining us!
Our mission-aligned Girl Geek X partners are hiring!
- See open jobs at Zendesk and check out open jobs at our trusted partner companies.
- Does your company want to sponsor a Girl Geek Dinner in 2021? Talk to us!
