VIDEO
Suzanna Khatchatrian (Senior Engineering Manager, Product Security at Slack) covers how Slack innovates in proactive security approaches and delivers secure by default libraries and services.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
- See open jobs at Slack and check out open jobs at our trusted partner companies.
- Watch all Elevate 2021 conference video replays!
- Does your company want to sponsor a Girl Geek Dinner? Talk to us!
Transcript
Angie Chang: So we have Suzanna joining us — she is a Senior Manager for the Product Security Foundations team and at Slack. She’s built a brand new team within Product Security that’s focused more on proactive security measures by delivering secure by default libraries and services for Slack. So, welcome Suzanna.
Suzanna Khatchatrian: Thank you, Angie. Thank you for nice intro. Hi everyone. Nice to meet you virtually. I’m so excited to be here. Before I start talking about Proactive Security at Slack, I want to also join all the other ladies who spoke before to congratulate you with International Women’s Day. I’m so happy to see that it’s becoming a bigger and bigger event in United States. When I moved to US 25 years ago from Armenia, it was a big event there and it is still there big event, but now I see it getting more and more recognized internationally in this special United States. It’s really special to me, so, Happy International Women’s day to all of you. So as Angie said, I am the Manager for Product Security Foundations Team at Slack, and here, I’m going to talk the next 20 minutes, what we do at Slack in terms of a proactive security measurements especially around product security.
Suzanna Khatchatrian: Quick introduction and background, just in case, we… By the way, I heard lots of great love and feedback in our chat talking about how much you love using Slack, the tool. That’s great. Thank you so much. So, in case of some of you never heard about Slack, quick intro. Slack is basically a collaboration tool that makes your work life much simple, much pleasant and much productive… And more work life, meaning not just your corporate work, it can be anything to do with nonprofit organization, your fun activities, you are planning to do something with your friends and the use cases go forever. So basically, it’s an amazing collaboration tool to make your work done, right? And what we do here in the Slack security team, we want to make sure that we keep that tool for you to be safe, secure, and make sure that your data is safe with you. So basically, we are providing the collaboration platform to make sure that everything, what you do, it is Slack secure and pleasant.
Suzanna Khatchatrian: And what’s our strategy? We provide lots of best spoken reviews, we do lots of automation, ongoing education and production ready components. Basically, anytime we release something for our engineers to start using it and I have a use cases to go through one by one how we do that. Before that also quick history. Today… This year, we are celebrating Slack’s seventh anniversary and as we celebrate Slack’s seventh anniversary, we also have a great history of the security, right? So first actually security, the way it launched our first security engineer was our CTO, Cal Henderson. In 2014, he basically aligned the Bug Bounty program for Slack via HackerOne and he was the first person who was actually triaging the security issues. And then in 2015, as the Slack was growing, they hired the first Engineering Manager who basically started putting plan together and charter for the product security.
Suzanna Khatchatrian: He started hiring more engineers and doing some basically other automations around security development life cycle. As Slack start growing and team growing, in 2018, the current Director of Product Security, Larkin Ryder, took the ownership of the Product Security pillar and she realized that as we’re exponentially growing, we need to make sure we put more measurements in our proactive security, so it’s important to have our traditional product security components in place, such as reviewing, providing consultations, providing some scanning and automations, but we need to make sure that we also look for classes of vulnerabilities and come up with proactive measurements. With that, they hired me in the late of 2019. They hired me as a founding manager for Product Security Foundations team.
Suzanna Khatchatrian: So, what is Foundations team? Just a quick thing is, as I mentioned, we are basically building secure by default libraries and services. We’re providing future proof for Slack, the product, and how we do it, we basically write the code as any other development organization, and we work with other engineers to make sure the code is utilized and used safely by our engineers.
Suzanna Khatchatrian: And how we decide to what kind of work we want to work, what classes of vulnerabilities we want to address. So that’s pretty straight forward. In the beginning, obviously we wanted to make sure that we look at our code base, we understand our potential big classes of vulnerabilities and figure out the best approach, how to build new libraries and services.
Suzanna Khatchatrian: As the team became bigger, as our services became more mature, we also done lots of data analysis, gathering lots of interesting information so to make better data-driven decisions such as our incidents that our back boundary reports, our data that comes from our tooling automations to better understand the classes of vulnerabilities and come up with ways how to solve it and we also marrying that with OWASP top 10. So OWASP top 10, which is The Open Web Application Security Project, those are the known vulnerabilities and we wanted to make sure that we have the right services and libraries to address those issues. And obviously, sometimes also customer demands come our way too, but we’re not feature-facing team, we are more backend team basically working on a future proof and that’s why we are not doing lots of… We don’t have product management working with us, basically we act as a product manager for our team.
Suzanna Khatchatrian: But with that, we also use Slack’s design principles. The most important things, which is very… To make sure that we align with our organization, same way as other engineering teams do, we want to make sure all the libraries and services we’ve write, they don’t make… They are very simple and engineers don’t have to think how they code the service or the library. It has to have great serviceability components in terms of, for us, making sure that our reliability is always up and we don’t want to boil an ocean, we want to make sure we have small, rapid development, prototype the way, do the experiments, learn from it and then move on.
Suzanna Khatchatrian: And most importantly, especially in the security team, we don’t want to reinvent the wheel. If there are all these tools and services there that we can reuse, we definitely going to do it instead of writing from scratch on our own. And the biggest one I want to highlight is the take bigger and bolder steps and that’s basically figuring out like, what can we do? Maybe something that we can invent, we can put lots of efforts to, but if in return we’ll get lots of great benefits securing our product.
Suzanna Khatchatrian: Okay. So as I mentioned, I’m going to talk about the top OWASP top 10 vulnerabilities that we’re trying to prevent with our new secure by default libraries and services. The first one I want to talk about, Broken Authentication and Broken Access Control. That’s a big one. Authentication is very important. It’s also scary. We need to make sure the right people are authenticated to our platform and we make sure that people don’t automatically get these and get reset their passwords or get their password to get stolen or hacked. So what we do, so, first thing about our Crypto library, that was actually the first library that we released for the foundations team. And the… Basically the purpose of this, we know cryptography is very complex, it’s very challenging and we don’t want our engineers, backend engineers, to think about like what type of hashing algorithm to use or what type of encryption algorithm to use.
Suzanna Khatchatrian: We want to provide a library with the right help or functions to do all the job for them. So they don’t have to think about it. They don’t have to unintentionally introduce some potential problems and bugs, and that’s how we introduced our Crypto Library. This is a also great example as an impact for our Slack’s Better Engineering Initiative. So basically, what is a Slack’s Better Engineering Initiative? Whenever you go, you touch a code, you leave that code much better and you don’t try to fix it like small thing, you want to fix it up a very bigger way so it can be used for other engineers, much better and much safer and secure way. So that’s the great impact of Crypto Library. We also made sure we have linters so all the new engineers who come, who trying to use the old libraries that have a linter in place, or they know they have to use the new one.
Suzanna Khatchatrian: The next one is about Magic Logins. So what is Magic Logins? Magic Logins is a very… It’s a unique thing for Slack, but also for some other web applications. But for Slack, basically, if you don’t want to use your password and you can use this Magic Login to authenticate yourself. Obviously, it’s a powerful tool, but potentially also can have a security problem. So when we were looking at our code base, we realized that it has some [inaudible] and legacy code patterns that we wanted to fix it and make it better. And that’s what team actually done. So basically first step was reviewing the code, understanding what’s happening and then providing better documentation, providing better test case coverage, and also framework, which will be much simpler and easier for our engineers to use. This was also amazing core foundation for other future authentication hardening efforts.
Suzanna Khatchatrian: Let’s talk about Injection and Cross Site Scripting. The first example, which is like basically a very important project that our team worked on is Magical Images. So what this does basically, anytime you go to Slack, you upload a video, you upload any type of image, right? A PDF file, your Excel documents, et cetera. It going, it is going to thumbnail that image or that document and basically it goes through Image Magic Library, which used to be called directly in our webapp. So what our team has done, basically, isolated that call to a separate service. That was the brand new service that the team put together last year, beginning of last year. And that was right before we ended… Like we were entering the COVID era. Basically everyone started using Slack, so the usage of slack was exponentially increasing.
Suzanna Khatchatrian: And that was even more scarier to make sure that our brand new service scales and doesn’t produce any serviceability issues and definitely, we didn’t want to make sure we produce any type of outages. So, amazingly, as we launch this service last February and March, it was already in full protection, especially whenever you are uploading files and thumbnailing and we had zero incidents. And the reason why, because the team really put lots of efforts, making sure we had extra test case coverages, making sure we have feature flags and making sure we are going to the production on gradual steps. So right now the biggest impact is basically, we’re processing approximately 300 requests at the peak and image thumbnailing, or any other forms, and which is equivalent to 15 million requests per day. What it does, it dramatically reduces security risk when you are thumbnailing images.
Suzanna Khatchatrian: So what HTML Sanitizer does, basically, whenever you go and post a link in your Slack channel or somewhere, it unfurls the information. So that unfurling basically it has HTML tag. So our service basically makes sure that all these tags are properly sanitized to prevent any potential Cross Site Scripting or Injection Security vulnerabilities. The good thing about this library was that it actually got open source. So other people can come now to Slack HQ repository and use that library if, especially if they are using Hacklang … And I know it’s not many companies use Hacklang, it’s probably Facebook and Slack, but anyways, so it’s there, you can use it and you can contribute it. And this was our first virtual internship project to basically this done by our interns and if you search, use blog on virtual internship at Slack and security, you’ll find all the details about this project and how this project was impactful and meaningful for our interns who are coming back as full-timers to Slack and work for our team this summer.
Suzanna Khatchatrian: Sensitive Data Exposure, that’s another very important security type of vulnerability that, especially with GDPR that came in two years, couple of years ago, and the California data privacy act, European data privacy acts. We know we need to make sure that our customer data stays with them and secure and logs obviously are very powerful tool for our engineers to understand if something goes wrong, but we want to make sure we don’t unintentionally log something that we didn’t want to log. So sensitive information can be channel names, file names. All these information, we don’t want to make sure they are not in our logs. So basically, we wrote a tool to do that, to detect automatically we unintentionally log this type of data and we make sure that it is out of our serach, we get an alert and we take care of it immediately. And there was an amazing talk about just this project by Ryan Slama, who is the engineer in my team.
Suzanna Khatchatrian: So if you are interested in, he did this talk in the Loco Moco Security Conference, so feel free to go and search and look the details about this project. Security Misconfiguration, that’s another important thing. Like you can unintentionally do lots of mistakes and we want to make sure we detect that. For these type of vulnerabilities, we have tons of tools that we actually purchased, or we use as an open source. So, but I’m using just one example, which is a tool the team wrote. Obviously, Slack is on AWS infrastructure and provisioning process can be very complex. So people can unintentionally, basically, bring their own points without proper authentication.
Suzanna Khatchatrian: Obviously, all our sensitive services are behind authentication services, point authentication, but sometimes for some testing purposes, this and that, developers can make mistakes. So basically a range is built to prevent that kind of mistakes because this type of defects we we’re getting from our bug bounty reports all the time, the reports coming… So the bug bounty researchers, this was the easy way, very easy way to detect and find the open end points without proper authentication and we were giving lots of bounties to the researcher, so this was… We knew if we put the tool together, we can easily fix this problem.
Suzanna Khatchatrian: Okay. So the last one I want to quickly talk about, Using Components with Known Vulnerabilities. Again, this is… I’ll talk a little bit about Twistlock. So Twistlock is a tool we purchase from Palo Alto Networks, which basically scans for vulnerabilities in our Kubernetes infrastructure and we didn’t want to invent, we knew, we actually spent lots of time putting pros and cons of all the existing tools there in the even open source, free tools that we can use and we came out with the Twistlock, which provides amazing infrastructure. It provides good reporting that we could easily integrate to our CI pipeline and make sure that we have our infrastructure, Kubernetes infrastructure, without a third party vulnerability. So as we launched this product, we already found dozens of vulnerabilities that we prevented, right? Which could bring buffer overflows or Denial of Service attacks.
Suzanna Khatchatrian: Ossify is another tool. This is another internship project was done on vulnerabilities, but this was the vulnerabilities in our code libraries, not the infrastructure. So this was internship project done by our interns, two summers ago. Another great conference talk about that, 10,000 Dependencies Under the Sea. So I will highly encourage you to go look at it. This was taught, presented at DevCon last summer and with that, I know I only have one more minute. So I just want to say, okay, what’s next? What we are planning to do. There are still… We’ve, all the tools that I talked about or services libraries. It’s like probably 50%. We’ve done more than that. But I wanted just to highlight a few key of them and the most important thing we want to continue our working on the project that produce big impact.
Suzanna Khatchatrian: So we want to harden our authentication, authorization, code base, or we are working on the 2FA hardening close right now and client hardening. We’ve been… The first few years, we’re focusing on [inaudible] backend. Now, we want to start to put some focus on the client side from the security perspective and also look for more types proactive security, such as Early Cross Site warning systems and do… Continue our education and evangelism both internally to our engineers in the backend, as well as to the external community and put lots of focus on efforts on Better Engineering, because all the projects for Foundations team is all around Better Engineering, making the code much better, safer, and proficient to use.
Suzanna Khatchatrian: With that, I want to thank you all for your great time and I’ll go and look, if you have any questions, I’ll answer them. Thank you so much and we are hiring.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
- See open jobs at Slack and check out open jobs at our trusted partner companies.
- Watch all Elevate 2021 conference video replays!
- Does your company want to sponsor a Girl Geek Dinner? Talk to us!
