Sukrutha Bhadouria: Faylene is Director of digital marketing platform operations at NVIDIA. She provides global leadership on web technology, e-commerce, personalization, and new digital experiences. Faylene worked with the leading corporations, such as AT&T, Digitas, American Express, American Cancer Society, and much, much more.
Sukrutha Bhadouria: Welcome, Faylene.
Faylene Bell: I am trying to tee this up, so hopefully you all can hear me okay and see me.
Sukrutha Bhadouria: Yes.
Faylene Bell: Alrighty. So that was awesome. I enjoyed Anu’s motivational speak. It was about self-care, strengths, paying it forward. I was like, “Yeah! That’s awesome.”
Faylene Bell: I’m going to actually do a little pivot and talk a little bit about marketing and some digital marketing best practices.
Faylene Bell: So a little bit about me and who I am and why I’m here. I’m actually Faylene Bell. And I work for a company called NVIDIA. We are known very much in the PC gaming space. We also do quite a bit when it comes to high powered computers, and I am a director of digital marketing platform operations. And you’re like, “What is that?”
Faylene Bell: So when you think about nvidia.com, our digital marketing footprint from a website perspective, I’m responsible for the operations behind that, keeping our site up and running, efficient, smooth sailing, 24-7. So that’s my role here at NVIDIA.
Faylene Bell: I have many years of experience in marketing, working across many different brands, really working on digital strategies, marketing technology stacks, providing advice, consulting, running campaigns, running results. You name it, I’ve done it.
Faylene Bell: And I have a whole lot of experience to share about lessons learned in the corporate workspace, being an African-American woman in a male-dominated world. But that’s another talk. Today, I’m going to kind of focus on the marketing aspect and some tips and suggestions that I’ve picked up along the way.
Faylene Bell: I got my credentials in marketing way back when, and as I’ve mentioned, I’ve worked at some large brands. I’ve had really the benefit of living in multiple cities across the US. I’ve spent some time in New York. I worked at American Express. I started my career off in Chicago. I was in Atlanta, most recently. Worked at AT&T, American Cancer Society, a digital agency, and now I’m here at NVIDIA, and I’ve been here in the San Francisco Bay Area for two years now. Been married almost 19 years, and I have two teenagers. So that’s a little bit about me and my background.
Faylene Bell: So I’m going to kind of talk about the topics that I’m going to share today. One, I want to just share more about what is the optimal digital journey from a customer perspective and some best practices and tips and suggestions to align your strategies and then also sprinkle in some industry examples.
Faylene Bell: So McKinsey & Company did this study and released this. And this is probably not new news to many of you because we’re still in a pandemic, even though the vaccinations are out there, and I’m ready to raise my hand and get mine. Really, a lot of companies are still trying to accelerate how fast they should go as part of this digital transformation. How can they take their brick and mortar approach and strategy, make sure it’s online and just be there for customers across all different types of platforms and channels.
Faylene Bell: The one thing I can say is some of the companies who haven’t really focused on this as a strategy for the last couple of years, they’re scrambling. And so they’re also having to figure out how to prioritize all of this work and make this shift. So it was really important to have a strategy in place because you are up against time. You probably have limited resources. So if you’re not necessarily in the digital space, you probably are feeling a lot more constraints than other industries that have always been focusing on a digital platform strategy.
Faylene Bell: Fortunately, I’ve always been in the digital marketing space, so I was probably ahead of my time way back when, and I’ve enjoyed the reaped benefits of us being in this virtual environment because I’ve been busy as ever and being able to push a lot more.
Faylene Bell: So what is the optimal digital customer experience? It’s really about five key pillars that I’ll talk about.
Faylene Bell: One, I think you need to make sure you have a scalable and a fluid web platform. You also need to make sure customer navigation is seamless. No friction. Have a very easy purchasing journey, dynamic and digestible content, and top notch customer service.
Faylene Bell: Getting background noise from somebody on the panel. Hopefully, they mute it. It’s hard with digital, and I can’t see you all.
Faylene Bell: So those are five pillars that I feel really encompass the optimal digital customer journey.
Faylene Bell: So first, I want to talk about web development. Prioritizing your website design and development is key, and it’s really the foundation of anything when you think about digital marketing, digital strategies, having a digital presence. Mobile-first design, no brainer, but you’d be surprised how many brands still build templates, websites, experiences with a desktop mentality first.
Faylene Bell: You need to think about how customers are looking at your experiences on their mobile device. And that’s where you start. Then you supplement with desktop, but mobile-first is critical. Responsive web is how many companies are doing that from a web platform perspective.
Faylene Bell: The second thing, chatbots. So you might get these little irritating pop-up chat. You’re like, “What is this?” It’s actually a service. It’s managed by rules behind the scenes, and there’s logic behind that. And companies use strategies to inject the chat at places where it makes the most sense within the customer journey.
Faylene Bell: If you’re on the billing page, that might be a great place to inject a chatbot because customers may have questions about their bills, or they might need help with troubleshooting. Well, it may make more sense to inject that dialogue there to help customers overcome their task, whatever it is they’re trying to achieve versus just having a generic chat on the home page.
Faylene Bell: I’m not really trying to do much on the homepage other than look around and try to figure out the navigation. So there’s strategies behind when the chatbot should appear. All the logic behind the scenes or rules in place, it’s very critical to make sure that’s included in your strategy on both mobile and desktop.
Faylene Bell: The third thing is when you think about the web, your design is really key. Of course, you need to have bold colors. You need to also make sure all of your design elements from a UX perspective are accessibility approved.
Faylene Bell: Think about… there are millions of people that are visually impaired or hearing impaired. They actually use devices such as screen readers to help read out loud all the content on your site. So you actually have to design and develop tools so that accessibility is not an obstacle for customers to engage with you on your site.
Faylene Bell: Push notifications. We get a lot of these sometimes when you’re shopping. It’s a strategy behind that as well just like chatbots. When does it make sense? When is it too much? What kind of material, what kind of information should be pushed to customers? Opting in is absolutely critical and to that entire strategy behind push notifications.
Faylene Bell: And the last thing I’ll just mention, GDPR, and you may be like, “What is that?” So this is really something that was driven out of the European environment. When you think about data protection, making sure me, as a consumer, I can opt in, and I can say, “Hey, I want you to share all my data with me that you have on me. And guess what? I want you to delete that.”
Faylene Bell: That’s, at a high level, what GDPR is all about, and companies have to adhere to that. You’ll see those banners showing up on the sides. Do you want us to track you? Can you accept the cookies? Delete the cookies? All that’s connected. You need to make sure you’ve got the infrastructure in place to support that.
Faylene Bell: In Europe, there’s laws. There’s hefty fines. Here in the States, we are adopting it more and more and eventually, it will be global. Everyone will be doing it.
Faylene Bell: Just some industry examples. I’m not going to go too deep. I do have a limited window here on timing, but Disney really does a great job when you think about responsive web design and their experiences across multiple platforms and what they’ve been able to actually put out there to date.
Faylene Bell: The next thing I’m going to talk about is seamless experiences. Creating a seamless digital experience, again, more so of an emphasis now because it’s less about brick and mortar. It’s more about the online presence and what you’re doing.
Faylene Bell: So companies have to build a cohesive presence and make sure that you’re able to pick up the customer journey regardless of what channel they’re tapping into. Many people… I’ll start off on my mobile device, looking at Neiman Marcus, shopping, which I shouldn’t be doing, but I do a lot of that because it’s COVID. I then log into my laptop, join a bunch of Webex calls for work. During lunch break, I might go to the Neiman Marcus site, get busy, distracted, back to working. In the evening. I’m picking up my iPad, back on Neiman Marcus. I’m one customer, and I’ve just hit up Neiman Marcus on three different platforms.
Faylene Bell: It’s important to understand where I am in my journey and all the different touch points that I might be using on a regular basis. So that’s important when you think about the seamless experience across those.
Faylene Bell: Some players in the market that are killing it when it comes to seamless experience, to me, Apple. Just look at the experience offline, online, integration, definitely things to think about.
Faylene Bell: Purchasing journey. So I’ve had the pleasure of working for many brands that e-commerce was a very big initiative. It was about how we can generate online sales and make conversions from visitors, prospects, and even customers. So there’s four key things, I think, when you’re thinking about the digital experience, the journey, and things that companies need to make sure they’ve incorporated.
Faylene Bell: When you’re shopping online, you want to have trust, reliability, those basic things like assurance that where I’m trying to purchase is legitimately your company. Don’t take me off to some wacky little domain that has nothing with your URL on it. You need to be consistent with those experiences.
Faylene Bell: Clear action items in the checkout cart, adding a cart, simple UI thing that so many brands miss the mark. Make it very, very easy for people to know, “This is what I need to do. This is how many items I have.”
Faylene Bell: And connecting with customers, and I’ll say this over and over again. Do not hide customer contact information. Make it very easy for customers to call you, email you, chat with you, something, but a lot of brands hide it. And there’s cost reasons behind that. It’s expensive to have call centers up and running and providing all the support, even now. So that’s been the strategy behind why you don’t find it as available on a lot of sites.
Faylene Bell: Progress bar. When you’re going through that purchasing journey, it’s helpful to know where you are. It’s also helpful to have a lot of auto-completion logic built in. Why am I having to enter the same information or my shipping when I already told you and clicked the box that said, “It’s the same as my billing.” Auto-populate all that information.
Faylene Bell: And then, offering a guest checkout. Not everyone wants to sign up and register as a customer, so give customers and prospects that opportunity to do that on the end after they’ve made the purchase.
Faylene Bell: Examples of players that are really doing this really well online, Amazon.
Faylene Bell: Let me talk about dynamic content. So it’s important to have content that’s digestible. People do not want to read a bunch of stuff. They want to watch videos. They want to get bite-sized information from brands. Think about even TikTok. 15 seconds is enough. You don’t need necessarily a lot, a lot of content. So this is the trend. It’s going to continue to grow and definitely be more prominent.
Faylene Bell: Metrics from… I’m a marketer, core to my heart, been doing it for many years, and you’ve got to have metrics and marketing logic and numbers behind what it is you’re doing and why and how you’re tracking it, and use that information to help make decisions about content that’s working, content that’s not working, more content that’s needed
Faylene Bell: Examples. Think about Netflix. You have Recommended for You. There’s content, there’s logic behind that. Why is it recommended for you? What did you watch last? What are those bite-sized pieces of information and content that you see, and you are just so accustomed to it, you don’t realize all the logic behind that. That’s great marketing.
Faylene Bell: Customer support. Last but not least, you’ve got to provide top notch customer service. Those that are best in class when it comes to customer service, they respond back to any inquiry within 3 hours or even sooner. Some, 12 hours, 24 hours. But even if you can’t respond back with a yes, no, what the option is, invest in automated, generated types of tools that can automatically send a message to anybody who responds or sends you an inquiry within a minute, “Got your information. We’ll be following up. Something, something, something.” Very simple things. You’d be amazed at how many brands and companies skip right over this.
Faylene Bell: I have been also fortunate to work with a lot of brands that have a lot of metrics behind tracking customer feedback, customer satisfaction scores, net promoter scores. You get those little surveys at the end. It’s like, “Hey, how are we today? Did we accomplish what we said we were going to do?” You scale it from 1 to 10. Half of you might have never realized there’s a lot of logic behind that. And a lot of that is feedback on how we adjust our customer service needs based upon the results from those surveys and the information that’s tracked. That’s important. It aligns to great customer service, based upon what you’re saying is not helpful or is helpful.
Faylene Bell: And the last point, don’t hide information. Make it available to customers. Ritz Carlton does some great customer service efforts online and offline.
Faylene Bell: So in summary, some key takeaway points. I think it’s important that you make sure you are profiling your points of differentiation across digital channels in a way where customers expect things from your brand. Enable customers to self-serve and solve common challenges. It will drive satisfaction and loyalty.
Faylene Bell: And then having a long-term mindset, building a framework for continually improving, adapting, and excelling in a changing world. And this is also a personal career tip I share with people that I mentor.
Faylene Bell: You need a flexible mindset as well when you’re working in these changing times and environments. So don’t start a job having that mental mindset where it has to be 1, 2, 3, 4, 5. Two weeks from now, I might need you to do 7, 8, 9, 10, and then I’ll give you a little bit more in another month. So you got to be able to adjust fluidly to change.
Faylene Bell: That’s all I have for you today. Thank you so much for your time and allowing me to come on your platform and share more. And I do have a website, faylenebell.com, if you guys want to reach out to me.
Angie Chang: Thank you, Faylene. I love your podcast as well.
Faylene Bell: Thank you.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Angie Chang: So we have Suzanna joining us — she is a Senior Manager for the Product Security Foundations team and at Slack. She’s built a brand new team within Product Security that’s focused more on proactive security measures by delivering secure by default libraries and services for Slack. So, welcome Suzanna.
Suzanna Khatchatrian: Thank you, Angie. Thank you for nice intro. Hi everyone. Nice to meet you virtually. I’m so excited to be here. Before I start talking about Proactive Security at Slack, I want to also join all the other ladies who spoke before to congratulate you with International Women’s Day. I’m so happy to see that it’s becoming a bigger and bigger event in United States. When I moved to US 25 years ago from Armenia, it was a big event there and it is still there big event, but now I see it getting more and more recognized internationally in this special United States. It’s really special to me, so, Happy International Women’s day to all of you. So as Angie said, I am the Manager for Product Security Foundations Team at Slack, and here, I’m going to talk the next 20 minutes, what we do at Slack in terms of a proactive security measurements especially around product security.
Suzanna Khatchatrian: Quick introduction and background, just in case, we… By the way, I heard lots of great love and feedback in our chat talking about how much you love using Slack, the tool. That’s great. Thank you so much. So, in case of some of you never heard about Slack, quick intro. Slack is basically a collaboration tool that makes your work life much simple, much pleasant and much productive… And more work life, meaning not just your corporate work, it can be anything to do with nonprofit organization, your fun activities, you are planning to do something with your friends and the use cases go forever. So basically, it’s an amazing collaboration tool to make your work done, right? And what we do here in the Slack security team, we want to make sure that we keep that tool for you to be safe, secure, and make sure that your data is safe with you. So basically, we are providing the collaboration platform to make sure that everything, what you do, it is Slack secure and pleasant.
Suzanna Khatchatrian: And what’s our strategy? We provide lots of best spoken reviews, we do lots of automation, ongoing education and production ready components. Basically, anytime we release something for our engineers to start using it and I have a use cases to go through one by one how we do that. Before that also quick history. Today… This year, we are celebrating Slack’s seventh anniversary and as we celebrate Slack’s seventh anniversary, we also have a great history of the security, right? So first actually security, the way it launched our first security engineer was our CTO, Cal Henderson. In 2014, he basically aligned the Bug Bounty program for Slack via HackerOne and he was the first person who was actually triaging the security issues. And then in 2015, as the Slack was growing, they hired the first Engineering Manager who basically started putting plan together and charter for the product security.
Suzanna Khatchatrian: He started hiring more engineers and doing some basically other automations around security development life cycle. As Slack start growing and team growing, in 2018, the current Director of Product Security, Larkin Ryder, took the ownership of the Product Security pillar and she realized that as we’re exponentially growing, we need to make sure we put more measurements in our proactive security, so it’s important to have our traditional product security components in place, such as reviewing, providing consultations, providing some scanning and automations, but we need to make sure that we also look for classes of vulnerabilities and come up with proactive measurements. With that, they hired me in the late of 2019. They hired me as a founding manager for Product Security Foundations team.
Suzanna Khatchatrian: So, what is Foundations team? Just a quick thing is, as I mentioned, we are basically building secure by default libraries and services. We’re providing future proof for Slack, the product, and how we do it, we basically write the code as any other development organization, and we work with other engineers to make sure the code is utilized and used safely by our engineers.
Suzanna Khatchatrian: And how we decide to what kind of work we want to work, what classes of vulnerabilities we want to address. So that’s pretty straight forward. In the beginning, obviously we wanted to make sure that we look at our code base, we understand our potential big classes of vulnerabilities and figure out the best approach, how to build new libraries and services.
Suzanna Khatchatrian: As the team became bigger, as our services became more mature, we also done lots of data analysis, gathering lots of interesting information so to make better data-driven decisions such as our incidents that our back boundary reports, our data that comes from our tooling automations to better understand the classes of vulnerabilities and come up with ways how to solve it and we also marrying that with OWASP top 10. So OWASP top 10, which is The Open Web Application Security Project, those are the known vulnerabilities and we wanted to make sure that we have the right services and libraries to address those issues. And obviously, sometimes also customer demands come our way too, but we’re not feature-facing team, we are more backend team basically working on a future proof and that’s why we are not doing lots of… We don’t have product management working with us, basically we act as a product manager for our team.
Suzanna Khatchatrian: But with that, we also use Slack’s design principles. The most important things, which is very… To make sure that we align with our organization, same way as other engineering teams do, we want to make sure all the libraries and services we’ve write, they don’t make… They are very simple and engineers don’t have to think how they code the service or the library. It has to have great serviceability components in terms of, for us, making sure that our reliability is always up and we don’t want to boil an ocean, we want to make sure we have small, rapid development, prototype the way, do the experiments, learn from it and then move on.
Suzanna Khatchatrian: And most importantly, especially in the security team, we don’t want to reinvent the wheel. If there are all these tools and services there that we can reuse, we definitely going to do it instead of writing from scratch on our own. And the biggest one I want to highlight is the take bigger and bolder steps and that’s basically figuring out like, what can we do? Maybe something that we can invent, we can put lots of efforts to, but if in return we’ll get lots of great benefits securing our product.
Suzanna Khatchatrian: Okay. So as I mentioned, I’m going to talk about the top OWASP top 10 vulnerabilities that we’re trying to prevent with our new secure by default libraries and services. The first one I want to talk about, Broken Authentication and Broken Access Control. That’s a big one. Authentication is very important. It’s also scary. We need to make sure the right people are authenticated to our platform and we make sure that people don’t automatically get these and get reset their passwords or get their password to get stolen or hacked. So what we do, so, first thing about our Crypto library, that was actually the first library that we released for the foundations team. And the… Basically the purpose of this, we know cryptography is very complex, it’s very challenging and we don’t want our engineers, backend engineers, to think about like what type of hashing algorithm to use or what type of encryption algorithm to use.
Suzanna Khatchatrian: We want to provide a library with the right help or functions to do all the job for them. So they don’t have to think about it. They don’t have to unintentionally introduce some potential problems and bugs, and that’s how we introduced our Crypto Library. This is a also great example as an impact for our Slack’s Better Engineering Initiative. So basically, what is a Slack’s Better Engineering Initiative? Whenever you go, you touch a code, you leave that code much better and you don’t try to fix it like small thing, you want to fix it up a very bigger way so it can be used for other engineers, much better and much safer and secure way. So that’s the great impact of Crypto Library. We also made sure we have linters so all the new engineers who come, who trying to use the old libraries that have a linter in place, or they know they have to use the new one.
Suzanna Khatchatrian: The next one is about Magic Logins. So what is Magic Logins? Magic Logins is a very… It’s a unique thing for Slack, but also for some other web applications. But for Slack, basically, if you don’t want to use your password and you can use this Magic Login to authenticate yourself. Obviously, it’s a powerful tool, but potentially also can have a security problem. So when we were looking at our code base, we realized that it has some [inaudible] and legacy code patterns that we wanted to fix it and make it better. And that’s what team actually done. So basically first step was reviewing the code, understanding what’s happening and then providing better documentation, providing better test case coverage, and also framework, which will be much simpler and easier for our engineers to use. This was also amazing core foundation for other future authentication hardening efforts.
Suzanna Khatchatrian: Let’s talk about Injection and Cross Site Scripting. The first example, which is like basically a very important project that our team worked on is Magical Images. So what this does basically, anytime you go to Slack, you upload a video, you upload any type of image, right? A PDF file, your Excel documents, et cetera. It going, it is going to thumbnail that image or that document and basically it goes through Image Magic Library, which used to be called directly in our webapp. So what our team has done, basically, isolated that call to a separate service. That was the brand new service that the team put together last year, beginning of last year. And that was right before we ended… Like we were entering the COVID era. Basically everyone started using Slack, so the usage of slack was exponentially increasing.
Suzanna Khatchatrian: And that was even more scarier to make sure that our brand new service scales and doesn’t produce any serviceability issues and definitely, we didn’t want to make sure we produce any type of outages. So, amazingly, as we launch this service last February and March, it was already in full protection, especially whenever you are uploading files and thumbnailing and we had zero incidents. And the reason why, because the team really put lots of efforts, making sure we had extra test case coverages, making sure we have feature flags and making sure we are going to the production on gradual steps. So right now the biggest impact is basically, we’re processing approximately 300 requests at the peak and image thumbnailing, or any other forms, and which is equivalent to 15 million requests per day. What it does, it dramatically reduces security risk when you are thumbnailing images.
Suzanna Khatchatrian: So what HTML Sanitizer does, basically, whenever you go and post a link in your Slack channel or somewhere, it unfurls the information. So that unfurling basically it has HTML tag. So our service basically makes sure that all these tags are properly sanitized to prevent any potential Cross Site Scripting or Injection Security vulnerabilities. The good thing about this library was that it actually got open source. So other people can come now to Slack HQ repository and use that library if, especially if they are using Hacklang … And I know it’s not many companies use Hacklang, it’s probably Facebook and Slack, but anyways, so it’s there, you can use it and you can contribute it. And this was our first virtual internship project to basically this done by our interns and if you search, use blog on virtual internship at Slack and security, you’ll find all the details about this project and how this project was impactful and meaningful for our interns who are coming back as full-timers to Slack and work for our team this summer.
Suzanna Khatchatrian: Sensitive Data Exposure, that’s another very important security type of vulnerability that, especially with GDPR that came in two years, couple of years ago, and the California data privacy act, European data privacy acts. We know we need to make sure that our customer data stays with them and secure and logs obviously are very powerful tool for our engineers to understand if something goes wrong, but we want to make sure we don’t unintentionally log something that we didn’t want to log. So sensitive information can be channel names, file names. All these information, we don’t want to make sure they are not in our logs. So basically, we wrote a tool to do that, to detect automatically we unintentionally log this type of data and we make sure that it is out of our serach, we get an alert and we take care of it immediately. And there was an amazing talk about just this project by Ryan Slama, who is the engineer in my team.
Suzanna Khatchatrian: So if you are interested in, he did this talk in the Loco Moco Security Conference, so feel free to go and search and look the details about this project. Security Misconfiguration, that’s another important thing. Like you can unintentionally do lots of mistakes and we want to make sure we detect that. For these type of vulnerabilities, we have tons of tools that we actually purchased, or we use as an open source. So, but I’m using just one example, which is a tool the team wrote. Obviously, Slack is on AWS infrastructure and provisioning process can be very complex. So people can unintentionally, basically, bring their own points without proper authentication.
Suzanna Khatchatrian: Obviously, all our sensitive services are behind authentication services, point authentication, but sometimes for some testing purposes, this and that, developers can make mistakes. So basically a range is built to prevent that kind of mistakes because this type of defects we we’re getting from our bug bounty reports all the time, the reports coming… So the bug bounty researchers, this was the easy way, very easy way to detect and find the open end points without proper authentication and we were giving lots of bounties to the researcher, so this was… We knew if we put the tool together, we can easily fix this problem.
Suzanna Khatchatrian: Okay. So the last one I want to quickly talk about, Using Components with Known Vulnerabilities. Again, this is… I’ll talk a little bit about Twistlock. So Twistlock is a tool we purchase from Palo Alto Networks, which basically scans for vulnerabilities in our Kubernetes infrastructure and we didn’t want to invent, we knew, we actually spent lots of time putting pros and cons of all the existing tools there in the even open source, free tools that we can use and we came out with the Twistlock, which provides amazing infrastructure. It provides good reporting that we could easily integrate to our CI pipeline and make sure that we have our infrastructure, Kubernetes infrastructure, without a third party vulnerability. So as we launched this product, we already found dozens of vulnerabilities that we prevented, right? Which could bring buffer overflows or Denial of Service attacks.
Suzanna Khatchatrian: Ossify is another tool. This is another internship project was done on vulnerabilities, but this was the vulnerabilities in our code libraries, not the infrastructure. So this was internship project done by our interns, two summers ago. Another great conference talk about that, 10,000 Dependencies Under the Sea. So I will highly encourage you to go look at it. This was taught, presented at DevCon last summer and with that, I know I only have one more minute. So I just want to say, okay, what’s next? What we are planning to do. There are still… We’ve, all the tools that I talked about or services libraries. It’s like probably 50%. We’ve done more than that. But I wanted just to highlight a few key of them and the most important thing we want to continue our working on the project that produce big impact.
Suzanna Khatchatrian: So we want to harden our authentication, authorization, code base, or we are working on the 2FA hardening close right now and client hardening. We’ve been… The first few years, we’re focusing on [inaudible] backend. Now, we want to start to put some focus on the client side from the security perspective and also look for more types proactive security, such as Early Cross Site warning systems and do… Continue our education and evangelism both internally to our engineers in the backend, as well as to the external community and put lots of focus on efforts on Better Engineering, because all the projects for Foundations team is all around Better Engineering, making the code much better, safer, and proficient to use.
Suzanna Khatchatrian: With that, I want to thank you all for your great time and I’ll go and look, if you have any questions, I’ll answer them. Thank you so much and we are hiring.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Angie Chang: Our next speaker is Vrushali, who is a VP of product at Carta. She’s building the operating system for venture capitalists. She previously led product at Rocket Lawyer, and she’s passionate about making the world more equitable.
Vrushali Paunikar: Thanks, Angie. All right, Girl Geeks, to start off I have some questions for all of you. How do you find and nurture the best product managers? How do you stand out when you feel like you’re being overlooked or undervalued? How do you thrive in a political environment without losing your sense of self? Fellow leaders, I imagine these are all questions you’ve grappled with. I know I certainly did, and that’s what I want to talk to you about today.
Vrushali Paunikar: All right, so I’m Vrushali Pauniker and I lead the investor services product team here at Carta. My answer to these questions is simple. Focus on solving problems. I will share with you three stories of how this approach has helped me navigate a career into product management and scale to vice president. I will also talk to you about how these experiences have shaped how I think about building teams today.
Vrushali Paunikar: All right, the first story is about how I got into product management. When I graduated from college, I did what a lot of business majors do who don’t know what to do with their lives. I became a strategy consultant. I joined a boutique practice that was developing its own software. As a consultant, I worked on complex analyses to help drive insights for our clients. As a power user of the software, I had a lot of opinions about the improvements that could make my life better. I signed up to be a part-time product analyst, helping our engineering team scope and build features.
Vrushali Paunikar: Many of the features I was assigned were problems I had been solving for our customers. I would spend an hour of every morning working with our engineering team, where I would share my SQL queries and the outputs that we presented to customers. We would talk through user flows, brainstorm the user experience. We would talk about the math and the validation strategies.
Vrushali Paunikar: And then after that hour, I would switch back to my consulting work. This quickly became my favorite hour of the day. It also changed how I thought about my day job. Every predictable and deterministic task was one that could be automated through software. Every complex analytics flow, I started imagining the software workflow that would streamline it.
Vrushali Paunikar: I became obsessed about solving more problems for more customers through software. I was becoming what Jeff Lawson from his 2013 unSEXY Conference talk would call a software person, someone who solves problems with magnetic particles.
Vrushali Paunikar: Two years later, when the firm decided to start its own full-time product management team, I was asked to be one of its seed members. It was this obsession of connecting customer problems to software, and the ability to execute, that made me a clear choice for this team.
Vrushali Paunikar: I had found my calling. So think about the Vrushalis lurking in your org. How might you spot them? How might you create opportunities for them? I’m a big believer in creating internal mobility into product management. Create the opportunity that was created for me. Not only is it my way of paying it forward, it’s also good for business.
Vrushali Paunikar: If you’ve been a hiring manager for product roles, you know that the competition for the best talent is steep. With a little investment, you can nurture the best PMs of tomorrow. By hiring product managers from within your company, you get the advantage of asymmetric information. You can look for the people who demonstrate the skills that lend themselves to a future in product management, the systems thinkers, the structured communicators, the resourceful, and the creative problem solvers.
Vrushali Paunikar: You can also hire from areas of the business where domain knowledge is especially valuable to a particular role in product. Over 50% of my team today came into product management from another role at Carta. Many of them came from services teams and have a deep knowledge of our customer and the venture industry.
Vrushali Paunikar: All right, my second story is one about Rocket Lawyer. So I joined Rocket Lawyer to help democratize the access to justice. About a year into the job, our leadership gathered all the product managers in a room to do our quarterly planning. They had listed all of the R&D priorities, in order, on the whiteboard. And they started assigning each priority to a product manager.
Vrushali Paunikar: Number one, Stan. Number two, Stephanie, Vanessa, Jeff. I held my breath as each name got called out that wasn’t mine. Sure enough, my name got called out last. It was a project to redesign our logged in dashboard. What made matters worse was that it was a project that had had a lot of starts and stops. PMs before me had attempted to tackle this project unsuccessfully. Many of them were no longer at the company.
Vrushali Paunikar: After some private sulking, I decided to embrace the problem. I focused on delivering value to the user, the person who needs legal help. I had a small team, as a function of being the lowest priority project, of just one engineer and one designer. With this team, we decided to take a different approach than our predecessors. We skipped the months of research and planning. Instead, we took a rapid prototyping approach to validate our hypotheses and build towards a minimum lovable product.
Vrushali Paunikar: We learned to do a lot with a little. More importantly, I learned how to get people excited about solving problems with me. My small squad felt a little rebellious, a little bad, but we were the good guys fighting for the user. Slowly, we were drumming up excitement at the company. Our experience was sexy. We were able to show progress instead of just talking about it.
Vrushali Paunikar: Within two months, we launched. We measured the impact and engagement. It was up. After resolving some early performance issues, our experience also improved trial conversion rate. That was unexpected. That project put me on the map. It was a win that no one was counting on. I became known as the product manager who could solve any problem you threw at her.
Vrushali Paunikar: It also won me the right to ask for the projects that I wanted. This credibility and growing track record of solving problems helped me go from an IC to a director of product management at Rocket Lawyer. You don’t have to be put on the most important projects at work. Be a steward to the problem and the user. It will not go unnoticed. Vrushali Paunikar: Today, as a manager of several PM teams, I invest in the teams that show a track record of solving problems. Carta’s CEO, Henry Ward, talks a lot about progress, not activity. “Don’t fall into the activity trap,” he says, “Where people take on a lot, but make little progress on any front.”
Vrushali Paunikar: Oftentimes a team or a manager will collect a bunch of problems and then ask for resources to go solve all of these problems. That’s counterproductive. The signal for investment are the teams that have already shown a high return on investment. For product managers, success is not just about shipping. It is about driving an impact. A person’s problem solving track record is also a big part of how we do performance reviews, promotions, and staffing conversations at Carta. We’re in the process of actually rolling out internal resumes, problem resumes, on an application called Confirm HR.
Vrushali Paunikar: All right, so the third story is of how I withstood the turmoil of a rapidly growing startup and flourished. In 2017, I started talking to a company called eShares. They were set out to fix the income inequality gap by creating more owners. Henry Ward, the CEO, spoke and wrote about building a special type of company driven by first principles.
Vrushali Paunikar: They were deliberate about the way they wanted to run their business. Their manifesto, called eShares 101, laid out the guiding principles for the company. They ranged from inspirational to kooky. I was in. By the time I joined the company, they rebranded to Carta, and our series C had closed. Never had I met a group of people who were so passionate about the problems they were solving. The company’s core principle of, “Always be helpful” was so prevalent. Everyone went out of their way to help me onboard. We would also recall other company principles like do the right thing, data models first, cage match everything, when we ran into day-to-day challenges.
Vrushali Paunikar: Nine months into the job, we closed our series D at an $800 million valuation. This is when things started getting chaotic. We were on the map, on the verge of unicorn status. We started hiring outside execs that told us and Henry to reform our kooky ways. At one point, we removed the eShares principles from the doors of our office conference rooms. Politics started seeping through the various cracks of our rocket ship.
Vrushali Paunikar: At times, it felt like perception mattered more than reality. I felt overwhelmed. I felt overlooked and undervalued. But there was one thing that no one could take away from me. It was the one thing I would hold on to tighter when I felt confused, lost, or sad. It was the ability to wake up every day and solve problems for our customers. I was at Carta to build a platform for venture capital. It would help me shape the future of the industry and its players. It would also help us create more owners.
Vrushali Paunikar: Luckily, this period at Carta did not last long. After a troubling year of execution, Henry took us back to the first principles approach that made us the special company that we were. He published a new version of the eShares 101, this time calling it the Carta identity traits and operating principles. “We are helpful, relentless, unconventional, and kind,” he told us. The atomic unit of Carta, the company, is a problem.
Vrushali Paunikar: He did an audit of the company’s best problem solvers and elevated them. Carta promoted me twice in a matter of months. I now get to play a big part in shaping the company, our culture, and our practices. You too can establish a problem oriented culture at your company. Safi Bahcall in his book, Moonshots, talks about how as an organization grows, employees are put in a position where they are deciding how to best use the units of time.
Vrushali Paunikar: Given an hour, is that hour best spent on one, solving problems, or two, networking and getting ahead? You always want the answer to be number one. That is the higher value activity for your company. Establishing problem oriented culture starts with attracting the right people.
Vrushali Paunikar: To that end, we found that traditional job descriptions just weren’t cutting it. They don’t tell people what problems they’d be solving. They don’t give people a sense of what it’s like to be at the company or work with a particular team. Data also shows that traditional job descriptions often filter out women and underrepresented minorities when they feel like they don’t meet all the criteria.
Vrushali Paunikar: So last year, along with our Chief People Officer and the person who is now our Chief Marketing Officer, worked together to roll out problem descriptions. Problem descriptions focus on problems, not qualifications. They represent the team and the company in an authentic way. And they also remove all language we know that filters out underrepresented minorities and women. For example, years of experience. Now we tend to hire candidates who are more focused on what problems they’d have the opportunity to solve at Carta versus what their title would be.
Vrushali Paunikar: All right, you also need to make success problem-oriented. When new hires join Carta, and I tell them about what it takes to succeed here, I tell them three simple things. Find the right problem to solve, solve that problem, and tell people you solved that problem. On step one, the emphasis is on, right. The right problem is important, urgent, and one that the company’s willing to trust you with.
Vrushali Paunikar: Too often, I see people run towards shiny objects. Instead, find the problems that need solving, but aren’t getting attention. Remember my experience at Rocket Lawyer where I wasn’t solving the sexiest problem, but still was able to make the most of it. If you’re new to the organization, it’s important to establish trust. You don’t want the first problem you take on to be large and ambiguous. If you fail, the company won’t know if your failure was due to bad execution or because the problem is hard. Plus, you’ll know very little about what it takes to get things done at the company.
Vrushali Paunikar: So start with small and well-defined and work your way up to hard and ambiguous, and the rate at which you make that climb will depend on your seniority. Sometimes you’re going to fail at solving problems. That’s okay. It’s part of the game. But if you’ve dug deep, formed a solid hypothesis, and executed well, a failure is going to be full of learning and it will help you improve your next hypothesis.
Vrushali Paunikar: Which takes me to step number two, execution matters. Always lead with the problem. Make sure that your solution hypothesis matches the problem. On the product team, we write starts with why briefs to explain the problem, the solution hypothesis, and what success might look like. Bring your stakeholders along on this journey with you. This is also the step where you master the craft. The better you get at solving these problems, the higher quality outputs you’re going to produce.
Vrushali Paunikar: Step three, share that you’ve solved the problem or what you learned by trying. This is important. I often see women shy away from this step, but you won’t win the right to solve bigger problems without it. At Carta, we have weekly show and tells where anyone at the company can share what they’ve been working on. It’s one of the best ways to get visibility at the company.
Vrushali Paunikar: If you’re a leader at your company, create forums where people can share their problem solving journeys. In addition to show and tell in my business unit, we have weekly one hour sessions called investor services, IS, got talent, where people across the business shared their problem solving journeys at critical stages, problem, definition, hypothesis, demo, and report back. This time, for me, is sacred.
Vrushali Paunikar: And this is the success flywheel at Carta. If you do one, two and three, you win the right to solve bigger and more complex problems. That is growth. I know there’s someone out there right now who’s saying, “Hey, I do all of these things and I’m not being recognized.” Have patience. It will pay off. And I’ll let you in on a little secret. When you reach the higher levels of your organization through problem solving, you’re better equipped to handle all of the curve balls thrown at you versus someone who got there by other means.
Vrushali Paunikar: So steadfast, problem solvers, go forth and change the world.
Sukrutha Bhadouria: I see a question from Christine. How do you balance the possibility that organization will change and people will move around?
Vrushali Paunikar: Okay, I’m going to answer that question the way I interpret it, which, one is, how do you balance the possibility that organizations do change? And especially at a rapidly growing startup, that’s going to happen. I actually just like to embrace that change and lean into it because it’s an opportunity to learn.
Vrushali Paunikar: And then people will move around. That is especially if you’re creating internal mobility, that’s a function of that. And I think it’s … I like to encourage people to move around and go forth and like follow their passions. And within my own team, I’ve had someone who was like in product and then decided they actually wanted to pursue a career in design. And I try to be supportive of people’s passions because one of the things about building problem-oriented teams, and again, going back to the Safi Bahcall sort of problem where if with a unit of time, is that time best spent on networking or problem solving?
Vrushali Paunikar: One of the things that’s really important about that is person, problem, fit. So you want the person’s skills to fit sort of what the value they’re able to produce on a particular type of problem. So I think you need to let people sort of explore their passions and where their skill sets are best suited.
Sukrutha Bhadouria: Yeah. Well, with that, let’s wrap. Thank you so much, Vrushali. That was really insightful. When you do get the chance to look at the chat, people have really been resonating with your talk.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Angie Chang: So, our next speaker is Amrisha, who is the Infrastructure Engineering Team Lead at MaestroQA or Maestro. She works on all aspects of the application for infrastructure and security, along with deployment pipeline. And before Maestro, she was a senior network DevOps engineer. She’s based in New York City and has been for 15 years. Welcome.
Amrisha Sinha: Thank you. Let me just share my screen here. Hi, thank you so much for having me. That story before me was so inspiring. Thank you, Ashley. It was great for me to hear as someone halfway through my career and someone who’s just had a baby. It was a lot of advice that I will definitely action.
Amrisha Sinha: So, I’m here to talk about DevOps and how it’s a product as well within the organization that the DevOps engineers work with. As Angie said, I’m the Infrastructure Engineering Team Lead at MaestroQA, but I’m also a DevOps engineer. I just happen to do both at our company. And what I’m going to cover today is really five insights to help you empower your development team, your product team, and DevOps all at once, if this is a team that you have at your company.
Amrisha Sinha: A little bit about me. I’m a diplobrat. And for those who don’t know what that means, my father is a diplomat, and I’m Indian, but I was raised all over the world. We moved every three or four years. And the main thing that I learned from that is how to figure out a new school, a new environment, and where I fit into that, and really making the best of it. And that’s what led me to Cooper Union and Cornell, where I got my degrees, and have been in New York ever since.
Amrisha Sinha: I focused on various types of systems engineering, and really looking at the whole system and how to make it work better, what are the things that go into making all the little pieces, all the little cogs work together. And what I’ve learned about myself is that I’m just a problem solver. I do logic puzzles for fun. I have a puzzle a day calendar at my desk that I do to get the juices flowing, and I really enjoy them.
Amrisha Sinha: And over the course of my career, I’ve developed a real passion for engineering automated solutions for the teams that I work in and help everyone else get so much more out of the work that they do, because they don’t have to worry about fitting pieces together.
Amrisha Sinha: A little bit about MaestroQA. MaestroQA makes an omnichannel quality assurance software for modern support teams. And some of our largest customers are Etsy, MailChimp, Peloton. And one of our partners is Zendesk.
Amrisha Sinha: All of these companies, all of our partners, use our software to improve agent performance, optimize their customer experience processes, and we give them a platform where they can unlock business level insights, look at analytics about how their support teams are performing and improve metrics like retention, revenue, CSAT. And yeah, I have the privilege of working here for the last year.
Amrisha Sinha: Let’s dive into this. What’s DevOps? It’s one of those terms that a lot of people hear about, but not everyone really understands how to define it, and something that I’ve been sort of working on as well. DevOps is development operations. It’s the practice of getting your code commits from the repository into production, like everything that has to happen in the process. Which, if you lead a product team, you know there’s so many little pieces that all have to work together seamlessly in order to make sure that the deployments happen.
Amrisha Sinha: DevOps can be broken into three main functional areas: continuous delivery, which is how do I get every commit, every margin to master into production, continuously without a break, without waiting for something else to happen. Infrastructure automation. Frequently, the changes that you make will require some changes in the infrastructure, in the stack, in the server, in the load balancer. Something has to happen in order to facilitate this new feature you’re rolling out.
Amrisha Sinha: So, how do we automate that and make it part of the testing and the development? And monitoring and orchestration. Once your changes get into production, collecting feedback on how it’s performing, metrics on how the CPU is working or how customers are, what the end user experience is. Collecting all of that and feeding it back to your product team so that they can action on it and then deploy new changes. At the end, DevOps really makes your deployment process look like this infinity sign that we’ve got here. It’s making all of these pieces work together in ad infinitum.
Amrisha Sinha: As I mentioned, deploying software requires a lot of coordination. I’ve seen many different versions of this kind of flow in various organizations that I’ve worked in, and you need a development team, you need an infrastructure team. If you get large enough, a release or a change management team, a QA team to do testing, and product management to kind of make sure that there’s new stuff going on and all the things that product managers do.
Amrisha Sinha: And it requires a lot of coordination across all of these teams, which introduces delays, opportunities for misunderstanding, or having to spend time educating each other, translating what you need. And that takes up a lot of time, and it takes time and efficiency and interest away from your existing teams. Having to understand how to communicate your needs to an infrastructure engineer when you don’t have that knowledge, takes time away from development ,time from change management, time from testing.
Amrisha Sinha: So, it really is a cost to your teams in order to have to figure out how to get your changes across, right? DevOps kind of brings the cost of how much time your team spend communicating with each other down because the DevOps engineer’s job is to simplify this pipeline and make it happen more efficiently and more smoothly and consistently.
Amrisha Sinha: Where does a DevOps team fit? If you look at your team and you realize that it’s time to introduce DevOps, not something that a small company necessarily needs, but if your company’s growing, your product’s growing, and you find that you need DevOps, where would a DevOps engineer typically fit?
Amrisha Sinha: DevOps engineers aren’t developers and they aren’t infrastructure engineers, but they do require enough proficiency in both to be able to communicate with both teams. And so, they fit between these two teams, and are typically as an extension of release management teams or change management teams, and are kind of a missing piece between tying everything together.
Amrisha Sinha: We have the skills to build you a custom delivery pipeline, which is essentially software. And this pipeline is operated by your existing release team, and it interacts with their infrastructure. DevOps engineers, our goal is to build you a custom solution that fits, and stay there to continue to maintain it as your needs evolve, as your product evolves, and as your organization evolves. A DevOps team is really there to build you this product and continue maintaining it.
Amrisha Sinha: Let’s get into five insights on what will help you best work with DevOps engineers and empower them, if you’re managing a team with a DevOps function, and how to really make the best out of this really special skillset that DevOps engineers bring.
Amrisha Sinha: The first insight is that the team that’s responsible for doing your deployments, you have some sort of function there, you’ll either have a change management team or a senior developer that’s very knowledgeable about the deployment process, they should be involved in the pipeline design process. They’re giving you the user feedback, the initial discovery of what does the team actually need in terms of a pipeline. They should be heavily involved. They should be encouraged to interact with DevOps thoroughly, be demanding, and talk about what their pain points are.
Amrisha Sinha: This pipeline that you see here. Some version of it exists in every organization. A DevOps engineer needs to come in and figure out what this looks like and how to start automating it and how to solve the pain points so that everyone can go back to doing what they actually love doing.
Amrisha Sinha: So, make sure that your release team is talking to your DevOps team, providing feedback and looking at designs and fixing what needs to be fixed and making sure that the problem is well understood.
Amrisha Sinha: The second insight is, once a pipeline is done, this can take weeks depending on how automated you already are, to months to years. Once a pipeline is done, the DevOps engineer isn’t going to sit there and run it. DevOps engineers don’t replace a release management functionality or an infrastructure functionality. They add to it. It’s like you’re buying software, except you are hiring the person to make the software for you.
Amrisha Sinha: You wouldn’t expect to make software and then use it yourself. You would expect to make software, have an end user use it, and then keep deploying patches. DevOps engineers do the same thing. Once you’ve built a pipeline, hand it off to the team that’s actually going to run the releases.
Amrisha Sinha: And then look at new technology, look at what direction tools are going, and see if the problem’s changing… the problem that they’re solving for, the deployment pipeline is changing, and make up bits to it and make adjustments to the pipeline. Bring in new technology. It helps DevOps engineers also feel empowered to stay on the cutting edge of technology and make sure that you’re keeping your end users happy and not having them go two steps back when the entire process changes. DevOps engineers make sure that the pipeline updates along with it.
Amrisha Sinha: The third insight is to use the best tools for the job. Emphasis here is on plural. You’ll be able to find a lot of DevOps tools that kind of do everything for you, “Plug me in and I’ll do everything.” And that’s great, but not every organization has a canned deployment system. And if you have a DevOps engineer, you have the person there that will integrate five different tools into one seamless pipeline for you. There’s no need to compromise on the experience for your developers or for your end users or for anyone else involved by using a singular tool. You have a person on hand who has the skills to figure out how to fit in the right tool for the right job in the pipeline and write the scripts or the programs necessary to make sure that it’s a seamless integration. You have a person to do the integration for you, and don’t discount the impact that that can have on the end user experience.
Amrisha Sinha: Fourth insight is to use an iterative approach. Integrate automation deployment workflows into the pipeline early, and then provide feedback on how it’s going. I’m sure everyone here has had some sort of experience where you’ve worked on a feature or deployed some software that didn’t quite land. And it’s discouraging to go through a long design process and a long implementation process, and have them not land. The way we’ve all figured out solving that is by using an agile approach. Iterating early, providing a minimum viable product, and then seeing how that goes, collecting feedback and improving the end user experience.
Amrisha Sinha: It’s the same for DevOps. We can spend months building your pipeline, and if it doesn’t land, it’s discouraging for everyone involved. So, automate one part early, as you can see the infrastructure deployment, see how it works. If it works, go ahead, automate a different part, then string them all together. It’s very useful to provide feedback early, to figure out if the problem being solved is correct, if the pain points that have been identified are actually pain points or not. Same thing as you’re building a product. If you get feedback early, you do better, you have better end results, right?
Amrisha Sinha: The last insight is that we’re problem solvers, as I said at the beginning. We want to solve your pain points. We want to build a solution that works for this particular team. We want to fill any gaps, and not really sell a previously sold solution or come back and just reimplement the same pipeline over and over again. We’re engineers with unique skills that live in this gray area between all of these teams.
Amrisha Sinha: So, if the puzzle piece doesn’t quite fit correctly, let us know. We’ll find a better solution. We’ll make whatever changes are needed. That’s how I see my role. It’s really coming into a functioning team, pieces that fit together, and tying it all together into a picture that works, that’s efficient.
Amrisha Sinha: Bringing in DevOps means allowing your release team to focus on actually releasing software and not coordinating with changes, with infrastructure, and making sure development changes are done, making sure developers are working on code and building the next great feature, not worrying about whether their changes are actually going to get into production properly. That’s why you bring in DevOps. Yeah. Tell us if something’s not working. Tell us. We’ll find a better solution. Until you give us that feedback, how would we know, right?
Amrisha Sinha: For anyone who’s watching and taking screenshots, I summarized everything into this slide for you. These are your five main takeaways on how to empower your team when you bring in a DevOps engineer, how to take some of that load off your team, and how to make your DevOps engineer happy. These are things that I’ve learned over the course of my career, and I think are good to know for managers, for leaders within the company.
Amrisha Sinha: We’re unique. We’re unicorns. We have skills that go across many different functional areas. So, letting us solve your problems and let everyone else do the work is kind of fun for us. And staying on top of technology, also fun for us. Hopefully these insights will help you get the most out of your DevOps engineers. It’s really exciting to see what DevOps is doing in the last five years. It was a very nascent field when I entered it about six years ago, and now it’s really blowing up and it’s very exciting for me to see.
Amrisha Sinha: Lastly, we’re hiring. We’ve been awarded best place to work by BuiltIn NYC in both small companies and in New York City. As Angie mentioned, we’re located in NYC, but remote first. We’ve been making a lot of out of state hires lately, some incredible team members. These are the roles we’re currently hiring for, but find me on LinkedIn or send me an email. It’s just amrisha.sinha@maestroqa.
Amrisha Sinha: Send me an email for referrals. We’d be happy to have a conversation to see about where you fit.
Amrisha Sinha: We’re growing very rapidly. If I had to count, I’d say we’ve probably added about five people this year alone so far. And we serve the biggest brands, and we’re just looking for people to come join us and help us make excellent software.
Amrisha Sinha: And just on a personal note, I had a child in July, 2019, and I came to Maestro about six months after that, and I’ve found that it’s an amazing place to work. I feel very supported as a woman, as a working mother. And we are trying very hard to be a great place to work, which includes having conversations around race, gender discrimination, and we do this on a weekly basis.
Amrisha Sinha: So, please connect with me on LinkedIn. Happy to talk further about my experience, and I can talk about DevOps for hours. Happy to do that as well. Thank you. Any questions?
Sukrutha Bhadouria: Hi, Amrisha. Thank you so much for that amazing talk. I think it’s been really insightful for all of us to hear about MaestroQA, and of course the DevOps journey as well. But with that, we are going to just call out that people can post questions in the Q&A section, and post comments in the chat. Amrisha, if you can get the chance, you’ll see some questions and comments that you can respond to. We will now switch over to the next session, which is the panel. So, thank you so much, Amrisha.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Sukrutha Bhadouria: We’re going to now go into our next session. Like I said, it’s an amazing panel. The topic is “building high performance teams in a pandemic. I’m going to be joined by Rachana Kumar, who is the VP of Engineering [at Etsy], Elaine who’s the CTO at Change.org, Tina who is CTO and Founder at Transposit.
Sukrutha Bhadouria: Welcome ladies, I’m really excited to get us started. So first, because you’re such accomplished women, I don’t want to do you the injustice of trying to introduce you all. You will do a much better job yourselves. So please do go ahead and introduce yourselves and explain the different size of engineering teams that you’ll need. We’ll start with you, Elaine.
Elaine Zhao: Thank you. Thanks and happy International Women’s Day and I’m really glad that I have the chance to talk to you all and then share my ideas here. My name is Elaine Zhao and I am the CTO of Change.org, so my entire career is probably very similar to many of you, the start off as engineers and move up to the manager rank so I’m super excited to be here and share some of the ideas with you all, thanks.
Sukrutha Bhadouria: Tina?
Tina Huang: Hi. As Sukrutha mentioned, I am Tina Huang, I am the Founder and CTO of Transposit and we are a company that works on DevOps process orchestration so it’s great to hear a little bit of that last presentation on DevOps and this sort of up-and-coming nature. As a company, we were founded in 2016 and we are currently a Series B startup and around 50 or so employees total, about 20 something in engineering.
Sukrutha Bhadouria: Rachana?
Rachana Kumar: Hi all. First of all, happy International Women’s Day. I’m Rachana Kumar, I’m currently VP of Engineering and Managing Director for Etsy Mexico. So I have been at Etsy for about seven years so I’ve kind of seen now the whole growth from a smaller startup to a larger company now. Currently we have about 600 engineers and my org has about 150, 200 engineers and yeah, I also have a five-year-old son.
Sukrutha Bhadouria: That’s awesome. I mean, we should get right started. I kept getting questions like, “How does everyone define high performing teams?” Because that being the topic of this panel, let’s level set to begin with. How do each of you define high performing teams and how does this end up influencing the kind of engineers you hire for? Tina?
Tina Huang: Yeah, so Transposit has a few different core values but the one that is sort of nearest and dearest to my heart is we are a culture of pragmatism. And this really came about because at a number of different tech companies I worked for before, I worked at Apple, Google, Twitter, some of these kind of companies that really led the way for how we think about kind of engineering cultures, I found a lot of their tech ladders and the way they thought about what high performance meant is really like I could write a really, really strong and fast algorithm.
Tina Huang: That affected not only hiring and tech ladder and promotion but just the entire culture. I thought that was really interesting because what I saw was a lot of engineers who would hyper optimize a piece that was just not actually important to the holistic picture. So I wanted to build something different at Transposit and so we grounded this on this culture of pragmatism.
Tina Huang: What that means is that I really value everyone on my team to be able to really understand and empathize with the customer and the business value first and foremost, and then decide what is the appropriate amount of engineering to actually build for a particular piece of technology. Some people take that as when I say we are a culture of pragmatism, that you know we want hacky engineers.
Tina Huang: I think that there’s this perfect sweet spot with pragmatism and it’s really being able to understand is this piece of code something that is core to our system and is going to be run a hundred thousand times and we’re pretty certain it’s going to be part of our final product or is this piece of code for a prototype and it’ll be better to get something out to the customer sooner and get feedback and iterate more progressively.
Sukrutha Bhadouria: Yeah, and going from a startup like yours, I’d be interested to hear your point of view, Elaine. How do you define high performing teams and how do you do it at your company when you’re trying to build a high performing team? How does that influence your hiring process?
Elaine Zhao: I think probably no surprise that every engineering leader at all levels want to be a high performance team. That’s the ultimate goal. So I think the definition is really based on the why, why we’re doing that? And I think that have to take into account for the company strategy, the team strategy and goals and visions. At Change.org, the biggest social change platform in the world and under this umbrella, our goal and the vision is to become the undeniable leader in digital activism, right?
Elaine Zhao: We want people come, engineers come, to become the great engineers. And while they’re doing good for the world. So that really focused on couple areas and tied it up to our mission and vision then also tied up the two area that we want to bring and it’s one is higher experience engineers and continue to learn and improve the existing team that they both have to happen, right?
Elaine Zhao: We want to put a real emphasis on our user first because it doesn’t matter what we want to do, what we say we are good or not good, if our users not being served, then we cannot call ourselves a high performance team, so we really want to switch the emphasis to ours and focus on the user’s impact first, then come back and we drive the velocity, focus on learning and candor, that’s another thing that’s quite unique to Change or I would imagine many social mission focus engineering team as well that would tend to have a very strong culture already and how do we cultivate that we’re very caring.
Elaine Zhao: But when we so care about each other, then we sometimes forget about the best caring that we can have is actually make sure everyone continue to improve, right, we challenge each other to learn new skills and go up to the next level and that’s, whether or not you use the the words radical candor or be trusted, helping each other to succeed and that’s what we focus on and slightly different than many other companies in the world do, yeah.
Sukrutha Bhadouria: And then how’s your experience, Rachana, through seeing Etsy grow and then now expanding in Mexico?
Rachana Kumar: Yeah, I think both Tina and Elaine covered great points from customer focus to pragmatism to radical candor. So I’ll focus more on, as we’ve been scaling, how we are kind of thinking of scaling high performing teams. So in the past, we predominantly have had engineering teams in New York, San Francisco, and remote within the US. And over the last year or so, we have been expanding in Dublin and Mexico a lot.
Rachana Kumar: And for me, as we are ramping up hiring, they are starting to hire in these locations, one thing we are looking at is how do we take the aspects of starting from sourcing to hiring to onboarding to then forming your team, how do you then select projects in these new locations and countries for them to work independently? What kind of projects and what framework do we come up with?
Rachana Kumar: I’m looking with people at Etsy across the board. For me, we have not perfected it, we’re certainly just started figuring it out, how to scale. And there are a lot of other global teams that do that but I think Tina covered a little bit of this, right, what is our values that matters the most? Etsy is a pretty… other than specific skillsets like iOS and Android or DSML and things like that we are a pretty language agnostic company.
Rachana Kumar: We look at predominantly full stack and as long as you are interested and you’re a good engineer and the quality is great, other qualities like are you empathetic, are you kind of good at communicating when things don’t go well, all these basic things that are important for us here will be important irrespective of whatever the location is to build a high performing team, but also to bring all these people together.
Rachana Kumar:One thing I’m really trying to be mindful of is… because a culture has worked for us really well for over a decade, how do we be mindful of new cultures as we enter new countries and what is the best way to merge both of them and build teams together is what I’m trying to be more mindful about, talk more actively, yeah.
Sukrutha Bhadouria: Those are all great perspectives and it’s interesting how there is a lot of overlap no matter the size of the company and no matter where the company is expanding. But another aspect of building a high-performing team is getting the right leaders in place, whether it’s from a management standpoint or from IC leadership or technical leadership, right?
Sukrutha Bhadouria: There are times that you reach a crossroad where you’re like, “Should I invest in growing somebody to that leadership role, whether it’s a manager or a an IC, or should I be like hiring externally?” And different choices are made at different stages because building a team is literally like building a dynamic puzzle. You add something, it fits and then it changes the whole puzzle all over again, right?
Sukrutha Bhadouria: So, Elaine, what has been your experience in that whole aspect of choosing to hire versus grow when it comes to IC technical leadership roles? And when you when have you made the choice to do one versus another?
Elaine Zhao: All right, that’s always a a tough question to answer and leaders have to make that choice and decisions that are sometimes it’s external constraints. But what I also want to focus on, we have to look into, I like to focus on at that current stage of the my team and the company, what can we do and cannot do, right? At the current stage, I’m putting all the emphasis in hiring from outside. Part of that is when a huge growth phase, we need more people to the team and every time we need more people and it’s a perfect opportunity to bring in the right skill set into cultural add to the existing team.
Elaine Zhao: And also the other things that I want to highlight is a lot of time I participate a lot of conversations is about a building team and growing internally, we forget to think about whether or not the team itself, the company itself have the bandwidth, have the right mentorships and mentors in place to provide those mentorship to the ICs and all levels.
Elaine Zhao: I also want to focus on from hire outside there’s multiple things. However, of course, most the time we talk about hiring is hiring full time, but you can hire in fixed terms or short-term contractors, bringing the specific skill set to pair with your existing team to help them to fast-track the learning as well. You can also bring in specific skills in that way. Consultants is also a great way to do that.
Elaine Zhao: I cannot have emphasis enough about bringing in the external help to bring in those mentors only that is fair because a lot of time it’s not fair for the for the ICs and they want to learn but there’s no one for them to learn from. It becomes such a frustrating experience in my experience.
Sukrutha Bhadouria: Yeah and a parallel story is yours, Tina, where you were at the start of building your company, you have spoken about how you didn’t have management experience and then you had managers who didn’t necessarily have management experience as well. So how did you end up growing them in order to build the leaders that you needed?
Tina Huang: Yeah. I mean, I feel like Elaine is speaking exactly my language. I think that my experience is really for all of you who are out there building early stage startups, but I feel like one of the superpowers that women have is self-awareness and the willingness to kind of put your ego aside. So I was founder and CTO of the company but I knew very well that I had never been a manager, and as much as I’d like to think that I am an empathic person and I could learn that job, that’s very, very different than saying that I have had that experience and I place a high value on just domain experience that you can only really learn from doing that job.
Tina Huang: I saw early on, even though I really wanted the culture to sort of be grown internally and all that value from growing ICs into those leadership roles, well, I’ve seen just from observation a lot of my friends’ startups is they will grow their ICs internally, ignore the management problem, it’s always high risk to bring in your first VP of engineering, it could have a really, really catastrophic effect on your engineering culture, but the way they handle it is to just ignore that problem, right? So they are just growing their startup engineers into kind of quasi leadership, and then they hit a cliff, and all of a sudden there’s an emergency and a fire where you realize, we have a bunch of fairly… We just lacked that leadership experience, we needed it yesterday, and so then the company just gets external leadership and suddenly there’s this really, really strong culture shock to the startup culture.
Tina Huang: So we did exactly what Elaine’s talking about at a much larger org early, early on at Transposit. We hired an external contractor, this team, Bill and Amanda at this consultancy called Thrive Consultancy, and they worked with us and some of our ICs that were trying to grow into eng managers to sort of help understand our culture and build out that leadership internally.
Tina Huang: And I was very honest with the team, I said, “Look, if we grow at a fast enough rate, we will likely need to bring in outside leadership, but having consultants come in and help you all grow at least gives you all the opportunity. And that’s no guarantee that we won’t need that leadership, but at least you stand a chance. If we don’t bring in the consultants early on, we will inevitably have to bring in outside leadership and you will not have even had a chance to grow.”
Tina Huang: We were very fortunate that the internal eng managers, they had that hankering for mentorship, exactly what Elaine’s talking about. So they were really wanting that mentorship that I couldn’t provide for them, and at the same time, Bill over at Thrive that we’re consulting for, he likes to say, he got a little bit too close to the pond and just sort of slipped in. He fell so in love with our team and our culture that he agreed to come on as our VP of Engineering and really get deep into our team and our company.
Tina Huang: And that was really the best case scenario for us because we had the seniority of someone who honestly was too good for our company. He had previously been the VP of Engineering at Venmo and then was retired and we pulled him out of retirement to come work with us. So there was no culture shock there and we got a ton of great, great expertise added to our leadership team.
Sukrutha Bhadouria: That’s great. It’s always a win when you’re pulling someone out of retirement. Rachana, I’m curious to know your thoughts because IC leadership track versus the people manager track is something that people talk about in parallel but not enough of an emphasis on one versus another. So what are your thoughts on whether or not we should hire versus growth for leadership in general?
Rachana Kumar: Yeah I’ve not yet pulled anyone out of retirement so I don’t have Tina’s super powers, but yeah, I think it’s a balance, right? But to reach that balance, you need to reach a certain scale. After we reached a certain scale at Etsy, actually I can speak for my personal belief is if people want to get into management, that’s great, but that shouldn’t be the only way engineers can grow, right? And they’re very deliberate about having a very clear individual contributor growth track and a management growth track.
Rachana Kumar: And even within management, having said that, if someone is interested in becoming a manager, we have kind of process around that, they can try it for three months or they can interview for a role internally then it’s kind of like getting a new job within the company but there’s so much mentorship and support at this scale to make that happen for them.
Rachana Kumar: But also one way I kind of evaluate the question whether, let people grow internally or hire externally is… So at least with me and my directs or the whole manager… There are about 20 managers in my group, the cohort I think of as, say, between me and my directs what are the skill sets that we have significant gaps in right now, right?
Rachana Kumar: It might be technical. It might be industry experience. Or it might be just leadership experience. Like last year as a specific example, I hired a director of engineering for a initiative or a group within my larger group and there were already about four engineering managers and two of them were senior engineering managers and they both had absolute tremendous potential to grow into a director but between what my responsibility was and what they were doing, we could see there was a clear gap in terms of someone with significant management experience coming from the outside and being able to mentor both experienced managers and manage also more junior managers.
Rachana Kumar: And it was a hard conversation because the team had really talented managers within the group and so what was helpful was kind of laying out what are the gaps and can any of us fill that right now in talking about it actively.
Rachana Kumar: I feel like the conversation doesn’t happen enough similarly in the IC track until it’s very obvious that it’s a specialized skill set that we don’t have right now. And because our managers are also hiring managers in that sense, they understand when you sit down with them and I feel like because of that, at least most companies have seen, in the IC track, if they have a robust IC track, they end up promoting internally much more than hiring externally because we’re talking about technical skills here.
Rachana Kumar: In management, it’s easy to say someone at a director we need eight years for these reasons, so I feel like most companies, even if they have a clear IC track, end up promoting more internally.
Sukrutha Bhadouria: Yeah, that makes sense. And I mean all of your stories do resonate. I’ve worked in larger companies for most of my career. So now that we’ve talked about hiring and building, let’s talk about the dreaded f-word, firing. That is a really, really hard decision to make and it’s something everyone hates doing. It’s not enjoyable by any means, but it’s also not something to be afraid of. So, Tina, I’d love your perspective, let’s start with you on why it is an important skill for managers to learn?
Tina Huang: Yeah, this is something that came up recently, I think actually, as we were meeting up to talk about this panel here where, especially during the pandemic when we know that everyone’s struggling, it’s like how do you have hard conversations and then the most extreme of those is the conversation to terminate employment. And I was thinking about having to have those conversations as well as the follow-on, which is explaining to your team the necessity for these.
Tina Huang: I had previously always thought of the need for firing as part of–I think if you all have read the book, Netflix: No Rules Rules, the need to preserve talent density. And that always had a lot of resonance to me because, as an engineer, I really appreciated working on teams with very, very high talent density.
Tina Huang: But from the leadership perspective, it always struck me as almost a bit of arrogance, and so I sort of went on to a little bit of a philosophical journey to sort of reflect and think, why do I think that it’s so important or what’s another way of thinking about why it’s important to fire?
Tina Huang: One of the things that I realized is, for a lot of companies out there that I’ve seen, there’s a huge difficulty in firing and a huge sort of aversion to doing so. And so you put people on performance plans, you go through a whole rigamarole, oftentimes just transition them to another team. But what this actually ends up happening is this corollary on the hiring side.
Tina Huang: Because we’re so afraid of letting people go and oftentimes just because they aren’t a good fit for the role, not because they aren’t a smart, talented, wonderful person, that leads us to set these ideas on the hiring side which is, when in doubt, don’t hire. And this is something that was very, very common when I was working at Google, which at any hiring panel, if you weren’t 100% sure, you were told vote no on the hire. And so what does that mean?
Tina Huang: Well, if you have this notion that says when there’s any sort of doubt don’t hire, you’re going to look for high pedigree, people who are from known goods, go with your previous pattern matching, go for elite universities, for brand name tech companies, because it reduces doubt. And this is a way that you actually end up with a lot more bias in your hiring process because you’re unwilling to take a risk on anyone who adds a diversity to your company.
Tina Huang: So I started having this framework of thinking about what I told my engineers for how to think about hiring, which is this risk versus reward framework. Rather than thinking of just purely doubt, it’s like, we try to think about what is the risk of this hire but what is the upside that you can have by doing this hire and diversity of experience, diversity of thought, is a huge upside, in our perspective and so we’re willing to take risks if we see a potential hire has high upside but sometimes that risk comes at a cost.
Tina Huang: And so as an organization, I’ve been trying to coach my engineers to feel more okay about giving negative performance reviews when necessary because that’s the only way that we can continue a culture that allows us to take high risks and have a very, very diverse team.
Sukrutha Bhadouria: Yeah, these are really important thoughts and you see, when you get the chance, that people are resonating with what you’re saying and they’re adding their thoughts as well in the chat. I do want to take a little bit of a tangent because, Elaine, your stories are not so much about firing but when you have this mutual agreement that the path or the partnership that you had with somebody may not be the right one anymore, so not so much a firing but a polite handshake, let’s meet again, maybe.
Elaine Zhao: Yeah. No, absolutely right. Instead of just talking about firing and I prefer to use the word “part ways” with the staff that the core of the any of these issues is that working relationship no longer works, no longer right, but whether or not it’s performance related, it is skill set related, it’s the drive, the desire, the company wants to go to one direction, the employees career path go a different direction, whatever it is, we need to face that both sides need to face it and then treat it as fully grown adults.
Elaine Zhao: You don’t force a relationship nowadays if we all know it doesn’t work, right? People know and employees know it too. For us to so afraid to talk about performance-related firing or whatever reason that are no longer the right fit and just work around it, we don’t treat each other with the respect in a basic, we don’t believe that employees can understand that we try to treat them as a kids. And that is something that I fundamentally disagree with, we need to treat each other with respect that we’re fully grown adults. Let’s talk about it, right?
Elaine Zhao: What is the best way out? I have heard stories these days and the pandemic people challenge question about we shouldn’t fire people because due to performance because it’s pandemic or shouldn’t be a performance plans, whatever it is. And I think that is really not doing those employees the justice. Instead, we need to open the conversation. If the concerns about because the pandemic, financial related issues, there’s so many way that we can do it right with empathy see and I will take care of that.
Elaine Zhao: Let’s face, it what is the best way to help forward or not that’s a layoff situation, firing situation, a transition. And one last thing sort of talk about not talking about these topics objectively is when we’re avoid doing, make the tough decisions, we actually treated the best performers, our high performance, the worst. Those who deserve the most from the leaders, they end up received at least, right?
Elaine Zhao: When we’re kind of letting the low performance hanging around when there’s no longer the right relationship, right fit there, we actually treat our best performance worse. And guess what, we’re going to lose those top performers or we turn the top performance to mediocre performance, and I think that’s exactly the opposite that we wanted to see, right?
Sukrutha Bhadouria: Yeah, and I mean oftentimes letting go of someone feels like a difficult decision not just because of how they’re doing but because of how that can cause an impact on the project deliverable. And, Tina, you spoke about encouraging your employees to not be afraid to give negative feedback so I want to go to you now, Rachana, because you were that person who wasn’t afraid to give that feedback and then you saw a difficult decision being made based on that. So do you want to go into that?
Rachana Kumar: Yeah sure. I think both Elaine and Tina made great points. At any given point in time, letting go of anyone is a hard decision. But for me, this was very early on in my career, I was a tech lead and we were working on a project that had like crazy tight deadlines and my manager said, “Can you peer program with another engineer and what if you try to finish something we were struggling with as soon as possible?”
Rachana Kumar: And I went and sat next to him and we were peer programming and I made a suggestion on how we both can best… A technical suggestion for whatever we were working on and he turned towards me and he basically said, “I don’t want a woman telling me how to code. I know how to do my job well.”
Rachana Kumar: And it of course it made it very challenging for me to continue peer programming at this given point in time. I just got up and I realized I had to tell this to my manager because this expectation was we are peer programming and we are going to finish it and after a comment like that, continuing peer programming and I was also new to the US and I said person new to a country and understand still understanding the culture and all those things I was very afraid to say anything. But I felt like I had to tell my manager what he said even because I had no idea what the outcomes were and I was certainly very nervous. I just went and told my manager. “I don’t think we can continue to peer program because this is what he told me.” And my manager actually let him go immediately.
Rachana Kumar: And for me, I have been manager for over a decade but the learning from that incident was… That put the project at risk and the team’s execution at risk but he chose not creating a toxic work environment, especially for minorities, over the project outcomes, which for me was a really good learning experience as a leader and that’s something I always keep in mind because it’s not just about what business and customer outcomes you’re driving, it’s about the culture you’re driving, and it’s like, even if someone sometimes a high performer, if they’re toxic to the culture, it’s a really hard decision to make but it’s an important one.
Sukrutha Bhadouria: Yeah, that’s a very brave story and I’m glad it ended up in a positive outcome and not just for you but the lesson and the message that was being sent that a toxic environment is absolutely not okay by any means. So now we’ve talked about hiring externally versus growing people versus identifying when it’s time to part ways. We all are very aware that the traditional tech promotion systems, we haven’t figured it out. So let’s talk about the problems with it, right? So what are some of the challenges we see with how it is today? Elaine, let’s start with you.
Elaine Zhao: Talking about getting the opportunity for us, I think couple of things here is, first thing first, you have to, my experience, that have to focus on the opportunity in front of me, which is my current job, right, and actually do a good job in it and I know there’s a lot to talk these days about people not getting noticed but I can tell you if you’re not doing a good job you absolutely would not get noticed. All right. So focus on that one first.
Elaine Zhao: And the other area is really I learned my lesson that I really set the expectations with my manager, I also recognized a very fortunate that I have earlier my career I have one leader, manager hired me four times at four different companies so he become my mentor. The key thing is to really understand the manager’s perspective, and in my case, what he sees I should focus on and go from there. Now there are situations which I also experience my career, especially later in my career, when I get in the more upper management level. It’s just a misalignment.
Elaine Zhao: At that time though, I got bigger responsibility that my focus is actually to talk to peers and talk to different folks within the company to understand a higher level, not just engineering, not just my team, but the overall challenges are other leaders facing and then see what I can help whether or not a [inaudible] standpoint or just technology strategy standpoint that I can meet my team and collaborate with them and that is actually really for me have been served me really well, because the ultimate goal is not just whether or not me getting noticed or do a good job is whether or not we solve a problem, solve our customer’s problem, and I think that’s the most important thing that we need to focus on, yeah.
Sukrutha Bhadouria: Yeah. And for you, Tina, what do you think are the challenges with the current situation, in terms of how we have the traditional promotion expectations that we have in the tech industry?
Tina Huang: Yeah, so one of the things I noticed from my experience, primarily at Twitter, was I realized that a lot of times one of the challenges is we think a lot of aspects of technical, what was on the tech ladder, is highly quantitative when it really is fairly subjective. And the example that I often give is around code quality.
Tina Huang: Code quality and even the fact that we use it as a metric makes it sound like it is a quantifiably measurable thing that you can look at someone’s code. When in practice, if you think about it, code quality is highly subjective depending on what was that purpose for the code. If the code was to try to get a prototype out or meet a very, very aggressive customer deadline, the best quality code is the code that will actually ship on time. Whereas, if it’s part of a long-term project, the right quality of code might be higher performance or higher reliability, if those are aspects of the system that you really care about. Similarly, when you talk about something like code quality, there’s also, I almost say a historical lens that people try to put towards it, where they evaluate the code without any sort of thinking around what was the context that the code was written in?
Tina Huang: So I was at Twitter in the very, very early days and so there would be questions around, “Why was this code not written on top of this other library?” And no one bothered to look at it and say, “Oh, that library didn’t even exist when we actually wrote that code.” And so one of the things that I try to do very differently at Transposit is I try to think about, set very good high level value metrics that the company should be driving for, especially on the customer and product side and rooted in that aspect of pragmatism that I talked about earlier, and then use that to sort of drive the metrics. What are the product deliverables, the guidelines, et cetera, and then have that sort of cascade into what is the appropriate quality of code to evaluate on?
Sukrutha Bhadouria: That’s awesome. Yeah. I mean, I don’t think people are really realizing how difficult it is when it’s subjective or not, right, in terms of evaluation. So I’m curious now next, I mean, Tina, you’ve had an amazing career, all of you have, and not everybody has had opportunities handed to them. Sometimes we’ve had to make it for that for ourselves. So how have you done that for yourself, Tina?
Tina Huang: Yeah, one of the things that I hear a lot from various engineers is they come to me and they say, “Hey, it’s not my fault, I’m not given the right projects in order to hit that next technical milestone, to get the leadership or the technical kind of benchmarks necessary for that promotion.” I often talk about how when given a project, you can take it very, very literally and just execute on that project or you can try to think bigger picture around what are the actual goals and the necessities there?
Tina Huang: I think people often look at my experience and they could say, “Hey, Tina, you became a CTO by creating your own company that feels highly, highly unapproachable to me.” I often like to talk about some of my experiences navigating larger companies at Google or Twitter to sort of better ground that into real world examples or examples that feel a little bit more approachable.
Tina Huang: One example was at Google, we were asked to do a front-end redesign. And so we were given some mocks and the obvious solution would be, let me just write a new front-end for Google News that executes against these mocks. Instead, I looked at the actual architecture and what it would take to build that. And at Google, believe it or not, back in the day, for all of their search infrastructure, including Google News, the front-ends were written in C++.
Tina Huang: And I said, “Hey, this is highly unscalable to have and even hire for engineers that both know C++ and also know Javascript and to be able to build a modern web front end with all of these bells and whistles there. So why don’t we take an approach that actually shards that old C++ front-end and make it into an API server and build a Java front-end on top of it?” Because we had a lot of Java expertise at the company, because all of the Google Apps were built on top of Java.
Tina Huang: And so I was a fairly junior to mid-level engineer at the time but I championed this to a bunch of the senior technical team members and pushed forward to actually re-architect the system. And that’s how you take a project that is, on face value, not technically sophisticated and actually turn it into something that is worthy of a promotion.
Sukrutha Bhadouria: Yeah, it’s really inspiring to hear how one shouldn’t just stop at what’s not in front of them. All right, so on that note, do you, any of you have any closing thoughts you’d like to share, Rachana, let’s start with you. What are your thoughts you’d like to close with in a pandemic situation for everyone?
Rachana Kumar: Yeah, I would just add to what Tina said because I think that’s so interesting about making opportunities for yourself and even for me, throughout my career, it’s been like kind of looking for things that no one is in my peer group, there are obvious problems, but either it’s not glamorous or it’s too risky and no one wants to own and kind of seeing…
Rachana Kumar: Because I know it’s an important problem but no one thinks it’s glamorous enough or exciting enough, owning those and working on them it’s been how I’ve created opportunity and that ties to also the promotion conversation. A lot of my promotions have come because I found a pattern and a problem, either small or big, depending on the stage of my career, and kind of paid attention to it.
Rachana Kumar: During the pandemic, with everything from child care to everyone going remote suddenly, how do you make mind space for that has been really challenging for me and the Mexico job was an interesting one. When they offered me that, it’s again a risky one, we had to move there so in the next few months and things like that. But I’m trying to still keep an open mind to what has made my career successful even during the pandemic, even though it’s hard.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Sukrutha Bhadouria: Tracy’s a senior staff engineer at Slack. Prior to Slack, Tracy worked at Adobe for many, many years, specializing in client networking and support for streaming audio, video, and screen sharing. Tracy lives in San Francisco with her husband and two kids. Welcome, Tracy.
Tracy Stampfli: Yes. My name is Tracy Stampfli. I’m a senior staff engineer at Slack. I lead the iOS infrastructure team. iOS infrastructure basically handles things like networking, and data syncing, and all the stuff that’s a bit under the hood of the app, supporting the UI and the features.
Tracy Stampfli: I’m going to talk about modernizing mobile codebases, but I’m really going to talk about modernizing and refactoring codebases in general. I hope that a lot of this talk is actually generally applicable and not just about mobile.
Tracy Stampfli: I’m going to talk about some things we’ve learned from doing a rewrite, or at least a partial rewrite, of both our iOS and Android codebases at Slack. Why did we think this was necessary? What decisions did we make about how to rearchitect our code? What has made this successful, at least so far? It’s still in progress.
Tracy Stampfli: Why did we decide to do this? Well, Slack’s mobile codebases had not really had a big rewrite or refactor in a long time. Slack’s been around for about seven years, and on iOS, we hadn’t actually done a refactor in that entire time, or a major refactor. Android had done one about five years ago, but that’s still a while.
Tracy Stampfli: Both of these codebases had a lot of tech debt. They had a lot of obsolete design patterns that didn’t match up with how we do things now. There were a lot of inconsistencies, you know, five different ways to do the same thing.
Tracy Stampfli: That made it really confusing for new engineers and hard to onboard, because it was hard to figure out what the right way to do things was. The code was fragile and too tightly coupled, which made it easy to inject bugs.
Tracy Stampfli: All of these things added up to slow down feature development. That became a really big problem. Mobile development was starting to be a drag on feature development in general at Slack. That was something we knew that we had to tackle and do something about.
Tracy Stampfli: We decided we really needed to prioritize addressing this tech debt, and invest in more modern architecture, and improve our development practices to speed things up.
Tracy Stampfli: Where deciding how to do this, there are a number of options. We’re going to do a big refactor, so how are we going to do this? We considered a few things. We could do just a full rewrite of our mobile apps. There’s some nice aspects of that. You get to just toss out the legacy code and start afresh, new project, just start going with new patterns, do it the right way that you’d like to rewrite all the features.
Tracy Stampfli: But obviously, there’s some risks with this as well. If you’re going to start from scratch and just fully rewrite the app, you have to bring it all the way up to feature parity with your existing code. For a product like Slack, that’s been around for a long time and has a lot of features, that’s going to be a really massive task.
Tracy Stampfli: While you’re doing that, you have to maintain two codebases. You have to keep the new codebase … There’s going to be feature development going on, so you have to be doing dual feature development in the old codebase and the new codebase.
Tracy Stampfli: Overall, this is just a very big, risky bet. What happens if you don’t ship that new, modern codebase for some reason? You’ve wasted a huge amount of work. We decided this was a bit too risky for us.
Tracy Stampfli: We considered some other options. When you talk about speeding up mobile development, one option that comes up a lot is sharing code. There’s a number of ways to do this. We could share code between iOS and Android. We could share code between the mobile apps and the desktop. There’s different frameworks for sharing code. You can share UI code or business logic. Slack actually tried to do some shared business logic a few years ago across the clients. It didn’t actually work out too well.
Tracy Stampfli: There are, again, some benefits here. Obviously, there’s this benefit of not rewriting the same feature or the same logic three times, or however many clients you have. You can also, hopefully, maybe share some developer resources across platforms if you go this route.
Tracy Stampfli: But there’s also, again, downsides. Shared code complicates your tooling. It makes things like just debugging, building the CI system … All of those things become more complicated. Also, the big danger is that, if you share code, you might lose native look and feel or native performance. This is a big issue for mobile.
Tracy Stampfli: We really want to take advantage of the latest and greatest features that each of the platforms has to offer. We want our iOS apps to feel like great native iOS apps, and same with Android. We want performance to be great on our mobile platforms. This was something that we were definitely worried about.
Tracy Stampfli: Also developer sentiment about this idea was just not very good. Our developers were just not excited about doing shared code. If the developers aren’t excited about it, it’s unlikely to be successful.
Tracy Stampfli: What’s left, basically? Well, you could refactor your existing codebase in place, essentially rebuild the airplane as you’re flying it. This is the option we ended up going with. There’s some, again, benefits and downsides to it.
Tracy Stampfli: Benefit. Reduced risk. We were basically having to refactor and ship this codebase continually. We just have one codebase, so at any given point in time, it always has to be shippable, because we have to keep releasing it.
Tracy Stampfli: We get faster payoff, because as we’re modernizing that codebase, it is continually being improved, and everyone working on the team is getting the benefit of that improved codebase as we go along. We don’t have to do this dual codebases, try and develop things in both thing, which seems very challenging.
Tracy Stampfli: But it does mean that we don’t get to just get rid of our legacy code. We have to actually deal with it. We can’t get rid of the tech debt. We have to migrate and rearchitect our existing codebase.
Tracy Stampfli: We launched this thing called Project Duplo. You may have noticed, there’s a Lego Duplo theme to this presentation. That’s because, as I’ll talk about a bit more later, one of the big themes of Project Duplo was modularization, or breaking down the codebase into smaller building blocks. Duplo are the larger Lego bricks, so yes, basically a big theme about breaking down the app monolith.
Tracy Stampfli: This was a big rearchitecture of both of our mobile code basis that we launched a little while ago. It was coordinated across both mobile platforms. iOS and Android came up with proposals together, did an investigation together, scoped this effort together. We’re now running the project together. This was all a coordinated effort.
Tracy Stampfli: The goals were, number one, improved developer velocity. Start shipping features faster. But also adopt modern design patterns, bring some of the patterns in our app to more in keeping with current mobile development practices.
Tracy Stampfli: And really enable larger teams. Obviously, the mobile teams of Slack have grown a great deal since the company started. We hope they’re going to grow a bunch more. The patterns that work for smaller development teams don’t necessarily work for a team of 40, and what works for a team of 40 may not work for a team of a hundred. So we really wanted to enable our growing development team.
Tracy Stampfli: And also set us up to adopt future technologies we might be interested in. Like right now, we aren’t actually using SwiftUI, but we might want to in the future. We want to have an architecture that keeps that door open, and allows us to have that possibility.
Tracy Stampfli: This project had three phases. The first phase we called stabilization. The idea here was to complete a bunch of ongoing migrations and refactors that we already had in flight, things that we’d started, but then didn’t have the resources to finish. Again, that was really leading to inconsistency, so we wanted to remove the worst of the tech debt, the worst of the anti-patterns, and just clean the codebase up to set us up for the rest of the project. This phase had very well-defined work streams and really clear metrics for success, so we really could track our work very well.
Tracy Stampfli: I’m not going to get too deep into the details of what we actually did on each platform, due to lack of time. But this is just some highlights of some of the top goals on each platform.
Tracy Stampfli: On iOS, we wanted to move to being 100% Swift. We were already about 80% before this project started. And we wanted to finish some migrations onto some of our infra frameworks. Similarly on Android, you’ll see these common themes of finishing migrations, finishing adoptions of different patterns and different frameworks, and breaking down some of the infra frameworks to be more usable.
Tracy Stampfli: The second phase here was modularization. Here, we’re getting into the building blocks. The idea here is that we had already somewhat modularized our code. We had on both iOS and Android. We had some code that was split off into frameworks, but we wanted to go a lot further in that direction, because there’s a lot of benefits to this.
Tracy Stampfli: By breaking the code apart into smaller frameworks, and breaking up that big app target, you reduce interdependencies, you remove this tight coupling between features that can make things more fragile and introduce bugs. You enable …
Tracy Stampfli: You have to have separation concerns, because you’re actually separating the code out. This enables developers to work better, independently from each other, and also improves the build times, because if you’re building your feature in a framework, as you make changes, you only have to rebuild that framework and not the entire app target.
Tracy Stampfli: The third phase was modernization. This is the one where, again, we’re really trying to look forward and think about what architecture patterns do we want to adopt that will make us compatible with current industry trends, but also will set us up for the next five years of app development.
Tracy Stampfli: We got into this project. As we got started and were going through the stabilization phase, we realized that actually this whole three phases thing wasn’t such a great idea. We realized that we should actually combine our three phases back down into two, and combine the modularization and modernization phases, because we realized that, if we’re doing these separately, you would refactor some code to split it out from the app, then modularize it, and then you would refactor it again to modernize it. That was going to entail a lot of wasted effort. To avoid refactoring things twice, we combined these phases and just made it a two phase project. So now it’s just stabilization and modernization.
Tracy Stampfli: As we were deciding what we were going to do for our modernization, and trying to decide what patterns we should pick, we did a lot of research with our developers, talking to our developers, doing focus groups, doing polling, trying to figure out what are the current pain points, and what did developers want to see out of this project. What did they really want us to do?
Tracy Stampfli: What we heard from them was interesting, because we didn’t hear that they felt super strongly about one particular feature architecture, or exactly how we did dependency injection. What we heard from them was that they wanted us to increase what you see here, CPR is the acronym we came up with, consistency, predictability, reliability. That was what was important to them, improving these things.
Tracy Stampfli: Consistency. Making it so that all the features all are built in the same way. So it’s easy to tell, if you look at another feature, that is built the same way the other ones are, the ones that you have built. So you’re familiar with the code. It all works the same way.
Tracy Stampfli: Predictability. Making things like routing and deep linking understandable. Making sure that the code actually works the way you think it’s going to work. Removing unintended effects where making a change somewhere in the product for some reason breaks something somewhere else.
Tracy Stampfli: Reliability. Again, making the code less fragile, so we’re spending less time on regressions and incidents and things like that.
Tracy Stampfli: So this was really what we wanted to focus on. How do you do that? Well, we realized that if we wanted to increase consistency, we couldn’t just do that by coming up with some feature architecture pattern and expecting every developer in the team to implement it exactly the same way. You can’t just hope that it’ll work.
Tracy Stampfli: You have to enforce these things through things like templates, and linting, and code generation, and basically making it very easy for developers to know what the right thing is to do, and to do that, and much harder for them to do things that we don’t want to do, and re-introduce inconsistencies and anti-patterns. If we’ve decided we don’t want developers to use singletons, we need to add a linting rule that actually prevents that, versus just hoping that they won’t.
Tracy Stampfli: What did we end up deciding to as our main goals for modernization? Again, not going to get very deeply into this, but on iOS, this big push to break down the app into, again, service and feature modules. So increased modularization. We did decide to adopt a new feature architecture that’s VIPER-like. We’re adopting Combine.
Tracy Stampfli: This last one is actually pretty important. We’re switching to using Bazel, which I don’t know if folks are familiar with it, but Bazel is a build system that you can use in place of Xcode or alongside Xcode. It deals better with highly modularized projects, projects where there is many, many frameworks. Bazel handles that pretty well. It also has a build cache, which means, hopefully, again, we’re going to see improved build times, both locally and in CI, by having a better caching of build artifacts.
Tracy Stampfli: On Android, again, similar themes. Completing modularization on that side as well. Adopting some new frameworks and libraries to do things like networking, and JSON parsing, and configuration changes. And building abstractions around startActivity and onActivityResult.
Tracy Stampfli: Why has this project been successful, or has it been successful? Well, we started off with the stabilization phase. All goals of that phase were completed on time for both platforms, 100% completed everything. So that was certainly successful.
Tracy Stampfli: We are now in the middle of the second phase, the modernization phase. That’s still in progress, so there’s a bit of a caveat here. We’re on track, but there’s still a ways to go. But we’re successful so far.
Tracy Stampfli: Why has this been successful? Number one, it was an engineering-led initiative with executive prioritization and resourcing. I think both of these things were really important and really key to the success.
Tracy Stampfli: This has really been an engineering-driven, IC-driven driven project. It was engineers who came up with the initial proposals, who did the research, who figured out what the problems were and how that we should solve them, and who came up with all of the scoping and projects and proposals that have led to what we were actually doing as part of the project. So very much not a top-down project.
Tracy Stampfli: It was very much driven by engineers. But it had executive prioritization and resourcing, which is equally key. This isn’t something that you can do as a side project. It really needs dedicated resources. It needs executives to sign onto the fact that we’re going to do this in place of doing something else, in place of doing feature work or whatever. So we have to have that buy in that this is the right thing to do, and we’re going to get the resources to do it properly.
Tracy Stampfli: Along with that, sponsorship from key engineering leaders outside of mobile helped us get that buy in. We had principal engineers and key managers who were willing to say, “Yes. This is super important. This is the right thing to do. We need to do it now.” That helped us get this backing that we got to actually do the project correctly.
Tracy Stampfli: Additionally, splitting the project into a couple of phases was really helpful, because having this initial stabilization phase, we just knew what we had to do. We were getting rid of tech debt. Things were really well-defined. We just had to execute on it. That gave us time to do R&D for the later phase.
Tracy Stampfli: So during the stabilization phase, we were able to do the investigation, and the prototyping, and figure out what should we do for modernization, and make sure that we had all of those things set up, and all of our scoping and all of that by the time we hit the modernization phase.
Tracy Stampfli: Finally … This one is very, very important. This has been a very metric and data-driven effort. We really wanted to have very clear metrics to measure our progress, what we’re doing, how successful we’re being. It wasn’t just tracking with Jira or whatever. We were actually running scripts on every file in the codebase to track, how are we doing with these migrations, what progress have we made, where are we. We had a lot of dashboards and graphs.
Tracy Stampfli: This is an example of one of … For each of the goals that we had, we would have a dashboard saying where are we with this in terms of progress on this measure. That allowed us to do things like see if we were falling behind or we’re doing fine. We could switch resources between different work streams if we needed to, if one of them wasn’t doing as well.
Tracy Stampfli: On this particular graph, you could see it doesn’t quite go to zero at the end. That was because we had ported the code to Swift, but some of it was still behind feature flags, as we were rolling out those feature flags when this snapshot was taken.
Tracy Stampfli: But basically, this was very, very helpful for the initial phase of the project. We are continuing that in the modernization phase. We’re really trying to come up with very clear metrics for each of our goals, so that we can track it and see that we are being successful. That enables us to do resourcing planning and all sorts of things that are very, very helpful for the success of the project.
Tracy Stampfli: That is the end of my presentation. If you have … I don’t think we’re going to have time for questions, but feel free to reach out to me. I’ll try to respond in the chat if folks have questions. Feel free to connect with me on LinkedIn if you want.
Tracy Stampfli: Also, as other folks have mentioned, Slack is also hiring. We’re hiring on the iOS team. We’re hiring on Android. We’re hiring on other teams as well. If you’re interested, please check out the careers page at Slack.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Angie Chang: Our next speaker is Nicole. She is a product manager at Zumba Fitness, where she’s leading product growth and instructor education in Miami, Florida. Before joining Zumba, she was in product at Everlywell, The Knot Worldwide, and Box. She’s passionate about creating products that solve important problems for people and makes a positive impact in their lives. Welcome, Nicole.
Nicole Salzman Page: Thank you, Angie. And thank you so much for hosting this event, to the whole Girl Geek team, and for everyone too for being here. It’s been really inspiring so far with all of the amazing speakers so I’m definitely honored to be among this group. Let me present.
Nicole Salzman Page: Again, thank you for having me. My name is Nicole Salzman Page, and I’m currently a product manager at Zumba. Today I want to talk to you about remote communications strategy for working from home and beyond. This talk is for you if you’ve ever been standing in front of your company at an all-hands meeting, talking about an exciting new game-changing product you’re releasing for your customers, only to look out and see a sea of blank faces and, potentially, some people looking at their phone.
Nicole Salzman Page: Or, this talk is for you if you’ve ever led a go-to-market meeting for a new product launch, feeling like everyone knew exactly what to do next when they left the room, only to find a few days later that no one knew what to do next. And I want you to walk away today with the tools you need to ensure that that doesn’t happen and to communicate like a pro. And I know what it takes to communicate like a pro because I was a communications professional.
Nicole Salzman Page: Before I moved into tech, I was a communications professional, working in a variety of industries from a public utility company doing crisis communications to a nonprofit children’s hospital, where I led communications with a number of our donor groups. And when I moved into tech, I quickly realized that the strategies and frameworks we used in communications were also essential for technology teams. And I worked in product for a variety of technology teams, in companies with different products, team sizes, and customer groups, from enterprise software at Box to growth at Zumba.
Nicole Salzman Page: And I found that these communications strategies that we used as communications professionals really apply globally across industries and team sizes.
Nicole Salzman Page: Communication was important when many of us were in an office, seeing each other face to face and communicating in person all the time. But it is critical while we’re working at home remotely, and you don’t need to be a communications professional. You don’t need a communications background in order to communicate effectively. You just need a little bit of prep and a thoughtful strategy.
Nicole Salzman Page: I want you to walk away today from this session with a framework to be able to do this effectively in your own organization, both while we’re working from home, but also when you go back to the office, if you go back to the office. Being an effective communicator will help you in whatever role you’re in. As long as you work with people, having a communications strategy will enhance your ability to be effective.
Nicole Salzman Page: The strategy has five parts. I’ll walk you through them here and then deep dive. The first is to tailor the information. This is all about thinking about who the information is for. The second is to cut it for time. Consider what amount of time the person who’s on the receiving end of your communication has to process that information and make a decision. The third is to make it easy to find. Make sure that your stakeholders knows exactly where they can find it. The fourth is to make timing predictable. Make sure they know when they can expect it. And then the fifth is to automate it. Make sure that your stakeholders know exactly how it will be delivered. I encourage you to think about this as a five-part strategy. When you go to communicate to your stakeholders, think about all five of these things in order to make sure that you have that effective connection.
Nicole Salzman Page: The first one, tailoring the information, thinking about who the information is for. This is all about making sure that you’re giving the right information to the person who’s receiving it. For example, let’s say I’m a product manager, and I am releasing a new feature. As the product manager for this feature, I have a whole bunch of things in my head. I have the launch plan. I have this strategy, how it’s making money, what the goals are, how we’re going to measure success, how we’re going to test it, how are we going to tell people about it, how it works. And I have all of these things in my head because I work on it every single day. I’m in this all day, every day.
Nicole Salzman Page: What I need to understand to be an effective communicator is that when I communicate this information out, the people who I’m communicating to are not going to be in the same situation as me. In order to tailor the information for the right group, think about that group and what information they need. For example, going back to that all-hands example from the beginning, if I were to do a launch announcement of this new feature at an all-hands, perhaps what information might be most helpful for that group could potentially only be the goal and how we’re going to tell people about it. Those might be the two pieces that are going to get the company excited to rally behind this product without actually having to tell them about every other single thing about the product, because that’s where you’re going to start to get those blank stares.
Nicole Salzman Page: Another example, same product, same feature launch. Let’s say I’m taking this to an executive update. The executives definitely aren’t thinking about your one product all day, every day. They might be thinking about five different products. They might have heard about five different new projects going on that day. So in order to bring them the most essential information from them, they might only need to know about the goal and how we’re going to measure success.
Nicole Salzman Page: My last example on this point is go-to-market planning. This is where you might have the customer success team, your customer care team, your marketing team, your product marketing team. And in order for that meeting to be successful, the information that you might need to bring to that meeting could potentially just be the launch plan, the timeline, and how it works.
Nicole Salzman Page: Think about the audience and what information they need, and pull that information out of your knowledge base to provide to them. It seems simple, but it does take practice to make sure you’re providing that right information for your stakeholders so that they can take the information and do what they need to do in their jobs with it.
Nicole Salzman Page: For example, the executives, they’re going to be needing to make some decisions on the information, potentially at a high level, whereas your go-to-market team might need that information to go and do their jobs, like creating marketing materials.
Nicole Salzman Page: The second one, and this goes hand in hand with tailoring the information. This is cut it for time. Think about what amount of information your stakeholders need. Once you have the right amount of information, this one really has to do with how much time the person on the receiving end of the communication has to process and take in that information. And this is going to depend on who the team is.
Nicole Salzman Page: For example, engineering team. As a product manager, I will work with the engineering team, often in really big chunks of time, like a four-hour sprint planning or potentially multiple sprint planning sessions. In this case, the engineers might need about 80% of the information that’s available. They’re going to need a lot of information. They’re making a lot of detailed decisions based on the information that I have about the product. So we need to make sure that we’re aligned and very closely in step, with a really big chunk of that information.
Nicole Salzman Page: Next, a business stakeholder or project sponsor. They might, just similar to the engineering team, who might be dedicated to the product just like I am as a product manager, potentially a business stakeholder or project sponsor has a small handful of projects that they’re responsible for making hiring decisions on, resourcing decisions on. So they do need a good amount of information, but potentially not as much as the engineers need. In this example, the business project stakeholder might only need about 40% of the information.
Nicole Salzman Page: And lastly, going back to that executive update, they might only have just a few minutes to read an email or hear the project update in a meeting. In this case, the executives potentially only need about 5% of the information. And I call this two sides of the same coin because when you think about cutting for time, you can’t really just take one communication and chop off different pieces of it, depending who you’re speaking with. It’s really important to make sure that you’re not just choosing the right amount of information, but the information that you choose is really the right information for that person that you’re communicating with.
Nicole Salzman Page: The third one is to make it easy to find. Make sure your stakeholders know exactly where they can find the information. If you’re like me, I’ve definitely spent hours before, trying to find information, using all kinds of advanced Google features, Google Gmail searches to try to find some data that my BI team sent me a few months ago, or maybe it was weeks ago.
Nicole Salzman Page: Your stakeholders are probably doing the same thing. So make sure that when you send out an update, you figure out exactly what the best place is for your stakeholders, and stick to that place. It might take a little bit of trial and error. You might need to do some research and figure out where your stakeholders will be.
Nicole Salzman Page: For example, maybe your executives like to work in email, and they like to open up PDFs from an email. Maybe your business stakeholders or project sponsors like to be in Confluence and like to open up links to a team workspace like Confluence. Or potentially, you are working with engineers, and you’re in Slack, and you’re day-to day talking back and forth in Slack.
Nicole Salzman Page: So wherever it is that you find that your stakeholders are going to be, make sure that you pick that place, and put it in the right spot. Put it in that spot every time so that way, when your stakeholders are thinking about where it is that they saw that update from you, they know exactly where to go to find it. They know that they can go find it in Confluence on a certain page, or in their email with a certain subject line.
Nicole Salzman Page: The next one is to make timing predictable. Make sure that your stakeholders know when they can expect it. This will help make sure that stakeholders aren’t chasing information down throughout the day. Make sure that there’s fewer emails being sent and efforts to try to find the information. They know exactly when they can expect it.
Nicole Salzman Page: And to do this, I would say to pick the best day to send your communications. And this is going to depend on what your communication is. For example, if you have a communication where you’re doing a wrap-up of the week, of the week prior, and your stakeholders need to act on this information to do their job throughout the week, this might be one where you send out an update on Mondays. For me, I wrap up information from the week before, related to traffic and users on the site. So, I send my updates on Mondays.
Nicole Salzman Page: Similarly, if you’re sending updates that wrap up the week, but your stakeholders don’t necessarily need to act on it right away because a lot of us have offices closed on the weekends, this might be a communication that you send out, potentially, at the end of the day on Friday. You can wrap up the week, give any updates, status updates, talk about next steps that are going to come up the next week.
Nicole Salzman Page: You also might have communication that turns over more quickly throughout the week. Maybe you have some data that moves really quickly throughout the week, that is going to be changing, and that your stakeholders that receive the communication need to act on multiple times throughout the week. You might choose two different days to send this communication out.
Nicole Salzman Page: And the last piece of this communication strategy is to automate it. Think about how it will be delivered. I’m sure many of you are familiar with Google Sheets.
Nicole Salzman Page: You can use Google Sheets as a way to save you time to put together these updates and these communications, especially if you’re new to having a communication strategy in your role. This might take a lot of setup on the front end. It might take you some time to figure out how you’re going to send out your communications, what the best place that it’s going to be. But all of that work will pay off as you start to see more alignment, decisions being made faster, products being released faster.
Nicole Salzman Page: Anywhere where you can automate, once you have this all set up, will help you as a product manager or whatever role you’re in that requires good communication, move a little bit more quickly. In the Google Sheets example, if you are grabbing data from multiple sources, you can actually port data from multiple sources into Google Sheets and manipulate the data there and send it out from there.
Nicole Salzman Page: Maybe you get traffic information from Google Analytics. Maybe you get sales information from a database that you use. You can get information from multiple sources and port it into Sheets so that you can use it in one spot.
Nicole Salzman Page: You can even go one step further and use an add-on called Email Spreadsheets. Email Spreadsheets will allow you to put together an automated spreadsheet, where you can pick any of the tabs in your sheet and automatically send it out on a certain cadence. For example, if you send out trend charts of conversion or registrations or anything you would need to show, you can take the trend chart as a PDF and automatically send it out to your stakeholders on a regular basis.
Nicole Salzman Page: And if you’re not comfortable with sending something automatically from Google Sheets directly to your stakeholders, you can even send it to yourself, and then forward it on to your stakeholders with your own commentary. So, lots of options.
Nicole Salzman Page: I encourage you to check out the different ways that you can create these efficiencies, because they’ll hold you accountable to those communications, and will help you keep them on a regular cadence, and help you keep up with them.
Nicole Salzman Page: And a quick note here is we really focused in this talk on one-way communication, giving updates on projects, aligning stakeholders to do things like product releases or feature launches. But there’s also two-way communication that you can create a communication strategy around. For example, project retros, issue postmortems, design sprints. There’s a lot of different two-way communication that you can use these principles to put a thoughtful strategy behind.
Nicole Salzman Page: The communications toolkit that I want to leave you here with today: tailor the information, make sure that you really think about who it’s for, cut it for time, consider what amount of information your stakeholders need, make it easy to find, make sure your stakeholders know exactly where they can find it, make the timing predictable, make sure your stakeholders know when they can expect it, and automate it. Save yourself time by thinking about the best way for it to be delivered.
Nicole Salzman Page: For me, I’ve seen this framework result in fewer emails, less of a need for constant one-off check-ins, more alignment, and the ability to get decisions made faster in order to make products, in order to release products more quickly and with higher quality. I hope that that works for you and your organization as well. I encourage you to bring this new communications strategy to your organization, and try it out. And please feel free to reach out with any questions. And if you try this in your own organization, I’d love to hear how it’s going for you so feel free to connect. I would love to hear from you. And thank you again.
Sukrutha Bhadouria: Thank you, Nicole. This was a great presentation and a lot of great takeaways for all of us to implement, especially with Google Sheets step. That was new for me and new for a lot of people who are a lot of our audience, who’ve been commenting in the chat.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Sukrutha Bhadouria: Hema is the Director of Technology Advocacy with AI for Good at Intel. She’s involved with Intel’s pandemic response technology initiative to collaborate with customers and partners to address COVID-19 challenges. This is a really important effort. So thank you, Hema. She has a passion for technology and for the role of AI in addressing issues in the healthcare and the life sciences sector. Welcome, Hema.
Hema Chamraj: Thank you. Thank you, Sukrutha. Super. Hey, I’m really excited to be here to talk about this topic, Responsible AI for Healthcare. Before I talk about responsible AI, I want to go first into health care and then I’ll talk about AI for healthcare and then I’ll get to responsible AI for healthcare. So as you know, all of us here as patients, as consumers of healthcare, this is something that we all understand very well.
Hema Chamraj: Our health care system is really not in a great shape. As a nation, we spend the most, almost 20% of GDP on our healthcare system, but we have one of the largest sickness burdens. If you think about, at least nearly 50% of our population has at least one chronic condition. And also life expectancy also is lesser than Canada and Japan.
Hema Chamraj: So we are not doing that great. And as we age and as a population we’re aging, and so as we age it compounds the problem. And so what we ended up is with more sickness and more costs, and one of the things also that is not helping is the fact that we’ll have shortage of clinicians. So we’ll have less people to help us take care of ourselves.
Hema Chamraj: And so this is the state of healthcare we are in when it comes to cost, access, and quality. Really, it’s not looking good. And this is a form I’ve used multiple times over the years. I just have to change the numbers and it’s not trending in the right direction because in reality, we’re in trouble in terms of the healthcare system, and we need help from many, many, many directions, and we need innovations and tools.
Hema Chamraj: And I’m very, very enthusiastic about AI because of some of these. And here are some examples where I can try to explain why and how AI can help. When we think about healthcare, right? Just like every other sector we are collecting data and the data tsunami.
Hema Chamraj: Health is almost, 30% of the world’s data is healthcare. And it’s hard to think about, okay, how much, if we have all this data, how much, really, are we going to be using? It’s actually 1-2% of the data is really being put to work. So, and it’s humanly impossible really to be looking at this tsunami of data and make sense of it.
Hema Chamraj: And so in that sense, AI has made some real magic, especially in medical imaging and some of the areas where it has found all this hidden insights and hidden insights humanly not possible, actually. So that gets me all excited about, we need some tools to think of it. If we could really go after this 90+ percent of the data that is today not being put to work, if we could look at it, how much more problems we could be solving. And so that gets me excited.
Hema Chamraj: And also the other example here I want to call out is I talked about clinician shortage, right? This one is actually, as you look at developed nations, this problem is much more of a dire situation, because if you think about developing nations with a billion plus population and less than 80,000 radiologists to kind of serve this population, right?
Hema Chamraj: And those numbers actually could be, official numbers could be much lower than that in China. And think about that. This is the problem. And then if you have something, imagine that AI can look at all these cases and say, 70% of these cases actually doesn’t need to be looked at because they’re normal and then 30% should be actually be looked at by the physician because it needs attention, right? And so just imagine that, right?
Hema Chamraj: Now actually that is reality because we have worked with some, our customers and partners. Actually in China, it’s happening and not for everything, but just for the lung nodule detection is one of the problems that we were able to address. And there’s no reason to think we cannot address other conditions, also. And so that is one example. And the other one is about drug development.
Hema Chamraj: Drug development is really needs, badly needs to be some kind of disruption because you’re talking about billions of dollars and sometimes a decade for each drug and only 10% of these drugs actually make it to the final. So we have this really tough problem. And already we are seeing how we say with the recent vaccines that we are seeing, we’re already making progress and AI is playing a big role in it already.
Hema Chamraj: And if you think about Moderna, one thing I feel like people may not have noticed it, but Moderna actually were able to put out a booster vaccine for the latest COVID variants in less than 30 days. And that’s unheard of. And so that gets me really excited about what all AI can do. And of course, in AI, I mean, if you ask people about AI for COVID, depending on who you ask, people may say, well, it didn’t really fulfill the promise, but I am on the other camp. I say AI has actually played a role here because we are seeing first time with our own partners and customers.
Hema Chamraj: And I just put in a few examples here, looking from the left-hand side on the disinfecting robots to telehealth. And then I talked about medical imaging being one area where we have seen a lot of promise, especially when we had COVID testing shortage, actually a lot of the MRI and the X-ray, the tools were very helpful to detect COVID. And then you have on the bottom section, you have virtual ICUs. We had ICUs to complement to make sure that the ICU beds are going low. There were some solutions there.
Hema Chamraj: And then there are more around genomic sequencing of this virus strains, and then AI repurposing, drug repurposing. This is an example where drugs that were created for a different purpose, but repurposed – Dexamethasone, Remdesivir, the one that President Trump took when he was sick, that was also, it involved some level of AI for drug repurposing.
Hema Chamraj: So if you ask somebody like me, who’s a AI enthusiast, has AI been helpful? And is it a important tool? My answer is a hundred percent yes, right? And of course, people who have been at AI, they’ve understood that it has to be about a responsible AI. That’s not something that they take lightly, but what has happened for us in this year has been a spotlight, right?
Hema Chamraj: The COVID has put a spotlight on really the disproportionate burden and the inequities in healthcare, right? That’s come to light in a way that we are all forced to step back and take a look. If you think about the 4.5 times the hospitalization rate, or the two times the mortality rate of black versus white, it’s data that you cannot dispute, right? It’s something that we have to, and everybody’s sitting and thinking, the first question is who’s to blame, right? That’s where most of us aim to do, but the questions more than that is, where do we start? How do we fix? These are all questions that we’re all asking.
Hema Chamraj: So we don’t need to go to AI immediately because that’s one thing I would like to see is that not people running towards AI as somehow being the problem, we should start with the system that has been in place and the processes that have been in place for nearly a century now. And I want to talk about that as a starting point.
Hema Chamraj: Now, before I say anything, I want to say that our clinicians are the best. We need more of them. I mean, they are in this service to help us. So it’s less about the clinicians. It’s about the tools that clinicians need as they look at their patients to make decisions, that they need tools. And let me see, am I progressing? Yeah.
Hema Chamraj: So I just pulled out a few of these risk scores that have been in use for a long time. And these are used by clinicians to make decisions on diagnosis and referrals and so on. There’s the AHA, the American Heart Association, there’s a score and a study just as recent as 2019, there’s a study that showed that there’s the black and Latinx patients with heart failure go to emergency department in Boston. This is in 2019 and they’ve provided lower levels of care.
Hema Chamraj: And it’s somehow it was actually built into some of the scores from AHA. And then there is another score called VBAC, it’s a Vaginal Birth After Cesarean. And that score has been attributed to increased rates of cesarean sections for non-whites. And so I could go after each one of them, there’s EGFR for kidney failure. And so every disease category, there are so many scores that are embedded in the system that are being used on a daily basis, right? And so the question then is what are these scores? I mean, what’s the basis of these scores and it’s not all very clear. I mean, clearly they were done for a reason, but it’s not very clear.
Hema Chamraj: The assumptions are very invisible and race is often used as a proxy to explain the genetic differences and unconsciously, it has been propagating bias in the system. Now, one thing that I strongly and all of us, we believe that we look at the world based on our experiences. And so one example for all of us women here is that if you, I don’t know if you experienced this, but when women tend to go to the hospitals, and explain and talk about pain, that usually it’s chocked out as something psychological, usually.
Hema Chamraj: And that is not just something that happens once in a while. It happens routinely. And it is about something about a “brave men” and “emotional women” kind of a syndrome. So there are some things that are very embedded that propagates physician bias without them knowing it, right?
Hema Chamraj: And so this is the system there is. And then it happens more for people from the underrepresented and minorities, right? And so it requires all of us to look and understand what they’re experiencing. In order to even understand AI, we have to understand the experience. And so, there is the PBS documentary that I’ve started to look at, and this here is a story of one lady.
Hema Chamraj: She goes into for a gallbladder surgery, and then she is discharged. And then she complains of pain. And then she repeatedly and they repeatedly said, nothing is really at fault. And then when she collapses, she comes in and it is known that she has a septic infection. And that which is really a hospital acquired infection, which can be deadly. So, that’s just one example.
Hema Chamraj: And you really need to understand and look at, hear the experiences of people to really understand the level of fear. The level of distrust is in the system, right? And why does it matter? It matters because it translates to data that’s in the system. If you are so fearful that you don’t show up to the healthcare system, then there is the data is missing, that’s misrepresented, right?
Hema Chamraj: And so it’s an incomplete picture, but it has other things that are based on bias and that are based on beliefs. And the other, the additional thing we are seeing these days is this social determinants. These are things beyond just the clinical score, right? It talks about where do you live? How do you have access to healthcare?
Hema Chamraj: And a lot of these questions which are called social determinants is largely missing. I mean, we are doing, we are working towards it, but so what we end up having is this incomplete data that is then being used to develop algorithms as input, right?
Hema Chamraj: Just imagine what happens with that. You have data that is incomplete. So it’s garbage in, garbage out. And so there is one study that in 2019, this was highly publicized study that everybody, people with good intentions, try to build this algorithm and said, “Let’s look at the data and say, who is more sick. And so we can find out who should get the highest level of care.”
Hema Chamraj: And it turns out that the white population were determined to be more sick. And obviously it didn’t add up and turned out that the algorithm was flawed in terms of the design, because it was looking at cost as a proxy to say who are more sick. So it’s a flawed design looking at incomplete data from a distressed system. And obviously it leads only to a flawed diagnosis and hence wrong outcomes.
Hema Chamraj: So AI is not the where the problem starts, but it does do a few things. It reinforces or amplifies the bias and the power imbalances, right? And the people who develop the algorithms. I mean, it’s hard to imagine that they understood the experience of the people for whom this will be used against, right? I mean used towards. I mean, so there is the voices of the people who got the outcomes were not part of this, their voices were not represented.
Hema Chamraj: So this power imbalance continues. It becomes this vicious cycle, unless we stop and say, we really need to step carefully and see how we are developing AI. Are we developing AI responsibly? Is it equitable AI? Is it ethical AI? Is it based on trustworthy systems? So this is why I feel like responsible AI now is important, because as AI enthusiasts, right, we tend to get all excited, including me.
Hema Chamraj: I mean, I’m one of those people who says that we cannot do, we cannot fix the healthcare system if we don’t have things like AI, right? But I think it requires us to take a step back and see how we should be thinking about responsible AI. And there are so many things, it requires an hour on its own, but I would say we should go back to saying design AI, the design element of it is, let’s start with asking the questions.
Hema Chamraj: Why are we developing this algorithm? Who is it going to impact? How is it going to impact? What are the intended consequences or what are the unintended consequences? And then how do we remedy it? So these are the questions that we should start thinking about and start with designing AI for good and having a very comprehensive approach of the people, process, technology, systems, data, and algorithms.
Hema Chamraj: I mean, we have done some of this already, and this comprehensive approach is something that we are used to, and we need to apply that to AI because as technologists, which I am one of them, we tend to get all enthusiastic when we run towards AI. The questions we have, we ask are what is the speed, where’s the accuracy, what’s the sensitivity, what’s the specificity and the area under the curve. All of these questions are where people tend to, when they look on an algorithm to say, is it to measure the performance or the efficiency of it, right? But I think the question we need to focus on is how do we design AI for good?
Hema Chamraj: And there are many frameworks, but I’ve looked at these different lenses through which we should be looking at. Data and algorithm is very important because it’s about the data there. That’s where it manifests, AI manifests, and we have to understand the data governance. Where does the data coming from? How is it going to be used? The security, the privacy, and so on. Those are all truly very important.
Hema Chamraj: But, I would start at the beginning. At the people level, right? It has to be human centered, the human agency. And it is about not just the developers who are developing it. It has to have the voices of a diverse community of social scientists, physicians, the patients, and even the developer voices should include all of us.
Hema Chamraj: People who will have AI algorithms used upon. We need to develop the education pipeline so that we can be part of those developer voices. And so there’s a lot to be talked about here, but I would also, I would just say about the policy and the regulation, those are two areas that we are just starting out. I think we should pay more attention there because we need accountability and so on. But having said all of this are I still feel we have so much we can do together.
Hema Chamraj: We can lay this foundation for good and bring the best outcomes for all. So, that’s where I think I should be stopping. I’m getting a signal, guess I’m stopping there. Just saying that healthcare, we are all in here for healthcare. It’s not in a good state. AI can help, but we need responsible AI for ethical and equitable AI. And together we can do it.
Angie Chang: Absolutely. Thank you so much, Hema. That was an excellent talk.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Angie Chang: We’re going to have Swati from Atlassian join us here. And she will be sharing about building resilient products while we all grab a coffee or a tea and kind of get re-energized for the afternoon. I know it’s a long day of talks, so yeah. Please help me welcome Swati. She is the head of Confluence Engineering at Atlassian.
Swati Raju: All right, good afternoon everyone, just want to clarify, I head up Confluence Experience Engineering, which is a subdivision within Confluence Engineering. But anyway, thank you so much for having me here today. I work at Atlassian and like Angie said, I’m going to be speaking about building resilient products. Before I get started, I would love to take a moment to wish you all a very happy International Women’s Day. I also want to acknowledge our female predecessors in science and technology have really opened the doors for us and paved the path for each one of us. I can’t think of a better way to spend International Women’s Day than in this group with all you wonderful women engineers. And thank you for folks who are hanging on for the coffee break to spend your coffee break with me and to learn about building resilient products. So today we’ll be talking throughout this conference a lot about resilience in ourselves and in our teams.
Swati Raju: Those ideas are all very close to my heart, but I would like to speak to you about a different type of resilience. Resilience and reliability in cloud products. So the learnings that I’m about to share are really based off of some of the work that my colleagues and I have been working on in the last year or so. But just to set the context since I’m not sure how many of you are familiar with the name Atlassian. Atlassian builds software tools for team collaboration. The tools that we build help teams of all types. So it could be like sending the Mars rover up to Mars, or it could be startups that build the next concept electric car. So you may have used some of our products like Jira, Trello, Bitbucket, Opsgenie, or perhaps Confluence Cloud, which is a product that I work on.
Swati Raju: The organizers asked me to briefly touch upon my own background, and I would love to share with you real quick before we deep dive on our topic. I have a bit of an unconventional background because I studied architecture in my undergrad. After that I moved to the United States to do my Master’s in Design Knowledge and Computation. Worked in a few companies in the Valley from Yahoo! Search to Groupon. And then most recently I was heading up engineering for a small startup called Traveling Spoon, which incorporated three of my absolute passions technology, food, and travel. And then now I am at Atlassian where I work on Confluence Cloud. For those of you not familiar with Confluence Cloud, it’s a space for teams to do knowledge sharing and collaboration. So when I’m not working, I’m usually busy being a mom of two young boys and pretty much chasing them around all day long.
Swati Raju: Some of you might relate to that. So the common thread in my career from studying architecture to working on Search, to working on Confluence now, has been this idea of building something that’s enabling, empowering and really meaningful to the human experience. So when we talk about products that are truly critical to your day to day, resilience and stability are need to be part and parcel of those products. Let me give you some concrete examples.
Swati Raju: So on the first working day of 2021, Slack had a three hour outage. No disrespect to Slack, but if any of you were impacted, you recall how disruptive it can be when a critical tool like that goes down. Similarly, Confluence in 2019 had a bad outage. What we define as a severity 1 based on how many customers it impacted. What our users saw was this chilling screen and users could no longer access their wikis or collaborate with their teammates because of a code change that just didn’t work when it was pushed to significant load.
Swati Raju: The point is that if you are involved in building products that are critical to someone’s day-to-day working, the reliability of those systems become critically important. So I’m going to share with you three principles and practices that can help you improve the reliability of your systems.
Swati Raju: So let’s dissect each one of them. The very first one is accurately measure the customer pain. So if you’ve heard of Peter Drucker, arguably the most, one of the most influential thinkers in management, he said, “If you can’t measure it, you can’t improve it.” This is true in all aspects of life, especially true for reliability of our systems. So now traditionally, most measures of reliability were based on server-side metrics or what we call uptime. This is not always representative of what the customer is actually seeing. So reliability on the other hand measures what the customer is actually experiencing. A lot of cloud companies, including Atlassian, promise our customers very high reliability as well as uptime.
Swati Raju: So it’s a combination of both. But why is reliability and specifically like this idea of an overall reliability from the customer’s perspective versus uptime, a much more harder metric to nail, especially in distributed systems where you have a lot of depending microservices? Let me share an example here. So take for example, this lamp at my desk, right? It is dependent on the reliability of the bulb, the power cord, the lamp arm, the lamp plug, the shade. Hence, when you calculate the reliability of this lamp, it would be the reliability of each of its components multiplied by each other. The more components you have in the system, the greater likelihood of lowering the reliability.
Swati Raju: So, really we really need to think about kind of how to build systems that have great end to end reliability. Let me give you an example of how we have approached this at Confluence. So what we’ve defined is something called key user journeys. And these might be the most important things that a user does on your product. So for example, viewing a page is a really important key journey for us, and the team that owns that key user experience is responsible for understanding the operational metrics of the way the action, right from where the action the user takes to how the request gets sent and all the dependent services that then touches. So holding ourselves accountable for not only our own code but for the key user journeys that… and on how the code, what are the different aspects of the code that are being touched is critical.
Swati Raju: And then having visibility and alerting you to not only what the components that, say, your team owns, but also the underlying dependencies, becomes super key. What we ended up with then is a whole lot of rich dashboards and alerting for exactly what the customer was experiencing versus silverside reporting. So moving on, before I jump into the next principle.
Swati Raju: We collected data at Confluence and found that bugs and software are the majority of the root causes for incidents or what we call when a customer has an outage. As much as 50% of our outages were caused due to bugs in code. So this whole idea of move fast and break things does not always work. So don’t get me wrong. There might be situations and places where the mantra of move fast and break things works really well. When I was at a startup, trying to get something out into the market, where my goal was to prove my hypothesis on product market fit, move fast and break things absolutely works.
Swati Raju: However, when you’re working on a product that users depend on for their livelihood and revenue, this strategy needs some rethinking. For a B2B product like Confluence, where our users depend on us for timely collaboration and real-time reference documentation, taking such an approach is, dare I say, irresponsible. So in fact, Mark Zuckerberg, the CEO of Facebook, who famously coined this phrase, or at least as attributed to have coined this phrase, move fast and break things, he announced in 2014 at F8, which is Facebook’s annual developer summit, that they had changed their mantra. So wait for it, move fast with stable infra.
Swati Raju: So unfortunately this phrase, doesn’t back the punch of the original move fast and break things, but you get the point. It’s addressing the reality of a product that has to support its users at scale. So the point I want you to take away is that speed of change should always be balanced with the ability to detect and recover quickly and B, ability to limit the blast radius when something breaks.
Swati Raju: So some of the examples of how, that I’ve seen to successfully balance moving fast with ensuring reliability have been progressive rollouts where a very small percentage of users incrementally get changes, so we gain greater confidence on the reliability of that change. Another way it will be longer soak times where we leave the production, the newest production version, and our internal instances for enough times then we can exhaustively use it internally. And then the last one is really this idea of early detection, looking for anomalies in the production pipeline.
Swati Raju: Additionally, what I found incredibly useful is moving to this idea of shift left in our approach of software development, where we shift the effort for improving the quality of the software earlier in the development process. So what we found at Confluence is that 89% of our instances or incidents, 89% of our hots and incidents could have been avoided by just adding more detection in the predeployment testing.
Swati Raju: So that should make it really clear. Investing in prevention early in the development pipeline is absolutely crucial. This brings me to my next principle, when we shift changes, we must always expect the unexpected and anticipate and plan for failures. So we need to build systems that can embrace failure as a natural occurrence, even if we do not know what that failure might be. So some of the methods that we have been using have… it’s been around load testing.
Swati Raju: It’s a really good idea to load test significant features with what might be a simulation of peak traffic, so you can identify bottlenecks for example. Throttling and limiting. So when planning operations in the cloud, we want to know what are the upper bounds and limits that can be consumed. This can be critical for us to design our systems. And some cases, we want to do throttling so that there’s a small, very, very small percentage of users who might get a bad experience.
Swati Raju: The 0.0001% and the 99.99% get an awesome experience. And obviously you want to work with your product teams to figure what that best trade-off would be. And then the last one is around reducing blast radius. So it is important to manage components that are impacted without the need for the overall system to go down. So we need to develop this into our fundamental planning where failure occurrences, such… that impact the overall health. We never reached that point. It’s always very, very local. So I’ve shared a ton with you.
Swati Raju: You can do all of those things, but here’s some really bad news for you. Things will still break and shit will still happen. The best that we can do in these situations, A, is to recover our systems quickly. And B, is to show some customer empathy. So this is an example of designing and planning for error states really shows that you are bringing empathy in the most rotten situations.
Swati Raju: So here’s an example of what the failure pages looked like in Amazon Prime, 2018. When all else fails, bring the dogs and the cats, right? And then finally, every outage or incident is an opportunity to learn, to helps us think about how we could have avoided it in the first place, how we can mitigate its impact in the future and how we can reduce the blast radius in the future. So here’s a recap of the three principles.
Swati Raju: And hope you can remember these, accurately measure customer pain, move fast and break things does not always work, and expect the unexpected so you can anticipate and plan for failures. Finally, I want to leave you with this thought. So no matter what your role, you should be thinking about your reliability. If you are a developer, really think through what are all the unexpected things that can happen when your code goes to production. What are the absolute crazy wild things that could go wrong?
Swati Raju: Because let me tell you this, they will go wrong. And if you’re an engineering leader, go beyond just thinking about tooling, metrics, processes, and habits that your team needs to do for the stability of your product. And really think about building that culture of reliability, because that truly shows how much you care for your customer.
Swati Raju: With that, I will wrap up, but I will make a quick pluck from teams in Atlassian that are hiring. I put a link at the bottom there. If you’re interested in working to support teams that will tackle some of the next big challenges for humanity and really care about working in a culture of awesome diversity, do go ahead and check those out. Well, that’s all I had. Thank you so much for your time. Enjoy the rest of the conference.
Sukrutha Bhadouria: Thank you so much, Swati.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Angie Chang: She is a UX researcher at Google. She joined as a UX researcher, a senior UX researcher from Fitbit, which was acquired by Google. She has a background in digital media, entertainment, forensic, and health. Her specialization is in kids and family, digital products, and services. And with experience and years of research in curriculum analysis and evaluation, she worked at Nickelodeon, Televisa Foundation, and Sesame Workshop. So welcome, Kat.
Kat Chiluiza: Hi. Can everyone hear me?
Angie Chang: Yes.
Kat Chiluiza: Okay. Hi. Hi, everyone. Thank you for the introduction. And yes, so my name is Kat, and I’m a UX researcher.
Kat Chiluiza: And really, UX research is about speaking to users, in order to understand their needs, their motivations, their behaviors, and then, taking these insights, in order to better distill what it is that we need to learn and apply and adapt into your own products and services and features.
Kat Chiluiza: I am trying to find my deck, because this is normally the part in which I would then lead into the actual deck itself. So if you would just bear with me for a second.
Kat Chiluiza: Yes. So UX research is about making sure that we are the voice of the user. However, we know that sometimes that it isn’t always the case.
Kat Chiluiza: When we think about products that are out there, sometimes they aren’t accessible or inclusive of people of different needs. This is not only frustrating, but limiting, right?
Kat Chiluiza: If you’re unable to use something, you might actually decide to leave the product altogether. Oftentimes, what causes this is when users’ intersectionality isn’t taken into account. That’s really what I’ll be talking about today, inclusive design and research.
Kat Chiluiza: What exactly is inclusive research? Inclusive research is when research is conducted in a way that involves and respects people of diverse identities. This could be gender identity, your ethnicity, any disability that you might have, all of that fits into identities. When you think about that intersectionality, that’s where, oftentimes, products do tend to fall, and not fully fit the needs of the full person.
Kat Chiluiza: Most people commonly think that we can address this by making research and design more inclusive in diversifying our participant pools. However, the challenge really is that diversity for diversity’s sake isn’t automatically going to lead to inclusive research findings. Instead, inclusive products is an intentional act. Inclusive research is an intentional act, and something that occurs throughout the entire production cycle.
Kat Chiluiza: Today, I’d like to share some tips that empower folks in product and design, to think about ways that you can add more inclusivity in the production process. And I’ll talk briefly about, what are some things that you can do when preparing for research, and when delivering those findings, and I’ll spend most of the time sharing examples of how research questions can fail to address that full spectrum of a person’s identity, and give examples on how we can improve them.
Kat Chiluiza: So let’s start at the very beginning, preparing for research. What are some things that we can do at the start, in order to make sure that we’re starting off at the right foot?
Kat Chiluiza: When we start planning for research, it’s important to narrow down and really kind of understand what it is that we actually are trying to get to know. Three questions to ask yourself at this stage are, what do you want to know, what do you already know, and what are you hoping to do with this information?
Kat Chiluiza: These questions are both not just for the researchers, but for the stakeholders, as well. This can help inform who you recruit, identify potential types of edge cases, and then, also keep you from going beyond the scope of the work or the scope of the research, as it’s very easy to sort of get down different paths.
Kat Chiluiza: When recruiting for participants, think critically and use your own intuition about, what are the users that might be excluded when designing for this product or feature? Ask yourselves who are our marginalized users, or think about, who are the users that might be negatively impacted by a feature update?
Kat Chiluiza: Oftentimes, there might be trade offs when making a new feature update. And in those trade-offs, is there going to be a certain user base that is negatively impacted, or whose services might actually not be as supported as well as before?
Kat Chiluiza: You can also look at behavioral data. What are your analytics telling you, not about the patterns, but about the outliers? For example, if it were a standard deviation, who are those people at the margins? Once you’ve identified those groups, then you can have a better sense of who are the recruits, who are the people that you often don’t speak to, but might benefit from actually doing so?
Kat Chiluiza: So now let’s actually get into writing out your discussion guide or survey. Every time we conduct research, we’re looking for ways to ensure research rigor. We make sure to screen out any of those double-barreled questions or any questions that might be leading or loaded. And we’re also mindful about the type of bias respondents may have coming into a research project. In fact, that’s part of the reason why we are so thoughtful about the recruit itself.
Kat Chiluiza: But what about our own cultural bias? How often are we reflecting on the types of questions we ask, and looking to see if we’re being exclusionary or insensitive, as we’re asking these questions?
Kat Chiluiza: Researchers unfamiliar with working with diverse audiences may ask questions that inadvertently show bias, right, or stereotype, or even completely ignore context. Oftentimes, these are researchers with the best intentions, but it just so happens, all of us have our own bias, and that’s how these things sort of happen.
Kat Chiluiza: To put this in context, imagine this was a survey that was sent to young adults in the US, and it’s asking them to select their favorite sport. And now imagine that this is the survey, and these options are the ones that were presented.
Kat Chiluiza: Technically, the researcher did their due diligence when it came to writing out these this question. These options are actually the most popular sports in the world. And so, as a result, it would make sense to include this as, “What are your favorite sports?” available.
Kat Chiluiza: However, they didn’t take into account that this audience is from the United States and localized, what might be popular in that country. Had they done so, they could have adjusted the potential responses to make it more culturally competent and relevant. While this question may seem really obvious to a lot of folks from the US, the reality is that this type of bias exists in anything.
Kat Chiluiza: So let’s look at a question that might be heteronormative. “What is your gender?” Heteronormative questions are those that are written from a cisgender or heterosexual perspective. The reality is that sex, gender and sexual orientation are far more nuanced, and should be their own individual question.
Kat Chiluiza: Additionally, if we look at the options, we notice that there’s a specific option for cis female and cis male participants, yet all trans people are put together. It doesn’t actually give an option between trans women or trans men. And lastly, when we look at the word “Other” to use to describe an identity, this phrase itself, “Other,” is dehumanizing to folks that might not fall into the first three categories.
Kat Chiluiza: This is an updated version that actually breaks down the nuance between gender identity and sexual orientation. And since identities have evolved over time, there’s also some new options in there, added to reflect that nuance. And by breaking down these questions, there’s also the benefit that it can help researchers and stakeholders better pinpoint what it is that they want to learn, and tease out what’s not relevant in the project.
Kat Chiluiza: If we go back to the initial set of questions that you answered in the beginning, now you can actually see which of these questions actually is most relevant to your project. How are you going to use this information that’s been presented?
Kat Chiluiza: So let’s take a look at this other question. This was, again, these are all questions that have been out in the real world. All of these have happened. This is a survey that was directed towards women on a website. If we pause for a second, I’ll give you guys two or three seconds just to think about what might not be inclusive about this question.
Kat Chiluiza: While this survey possibly considered the participants’ sex, it didn’t consider their gender identity, or even that some folks might just have a broader range of shoe style preferences that can impact their answers. Not all women identified folks wear women’s shoes. But this question assumes that our respondent is going to give a woman’s size. But what about those that only wear men’s shoes, or only know their shoe sizes in men’s sizes?
Kat Chiluiza: It also only gives shoe sizes in the United States, which is an oversight, since many websites tend to have a broad international readership. By neglecting these factors, you’re now risking faulty data since participants don’t actually know what size to put down, or they have to convert the sizes. Now you’re asking your participants to do additional labor, and possibly feel ignored, since they have to do their own work in order to provide you with answers.
Kat Chiluiza: But if we are realistic, we know that users or participants in surveys don’t tend to take that time in order to make those types of conversions. What’s more likely is that they’ll probably just leave the survey. And so, now this group that was already underrepresented will continue to be so.
Kat Chiluiza: Let’s see how this can be addressed. This updated version has the sizes available in both men’s and women’s, and that also show conversion sizes across different countries.
Kat Chiluiza: So let’s take a look at this Eurocentric question. While the question about race is very common in surveys, often, the options don’t include some more common groups in the US. And this question, in particular, doesn’t reflect the most up to date language, when it comes to describing some specific races, such as with the Hispanic option.
Kat Chiluiza: Lastly, there isn’t an option for mixed race people. Before we jump into how we can address that one, let’s also take a second to look at this other Eurocentric question. This one in particular is one that’s been making trends in culture surveys across organizations. And really, while this question is well-meaning and comprehend, and appears to be comprehensive, the reality is that it conflates culture, religion, and skin colors all as one, and ignores that they potentially have different experiences, and potentially could be treated differently.
Kat Chiluiza: When we look at how these questions can be updated, again, we noticed that there’s this breakdown of different identities, right? Race, culture, skin color have all been broken down separately. There’s also more comprehensive options when it comes to the changing demographics in the U S. For example, with Hispanic, there’s now been added the Latina, O, and X.
Kat Chiluiza: Here’s a question that one might see when you’re applying to jobs. “Do you have a disability?” A few issues that commonly are raised with this question in particular is its vagueness. People don’t know how to answer because they aren’t really sure what’s considered a disability for the specific role.
Kat Chiluiza: Instead, it’s often suggested to contextualize this question, and clarify what it means when you’re saying a disability. And lastly, allude to the relevance of this question at all for your specific application. So in this updated version, it’s, “Do you have a physical disability that can impact your ability to lift 10 pounds unassisted?”
Kat Chiluiza: So what’s the takeaway? Be clear and explicit. Think about the context of your questions in a person’s everyday life. Because if not, you risk basing decisions off of bad data, alienating your user base, as they might want to leave, or just left with a negative taste in their mouth. Or you might have to spend more money and time redoing the research, if you realize that that the data that you’ve gotten was impacted by some faulty questions.
Kat Chiluiza: Now that we’ve seen a few different questions and thought about intention, intersectionality, you’re probably thinking, “When is it appropriate to ask these questions? Or what type of considerations should I take into account?” Or even, “How do I know if I’m going in the right direction? I do have the best intentions, but now I’m wondering if that, maybe, that just might not be enough.”
Kat Chiluiza: The first thing to do really is just to go back to what you were originally hoping to learn, and what you were hoping to do with this data. That information is going to be really key, as it’s going to help you narrow down, what are those relevant questions you actually need to ask?
Kat Chiluiza: Then once you have that sense of, “Okay, these are the things that I definitely need to know,” then you can begin writing the questions. And as you begin to write these questions, learn to look from others, research the topic online, see what are some trends or some updates or things that have changed along the way, such as, for example, with the Hispanic/Latinx option.
Kat Chiluiza: Or see what others have said about crafting questions to that specific target audience. For example, the HRC, a few years ago, launched their own recommendations for how to ask the question around sex and gender.
Kat Chiluiza: And lastly, check your ego. There’s going to be a lot that you’ll learn and unlearn along the way. As identity changes, as the way in which we speak about different marginalized groups changes, then that means that how we adapt is going to be very different, and what was relevant before might not be relevant now.
Kat Chiluiza: In fact, even that HRC survey that I mentioned, that was written in 2016, and since then, even I had to make some adaptions to it, just to make it a little bit more relevant for the 2020/2021 audience.
Kat Chiluiza: Once you actually have these questions, then you can begin drafting a survey, or piloting your research in some way. You can do this with a small sample. This way, you can make adjustments to language or approach, while still being able to retain that data integrity.
Kat Chiluiza: Even with just one or two people, you can see if there’s some areas where you veered off in the wrong direction, or maybe the participants themselves might call out something that doesn’t sound contextually correct. And then you can make those changes without having impacted your larger pool.
Kat Chiluiza: In the last few minutes, I want to briefly touch on data analysis and insights. So we know that in the data analysis stage, there might be times in which you’ll see some trends that are consistent with just the Latinx participants, or maybe just your LGBT participants, but aren’t trending across the entire sample.
Kat Chiluiza: Especially as you begin to increase your diversity pool of participants, you might be seeing certain trends in certain areas. But as we all know, sometimes stakeholders are resistant to making changes, especially if they feel that it’s only going to impact a small subset of users. While this can feel demoralizing, the reality is that you can advocate for these changes in the same way that you would do for any other type of UX or UI changes that you might be running across.
Kat Chiluiza: For example, you can conduct a second round of research for that specific demographic. Then you can continue to show that this is a pattern that impacts this group, and it’s very consistent that this impacts this group.
Kat Chiluiza: Or you can test design iterations to address the user concerns with a larger sample size. Oftentimes, changes that are made to impact one marginalized group tend to have a broader impact for the entire community.
Kat Chiluiza: If we think about on sidewalks, the small ramps that are there, initially, that was made to make, to make it accessible for people in wheelchairs. But now people with strollers, people who are on bikes or scooters, they all benefit from it. If you see in a larger sample size that other groups continue to benefit from it, then it’s clear that this is something that could have a potential powerful impact on your product.
Kat Chiluiza: You also want to include some examples from competitors implementing more inclusive solutions. You want to make sure your company is staying up to date. These are just the trends in the industry. And by taking these steps, you can show how your company can remain competitive, or become even more competitive. By tying it to some type of business need, whether it’s acquisition, retention, subscriptions, or so on, then you can further help boost that case.
Kat Chiluiza: This session really has only scratched the surface of what there is to learn and think about conducting research with diverse audiences. But I hope that you now have a greater understanding of how even a small change to a question can have a meaningful impact to the users answering them. It allows them that one moment to actually be reflected in the types of questions in the Internet in itself, that actually shows them for who they are.
Kat Chiluiza: We can all advocate for systemic change and user research. So let’s all fight for a world where tech reflects those needs and wants of all consumers, one user at a time. And so, thank you.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
