Over 100 girl geeks joined networking and lightning talks from women working in engineering, product, and design at the sold-out Grammarly Girl Geek Dinner at Grammarly’s office in downtown San Francisco, California on August 29, 2023.

Grammarly women shared lightning talks about building GrammarlyGO, Grammarly’s new contextually aware generative AI communication assistant that allows you to instantly compose, rewrite, ideate, and reply. Grammarly is hiring!

Table of Contents

1. Welcome – Angie Chang – Founder at Girl Geek X – watch her talk or read her words

2. Fireside Chat – Heidi Williams – Director of Engineering at Grammarly with – Charlandra Rachal – Technical Sourcer at Grammarly – watch the fireside chat or read their words

3. Building GrammarlyGO From Zero To One – Jennifer van Dam – Senior Product Manager at Grammarly – watch her talk or read her words
   

4. Engineering GrammarlyGO – Bhavana Ramachandra – Machine Learning Engineer at Grammarly – watch her talk or read her words
   

5. Designing GrammarlyGO – Sarah Jacczak – Brand Designer at Grammarly – watch her talk or read her words

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• See open jobs at Grammarly and check out open jobs at our trusted partner companies.
• More Grammarly Girl Geek Dinner photos from the event.
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!

Transcript of Grammarly Girl Geek Dinner – Lightning Talks:

Angie Chang: Is [this] your first Girl Geek Dinner? Wow, that’s a lot. How many of you have been to more than five Girl Geek Dinners? Yay! So good to see everyone. My name’s Angie Chang, in case you didn’t know, and you could tell by the t-shirt, I am the Girl Geek X Founder, and started Girl Geek Dinners in the Bay Area 15 years ago, so we’ve been doing events like this at hot tech startups up and down from San Francisco to San Jose, and I’m in the East Bay, so I wish there was more events over there as well. Tell your employers they need to have one of these showing off their amazing women in tech and product.

Girl Geek X founder Angie Chang welcomes the sold-out crowd to Grammarly Girl Geek Dinner on August 29, 2023 in San Francsico! (Watch on YouTube)

Angie Chang: I want to say thank you so much to everyone at Grammarly for helping put this event together. They have been so amazing and supportive and they’re definitely hiring, so please talk to someone here that has Grammarly on their shirt. They’re very friendly, so I’m going to say thank you for coming and hopefully you’ve made a lot of good connections. I know I’ve seen a lot of people talking to each other and I hope you have LinkedIn with each other or Facebook or whatever people are using these days, and continue to stay in touch.

Angie Chang: A lot of us are in this industry working to keep women in tech and I think that involves all of us together, so thank you. Keep coming back to events! Keep giving each other job leads! Keep poking other girl geek to get in the car ride together to get to that event after work when we’re all tired! Thank you for coming! I hope you learn something, make a new friend, and have a good night!

Charlandra Rachal: Thanks, Angie. I’m super excited to kick things off and host this fireside chat with director Heidi Williams, who’s been very involved in building our generative AI features for enterprise. Heidi, welcome!

Heidi Williams: Hi. Thanks for having me! Great to see you all here. It’s awesome. Full crowd!

Charlandra Rachal: Yeah. For those who aren’t super familiar with Grammarly, can you give us a quick overview of our company and our product?

Heidi Williams: Sure. I like to make a joke that either people have never heard of Grammarly or they love it! I know I talked to a few folks already that love it, but for folks who aren’t familiar, we are an AI-enabled writing assistance that helps with your communication wherever you write, and I do mean everywhere. Our mission is to improve lives by improving communication. Earlier this year, we also launched our first generative AI product to help you with even more writing and communication assistance beyond just revision, but also getting into ideation and brainstorming and composition and comprehension. It’s been really fun to see the product evolve in the time that I’ve been here.

Charlandra Rachal: I hear that you just celebrated three years here, so woo woo! Three years! Can you tell us what brought you here and what really keeps you here?

Grammarly Technical Sourcer Charlandra Rachal and Director of Engineering Heidi Williams welcome the audience at Grammarly Girl Geek Dinner. (Watch on YouTube)

Heidi Williams: When I was speaking about the mission, improving lives by improving communication, I do feel like I got to a point in my career, I’m a little farther along maybe than some of you, that I really wanted to work on something impactful and I feel like Grammarly more than any other place, it resonated with me that improving lives by improving communication is so real. It’s not a fake slogan because communication is what makes us uniquely human.

Heidi Williams: I was excited about the idea that we’re not just a platform to help you communicate more effectively, but also to help educate you along the way, especially thinking about things, like insensitive language or bias, there’s an opportunity to help educate people about the possible impact of their words that they may not even know is having a negative impact on someone, and so I got really inspired about the mission.

Heidi Williams: I’m also a word nerd, so that part was really fun as well. I think what keeps me here, is that everyone is so excited about the mission and the people. I think our values are amazing. We really live by our values, we hire and fire by them.

Heidi Williams: The last thing I’ll say is, we’re an amazing size company, where there’s still interesting problems to solve, but we’re small enough that people can really take the initiative if they see a problem that needs to be solved, or they want to advocate for something to change in some way, they’re really empowered to do that. I love being at that size company and our values really help us be successful doing that as well.

Charlandra Rachal: Nice. I like you mentioned initiative and impact. Do you have any sharing stories that you can share that where you seeing either yourself or someone else really make impact?

Heidi Williams: I have three examples if you’ll bear with me for a minute, but I see it all over the place, and it’s not just in the product. It’s about our organization, our culture. There’s an engineer on my team, her name is Lena, and she recognized on the product side that engineers were struggling with a certain pattern of ‘how do I reliably save settings about the individual, about their team and their organization about specific features. Then if I have all of these settings, how do I combine them and know which setting to apply at any time?’

Heidi Williams: She interviewed a bunch of engineers, realized it really was a problem for folks, and then proposed a new project called the Settings Registry, and advocated for it to be on our roadmap. It’s been exciting that she could spot an opportunity and a challenge for our developers and really advocate for that. That’s exciting!

Heidi Williams: The second one, I actually led an initiative where I noticed that I love our hiring process, but I noticed that we had one particular gap, which was that we didn’t necessarily have an interview where we asked people about their experiences. We ask about their knowledge, but we don’t ask, ‘what is the proudest thing that you ever built and tell me how it was designed and what did you learn?’

Heidi Williams: I noticed similarly that we weren’t necessarily getting the accept rates from underrepresented groups that I thought we should be getting, and advocated that this might give people an opportunity to talk about themselves, and for folks who aren’t used to bragging about themselves, that might not come out in a normal interview, but if you give them an opportunity to talk about themselves, then they can actually show off how good they are at stuff, which is exciting.

Heidi Williams: That pilot was successful, showed that we greatly increase the accept rates for folks from underrepresented groups to a really high degree, and now we’ve rolled that out as an interview across the engineering organization, so really proud of that.

Heidi Williams: The last one I’ll mention related to culture, Bhavana, who you’ll hear from later, identified an opportunity that folks were looking for mentorship inside of our women in tech group, and so she started a pilot with a few other folks to introduce an internal mentorship program for women in tech and we’re kicking that off in September.

Charlandra Rachal: I love that. Yes. I feel like the last two really spoke to me, especially being in recruiting so I love that a lot. Now, Grammarly continues to expand in its enterprise space. How do you drive value for Grammarly business with generative AI?

Heidi Williams: It was very exciting to see our generative AI product come out. A little bit of context: the part of the product that I’ve worked on is Grammarly Business, which is our B2B product for teams and organizations.

Heidi Williams: As we all know, communication is not a one person sport. There’s a team dynamic, there are team norms, there’s organizational knowledge that are part of the communication that you have at work. We looked at opportunities for how to incorporate organizational knowledge.

Heidi Williams: We have a feature called Knowledge Share that helps you define terms, definitions related links, key people, and then we can use that as part of the generative AI output to help you have something that knows something about your organization instead of maybe a more generic response.

Heidi Williams: We did things like that and then incorporated some of our Grammarly business features like style guides and brand tones, which help you speak with a consistent voice, and brand tones in particular, you can have a response from our generative AI product, and then choose ‘make it sound on brand to my company’.

Heidi Williams: That was a way that we could really make the information, both the information and the tone be tailored to your organization.

Charlandra Rachal: Nice. Well, I heard that there was some quick turnaround times. Can you tell us more about that?

Heidi Williams: It was definitely felt like this huge opportunity, this huge moment where a lot of folks are talking about generative AI and it’s an area (LLMs) we’ve been investigating for a long time and understanding what their capabilities and limitations were and whatnot, and so I think we really rallied as an engineering organization, and I think the way that we were able to turn things around quickly really came from our leadership approach, which is the idea that we really want to empower teams to make the best possible decisions on the ground.

Heidi Williams: The way to do that is to help with transparency and sharing context around ‘what are the business needs, what are the product needs, what are our customer needs, what problem are we solving for the user?’ Let me give you all of that information, all of that context. At the end of the day, if you need to choose, should this be a radio button or a dropdown or this should work this way or connect with that system, you can make that decision because you have all of that information. Really trying to be transparent and share context so that people are empowered to make decisions on the ground and not feel like they’re stuck with somebody else making decisions and kind of blocking them from things.

Charlandra Rachal: I hear you that you mentioned customer feedback. Do you have any feedback that you’re able to share with us?

Heidi Williams: Sure. You’ll hear more about it in one of the talks today. We did run a survey after launching GrammarlyGO and wanted to know how are people using the product and what’s working and what’s not working. Through that feedback, one of the themes that we heard was that ‘it didn’t sound like me’.

Heidi Williams: We started investigating – ‘how do you tailor the output to sound authentic to you?’ And it sounds, I see a lot of head nods, that resonates and what not. We invested in an area called My Voice and figuring out how to have your own voice profile and use that for all of the responses that are generated, so it’s more likely to sound like you than not and saves you an extra step for trying to even interpret what your own voice is. We can actually help you with that, so you’ll hear more about that when Jen talks about it.

Charlandra Rachal: Great. Well, I know this is one question that I know a lot of people probably want to ask but probably wouldn’t ask, but what would you say really sets us apart from our competitors?

Heidi Williams: Yeah, I was talking to someone ahead of time who asked this question, I’m like, oh, you’ll have to wait. <laughs> Great question. There’s one thing. First of all, I mentioned earlier, we work everywhere and that is one difference from some of the other products that are out there. We work in every writing surface, desktop web, and so we can be right in line for where you’re already doing your thinking, your writing your communication, so that’s certainly one.

Heidi Williams: The two I wanted to really call out, which I think are kind of reinforced by our engineering culture, is our important focus on security, trust and privacy, and also responsible AI. Because at the foundation of everything we do, we really want our customers and users to trust us with their writing and to feel like we can do things to make personalized experiences and what not, and so, what’s interesting to me, I feel like more than any engineering organization I’ve ever been at, because we are so mission-aligned, we recognize we have this huge responsibility to our users to be thoughtful about their data and their privacy, their security.

Heidi Williams: I feel like we care a lot about security maybe earlier than most engineering teams where at the very end before you ship security goes, ‘oh, not yet!’ And you’re like, ‘oh, I can’t’. The whole idea that engineers will advocate for, am I doing this right from the beginning, and wanting to make sure that’s so they’re proactive about asking for feedback about security and privacy. Or even there was a scenario where we had an idea about a feature and people are like, ‘That feels like it might invade privacy. Can we talk about that before we launch it?’

Heidi Williams: I really loved that people could bring that up and that we’re all trying to achieve the same thing, and so it’s a very fair question and let’s make sure we’re holding that to high regard.

Heidi Williams: Then on the responsible AI side, I think we’re so lucky to have an incredible team of linguists who can help us beyond what other competitors can do who don’t have a team of linguists where we can help sort of filter things like the inputs to generative AI to make sure that people are not asking for something harmful, but also that whatever they type in, they’re not getting harmful responses, which are either insensitive or inflammatory or traumatizing in some way.

Heidi Williams: I love the fact that we have the capabilities of being able to create these filters and create a safe environment for people to use these large language models, which have who-knows-what in them. Love that we are actually able to do that. We’ve also been able to build that not just through humans, but figuring out how to build automation and testing and all through the development process help you understand that you’re not going to create a feature that unintentionally create some sort of biased output or something like that, and so just tremendous examples over our long history in this of finding ways to make sure that we are building a product that is responsible and then also keeps everybody safe, secure, and all their information private as well.

Charlandra Rachal: Nice. Well that was fascinating, right, everybody? Alright, so we are going to dive deeper now to exactly how our generative AI features were built. As a heads up, we are going to ask for questions at the end and I’ll bring up all of the speakers including Heidi herself. For now, welcome Jennifer van Dam, who’s a senior product manager here!

Jennifer van Dam: Hey everyone. I’m Jennifer van Dam, product manager here at Grammarly. I’ve been here for three years and I worked on our features like emotional intelligence, tone detection, tone rewrites, inclusive language, and most recently I helped build out our generative AI product, GrammarlyGO, which I’ll be talking about today, so super excited to take you all through the journey.

Grammarly Senior Product Manager Jennifer van Dam talks about building the generative AI product GrammarlyGO from zero to one. (Watch on YouTube)

Jennifer van Dam: First off, I want to give a huge shout out to my fellow girl geek PMs that helped build GrammarlyGO together with me. We were a team of three PMs leading multiple product efforts. Specifically, my product focus was on the UX and also on the zero to one stage, so figuring out the UX framework and the zero to one building process. That’s what I’ll dive in deeper today. First off, I wanted to start with a refresh of Grammarly before GrammarlyGO.

Jennifer van Dam: What Grammarly has been focused on for many, many years is helping make your communication more effective by proof writing and proof reading and editing your writing. Anywhere you write, let’s say you’re writing an email, a message, a Google doc, Grammarly will read the text that you have written already and make sure it’s correct and clear and delivered in a way that you want to come across. But, we have a big mission of improving lives by improving communication, so we fully were aware that this is a small part of communication that we want to help with, and we’ve had many dreams beyond proof writing and editing.

Jennifer van Dam: One big user problem we always heard, for example, was the ‘blank page problem’. For years, we’ve heard that our users really struggle with the inception stage of communication – the writing, getting those initial ideas on paper – and it was a huge productivity blocker. That’s just an example from user problems we’ve been hearing for years, and we always dreamt about solving it, and we were super excited with this recent technological leap that with generative AI – now we have the technology to solve all those user problems we always dreamt about.

Jennifer van Dam: That’s how we built GrammarlyGO. We went from proofreading and editing towards helping solve composition, brainstorming, and all these new use cases, which was really, really complex, because we went from a decade of in-depth expertise of rewriting, towards composition and brainstorming, and we had a pretty aggressive timeline as well. This was super, super challenging.

Jennifer van Dam: What made it really challenging? First of all, it was zero to one. We had no prior experience how this would land with our users and there was no data we could rely on, so we had to make a really risky decisions because we went from a proven product concept with product-market-fit towards a huge uncertainty and risk area, which it was really exciting, but super, super challenging. How can we predict how it will be received with the absence of data?

Jennifer van Dam: Essentially we really had to take on a beginner’s mindset to solve these new use cases and almost operate like a startup again to build this new product from scratch, but we’re also an established company – pretty big – and we have millions of users, 30 million daily active users that have a super high bar of our product. We were building zero to one moving fast, but also had a very high bar we wanted to meet for our users in terms of quality, responsible AI, and security that we wanted to deliver.

Jennifer van Dam: How do you solve such a huge, huge problem? What we did was, let’s just start with the earliest draft possible, and get it out – get it out to users. What we did is we created this really highly-engaged alpha community, and we built very early prototype, and we shifted and we asked for continuous feedback, and it was really, really engaged community that would give us feedback super fast and inform next iterations. We really focused on the core experience before we wanted to invest in any type of polish or any type of design polish, we made a commitment – let’s not focus on that. Let’s figure out the UX framework.

Jennifer van Dam: We had a big challenge. How do we create a UX where someone can brainstorm and compose something from scratch? What is intuitive? What will land with our users? To give you an example of the fidelity of prototypes, what we did is we started in grayscale, because we made a commitment to figure out the framework, before deeply investing into building something out, because we weren’t sure if this is the version to commit to.

Jennifer van Dam: This turned out to be a great idea because we did end up throwing away a couple of prototypes, and the third prototype was the one that we felt landed the most and that we committed to building out and refining, which was of course a huge process as well and took us a lot of time. Sarah will actually be giving a fascinating talk later about all the design and brand work that went into polishing this prototype, so I won’t go too deep into that.

Jennifer van Dam: What was really cool about this prototyping stage is, the user empathy led to innovation. We came up with things that we didn’t necessarily plan from the start. One thing we kept on hearing when we were asking for feedback on the UX and was it intuitive to compose and brainstorm?

Jennifer van Dam: A lot of the feedback we were getting was ‘it just doesn’t really sound like me’. And that made people drop off. They would compose an email or a document, but it didn’t sound like something they would write or want to use, so this was a huge risk of people dropping off and also, it wasn’t of the quality we wanted to meet. This led us to come up with the voice feature that Heidi talked about with before.

Jennifer van Dam: This is a classic example – it wasn’t on our roadmap from the start, but it really, being in tune with the user made us come up with this feature. I remember when we launched our first basic version of it, how excited everyone was, and that made us realize how important voices in generative AI – and it led us to much deeply invest in this area, so we’re keeping investing in this and also it helped actually become an important competitive differentiation.

Jennifer van Dam: To take it even further, we would also hear from users, okay, now it sounds like me, but in this situation it doesn’t sound how I want to sound, which was also a really hard problem. We heard this a lot in the email reply use case, and what we came up with is harmonizing your voice preference with your audience as well. Let’s say, I prefer to sound casual maybe 80% of the time, but I got this super formal email, it would be a little bit awkward if I replied casually there.

Jennifer van Dam: We also created a model that looks at the context of your communication and your audience, and harmonize that with your voice preference so it doesn’t diverge too much, but lands somewhere in the middle. This was an awesome, awesome project and Bhava is going to do a much deeper dive into replies after this.

Jennifer van Dam: Looking back, this was a huge product, and when I reflect back on what were the things that made it successful, I think first of all the team was really, really important when we started this project because building in a zero to one, very high ambiguity, it’s not for everyone. It can be quite chaotic.

Jennifer van Dam: We started with a very small team that was comfortable with iterations and ambiguity and okay throwing away work for the sake of learning. We intentionally kept this team very, very small until we resolved the main ambiguities, we started to scale up the team a little bit or slowly.

Jennifer van Dam: We were very intentional about the initial zero to one stage, and then scaling the team, and we had a lot of high alignment and energy because of this, because the people on the team were excited about these problems.

Jennifer van Dam: We also learned that prototypes are huge to align leadership, because it’s easy to get stuck in discussions, discussing strategy or design flows, but there’s nothing like proving it with real concepts and real user feedback and real prototypes. And then also our transparency principle really helped. We had a ton of cross-functional collaborators and of course it’s inevitable zero to one, there’s going to be all these changes and all these teams are relying on you.

Jennifer van Dam: We were super, super transparent with changes and reasoning and this really helped us creatively problem solve. In the case when there were changes, we would come together and this basically set up a place for innovation with cross-functional collaboration as well.

Jennifer van Dam: What’s next for Grammarly? At Grammarly, we believe that AI is here to augment your intelligence. That is really our product philosophy. We believe that AI is not here to take over your life or dictate you, but it’s here as a superpower, to help you communicate more effectively.

Jennifer van Dam: This is the product philosophy we’ve taken building GrammarlyGO, and this is our philosophy with all our next products and features that we’ll be launching. I can’t share too much about it, but I can share that this is the philosophy we take in building the next features that we’ll be releasing. Thank you. Alright, next up is Bhavana who’s going to be talking about the fascinating project called Quick Replies.

Grammarly Machine Learning Engineer Bhavana Ramachandra talks about engineering the generative AI product GrammarlyGO at Grammarly Girl Geek Dinner. (Watch on YouTube)

Bhavana Ramachandra: Thanks Jen for that awesome overview. Jen spoke about how Grammarly expanded into the user’s writing journey, and we’re going to take a small detour into one of the features that we built, which was Quick Reply, or replying quickly to emails. My name is Bhavana, I’m an ML engineer at Grammarly. I’ve been here for about three years. I was one of the many engineering geeks on this project. There are a couple of folks here in the audience today – Jenny’s here, Yichen is here, yeah, wanted to give a shout out to the team.

Bhavana Ramachandra: Today, I’ll be really talking about foundations in motion and with respect to the quick reply feature. Jen touched upon this previously. We have invested quite a bit in understanding what our users want in terms of their writing, and we were looking at expanding into this user journey, so really we built a lot of fundamental understanding over the years that helped us accelerate into the new product areas that we wanted to go into.

Bhavana Ramachandra: Another shout out is that, the team that worked on Quick Reply, all the point people, all the cross-functional people, were women. We have an analytical linguist, computational linguist, ML engineer, senior PM, and interestingly this has not been my first project when we were all women but yeah. Four foundations deriving from two projects that I have worked on coming into this one. One was Tone. Jen mentioned she has been working on Tone as well.

Bhavana Ramachandra: I’ve been here for three years. She’s been here for three years. Tone was our first project together, so Tone was one of them as well as Recap, which is our investment in 2022 to really go beyond the writing phase and to also to the reading phase to help users read faster so that we can help them write better. With the respect to tone, this was the first version of this. We also have tone rewrites, but this one helps users identify the top three tones in their text so that they can reflect on if that’s exactly how they want to sound.

Bhavana Ramachandra: Zooming into what was the fundamental understanding we built in each of these projects. The first one I’ll cover is Tone, and the three areas we invested is product definition, quality and our AI. For product definition, some of you might be thinking like, ‘Hey, this sounds like sentiment analysis and that is a pretty well solved problem’ but really our product team tries to think about what is the user value of sentiment. If you look at user text, honestly, eight out of 10 times users sound positive. That is not helpful to know.

Bhavana Ramachandra: What the product team did was actually define 50 tones over different aspects of your writing that’s actually helpful for you to know. Do you sound optimistic? Do you sound direct? Do you sound confident? Do you sound worried? Do you sound concerned? The product team really came up with a wide range of tones. In terms of quality, we iterated quite a bit over it and during this phase we actually came up with three levels of hierarchy.

Bhavana Ramachandra: When you have 50 tones, especially if you’re building models for 50 tones, it’s a bit hard – one to get data and to make sure you’re iterating over quality of all of ’em. The way we tackle this is, we define three levels. We have the tone at the really granular level, we have the sentiment at the highest level, but we also came up with tone groups that was maybe around, eight tone groups, and that helped us identify quality at different levels. Then, we really try to nail quality in terms of what is the user values.

Bhavana Ramachandra: Now as an ML engineer, I like to see quality always improving, but is it really worth it to invest in taking one tone from 90 to 92% or is it better for us to improve on a certain tone group that is really valuable to our users?

Bhavana Ramachandra: That’s the kind of trade-off that we had to make and then we really derive over time. I also want to mention our AI is one of our biggest tenants as Heidi mentioned. In this case, this feature was one of the first few pieces to pilot our sensitivity process. The REI manager today was during this process shaping up our formal sensitivity process. We’ve always done it and I think she was making that a very formal process.

Bhavana Ramachandra: Apart from that, we also wanted to make sure that any tone suggestions we make – because we have a varying level of quality, we did want to understand – what are the sensitive cases, and what is our risk of quality with respect to sensitivity. That’s something that we understood during this project as well.

Bhavana Ramachandra: The second one is Recap, which was our comprehension project that we worked on in 2022. Here we were going beyond the writing journey into the reading journey of the user. We invested a lot in understanding the user problem. We had many, many discussions about certain areas that surprised me that I’ll get into. There were also technical challenges because now we needed to again look at the context outside of the text that you’re writing. Where is it? Where are we getting this context from? And then we had a whole new set of ML problems, which is exciting for me.

Bhavana Ramachandra: For the user problem, I wanted to touch on two things. Delight versus value. We wanted to provide summaries and so we identified emails and we wanted to provide summaries as well as to to-do items. But does it really make sense in all use cases? For example, if you have a one line email, it doesn’t make sense to summarize that.

Bhavana Ramachandra: Or, if you have a social promotional email that says, sign up now, that sounds like a task, but all of us know that’s not really a to-do item for any of us. These are the kind of gotchas that we were like, ‘oh, we have a model, but is it actually useful in all cases’ or ‘how long should a summary for a really long email be’ versus ‘a one paragraph email’ be? These are the kind of things that we iterated over quite a bit. And also understanding the context and intent of the user.

Bhavana Ramachandra: Imagine you have an email, an announcement to your entire organization. If you’re a manager versus if you are an engineer versus if you are in design, you might have different takeaways from that email. Trying to understand a bit more of what is that context and what is the intent of the user.

Bhavana Ramachandra: We also solved a lot of technical challenges. Again, shutting out like our AI is one of our biggest pillars. Privacy is also our biggest pillar. We are very, very cautious about what is it that we are asking users to share with us and are we really providing value from it? Before this we didn’t look at the context of the user because we looked at suggestions of what they were writing. Now we wanted to provide value from that, so then we had to update our privacy policy and we also had to update our client side logic to derive this context.

Bhavana Ramachandra: Coming down to the ML problems itself, like I said, there were two things that we were trying to provide – summarization – as well as – task extraction – or to-do list, but because we were talking about delight versus value, context and intent, we also invested in a couple of different areas, including signature detection, intent understanding, and email taxonomy. Trying to understand what is the context of the user – that was more of email taxonomy and intent understanding, and signature detection really helped us. When you look at emails sometimes especially short emails, if the signature is longer than the email, then the model sometimes gets tripped up.

Bhavana Ramachandra: This is true for generative AI as well because yeah, for many different reasons, sometimes models are not perfect, so it helps to help them along the way, and signature detection was one of those areas.

Bhavana Ramachandra: In all of these areas, we spend time annotating our own data sets because email is a space where data sets are not as public, so this was one where we had to understand what data sets existed, what were the things that we were trying to build. As Heidi said, we have a big internal team of analytical linguists, and they help us identify the data, identify what our guidelines are, and go get us annotations that we can build models with, and these were all the areas that we collaborated with them on.

Bhavana Ramachandra: Putting all of that together, from the Tone project, we knew tone was something that our users cared about that we wanted to bring into this feature. You’ll see it says, ‘Jason sounds caring’, but that’s not like models don’t know to think about that. That’s something we have to prompt them to think about.

Bhavana Ramachandra: All of the tone taxonomy that I spoke about, the 50 tones, that made it into the prompt as well. In terms of the recap project, we really built a reply user – like, who are the users? You might get a hundred emails, but you probably reply to 10. What are these reply use cases is something that we had built an understanding that came into this project as well. That really helped us understand quality for launch.

Bhavana Ramachandra: As Jen said, we were not trying to polish, but we were trying to aim for user value. That meant, are we comfortable with the quality for launch? We know that we’re going to iterate over it, but for launch, does this look good? It’s something we try to understand.

Bhavana Ramachandra: Then, on the client side, a lot of the logic, that we built for the earlier project, got repackaged and reused for this one as well. We were using a new protocol, we were using, so it wasn’t just copy paste, but repackage. And then our AI as always, because it’s a generative AI output, we want to be sure that any output that we’re sharing with our users does not have bias, and some high risk scenarios. That’s something as well that we made sure this feature and the output of this feature goes through.

Bhavana Ramachandra: This was one of the few features we built for launch, but it did get a couple of different shoutouts. I know WSJ called out, we had a lot of users sending us, this was awesome. I specifically wanted to, we had a segment on NBC where Courtney Naples, who is our director of language research, spoke about this and the host in fact called out the feature and mentioned how the output of GrammarlyGO sounds like him, versus OpenAI does not. And yeah, that was a really nice moment for us to see. That’s it. Next off, we have Sarah who will be talking to us through all the explorations that the brand design team did for launch as well as our product.

Grammarly Brand Designer Sarah Jacczak talks about designing the generative AI product GrammarlyGO at Grammarly Girl Geek Dinner. (Watch on YouTube)

Sarah Jacczak: Thanks so much, Bhavana. Hi everyone, my name is Sarah and I’m a brand designer at Grammarly. I’ve also worked here for three years and I’m so excited to share the brand design team’s work and show some of the behind the scenes process of the GrammarlyGO launch.

Sarah Jacczak: To start off, I want to intro the go-to-market design team. The team consisted of product brand designers, motion designers, content designers, brand writers, design researchers, and design operations. This was a complex launch and we were designing something completely new, and there were a lot of moving pieces and constant changes. On top of that, we needed to move fast. Having a team with a wide range of expertise, it allowed us to work quickly and collaboratively, and we were able to impact areas across product, brand, and marketing for this launch.

Sarah Jacczak: I want to give a special shout out to the brand and content designers and brand writers. I’ll be sharing some of their incredible work on the GrammarlyGO identity and campaign later on. To give a quick overview of the scope of work, the brand design team worked on in-product systems, a new brand identity that included a new logo and color palette, and a go-to-market campaign toolkit, which included guidance on how to design and write about Grammarly’s generative AI features.

Sarah Jacczak: To do this work, we had to consider how users will interact with this new experience and how we would differentiate Grammarly go from competitors. This required close collaboration with product and engineering teams as well.

Sarah Jacczak: When designing GrammarlyGO, one problem we identified early on was, we needed a way for users to access this new experience. We knew that users were familiar with clicking on the Grammarly icon to open the Assistant Panel and accept writing suggestions, but integrating GrammarlyGO features with this existing UI was not an option for the launch and it was something we would have to address in the future.

Sarah Jacczak: For the launch, we needed to keep these two experiences separate. and we decided to add a second entry point into the Grammarly widget, which would open the GrammarlyGO experience.

Sarah Jacczak: Here are some early explorations of the GrammarlyGO entry point. So on the left we tried two different button designs for the desktop app and browser extension, and we consider it a badge treatment on the desktop app, which has floating widget. The benefit here is that on desktop, the widget wouldn’t be much larger, so it wouldn’t interfere more with text fields.

Sarah Jacczak: However, the visual treatment, it felt kind of like a notification and because of its small size, we were worried it wouldn’t attract much attention. And so we moved on to another exploration. On the right is another exploration where we considered having multiple inline buttons with different icons, so there would be a new unique icon for composed reply and rewrite features, but when prototyping this design, we found that it was a little too cumbersome, and so we decided to simplify it down to one icon for all GrammarlyGO features. And this is what we launched with a single light bulb icon to open the GrammarlyGO assistant window.

Sarah Jacczak: Having one icon as the entry point gave us room to surface prompts that have unique icons. You can see on the example on the right, we have the improve it icon with the pencil, and this prompt appears when a user highlights their text and it gives them a quick and easy way to generate another version of their writing.

Sarah Jacczak: While we were designing how users would access GrammarlyGO, we’re also designing icons. We started exploring icons before we had a name, but we knew it needed to be unique and it would live next to the G icon. We explored a wide variety of approaches. Some were more literal and represented generative AI, like writing and pencils and sparkles and magic, and other explorations were more focused on abstract representations of speed and ideation. But yeah, we could have kept going and going, and this is not even all the explorations, but because of the tight timeline, we had to make a decision.

Sarah Jacczak: We went with the light bulb because we felt it was effective in conveying the new ideation capabilities of GrammarlyGO. We also saw an opportunity to design new product iconography for prompts. These icons would accompany the suggested prompts that appear based on a user’s writing.

Sarah Jacczak: Early testing showed that prompt writing is challenging, and so we prioritize these suggested prompts that are based on a user’s context, and we wanted to make this experience more visual and more delightful. Again, icon explorations range from abstract to literal, but we saw that these icons needed to convey meaning, and also support the prompt compi so we move forward with the literal direction.

Sarah Jacczak: Another discovery was that operating within the new limited color palette was challenging and it didn’t quite feel unified with the existing UI, so we looked to Grammarly’s tone detector iconography, and these emojis, they would appear in the same UI as the prompt icons, so it made sense to create a cohesive experience here.

Sarah Jacczak: We referenced the colors and styling of these emoji to create the foundation for the new prompt icons and here the prompt icons that we designed for launch. You can see they’re literal in that they depict the meaning of the prompt in a simple way, and keeping them simple also ensured that they could scale and be legible at small sizes. We also selected colors and subtle gradients that felt cohesive with the existing emoji icons. This resulted in an icon set that feels warm, friendly, and is hopefully fun to interact with.

Sarah Jacczak: We also needed to consider scalability. There would be hundreds of prompts and we wouldn’t be able to design an icon for each prompt, so we grouped them into categories. Each category has an icon, and within that category are prompts that share that icon. For example, any prompts about writing or composition, we’ll use the pen and paper icon and any prompts about ideation, we’ll use the light bulb and so on. We also identified which prompts we feel would be used frequently and created unique icons for those to add variety and more delight.

Sarah Jacczak: While some of the team was working on iconography and content design in the product, others were working on the identity and go-to-market campaign. Here are some of those explorations – a variety of logos were explored and taglines, as well as graphics for the campaign, and some visual explorations use gradient orbs while others focused on movement and transformation by using over layers of shapes and lines. And for the go-to-market campaign, we created a new tagline as well – ‘go beyond words’. It’s active and it conveys Grammarly’s ability to assist users beyond their writing.

Sarah Jacczak: We also designed a new logo that incorporates a bolder G with a circle forming the O as a nod to the classic Grammarly button. For the GrammarlyGO identity and campaign, the brand design team landed on a concept that uses overlapping shapes to convey transformation and the iterative process where one idea is built on the next. The softness of the gradients also speak to the human qualities, and they’re juxtaposed with hard edges to represent technology, and these overlapping shapes were further brought to life with animation.

Sarah Jacczak: The team also worked on a design toolkit, and this toolkit was shared across the company. The toolkit included logo, color palette, illustrations, photography, motion guidelines, and a library of product examples to be used across the campaign. A style and verbal direction guide was also created to ensure how we speak about GrammarlyGO is consistent. The brand writers provided headline examples based on themes. There was headlines about creativity, such as let your ideas take shape, headlines about productivity, such as ‘discover new ways to get things done’, and headlines about trust like ‘AI innovation with integrity at its center’.

Sarah Jacczak: This campaign was pretty large. We had a lot of requests and a lot of marketing channels to design for, but because the brand and brand writers and brand designers collaborated and built these systems and guidelines, we were able to move quickly and create consistency despite many people working on the campaign production.

Sarah Jacczak: Here are just a few examples of the work created for the campaign. The team created a series of demo videos and animated gifs that show product functionality, and these were used across marketing and PR. The team also worked on onboarding emails, landing pages, in product onboarding, blog posts, ads, and social assets,

Sarah Jacczak: To get a further sense of the scope of work, here’s some numbers from the naming and identity work. Over 500 names were considered, 188 Jira tickets were completed, over 105 taglines were explored, 45 videos explored, 39 product examples designed and animated, and over 230 logos were explored. And so, while these numbers don’t tell the full story, and we had challenges along the way, the team was able to overcome this and collaboratively design a new experience and produce a successful launch in a short amount of time. Thank you.

Charlandra Rachal: Thanks Sarah. To all the speakers that put together this incredible presentation, I learned a lot and I work here, so I hope you guys really enjoyed all of that. Let’s welcome back all of our speakers for Q&A. There is someone who is going to be in the audience with a mic, and I see it first hand already, so we will get a mic right over to you.

Audience Member: As mentioned, it was uncharted territory. I was curious how you went about ideating the first project. Was it based on existing user information you had? Was it academic papers? How’d you go about it?

Jennifer van Dam: That’s referencing my talk, so happy to talk more about it. What I mean with uncharted territory is the solution. We knew it was a problem – we’ve been hearing for years that since we started, we hear from our users, they struggle with these communication problems. What was the uncharted territory is the solution and delivering the product in a way that lends and resonate with our users.

Jennifer van Dam: The approach we decided to take is directly into the prototyping stage because we felt it was really important to connect the text and the product to the user. Let’s say, you want to compose an email, we can design and show you concepts, but we need that moment of you writing your text and seeing the output. That’s why we jumped right into the prototyping stage as our way to research the solutions and the design approach.

Charlandra Rachal: There was another hand right here…

Audience Member: Hello, my name is Kate, and probably question also to Jennifer because it was on one of your slides. When talking about prototyping, you were speaking about more empathy and I took a screenshot. Let me see how it looked like there.

Audience Member: ‘Deeper user empathy.’ Can you please elaborate a little bit more on that, how it worked? How did you do it while you were still prototyping, please?

Jennifer van Dam: Yeah, deep user empathy. What I really meant with that was to understand and dive into the types of things our users are trying to achieve. What are the types of use cases that, here’s a prototype, did you use it for rewrites? Did you use it for emails? What were those things?

Jennifer van Dam: We did so many sessions talking to people and getting their feedback to get empathy and then of course we had questions, but then the feedback we got was, ‘oh, but it didn’t sound like me’. This is what I mean by deep user empathy is really getting into the mindset of empathizing what is working and what is missing. That really helped us inform iterations and changes and new features or scrapping features.

Audience Member: Thank you. I would assume that the launch of ChatGPT definitely affected Grammarly. What would be the key learnings? What were the key learnings for you as product leaders from getting the LLMS viral and thanks for the presentations. My name is Maria.

Heidi Williams: One of the things which I think was interesting – we’ve been doing AI for a long time, we’ve used a lot of different technologies, whether it’s rule-based or machine learning, or all sorts of different technologies, exploring LLMs on our own.

Heidi Williams: The biggest thing about ChatGPT that ended up, was actually the discussion in the world about how to use AI as an augmentation tool. Before, you would have to convince people AI was okay and trusted and then all of a sudden overnight everyone’s like, ‘of course you trust it. Look at this’.

Heidi Williams: Now all of a sudden we don’t have to waste time talking about should you use AI? Now it’s about how can we be a trusted partner on how best to use AI and help you be more effective and help you succeed in your job or in your life. It has changed the conversation of ‘should I’ to ‘how should I’, and that’s been interesting and amazing that we can now focus on just solving real problems as opposed to convincing people they have a problem that AI can help with.

Audience Member: Thank you so much. First of all, want to say huge shout out for Girls Geek and Grammarly for putting together. Thank you. That’s a great event. I’m a huge fan of Grammarly. Go now. It finally sounds like Shakespeare in my emails and not like a broken machine. I want to ask you this question. I think Heidi and Bhavana, that’s questions for you. You said that one of the feedback was, ‘it doesn’t sound like me’.

Audience Member: Are you using LLMs to train data which your users are input? And if so, how do you also prevent some data security in terms of, for example, I’m putting something in my email as a product manager about revenue or about some specific of the product, which haven’t been on the market, and I’m always a little bit worried where this data is coming from. Yeah, I think it’s a good question. One is LLMS for make me sounds like it’s me. And the second one is the data privacy. Thank you.

Heidi Williams: I can talk about at least part of it and then if you have things to add as well. Because Grammarly has been around for so long, and that we are a trusted source, we were able to negotiate a really amazing contract with our LLM provider, which means that they don’t train on any data that we send to them. Not everybody could negotiate that, but because we’re Grammarly and we’ve been around so long and have such a big user base, we were able to do that.

Heidi Williams: I feel like that was a huge thing that is very differentiated from just using whatever’s on the market is that they’re not training on any of the data that we send. From that perspective of the data privacy, but if you want to talk more about the my voice and about how we do that and how do we then, if either of you want to add…

Bhavana Ramachandra: I’m going to pass to Jen.

Jennifer van Dam: Could you repeat the question?

Audience Member: Sure. How you make, if we’re not sending data to any LLMs, how you make it sound more like me, for example, GrammarlyGO, always suggesting me to be more assertive, which I think I’m already too much and then I’m like, no, no, no, let’s make it more positive. Yeah, how this happened, how it sounds more like me is like data being trained.

Jennifer van Dam: We look at context and communication patterns, so it doesn’t necessarily train on your data per se, but on the patterns of your communication and the context. That’s how we understand your voice profile across.

Bhavana Ramachandra: To add to that – we understand what your tones you prefer or you use are, but we don’t actually pass that on to LLMs. We had our tone detectors since 2019. We’ve been telling users how they sound for a bit now. We use that information to really update the writing rather than train the LLMs itself with your data.

Charlandra Rachal: I feel like I’ve been neglecting this side over here, so right here in the front.

Audience Member: More of a quick technical question. Do you list all of the tones that you detect for and measure somewhere publicly, or is that behind closed doors?

Jennifer van Dam: We have a homepage that lists a lot, but not all, of our tones. We feel it’s too competitive to reveal 50 plus. But yeah, you definitely can find information about a lot of our tones that we support with tone detection.

Bhavana Ramachandra: Maybe this is a challenge. Can you write enough with Grammarly to find all of them?

Audience Member: The tables have turned. I wanted to direct a question to the second speaker after Jennifer van Dam. Yes, you, um, why does my voice sound like this? Ah ha ha I see what you did there. The question I have more specifically was, when tone is being cited as a suggestion, when you write a sentence and it connotes that, ‘oh, your tone is serious and neutral’, and when you add a word or two and it changes the tone entirely, I’m curious, what quantitative scales do you use behind the scenes to make those on the spot judgements? You mentioned your team had a lot of linguists on it, and I was hoping you could expand on that because that has been an object of curiosity of mine for a while, possibly. Thank you.

Bhavana Ramachandra: Yeah, I’m just going to Jen about… Yeah, I think for, in terms of how do we decide, in fact, when we started looking at rewriting for tone, I think our initial exploration had just neutral and new tone versus in certain cases we were actually able to provide three levels, friend, friendlier, friendliest, but really depended on how much data we’d have. If you can actually, you can be neutral, but you can’t be more neutral. It really depended on the tone and how much data we have.

Bhavana Ramachandra: I think this is a part where our linguist really helped us really dive deep into this and look into each of the tone that we have, get data for each one of these tones. I think for our tone retype explorations, we started with the tones where we had most understanding and first started with two levels and then moved on to three.

Audience Member: Thank you. Hi, my name is Leanne and big fan of Grammarly. My question for whoever thinks that they’re best equipped to answer this is, could you tell us a bit more about how Grammarly is fighting bias, and what are some of those solutions that are currently in place? And maybe thinking about, in the future roadmap?

Bhavana Ramachandra: Yeah. Our AI is one of our biggest tenants, as Heidi said, and we’ve always invested in this area. The couple of things that we have done are actually very, very public. We have blog posts about how we look at pronouns or bias in gender bias in data, and how do we make sure our generative AI suggestions, how do we measure that and how do we prevent that as something that we do? And as Heidi said, this is part of the process.

Bhavana Ramachandra: This is not something that you think about at the end of the day, you plan for this. You plan to have a sensitivity analysis right from the get-go. The other part of this, we’ve published a couple of different papers this year. In fact, in the REI space in, I want to say ACL. Okay, thank you Dana. You’ll actually find a lot of public information. I don’t want to pretend that I know more than I do in this area. I am getting onboarded though. Definitely, we have blogs and papers out there that talk about what are the solutions we have implemented.

Heidi Williams: Maybe just one thing to add to that is part of it is actually cultivating a good data set because you could imagine that you just take, I think we’ve seen this with LLMs as well, you just take the words out there and you might see a bias of a gender bias around when you’re referring to a male, they might be more associated with certain words in the general public than a female. Then you would imagine that percentage wise, it might suggest like, oh, if you’re talking about a man, you must be referring to this, et cetera.

Heidi Williams: We’ve done a good job of cultivating our data sets to help ensure that the data sets themselves are not biased, and that’s a huge aspect of it is just making sure that we’re not having any gender weighting as one example, or it could be racial, whatever it is. There’s just making sure that you have a data set that’s representative and it’s not going to sort of skew things in one direction or another.

Bhavana Ramachandra: Do you want to add to that?

Jennifer van Dam: I wanted to add to that a little bit about how much we care about this investment. We also, besides of course all the deep investments in the modeling, we also have inclusive language suggestions for end users that help basically eliminate gender bias in your language while writing or talking to your coworkers or your team.

Jennifer van Dam:This is an area I also worked on and it’s a really great part of our product. For example, maybe you’re writing, ‘the businessmen are wearing suits’, we’ll underline ‘businessmen’ and we’ll ask if you’re writing to an audience that you want to be inclusive of everyone, maybe replace it with ‘business people’ rather than ‘businessmen’. We also tackled this from the end user standpoint and helping them communicate more inclusively and eliminate bias where they’d like.

Audience Member: <inaudible>

Charlandra Rachal: For those who didn’t hear, the question was, how much do we focus on educating, when people continue to make mistakes in their writing?

Jennifer van Dam: The inclusive language product, I encourage you all to check out because education was a huge part of the product UI and the way we wanted to position it. We always want to be educational rather than forcing you because at the end of the day, the user is in control. You have agency. That’s what we always believe in. We also realized we have to explain why are we saying ‘consider replacing businessmen with business people?’ How do we explain? Because some people don’t realize that it’s so ingrained, you just type it, you don’t really stand still.

Jennifer van Dam: Another example, like whitelist, blacklist, that suggestion, a lot of users didn’t understand why we were suggesting to replace blacklist with blocklist, so we actually focused our UI around education. Why is blacklist/whitelist is perpetuating certain stereotypes, so consider replacing it – and that was a real aha moment because when you say that in a training, it’s different than when in real life you’re writing a text and seeing it and applying it, so it’s actually been really powerful in educating people

Audience Member: To Jennifer and Bhavana. Earlier you mentioned deploying LLM models, initially, were you were skeptical how receptive the users would be and how they would perform.

Audience Member:Can you talk about your A/B testing strategies? Did you roll out to a part of your audience, I mean part of the user base first, and then started gradually increasing, rolling out the new features too, especially the generative AI features? Could you talk about your A/B testing strategy and how did you scale it to the whole user base? And after you employed gen-AI features and these new features that you earlier talked about, how did it impact the revenue subscription revenue and the user base?

Heidi Williams: Start?

Bhavana Ramachandra: I think Jen covered maybe some of the A/B tests. I’ll let her, I can maybe speak to the launch plans, the alpha testing. I can cover that a bit. Especially for projects like Tone where we did iterate over quality quite a bit, we would try to identify one. We had internal annotations. Every time we improve our quality, we do more internal annotations to understand how much of a bump is it? And once we have a fair understanding, we run experiments with Gen AI, we had to take a slightly different process.

Bhavana Ramachandra:As Jen said, we had more alpha testing with users, really deep conversations in terms of understanding what is a useful generative AI LLM feature because we’ve had rewrite features in terms of generative AI for the longest time, but what’s a useful composed feature? What’s a useful quick reply feature? All of that was not really A/B testing. We were building understanding in this case. That was a lot more alpha testing.

Bhavana Ramachandra: Then for the launch plan itself, we have 30 million users, we have five different surfaces, we have extension, we have desktop app, we have an editor, we have our website, and we’re in many different countries, so this to me firstly was the most impressive one, because we had to do a geo launch across many different clients that all have different release cycles, so all of them had to be in sync because we wanted feature parity. We started with certain countries to make sure, one, we can handle the traffic. Two, all the features are looking or performing as they should.

Jennifer van Dam: The difference with how we approach modeling quality, it depends on the maturity of the track. In the zero to one stage we do a lot of offline quality evaluations and make sure it meets our quality bar and the metrics. We don’t necessarily test out multiple models yet, but in the iteration stage we do. One example where we’ve A/B tested our improve it rewrite, which in one click will improve your text, and there was a lot of experimentation we did there with tone behind it and conciseness and what lands the best with improving my text with one click. Typically, we focus a lot on offline quality evaluation of our models, and then in the iteration stages we do a lot of A/B modeling.

Audience Member: After new features, did you see any bump in the overall user base?

Heidi Williams: I can’t talk about specific numbers, but I think obviously there was a lot of excitement and interest in this area. I think we did see that there was new interest, and then also just seeing interest and engagement from our existing users using the product maybe in a different pattern than they had been before as well. It definitely feels like there have been changes, but I can’t speak about specific numbers.

Audience Member: What’s your tech stack typically for the whole GrammarlyGO is hosted on?

Heidi Williams: The tech stack? A lot of different parts of it… it’s hard to answer with a really quick answer. Our particular LLMM provider is Azure OpenAI. And then there’s a variety of tech stacks above that – different things that we’re using for the linguistic side of things, and then there’s Java, there’s Closure, there’s all sorts of different technology stacks and then we run on AWS otherwise.

Charlandra Rachal: Thank you. Then I only have time. Oh, oh, oh. I was coming for you. I know you had your hand out. If you still wanted to answer, we would definitely break the mic over your way. The last I have one time for one more question. Alright, Nancy

Audience Member: Behind.

Charlandra Rachal: She’s coming. Yeah, she’s coming.

Audience Member: Oh hi. Alright. I’ve used Grammarly for a really long time and this may be more of a product manager question because I’m also, I can write circles around everybody, so I don’t really need GrammarlyGO. I’m actually wondering, I’m thinking about, the roadmap further down for advanced writers, people like me who write. What’s coming?

Audience Member: Because I will say now, I use Claude a lot just to be like, Hey Claude, this is what I wrote. What do you think? And then Claude will say, that’s really good or not or whatever. I’m just wondering it’s GrammarlyGO moving in that direction for people who don’t really need help getting stuff on paper or on screen.

Bhavana Ramachandra: These are the comprehension projects that I was talking about. They’re all about trying to understand what the user is reading or what the user has written. For example, tone is something, even if it’s not correction, if it’s not editorial, you still might want to understand how your tone is coming across, especially in cross-cultural communication.

Bhavana Ramachandra: That’s something that’s helpful and in general as well, especially for long writing, we’ve gotten a lot of feedback about, I think this is one area that we were investing in – how we, so we show top three tones and let’s say people use Grammarly to write books or their fictional books, and does it make sense to show top three tones? Then they want a different – so this is the kind of evolution of the features that we see.

Bhavana Ramachandra: Comprehension is one area. In our generative AI in GrammarlyGO, if you actually open it up in a document, we provide a lot of prompts around understanding the gaps in your document, identifying what are your main points. All of these are just comprehension. This is just not how to improve your writing. Rather like this is what’s there in your document. You can review it based on a couple of different dimensions.

Audience Member: I should use Claude and Grammarly. Yes.

Bhavana Ramachandra: That’s the answer.

Charlandra Rachal: Yes. Alright, I wanted to say thank you so much to all of the speakers here and all you wonderful guests. I’m going to give a shameless plug if you didn’t already see, I’m in recruiting and we are hiring! Definitely talk to us, talk to me. I know we’re going to send a link out as well.

Charlandra Rachal: I believe there are more refreshments in the back and everyone is welcome to kind of hang out, chat, network. If you have more questions, I feel like we got through a lot of them without telling all of our secrets, but feel free to pull them aside and ask more questions. I hope you have a great night. Thanks again for coming out.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• See open jobs at Grammarly and check out open jobs at our trusted partner companies.
• More Grammarly Girl Geek Dinner photos from the event.
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!

Breanna Carodine (People Operations Specialist at Vannevar Labs), Caitlin Stangland (Senior Talent Acquisition Specialist at Vannevar Labs) and Ann Zeng (Software Engineer at Vannevar Labs) talk about the company’s culture, values, and mission, as well as the opportunities for growth and development within the organization.

VANNEVAR LABS combines top software engineering talent with decades of mission experience to get state of the art technology to the people that keep us safe. Vannevar Decrypt is a foreign text workflow platform built for national security.

VANNEVAR LABS IS HIRING!

• Senior Software Engineer, Fullstack (Remote)
• Forward Deployed Software Engineer (Remote)
• Senior Software Engineer, Machine Learning (Remote)
• Technical Lead – Platform (Remote)
• Senior Software Engineer, Backend (Remote)

Check out open jobs at Vannevar Labs!

TRANSCRIPT

Angie Chang: We are here today to kick off our career fair session with Vannevar Labs, and I want to introduce Breanna.

Breanna Carodine: Hi everybody, super happy to be here and to talk to you guys all. My name is Breanna Carodine. I work for Vannevar Labs as the people operations specialist, and that just means I basically work with a lot of onboarding stuff and employee experience and a couple of other things like that. I’ve worked here since December of last year and it’s been such a great company so far.

Breanna Carodine: We’re growing so fast and we’re doing a lot of really great things. Some of my favorite parts about the company is the way that we value certain things. Transparency is really important to us. We want to make sure that everyone is aware of everything that’s going on behind the scenes. We’re not afraid to share some of our trials and tribulations with each other but also celebrate our successes.

Breanna Carodine: One of the things that we really love is that we are a team of Jedi, meaning that all of the people who work here are really good at what they do, and that means that we can accomplish so much more. And everyone has very similar mindsets around being user and mission focused on our work.

Breanna Carodine: Everyone comes together with the same mission, being really great at what they do so we accomplish so much. Also, a value that I think is really important as that we put your wellbeing first. Something I tell every new hire who meets me and everyone within the company, “You cannot do your best work if you’re not your best selves.” And we truly, truly believe that here.

Breanna Carodine: We hope that everybody who comes into this company understands that we have time that you can take off. We have mental health benefits so that you can always be working at your best selves because we truly, truly do care about that.

Breanna Carodine: We are also a remote-first environment so that you can create the environment around you the way that it’s going to be best for you to do your work. I know that’s really great for me because sometimes I don’t need a lot of people around to get a lot of work done. Sometimes, I remove myself from this space and go to a coffee shop and work in a space that I can have all that energy, so that’s really awesome.

Breanna Carodine: Just a little bit about Vannevar and our mission, our basic spiel is that we bring together multidisciplinary groups of people with a wide range of experience and over 40 years of military experiences, engineers from some of the top tech companies and startups, and a passion for delivering mission-critical tools to support public servants on the front lines of the country’s most important national security problem.

Breanna Carodine: A lot of the work that we do, we work right alongside with the Department of Defense and we do some really critical and important work for the country. So to get into a little bit more of those positions and even more details, I’m going to pass it over to Caitlin.

Caitlin Stangland: All right. Hi everyone, so excited to be here today. My name’s Caitlin, I’m based in Northern California, I am a senior recruiter here working on technical and non-technical positions. I’ve been with Vannevar now since January. It’s gone really quick, it goes fast here. And some of the open roles that we currently have, I’ll just give high level overview.

Caitlin Stangland: Currently, we have our mission team and mission roles. Those are broken up into groups, mission success and mission development. Mission success, think of it as customer success within an organization. So most profiles we’re looking for are people with military backgrounds and then have worked within a customer success team at a tech company. So they’re used to working and communicating with clients.

Caitlin Stangland: These positions, our employees on the mission success team are communicating with our customers within the Department of Defense. And there is frequent travel involved with these roles. Mission development, think of it as growth and sales. Usually, we’re looking for someone with military background, and active clearance, and have that sales background and mentality. We also have various engineering roles open.

Caitlin Stangland: We offer internships. Currently, we have deployment lead internship openings for this year. We’re actively recruiting on those roles. We also currently have an opening for a data researcher position. This role’s actually very highly confidential so I’m not able to say too much about what this position does. Then we also are currently looking to build out our finance department. We have a controller opening right now who would own all of the accounting function at Vannevar and would build out that team.

Caitlin Stangland: Those are currently high level overview of our current openings. You can see them all posted on our site and on LinkedIn. Usually at Vannevar, our interview process, we like to really keep it to the same process across all positions at the company. Always starts with a recruiter screen, then a hiring manager interview dependent on the role.

Caitlin Stangland: For mission roles, we usually like to include a decom, think of it as like a conversation prep document that we would send you prior to meeting with the hiring manager so you have a focus of that interview. From there, there’s usually homework or a technical assessment. And then the final part is a top grade interview that’s usually one hour long where someone at the company just gets really, it’s just a lot of questions about your background and goes kind of role by role. So we like to just streamline it and keep it as quick as possible.

Caitlin Stangland: And overall, like Breanna was mentioning, we have awesome benefits. The company culture I love, the startup environment, Vannevar, we’re fast-paced, it’s very collaborative, we’re growing like crazy. But first and foremost, I feel like everyone just genuinely really likes each other.

Caitlin Stangland: We do a really good job of hiring nice people. Every team, no matter who you’re working with, they just do a great job of collaborating with one another. When I joined Vannevar, I did not have a defense tech background, so this was new to me and there was a learning curve there. And I never felt like no question was off limits, no question’s a bad question here.

Caitlin Stangland: Everyone’s open to educating and just giving all the information that you need to start your position at Vannevar. And we have great benefits, competitive pay, equity, the flexible working environment, remote working environment. We do have the flexible PTO policy.

Caitlin Stangland: Mental health like you were mentioning earlier, we definitely prioritize that so we do have a mental health benefit. And overall, I just feel like we’ve done such a good job of creating such an amazing culture. And I’m excited to talk to more of you later. All right, and I’ll kick it off to Anne.

Ann Zeng: Thanks, Caitlin. Hey everyone, my name’s Ann, my pronouns are she, they, I’m a software engineer here at Vannevar Labs and I’ve been here for a little over a year now. Also, my internet has been going on and off so I hope I don’t cut out. Well, I’ll do my best.

Ann Zeng: I currently work on the team that focuses on collection and ingestion of the foreign media that powers the Decrypt platform. Like Breanna said, we’re focusing on solutions in the national security space. And one of our biggest products is called Decrypt.

Ann Zeng: Let me talk everyone a little bit about day-to-day life as an engineer. We’re at about 30 to 35 people on the engineering department among a couple of different teams. I wouldn’t say that the team boundaries are super solid. Somebody described it as a semi-permeable membrane. There’s lots of projects that are across teams. We really welcome people to advocate for themselves in terms of what they want to work on, what they want to learn, and to the extent that there’s room to have people work on things slightly outside of their job description. That’s definitely still there. I mean, we are a startup. No one’s going to say, “Hey, we don’t want you working on more stuff. Why wouldn’t we want people to work on more stuff?”

Ann Zeng: In terms of the tech that we use, it kind of depends on the team, but I would say broadly Python, Node, React, TypeScript, Postgres, OpenSearch. Our stuff is deployed in AWS so yeah, if you have any experience in any of those or are you interested in any of them, feel free to come by and say hi.

Ann Zeng: Because we are remote first. We’re pretty understanding about people’s time zones. There’s no hard start or stop times for working hours, like if someone messages me but it’s like 7:00 AM my time, nobody’s like, “Why aren’t you answering my messages?” For required meetings, at least on the engineering side, we’re pretty light on those, sprint rituals, standup retro planning refinement, one-on-ones with your manager. Those are usually the required ones.

Ann Zeng: We really encourage people to sit in on other meetings. So there’s an engineering department wide round table, there’s a company-wide demo space for engineers to show off like, “These are the things that we’re working on.” I love to lurk in different Slack channels just to see what’s going on. We really encourage people to get involved or to just spread transparency about the progress of work, that kind of stuff. Like Caitlin and Breanna said, we’re trying to solve problems in the national defense space and we want to do it with some full people. I’m really excited to meet everyone. Please feel free to come by to the booths. Thanks.

Angie Chang: Thank you so much for these great introduction to you all and the roles and the teams. We’re going to meet you in your booth in about 10 minutes. And now, we’re going to move on to our next session. Thank you so much, Vannevar ladies!

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner or Career Fair? Talk to us!

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner or Career Fair? Talk to us!

Sukrutha Bhadouria: Sanchika Gupta and Tutu. The talk’s title is “Beyond the Algorithm: the Human Element in Developing Trustworthy AI”. Welcome, ladies.

Yunwen Tu: Thank you. Thanks everyone for joining the session today. Today we will share our thoughts and learning about building trustworthy AI. First of all, we will share a little bit more about ourselves, ourselves. My name is Yunwen, go by Tutu. I’m a user experience designer. I enjoy using design as an approach to learn and solve problems for people in both digital and physical way.

Sanchika Gupta: Hey, I’m Sanchika. I’m a data scientist with experience in the field of technology, cybersecurity, and human-centered AI. In a former life, I was a computer science professor, though while we are building the human-centered trustworthy AI and have been working with each other. I’m generally curious about this thought of what is the human involvement in building the trustworthy AI.

Sanchika Gupta: There are the three topics that we are going to discuss in this talk today. The uncertainty and concerns of AI development. How do we build and maintain trust relationship in AI and the unique value of human expertise in the age of AI.

Yunwen Tu: What’s your reason, conversation, or search about AI? For me, I chatted with my designer friends, and we talk about how AI can streamline some of our work, and is there anything can be replaced by it so that we can do things faster, such as making icons or some simple websites.

Yunwen Tu: We also talk about the new design trends that brought by the new development of AI. Since the breakthrough of the large development in large language model, more people have exposed to this technology and learn about the possibility how AI can be applied to their work. Lately, I have even heard users, they share that they have the desire to add this and that function using AI in their product after their self-education.

Yunwen Tu: In general, I got a sense that many of us are concerned and uncertain about the impact that AI can bring to us or our society. We start wondering, has the time finally come, is finally replacing us now? As we already see AI in is applied in many fields such as healthcare, automobile and et cetera. I ask my co-speaker Sanchika, where do you feel AI has already made a big impact and what kind of role is AI playing?

Sanchika Gupta: Automation of AI may lead to replacement of certain human roles. However, AI also presents us with newer opportunities and creates our jobs easier. Let me talk about certain examples where I feel AI has been present around for so, so many decades now, and it already feels as a partner. First example that I would like to talk about is natural language processing.

Sanchika Gupta: AI has significantly improved natural language processing capabilities. There are virtual assistants like Google Assistant, Amazon Alexa, which utilize the AI algorithm to understand and respond to voice commands. Now, it may seem a little trivial to talk about these because they have become a part of our daily lives, and I personally use them on a daily basis to like set up reminders, ask about the weather, ask it to provide directions for a destination while in the car, making our daily lives more convenient and efficient.

Sanchika Gupta: Another example I would like to give is of natural language translation. There are platforms like Google Translate, which leverage AI algorithms to provide real time translations between different languages. If you go to a country where they don’t speak your language, you can still communicate effectively. I have done it myself so many times. With this, I want to say that AI may automate tasks that require basic skills while human can focus on higher level responsibilities, harnessing creativity and imagination.

Sanchika Gupta: The next question is how can we have more access to the system so that we can use AI as a partner? Generally speaking, education and awareness are crucial in fostering trust in ai. Trust is very essential for building a reliable, transparent, and available use of system. AI has been present in various forms for many decades now., even during my university studies, I delved into the neural networks topic and AI’s potential to replace job was already circulating at that time.

Sanchika Gupta: However, the conversation around trustworthy AI only gained prominence with the emergence of large language models, instances of AI generated hallucinations where the system just make stuff up, started gaining attention while getting a recommendation for an unwanted TV show may have minimal impact. A recent incident in which an AI system made a judgment on a legal case without the attorney’s verification highlighted the potential consequence. These recent repercussions have brought the issue of trustworthiness to the forefront, causing it to enter the collective consciousness of all of us.

Sanchika Gupta: Let me throw another example on you and you tell me which would you prefer. If we were to compare a trust in an AI driven car accident versus a human driven car accident, what would you choose the second time? My opinion as humans, we tend to trust other humans more than AI. How can we bridge the gap?

Sanchika Gupta: I believe that by focusing on AI literacy, upskilling collaboration, and ethical considerations, us individuals can also be empowered to embrace AI as a tool to enhance our skills, productivity and relevance in the job market. Now, I would like to ask Tutu, how do you as a designer build and maintain trust relationship in ai?

Yunwen Tu: As a designer, I started this journey by understanding the technology, especially why AI failed and why people don’t trust AI. The major distrust I learned in AI is the lack of transparency. We haven’t considered enough that trustworthiness is a high priority in building that.

Yunwen Tu: Many AI models feel like a giant black box sitting between the input and output. We don’t, we don’t know how it works and it just did it work. When handling very mundane tasks such as grammar correction, language translation, it’s great when they’re magically done by machines. But when it comes to like riskier cases with bigger impacts such as loan approval, it’s impossible to rely on a black box like this, which you don’t even know if can understand 10% of your problem. There are also potential ethical biases in a model that needs to be monitored closely.

Yunwen Tu: We help user increase the transparency, the observability and the visibility to make the process and the model more interpretable and explainable in their context work. That’s also baked in our design principles.

Yunwen Tu: And as part of our design principles, we also use design thinking method that to work with our users to understand what does trust mean to them, and discover how AI can solve their business problems in a trustworthy way. For here, I would like to give you two examples that we use the user interviews and other thinking methods to solve problems for our users.

Yunwen Tu: The first example that we are working with an insurance company to reduce their business loss. Through many rounds of discovery interviews with underwriters and their managers, we actually found their primary challenge is not about finding the best algorithm to analyze their internal data.

Yunwen Tu : Instead, they want to understand better how the events happened in the past 10 decades that had impact their current business performance. In the meanwhile, we feel everything moves way faster now. For our underwriters, they need to quickly catch up with all the new updates such as the regulation change, the new settlements of lawsuits in their professional area. In the end, we build a tool that use natural language processing to help our user to connect the dots and find the needle in the ocean of internet data, which is the result we would not expect if we didn’t spend that much time to talk with our users.

Yunwen Tu: The second example I would like to share is related to our ops platform. And as part of my UX research, I regularly chat with different users such as data scientists, business ops, and et cetera.

Yunwen Tu: I found the expectations from our data scientists on monitoring AI models are very different from other general business users. They’re not looking for a no code or a fully automated experience. Instead, their philosophy is not to trust the data, not to trust the model until they have seen enough evidence to take action. It’s very crucial for us to deliver those insights clearly and efficiently.

Yunwen Tu: From our users, I actually learned trust is not purely top performance, not the best performing model. Trust means making informative decisions after peeling off the complexity and the root causes. Now Sanchika, what does trust mean to you as a data scientist and how have you built trust in your practice?

Sanchika Gupta: Demystifying the AI systems and ensuring reliability helps human use them with confidence. There are certain visible limitations of AI, for example, drift observability, root cause analysis, bias and ethical use are important to establish trust. So let me explain with an example how I establish trust in AI.

Sanchika Gupta: We as data scientists do not tend to trust our model or data. Instead, try to gather enough evidence around it and then be able to trust it. Let me talk you through that process.

Sanchika Gupta: Let’s consider a case of an AI driven customer service chat bot used by an e-commerce company. The AI chat bot is deployed to handle customer inquiries and support requests. Over time, they noticed that there is a decrease in the customer satisfaction scores and there is an increase in the unresolved issues.

Sanchika Gupta: The first step at this point, which I as a data scientist would like to do is to check on the model performance, let’s say, model performance evaluation, reveal that there is a decline in the chatbot’s accuracy and performance compared to previous months indicating a potential issue. Then we can begin closely monitoring the chatbot’s interactions and collect data on input queries, chatbot responses and user feedback. Now by looking at all of this data, closely plotting it, we might be able to identify certain patterns or anomalies in the chat bot’s behavior. During all of this observability and analysis, let’s say that we were able to identify a set of queries that consistently receive incorrect or nonsensical response from the chat bot. Now these queries stand out as potential outliers as they significantly deviate from the expected behavior. Then the next step could be to, let’s say, look at drift analysis on the chat bots performance.

Sanchika Gupta: We can compare key metricses like customer satisfaction scores, response accuracy, and resolution rates over different time periods. During this analysis, we notice there is a significant decline in performance starting around the same time as there was an update in chatbot’s knowledge base. Based on all these findings, we start the root cause analysis and discover that the update to the chatbot’s knowledge base introduced some incorrect or incomplete information resulting in chatbots diminished performance.

Sanchika Gupta: Through this example, we observe how model performance analysis, observability, outlier detection and drift analysis collectively contribute to the identification of the root cause leading to targeted corrective actions for enhancing the chatbots performance. This provides a glimpse into the case that I mentioned above showcasing the methods employed to established trust in an AI system. This also demonstrates the importance of human involvement in analyzing and improving the AI system, reinforcing the notion that despite all of its capabilities, AI cannot fully replace human judgment and decision making.

Sanchika Gupta: Now this leads to my third question that we would like to discuss here. What is the unique value of human expertise in this age of AI? Now again, creativity, imagination, and diverse opinions are very unique to humans. If let’s say there were to be a discussion, we humans can participate in a discussion and arrive at different conclusions in the same situation

Sanchika Gupta: At the same time, AI lacks the ability to participate in a discussion as an equal lacking both opinions or any standing in human conversations. Now let me quote another example here. New neural networks founder Professor Jeffrey Hinton, six years back in 2016 said, we won’t need radiologists to analyze scans and image perceptional hams can do all the scanning and diagnosis by themselves. Six years have gone by and we are nowhere nearby. It is not because of compute power or resources because I feel that compute power and resources have only been growing in the last couple of years.

Sanchika Gupta: What I believe is that AI can only solve very well-defined problems. What happens when it is posed with ill-defined problems? That is where human ingenuity comes in. All AI attempts to do is to recreate memory and computation capability of human brain. But what makes human a human is not just being able to solve the task, but be able to synthesize the complexity of this world and make decisions on the basis of that. Now, at this point, I would like to ask Tutu to give her thought process around this topic.

Yunwen Tu: Thank you, Sanchika. Those are great takeaways and what you just shared also, remind me again the user interview process in our design method. We do that to understand user’s journey and the pain points, and then present a personal story that summarize our learnings and the synthesis. Sometimes I feel AI is like an abstract persona that is summarized in a way with a well-designed cover and well-defined title. However, when we are doing the interview, the persona story, it’s not about creating an abstract figure, but to emphasize with our users’ needs, their feelings and the the reason why they’re making those decisions. There are all, and also this process are all done through our communication and synthesis in person. But AI does not learn new insights as we do in those contexts. They also don’t understand the complexity of the world like us.

Yunwen Tu: For example, when I read the news, the debates on the news, when I also work with different people, design for different users, I always feel so, and this comes from our unique experience, our desire and the belief that makes us like diverse, unexpected. And sometimes we argue we also have conflicting ideas, but this also made the world and humans like very unique that AI cannot replace with. That covers all we want to share today. Thanks for staying with us. And if we have a little bit more time, please feel free to ask us question now or reach out to us afterward. Thank you all.

Sukrutha Bhadouria: Thank you. Thank you ladies. Yes I would definitely encourage everyone to reach out to you both on and ask their questions on LinkedIn. Let’s keep the conversations going and I really encourage everybody to rewatch this content, share this content as much as possible with everybody. We really appreciate the time that you’ve taken out Sanchika and Tutu. Thank you everyone for attending. Bye everyone.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner or Career Fair? Talk to us!

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner or Career Fair? Talk to us!

Angie Chang: With us today, we have Rahmira Rufus. She is a next-generation enterprise security architect, a university professor, academic and industry scientist. Welcome, Rahmira.

Rahmira Rufus: Thank you. Thank you very much, Angie. Hello everyone. Very honored to be here to share with all of the insights, all of the wonderful things that’s happening at ELEVATE this session. Like Angie said, my name is Dr. Rahmira Rufus. I’ve run a gamut of things that I’ve done in the cyberspace – lot of technical work with the elements that she spoke about, contracting areas, purely and academic. This element focuses more on a lot of the outreach work that I like to do that I’m not really able to dive into within, you know, places of employment and things of that nature. That’s what I’m gonna talk to you about today. I thought this would be a really good topic to bring up at a Career Fair, as folks are trying to find their way and navigate through the employment process, excuse me, workforce and academia and that whole gambit, that pipeline.

Rahmira Rufus: I’m gonna dive right in. Like I said, the title is Exposing Gaps in Cybersecurity Workforce and Education. I like to say that this particular work or slide deck is stemming from some work that had a different direction. This is more of a tangent of that work. I’ll start by addressing why are we even doing this analysis? Originally we wanted to find out what was happening within the cybersecurity paradigm as far as that pipeline for supply and demand. The workforce for the talent and things of that nature. What we discovered is that this is not a unique problem. This actually occurs in other industries. One in particular biomedical things of that nature, where you have very specialized kind of criteria for resourcing talent and staffing needs.

Rahmira Rufus: What we discovered here that’s the reason why we’re doing a gap analysis. A gap analysis is a very simplistic term. We’re trying to see what are those kind of hindrances, are those kind of blockers for where a particular environment or situation is at an actual level of performance? How do you get to a desired level? How that relates here through this series that I use when I work with my clients, how that relates here is that we’re trying to see what is this huge disparity against folks trying to either jump into the cyber field, maneuver within their career path, or transition to higher levels, or, even in some ways, try to make an entire pivot in another direction. What we discovered in this work was that there are some elements that are not being taken into consideration that makes the process a little more convoluted than it needs to be. That’s pretty much what we’re addressed. If there’s anything someone needs me to go back, just let me know. but other than that, I’m gonna dive right into it.

Rahmira Rufus: What we started was, we looked at a generalized problem,. What’s happening within the cybersecurity workforce and basically feeling that void of the workforce, of the talent that’s out there and the talent that’s more importantly needed. We started at a global scale, and the diagram to the left, the far left, is what we collected as data, globally. Every single place on the planet is not represented, but we try to get a proper representation of the skillset and those particular numbers that we would thought were relevant for the sample population that we were looking at.

Rahmira Rufus: We saw a disparity when in the US and wanted to focus there. Now, I do want to say the data that was collected for this very huge gap that shows for China, <laugh> and India, there is some emphasis there. However, that’s not a part of the scope of this talk, but that is something that as time progresses, we would like to see if we could journey into that gap.

Rahmira Rufus: We have some assumptions, but, you know, since we’re scientists, we’re not allowed to do that. We actually have to follow a process. Right now we’re in pretty early qualitative work with that study. If I dive over to the area that we’re looking at, we then wanted to focus in the United States. In my next slide, I want to show where we started seeing some disparities. I’m taking those two slides over the far right and they’re on the far right on this slide. We had looked at basically a percentage in a kind of you empirical actual value for each one of the data points for the job openings up against, compared against the employed workforce, right? The diagram on the right looks like it kind of flows in a normal fashion.

Rahmira Rufus: The problem is that the diagram in the middle, we saw little disparity around 2021, moving into this year. It didn’t quite match, it wasn’t like a huge disparity, but that triangle with that slant at the bottom, that that variation was really, really off. We said, okay, look, let’s dive into more granular elements. And we started looking at for state, and once again, every state ends represented here, but this is a sample population that we think would be a best representation. We noticed was by state that matched or was similar, kind of congruent to the, the diagram on the far right, but we were still concerned about that percentage. Why did we get a percentage that did not look like the two diagrams that basically are on the far left and the far li right of the middle one?

Rahmira Rufus: That’s where we started diving into what we considered to be this problem. And the reason why this is important for this particular forum is because in the next few slides, you’re gonna start to see some of the reasons why when folks are looking to streamline and understand exactly what they want to do in cyber, it’s a little more complex than than you actually thought from us just taking that tally that you saw on the previous slide, a very general approach to trying to solve or answer this question.

Rahmira Rufus: What we discovered was that approach was a little off because we didn’t look at the true nature or components that were involved with the supply and this demand. Now if we jump to this slide, we’re talking about a simple supply and demand problem, right? We then associated demand with the knowledge, the education that’s requested, as you see in the chart in the top, and then we looked at entry-level jobs and then against paired up against their requested education. If you notice in each one of these charts, it seems like folks in the bachelor’s degree area, they’re winning, right? For these particular cyber skills. Then, you have some luck in the kind of the sub, those are pretty much a lot of the associate degrees, junior college, but also a lot of the certifications and licensing that people engage in when they either have done a lot of higher education / learning, or they could be adult learners, it could be all kinds of different reasons, but they don’t necessarily go through a traditional degree program.

Rahmira Rufus: Then you see the graduate level of folks having that kind of smaller percentage of kind of matching a little bit of the sub bas. Now, what is funny, thank you. What is kind of funny here is that in actuality, because you know, I’m a teacher / professor as well in my background, I notice that’s not true to what’s happening here. Students, uh, adult learners, uh, folks in the field trying to figure out where they are, where their next step is gonna be, what’s their next direction, they’re not having the success that’s being represented in these charts. What we had then decided, okay, since we’re talking about supply and demand, well now let’s look at the supply side. Now let’s look at the workforce. What we had discovered was that this was a much more complicated problem than we thought, than someone just saying, Hey, I would like to get into the cyber field and I’m just gonna protect, I’m just gonna pick this particular area and I should be fine.

Rahmira Rufus: I’m just gonna grow here in years. Now an example that I will give, and not to shed any kind of negative light or anything on this particular field, but if we look at fields like psychology, most folks know this is something that’s been told by everybody in that field. Psychology, psychiatry. If you are not pursuing the master’s or PhD level, you’re gonna have a difficult time finding the type of placement that you thought you would being that breakthrough psychologist or psychiatrist. That’s something that’s a normal situation in that field. From what I’ve told, it’s been worked on over the years, but that’s usually something that’s been accepted in this space. That’s not necessarily the case. There are absolutely so many options and so many variations of what you can do, not only in cyber, but if you go down into its larger focus, like a computer science or pure network admin, or you name it, this becomes a little convoluted.

Rahmira Rufus: Now, I’m being very, very nice with this slide. Thank you. I’m being very, very nice with this slide because the next slide I’ll show is actually a very large data set that we had to work through so that I could bring you such a pretty visual right here as far as just some of the things that you have to take into consideration when you’re trying to think about what’s going to be your cybersecurity career path journey.

Rahmira Rufus: One thing really quick, I want to note if folks are paying attention right here to this graph over here, notice how low security intelligence is, I can give you a little hint of that. Remember, this is a trend, a growing field. Right now, the reason why this number is so small, that is actually being developed as a track.

Rahmira Rufus: All of the data analysis, all of the business intelligence, all of the different elements out there are now being able to be fueled and tunneled into a path that’s going to be security intelligence. Just letting you all know, look for that on horizon. However, these other tracks, they had experienced the same thing previously. That is another added complexity to this field, as things change, as things morph, as we’re talking about the next generation of computing, how do you know where to go, where you’re at, what you need to be? was able to break down some of this in some of these charts, but, I think everyone can kinda understand what’s going on here. Right here, you’re talking about, we’re, we’re usually using like as the little bit of a control, right?

Rahmira Rufus: The job opening, say, nationwide. Then you have to look at the entry-level jobs versus the knowledge that you would need. Then also you’d have to look at the skills and knowledge that would be required for particular sets. And then on top of that, what type of certifications and licensing would I need to perfectly be where I wanna be or where it is I’m trying to go see just that quick. On the next slide, as I said, this is just to give you an example. We have so many repositories that we pulled this information from, and I was only able to get 25 in these tiny little visuals on this screen. That is not something we want you to do. What does this mean to all of you? All right, here’s a little in the second slide you saw that I was gonna talk about dilemma, and then I’m gonna jump into the, what I think is a quick solution.

Rahmira Rufus: Here’s a method that you could help you without having to go through all of that granularity that we discovered in this process. One, be proactive, right here is a diagram of a chart, right? That’s all of the data that you see before, and I’m going to have this recorded, basically provided for you in the next two slides is from platforms like cyber seek.org, ISC squared, the, their platform, their education focus of the nice framework, things of that nature. You can go and get all of this data, get all of this information, and you can get metrics about these particular fields, right? Right here is a combined chart, a visual where you should go out there instead of waiting to go to an employer or waiting to figure out what’s out there or talk to your professor or whatever, kind of overindulgent process that you normally do.

Rahmira Rufus: You go out and be proactive and find out what these, what these particular roles and these skillsets are, and start diagramming them and them in a skillset similar to this slide. Now, to break it down even more, and I said to DIY – do it yourself – break them down into different skill sets that you find to be requested skills. And like I said, and these particular prac platforms, you can go to the go there to find this information. Right here, this is basically a broken down version of the slide you just saw. And you can pick particular skills and find out where you measure right now, right? And what is it that you’re gonna need to either increase that proficiency or, you know, whatever it is that you’re trying to kind of get at, right?

Rahmira Rufus: Be realistic with yourself, okay? Because this is about improving you and making yourself a much more viable product as you move forward. Use elements like that to set yourself up for success. Some real quick wins, as I said – I can tell you about – is do your own analysis. Like I said throughout this slide, you don’t have to dive deep like we did with the tangent that we found as a new area of research for the disparity in the path between the cybersecurity workforce and its talent and resourcing, but at least try to leverage different areas, like I said, cyberseek.org, these different platforms.

Rahmira Rufus: On top of that, when you actually capture all of these roles, develop proficiencies with analytic service or, data, data analysis… You can leverage different visualizations, power bi, you know, build your chops in those areas or create diagrams or metrics like these. I will let us say, since this is a Career Fair, these types of visualizations go very, very well in front of potential employers. Department heads, You know, folks that are trying to move up within an organization and even for folks that have their own endeavors, you know, clients like to be able to see the skillset and, proficiency of your talent, of your personnel, and being able to provide these types of metrics really, really set the right type of parameters for what you want to show as one, being proactive and kind of owning your area.

Rahmira Rufus: Some of the other things do self-assessments and skills inventory, slightly like the one that I just showed you in the previous two slides. Try to keep a skills inventory. If these things ever come up, you can tell people, “Hey, I’m at a level three or six at this particular coding element” (or this particular project management software, whatever it is). Gauge where you are and know where you’re headed, right? Or where you’re trying to go. Put your skills to the test battle, test your elements, okay. And things of that nature. I’m sorry, Angie, are you gonna say something?

Angie Chang: I was gonna say thank you so much for this talk. We’re at time, but we will say that you are on LinkedIn and people can connect with you there. Thank you so much, Rahmira.

Rahmira Rufus: Thank you. I hope everyone enjoy my talk and enjoy the career fair. Best of luck and be the best You.

Angie Chang: Thank you.

Rahmira Rufus: Thank you. Bye-bye.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner or Career Fair? Talk to us!

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner or Career Fair? Talk to us!

Sukrutha Bhadouria: Soundarya is a seasoned senior product manager currently leading growth for small and medium businesses on Instagram. Before Meta, she was a lead product manager at Yelp, where she’s spearheaded a team managing over 10,000 US restaurants. She launched Yelp waitlist kiosk, driving growth and innovation in the restaurant technology industry. Welcome, Soundarya.

Soundarya Chandar: Thanks. Excited to be here and really distill and share what I’ve learned over the years. I’ll kick off this talk on acing product management interviews with a little bit about me. This year is special because I’m celebrating a decade of being a PM in the Bay Area, which has meant that I’ve worked at different companies and I’ve been an interviewee over 200 times, and I’ve been on the hiring panel maybe about half of that time. As I was prepping for this talk, the stat actually surprised me too, and I was like, whoa, that is a lot of information and insights that I’ve gained over the years.

Soundarya Chandar: The goals for today’s talk are twofold. After being in over 200 interviews, I realized I did half of them quite mindlessly. I followed all the prep tips that you felt that you read on the internet, and I will be honest interviewing did feel like a second job. Sometimes the search took months and there would be times where I get through the onsite rounds and things don’t work out, and it really sucked because I was back to square one trying to look for the for the next gig. However, today I’m at a point where I actually love the interview process and the task, both being an interviewee, but also being on the side of an interview panel.

Soundarya Chandar: Today I wanna walk through a little bit of how that mindset shift happened for me and how that my day-to-day job played a big role in it. And the second goal I want to highlight for today is how developing this continuous learning mindset is so important, and it took me a long time to get there. I’d really want to help you all see the value of building your skillset, especially – this is so tough because as a PM our job day-to-day jobs are already so difficult. There’s so much context switching, there’s just so much your task to-do list is never ending, and there’s a life outside of work that we want to participate and be present in. There’s a way to actually get around it, especially as we think about time and balancing work and time and investing in our skill sets.

Soundarya Chandar: Before I jump in, I wanted to take a step back and talk about perspective – on things that we absolutely cannot control. Especially now, the market conditions are so tough. And, and all of this, just as we were starting to feel like we were recovering from a global pandemic crisis. There’s the economy, there’s the market conditions, there’s global factors, politics, policies that are changing. Every company seems to have a different hiring process. And of course, we are at the age of AI, and there’s technology advancements are happening so fast that it’s, it sometimes feels hard to really keep up with it. As PMs, we like to be in control a lot – so when all of these things are happening, it can feel pretty depressing and deplete our energy resources. Which brings me to my next point on what can be actually controlled here.

 Soundarya Chandar: In my opinion, the only thing that you can control is in investing in your skillset. This is something that you keep building growing and it sort of never leaves you. The closest analogy is, of course, if you regularly go to the gym, your physical fitness is going to get better. The more experience you gain, you in theory will get better at decision making. We often forget that how important and critical it is to keep developing our skillset. I really believe that by focusing on building our skillset, we can actually showcase all of that very easily in an interview setting.

Soundarya Chandar: Let’s talk a little bit about what these skill sets are. The first time I looked at this map, I felt pretty overwhelmed. This is a resource that I picked from this really amazing PM coach I follow on Twitter – Shreyas Doshi – who has been a PM leader at many different companies in the Bay Area. This skillset map looks so daunting – like where do you even start?

Soundarya Chandar: The good news is – if you’ve been a PM for some time, you’re already flexing a lot of these skills in many different ways. But, we don’t often take a step back to think about how we are calibrating. Are we actually building and sharpening these specific skill sets? Early on in my career when I was interviewing, I used my intuition a lot. That is just something that I’m naturally drawn to. I would research the companies, I would research the products, I would have all these ideas of how they can improve it. In a way, I was really getting good at building my product sense, and I would do really well in those interviews.

Soundarya Chandar: But when it came to some other aspects of interviewing, like analytics, storytelling, communication, something or the other would just wouldn’t work out. When I started later on interviewing at FANG companies where the interview structure is very closely aligned to how you measure your skillset, I worked with a program online, which I can share later that, that modeled the interview set up along the lines of the skillset. I was doing more mock [interview]s and getting more into the program, I realized the side effect, which was that I’m getting better at my day job.

Soundarya Chandar: I realized that I’m avoiding doing certain things at work because it was just not a skill that I felt really confident in. The a-ha moment for me was, oh, I’m doing these mocks. I’m improving the skillset. How do I do that in my daily job? A typical product sense question would look something like this. You are the PM for an amusement park. How do you improve the visitor experience? The old me would jump at this question. I would have all these ideas buzzing in my head, but I needed to learn to pause, breathe, remember, what am I really trying to answer here? Remember the user, remember the pain points? Remember what business outcome I’m trying to drive? Because that is the structure ideally that you follow in an interview, but you of course adapt it to your style.

Soundarya Chandar: If you actually think about it in your day job, you, you are constantly exercising and flexing these skills when you write a PRD or when you’re doing a product review or a strategy session where you are honing in on who the user is, who’s your audience, and why should that audience care? When you’re working with your designer, either whiteboarding sessions, you are mapping out the various happy parts and unhappy parts, or when you’re talking to your partner teams, if you wanna get some collaboration opportunities identified. At all times you are actually flexing your products and skills.

Soundarya Chandar: Similarly, a common question would be, you are the PM for talk’s feed. How would you set goals for your team? You may not. It’s a hypothetical scenario, of course. You start to think about, oh, what’s talk’s mission? Who are its users? What are their pain points? How do I measure the impact? Again? But again, you’re doing all of these things when you write your PRD when you’re setting up OKRs, maybe you’re debugging an issue, you noticed a dip in your metrics and you’re trying to figure out what’s going on. Or if you’re talking to XFN folks on tradeoffs, like, which features should we build? What has a better opportunity size? What will give us a more higher ROI?

Soundarya Chandar: To me, the shift really happened when I started to see the value of doing interview prep, the skills that it tests for, and how often we actually get to apply it in our day-to-day job. If an interview setting is so similar to a job setting, why aren’t we all acing our product management interviews? Well, obviously because there are some key differences, right? There’s time constraints. In an interview setting, you are you know, you’re given like 30 minutes, 40 minutes to solve that question. There is a lot at stake because you’re trying to prove who you are. There’s a lot about how you come across and how you are a fit for the role.

Soundarya Chandar: You’re interacting for the very first time with hiring panels, so you’re trying to make a great first impression and you’re evaluated on your ability to communicate clearly at all times. And this process can go on for many hours in a single day. But in your daily job, this is like a continuous process and you’re evaluated based on the success of the product. You’re interacting with customers, you’re interacting with your team members who you feel more comfortable with. There’s obviously some key differences, but I do believe that if we start to look at them slightly differently, start to look at how our daily job is preparing us for the next interview and how the interview process of prep preparation makes us better at our daily job. It can actually make the whole process much better give us more fulfillment from a career standpoint. And it does certainly work like that for me.

Soundarya Chandar: I think there’s two very important things of a mindset shift that needed to change. One is, how do I think about time and how intentional am I in taking the time to step back and think about how I did in my day job? It’s very rare that I’m actually questioning was my, this was my PRD number two better than PRD number one was, what was my opportunity sizing or my intuition on how a feature would perform? Is it improving over time?

Soundarya Chandar: A lot of times we do retros for the team to improve team processes, team culture, but very rarely do we apply that retro to our personal growth. The difference is, we certainly do it when we are prepping for an interview, but I really believe that not doing that on a regular basis, not allocating the time to check in and see how we are improving our own skillset, we’re missing out on the compounding effect it could have when it comes time to looking for the next gig.

Soundarya Chandar: This is a cool image I like to look at. This is a grid, from this blog “Wait But Why” – If you haven’t checked it out, I highly recommend that you do. This entire grid, both on the left and right is about 1780 hours in a day that is divided into 10 minute little squares. Obviously a big part of our day is occupied by work. All I’m saying is that we need to set aside three to four little blocks. It doesn’t have to be contiguous, but for this intentional practice of investing in ourselves and building our skillset and using that eight R chunk to actually keep flexing and checking in on how we’re building those skillset.

Soundarya Chandar: One of my mentors, she has a CPO role at a high growth company, two kids, does a lot to contribute and give back to the community, product community broadly. She was telling me how she finds time to journal every day. If you look at all the big leaders, they always talk about all these books and podcasts that they’re reading. It’s because I think somewhere along the lines, people have started to internalize how important it is to build this intentional practice of reading and really sharpening our skillset.

Soundarya Chandar: When it comes time to interview, when you’re actually ready to look for that next role, that intentional practice time block is probably occupied by mock interviews, which in my opinion is the most effective way to get conf to gain confidence while you’re interviewing. All the skills that you’ve been practicing in your work by recalling and being intentional about every time you have a conversation on who the user is, what their pain points are, by reflecting on it every single day or every now and then, that’s habit sort of naturally flows in to a mock interview time or in interview time when, when you’re just sort of then focused on keeping to the structure, keeping to the time constraint, but you’re more creative, you’re more insightful, you bring about net new ideas because you’ve been already practicing that previously.

Soundarya Chandar: Here are some ideas that has really worked for me about how to build that intentional practice, this image on the right, something I absolutely love. It used to be my screensaver for the longest time because it took me a long time to internalize this, the power of being consistent. This image on the right is about how the more consistent you are, the further you can get, but the more inconsistent you are, the harder it is to get to that next level. Certainly for me, when I started interview in my first early days of interviewing, I was pretty inconsistent. I would only care to reflect about what I learned, how the features, like how do I frame the success of the feature? How would I do it differently when it came time to interviewing? But now I’ve built a more consistent habit of writing, of occasionally sharing on the internet of what I think.

Soundarya Chandar: That has really helped change how I do my day-to-day job and also practice for interviews, pausing and critiquing products you see every day. This is an idea that I stole from Julie Zhuo, who’s a XPM design lead at Meta. She’s written a book on making the manager. She’s an amazing writer, highly recommend if you don’t already follow her to check her newsletter out. She talks about how there’s products all around us. All we need is to set that intentional time aside to think about, oh, what is it about this product that’s different?

Soundarya Chandar: The new Apple Vision Pro headset just launched yesterday. Did we take the time to think about, oh, it’s $3,500. Who would pay for it? What other use cases exist? Even if that’s just on a own, on your own personal notes, that is building intentional practice that really helps build a muscle. And the last thing about around numbers and execution. When I was prepping for execution rounds, I would study like what the population of the of United States is. What is that of California? And now when I look my look at my MAU, DAU numbers, or if I look at the stats of OpenAI and ChatGPT, I’m trying to actively put that in perspective.

Soundarya Chandar: Some closing thoughts on this. A mentor once told me that your next job is really a stepping stone for the job after that. That’s how you sort of build this practice of thinking about your career and you’re being more intentional about it, and you’re building the skillset that will take you to that next level. I’ve already talked a lot about making the time to invest in sharpening your skillset, but I also wanted to highlight the importance of, you know, occasionally paying for it.

Soundarya Chandar: Do courses, attend conferences to step back and gain some perspective. And then lastly, consider interviewing even if you’re not actively looking. One, you never know what job opportunity you may be passing up on. Two, it’ll quickly tell you where you stand and what you need to pay attention to in your daily job so that you can keep flexing that skillset muscle.

Soundarya Chandar: Lastly, I wanted to share this really cool visual on product manager competencies. It’s from Ravi Mehta’s blog. He’s also a very famous PM leader. If you notice that the top half of the section, as you start off early on in your PM career, your focus is on the top half of the circle, which is really about feature specification, data fluency, voice of the customer. As you grow, the competencies that you need to actually develop are completely different. By the time you get to a VP or CPO level, it flips to the other half of the circle.

Soundarya Chandar: This was pretty interesting for me to look at because it’s very easy for us to forget that when we want to get to that next level, we need to be intentionally building the skillset that’s going to be exercised at that next level. In closing, really making the time, being intentional using our day job to sharpen our skillset, which, trust me, will help you in your interview. I will set you up for success. That’s it. Good luck with your journey today.

Sukrutha Bhadouria: Thank you so much, Soundarya. This was wonderful. Thanks everyone who attended.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner or Career Fair? Talk to us!

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner or Career Fair? Talk to us!

Angie Chang: With us today, we have Cindy Deng who is a leadership coach working with leaders to in tech to succeed in their careers while enjoying life. She has been working for over 20 years in the tech industry and her experience includes over five years at Google. She has worked as a business analyst, program manager and a product manager. Welcome, Cindy.

Cindy Deng: Thanks Angie. It’s great to be here today. Hi everyone. I’m here in the San Francisco Bay Area. Hello to those of you also here, and everyone else in the world. All right, we’re gonna start with a little brief story.

Cindy Deng: It was really early in my career when I hit this one critical point. I remember doing this one offsite at this beach to staring at the ocean, feeling really overwhelmed. You see, before that point in my early career, I had amazing managers for a good stretch of time. I worked hard deliver results and there was felt like it was a clear path. My scope increased over time and I became a manager, and honestly I didn’t think much about career development at all that much at the time. Or if somebody were to ask me, I probably wouldn’t even really know what that meant. <laugh>

Cindy Deng: Hardworking seems to simply led to good outcomes at the time. It felt straightforward, and then one day things changed. We had new leadership in place and everything felt different. I was no longer working with managers that knew my history and my strength and many of my new colleagues had really opposite style from mine and they were excellent at advocating for their ideas, it’s just amazing. But me on the other hand, I prefer deep analysis and I was a bit shy at speaking up in meetings, especially because English was not my native language.

Cindy Deng: Even though I liked the people I was working with, I didn’t feel equipped to navigate the changes and I felt lost. That day, I was sitting on the beach, I felt like I was in the middle of that ocean and just like waiting for the waves to come crashing down on me, but all of a sudden down on me that I could sit there and felt sorry for myself or I could get up and learn how to surf that wave.

Cindy Deng: I didn’t actually learn how to surf, but I could learn how to navigate the unpredictable changes and ultimately my career was up to me. A lot has happened since then. After about 20 years of working with engineering teams, I realized, while I really like working on products, I actually enjoy working with people even more. I now primarily work with female leaders in tech to navigate career challenges and elevate their leadership skills. What I’ve noticed in the last few years is that while career development is on most people’s mind these days, unlike that younger version of it’s not always obvious what to do, which is what brought me here today.

Cindy Deng: Today we’ll talk about how we can apply product management thinking into managing our careers. It’s just one of the many possible approaches to tackle this, but it’s certainly not the only way. Hopefully this will become another tool for you on your career journey.

Cindy Deng: Where are you on your career journey? Which one of these fields that resonate more with where you’re at? Are you looking at taking the next leap to reach for the next big thing? Whether that thing is a big project or promotion or something else that you’re excited about or are you more on the crossroad trying to figure out where to go next? It could be a pretty significant shift of your career direction changing type of roles that you have been working as or you’re moving from IC to a manager or potentially vice versa, whatever it is.

Cindy Deng: You might be debating options and considering how to navigate from where you are to the future direction or perhaps you’ve accomplished a lot in your career so far and you’re looking at how do you carefully balance all things that you’ve accomplished and what’s important to you and what do you do to put on top of that stack of stones while maintaining that balance – whether it’s health, fun, family, relationships or something else.

Cindy Deng: Throughout the talk today, we’ll be kind of going through a number of questions, kind of like this, really for you to think about where you are at, where you wanna go, and if it serves you, feel free to put in your thoughts in the chat or grab a notepad and then jot things down along the way. Yeah, I see some people already starting doing that. That’s great. And as you continue, what we’re talking about today really is a general framework. You can apply the formula and adapt it yourself and regardless where you are at with your journey. Throughout my career I worked on products most of my time. One way or another. I enjoy working with customers to understand their needs, figure out solutions with designers and engineers and delivering products ultimately make users’ lives better.

Cindy Deng: Over the last few years, as I consider how people navigate their career journeys and reflect on my own, it hit me one day, what if we were to look at ourselves as products? If you think about a product leader that you admire, whether it’s somebody you work with or somebody you observe from far, far away that on the stage, how would they approach a product that they’re passionate about? What would be their vision for it and what would be the strategy and how they approach it for the next few years and what would be needed during this execution so that it can launch successfully?

Cindy Deng: How would they advocate for the product that they’re passionate about to the world? Sometimes especially when we might feel stuck kind of in our journey or sometimes have a hard time talking about ourselves and thinking about your career this way gives you a little bit distance. Just backing off a little bit and thinking about what if for you with the product can potentially make it easier to think about it a little bit more objectively. What would you do? Think about this product is truly one of the kind. Anybody would be lucky to have it and one that you care deeply about. I know that because otherwise you wouldn’t be here today.

Cindy Deng: What’s the vision? For those that are ready to engage in the chat, if you just think about what’s your dream vacation look like and just pop in the chat like this one place you would go if you don’t have to worry about time or money or anything else, what would be your dream vacation look like? Just like the images here on the slide. Some people would want to go to the top of mountain and some people would want to get on a hot balloon. Some people say beaches are my way of doing lots of different ways. Everyone is different. Yeah. Like some of the answers coming in, it’s a lot of fun and one person’s dream job, it could be nightmare for someone else. Yeah. But so often we compare ourselves with others. If you don’t worry about anybody else’s opinion, what would be your vision for a successful, successful career?

Cindy Deng: Think about at the peak of your professional career, where would you wanna be? What would you wanna be doing? What role would that be and what size of the company would that be? Do you wanna be in a large size international scale? Do you wanna mid-size or small startup? What industry would that be? Do you wanna be an entrepreneur? Everyone’s answer could be totally different and every single one of those answer is perfect cuz it’s for you.

Cindy Deng: Sometimes these answers could be vague. It’s not always clear for everyone. Some people have really clear answers to say, I want to be this CXO of this one company, but for most people it’s not that clear. And it’s okay to be vague., because often when we’re trying to, even when we’re trying to go onto the highest peak, you can’t always see the highest peak sometimes, until you get past some hills before that.

Cindy Deng: It’s okay to be vague, and if you have absolutely no idea like how I was when I was growing up, I really did not like the question of what do you wanna be when you grow up? I would just make it up if I had to make an essay for it. Some of you might take jobs in the future that don’t even exist today. Who knows, right? If you don’t yet know, it’s perfectly fine.

Cindy Deng: Consider some of these questions. Reflect back on your journey. What brought you joy and what drains your energy, and what have you tried that you did not like? Consider time when things feel really good and you were in the flow, what was really good about it. These kind of questions can help you surface what’s really important to you, so you can figure out like just general direction you wanna go and if it’s kinda hard to figure it out and it’s okay to say I don’t have my vision for the next 20 years, cuz most people don’t.

Cindy Deng: What would be the direction you’d like to grow in the next few years? And just focus on that. And ideally it’s something that’s both a little bit exciting and scary cuz that would indicate things that are drawing you and also would help you grow next with a little direction or maybe a good clear vision of where you wanna go. Now it’s time to think about how do you get there. Just like planning on the road trip, you know, we have general direction where we wanna go, but there are some questions that you wanna kinda understand and the principles along the way. What are some key sights you might want to see? For example, if we were a road trip and we’re all going to Las Vegas, some people will wanna spend time in casinos and some will like make sure they pick out the best buffets to go to, and I will not want to miss any concerts or circus shows.

Cindy Deng: What kind of experience and learning do you want to gain in the next few years? When you think about the strategy, think a little bit distant, maybe the next few years or so. Do you wanna go deeper in areas that you do already enjoy, or do you wanna get a like a more breadth in areas that you may not know yet? Get a better kind of broader perspective around things? And how fast do you want to grow? Do you want to really rapid growth, or you prefer it to be a little slow and steady considering other aspects of your lives to for a better balance? And either way, it’s possible to have a rough itinerary but you want to really detail one because things are changing so rapidly these days. You can’t really just control a lot of things and think about what options might be in front of you.

Cindy Deng: There’s almost always multiple ways to get to a place and some of the options might be obvious, some may not. Sometimes you have to talk to others and discover those doors that might be hidden behind another wall that you don’t know about. And when you’re comparing options, think about what excites you. Often I talked about question yourself to say if you are looking to move to somewhere, are you running towards that option? Or you are running away from your current option if you have the time and the luxury to choose. I know sometimes people don’t, but which way do you wanna go and how do you wanna think about this?

Cindy Deng: Ideally there are little sparks that are drawing you and don’t get stuck in analysis paralysis. Sometimes people ask, do I wanna do this? Do I wanna do that? And it feels like I have to have a perfect option and perfect choice to optimize it all. It’s not always possible. Trust yourself that you can walk through a door and you decide from there. You can build build and you design your f way forward as always. Think about along this way also, what are some of the missing pieces that you might need? Are there new skills that you need to learn to get to the next phase?

Cindy Deng: What type of experience or opportunities that you might need in order to unlock the next opportunity? Think about what type of relationship do you need to build, whether that’s potentially mentors or sponsorships or maybe even peer support, like all the wonderful people here today in our conference. And think about the summer learning, circles potentially that will help you support you along the way. And sometimes it could be something a little less tangible like a mindset, especially if you recently transitioned to a new level or a new role. Maybe as a new manager or manager or managers.

Cindy Deng: What is something that you need to kind of change the way you think in order to lead at this level? Sometimes it’s like, how do I delegate better? How do I be more comfortable of I don’t need to be the one doing everything? Where are you spending your time thinking about the most strategic areas? Remember, what got us here sometimes doesn’t get us there. Yeah. The mindsets are important as well on your journey.

Cindy Deng: And last but not least, thinking about where you want to go and planning how to get there. Make sure you embrace your unique values. I sometimes hear people say, oh, well I don’t have a traditional background of this role. How do I go out and talk about myself? I said, that’s great. What is the unique thing that you are bringing that other people don’t have? How do you own your story and be really clear about it and be proud about that too. We can change the past, but you can think about how do you position that and then use that forward.

All right, so execution time. Not a lot different than how you would approach a work project. First, think about kind of how we prioritize things. Breaking down large large projects into smaller milestones and smaller goals. Where would you want to focus maybe for the next year or the next month? Sometimes we have too many top priorities. If the list of things that you wanna do and things you need to do are fairly long, how do you identify the one or two things to to really just pursue first? Because if you have 10 top priorities for a single team, you don’t really have a top priority. That will be the the top couple things to pursue.

Cindy Deng: Think about what kind of support do you need around you. It can take a village to have a successful product. For you to succeed, who do you need around you? For example, do you have a solid relationship with your manager? If you’re currently in a job, do they know about your career aspirations and where do you want to go? If your manager has idea of those, then they could potentially create opportunities for you along the way and allow you to practice the skills or learn things that you want to learn. Most managers are doing that these days, but if your manager has not had career development conversations with you the last six month or a year, try to start that conversation. Use the one-on-one time and just bring it up. It is your time and to look for manager support around things like this. You can share what you’re interested in learning and see what they can do about it to align.

Cindy Deng: Thinking beyond your manager, what about your skip level or other senior stakeholders that you may not interact with on the frequent basis, but they may be involved in the interest in the projects that you are working on? How do you make your work a little bit more visible beyond the people that you are enacting on a daily basis? And if you’re thinking about kind of exploring what’s next, actively on the job search, think about what kind of roles are you most interested in and really have lots of conversations with people. Networking can feel kind of scary at times, but really is the best way for people to discover opportunities and to find those kind of hidden roles that may not even be published on the job boards yet

Cindy Deng: In this phase as we’re trying things out and learning new things all the time, it’s potential to make mistakes. It’s like likely for us to make mistakes when we’re learning something new. Experiment and try things when you can kind of control in a small kind of setting. It’s less risky, but don’t be afraid to try and don’t be afraid to kind of get out of that comfort zone. And really, successes come from failures. We all have that tendency. I don’t wanna fail. What if I fail? It feels scary, but that’s the best way to learn at times.

Cindy Deng: Let’s say we finish all execution steps and we are ready for lunch. Who would be your number one fan? Any guesses here? Thoughts during the chat? For some time I would say, my number one fan might be my mom in my life, but on the career sense, As I mentioned, I have really great managers in the past. And as we think forward forward, what we really want to be our number one fan is ourselves. This could be really difficult for some people. I found it difficult, especially, for a good amount of time in my career. I grew up learning that it’s good to be humble. When people compliment us, my parents would say, “oh no, no, no, you’re too kind” and brush it off. Talking about ourselves is bragging. I took me a long time to understand that I needed to shift my thinking here, especially working in the US and corporate America. If we’ve done it, it’s not bragging. If we could talk about it in a humble way, it is important to really acknowledge what we’ve accomplished and our strength and our ability.

Cindy Deng: The hard truth is, as much as we like it, we don’t have people constantly advocating for us all the time. Even if our managers try to do their best jobs, they often have multiple people on the team. They can’t do that for just us. We have to do our part and advocate for ourselves whenever we can. Doing excellent work is great, but it’s also important to make sure that’s visible and known to other people. All right, I’m gonna fly through the next couple slides a little faster cuz we’re short on time.

Cindy Deng: After launch, how do we continue to improve ourselves? Getting feedback, making, making sure you remember to keep taking small steps. Don’t forget to take care of yourself. Physical mental health are absolutely important, so we don’t build up a tech debt over time. The key takeaways… today’s talk is all about you. How do you invest in yourself, the next step, and how do you keep trying? Love to connect with everyone here. Feel free to reach out on LinkedIn and I have set aside time this year to help with folks on their career journey. If you like some additional one-on-one support, feel free to reach out.

Angie Chang: Thank you, Cindy. That was an excellent talk. Thank you so much for submitting this talk in the speaker call for proposals. Many of our speakers today came through that process. I encourage everyone in the audience to think about what is your TED talk and what can you speak about at the next girl Geek X ELEVATE. I did happily book a time with Cindy to talk and I’m so excited for that time. Thank you for being a leadership coach and helping women in tech. All right, we’re gonna close the session and hop to the next one, but thank you again, Cindy, and there’s a lot of chatter. I encourage you to look at the chat and we’ll see you on the other side. Bye.

Cindy Deng: Thank you, Angie. Bye everyone.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner or Career Fair? Talk to us!