Navigating College And Career As First-Gen Girl Geeks: Insights From CCPA Career Panel of Women in Tech

girl geek x ccpa career panel vanessa Vanessa magana an nguyen molly dubow bryanna valdivia elizabeth orpina hero

Girl Geek X continues its ongoing partnership with Oakland Public Education Fund to support Coliseum College Prep Academy. Girl Geek X volunteers actively engage with the local school community to illuminate the journeys of women in technology who have overcome diverse challenges to achieve successful careers in fields ranging from art and design to engineering and cybersecurity.

On Wednesday, October 4, 2023, Girl Geek X community members volunteered to share career advice with the senior class at CCPA in East Oakland, California.

An Nguyen (Lead Product Designer at Medallia), Molly Dubow (Customer Success Leader most recently at Webex), Bryanna Valdivia (Software Engineer at Flexport), and Elizabeth Orpina (Security Awareness & Education Manager at GitHub) spoke on a career panel to share their path from high school to successful careers in the tech sector. They offered valuable advice to students.

Below are key takeaways from the college and career panel discussion:

girl geek x ccpa career panel vanessa Vanessa magana an nguyen molly dubow bryanna valdivia elizabeth orpina

Girl Geek X CCPA Career Panel moderated by Vanessa Magaña with An Nguyen, Molly Dubow, Bryanna Valdivia, and Elizabeth Orpina speaking as first-generation students now working in the technology industry.

#1 – Diverse Backgrounds and Successful Paths to Tech

Panelists highlighted their diverse backgrounds and unconventional routes into the tech industry. The first-gen students shared their experiences with navigating financial aid and the school-to-tech-job trajectory!

An Nguyen shared her background of being a self-funded college student who pursued education while working multiple jobs. She emphasized the need to communicate and gain trust from family when pursuing non-traditional careers like design (specifically, UX design and now product design. She encouraged students to study what interests them, and believes “education is there for you to study to be as good as that ‘naturally good’ person.” 

Molly Dubow wishes she knew about paid tech internships during her high school days, stressing the significance of early awareness about opportunities. She encouraged students to take advantage of the college opportunity to leave home for a new experience, and pondered if junior college could have been a better step for her.

Bryanna Valdivia, the first in her family to attend college, described how a coding bootcamp propelled her into a tech role at a startup, emphasizing the effectiveness of networking through alumni at her bootcamp Hack Reactor that landed her first job in tech one month after completing the coding bootcamp. She has since paid off her student loans. Sstartups are a good way to break into the tech industry.

Elizabeth Orpina, whose entry into tech was facilitated by a volunteer opportunity while working at a foundation, revealed the importance of seizing unexpected pathways to success. She started working in tech at Autodesk, first as a contractor and then in a full-time capacity, before joining GitHub (a Microsoft company) where she is working now. in

#2 – Overcoming Financial Challenges and Scholarships

Addressing financial challenges, the panelists shared their experiences with loans and scholarships. Bryanna Valdivia explained that FAFSA gave a lot, and she took out loans to complete her education. She expressed the wish to have applied for more scholarships in high school and in every year of college.

Elizabeth Orpina advised students to opt for federally-funded loans over private ones, and highlighted the opportunity for additional grants for first-generation college students. Her loans covered books and housing, and her jobs “paid for the fun stuff” in college.

An Nguyen encouraged students to apply for all available scholarships, emphasizing the potential for unexpected opportunities.

Molly Dubow stressed the importance of seeking help and utilizing the myriad resources available to alleviate financial burdens. She underlined that students are welcome to connect with panelists on LinkedIn, the professional social network, so they can better ask questions and seek referrals in the future.

#3 – Students to Chart Unique Paths with Guidance, Pursue Interests

During the panel, the speakers offered advice to students navigating their paths. Molly Dubow encouraged students to consider junior colleges as an affordable and valuable starting point in higher education.

An Nguyen emphasized the importance of courage in pursuing one’s passion and interests from an early age. She underlined that education is just studying your interests, so pursue your interests! Her career in product design came after she initially pursuing a computer science degree; it was then she realized that she loved the design side of engineering, not the coding side.

Bryanna Valdivia advised against excessive stress during the transition from high school to college and emphasized focusing on long-term goals.

Elizabeth Orpina encouraged students to make decisions based on what’s best for them, urging them to listen to respected individuals rather than conforming to immediate circles.

#4 – Balancing Family and Career, Finding a Community

An Nguyen, currently on her maternity leave, highlighted the challenges faced by caregivers in the tech industry. She stressed the significance of support groups and employee resource groups for caretakers. Addressing ageism as well, she stressed the importance of finding allies and coworkers who understand and support one’s journey.

Both Molly Dubow and Bryanna Valdivia expressed feelings of isolation as first-generation college students and women. Molly Dubow underscored the importance of not allowing one’s voice to be silenced and encouraged finding ways to communicate effectively, even when faced with gender bias. Bryanna Valdivia advocated for joining employee resource groups (ERGs) and seeking connections beyond one’s immediate team to foster a sense of community and support.

Elizabeth Orpina emphasized the role of team and company diversity in creating an inclusive work environment, urging students to seek companies that value and promote diversity and inclusion.

In conclusion, the CCPA Career Panel offered invaluable insights into the unique journeys of successful tech professionals. The advice and experiences shared by the panelists will undoubtedly serve as a beacon of guidance for students and volunteers navigating their career paths in the tech industry.

girl geek x ccpa career panel vanessa magana molly dubow bryanna valdivia elizabeth orpina an nguyen

Girl Geek X CCPA Career Panel (from left): Molly Dubow, Vanessa Magana, Bryanna Valdivia, Elizabeth Orpina, and An Nguyen.

Special thank you to Shelly Lopez and Alicia Parise at Oakland Public Education Fund, the nonprofit that organizes programs with the Oakland public schools to bring students, educators and the local community together for maximum impact. You can still sign up for Girl Geek X Oakland Education Fund volunteer events this year – bring a friend!

Best of ELEVATE 2023: From Introverted Leaders to Technical Interviewing, Career Playbooks (Videos + Jobs)

girl geek x elevate fall conference speakers

Girl Geek X’s highly-anticipated ELEVATE Conference and Career Fair on September 6, 2023 hosted over 1.5k mid-to-senior women in tech around the world online for inspiration, connection, and learning. 

Thank you to our inspiring speakers & sponsors for helping make ELEVATE conference an incredible experience. Check out our sponsor’s remote/flexible jobs – they are actively hiring! Please spread the word and help a girl geek find her next role in tech!

Here are the most popular talks from September 6th’s ELEVATE 2023! You can watch (or re-watch) them at the links below:

  1. Why Companies Need More Introverted Leaders (Keynote) – Nicole Husain, Chief Operating Officer at Lighthouse Labs
  2. Career Fair Kickoff: Employer & Company Introductions – Bentley Systems  – Gen Taurand, Product Manager at Bentley Systems, Stephanie Robinson, Director of Services at Cohesive / Bentley System, & Meghan Goff, Manager of Talent Acquisition at Bentley Systems introduce themselves, the company, roles & hiring process.
  3. Level Up Your Technical Interviewing Techniques – Jessica Dene Earley-Cha, formerly Developer Relations Engineer at Google
  4. Company Introductions – The New Club  – Laura Du, CEO & Founder at The New Club, & Danielle McLaughlin, Founding Head of Talent at The New Club
  5. Things I Wish I Had Known Earlier in my Career – Rachel Rogers, VP of Industry & Product Marketing at Bentley Systems, & Natalie Plummer, Director of Diversity, Equity & Inclusion at Bentley Systems
  6. Rewriting the Leadership Manual: Playbook on Influencing for Non-Influencers – Karen Lo, Director of Engineering at JLL Technologies
  7. Cloud Migration Trends: What You Should Know – Whitney Stewart, Senior Cloud Solutions Specialist at Microsoft
  8. Mastering Effective Interviewing Skills  and Situational Interviews in a Professional Setting Sylvia Martin, Chief Nursing Officer at Kaiser Permanente
  9. Breaking into Product from Engineering – Rekha Venkatakrishnan, Amazon Head of Product
  10. The Many Facets of the Staff Engineer – Stacey Shkuratoff, Staff Software Engineer at Guild

Employers joined for introductions and virtual booths – see who joined below:

elevate june speakers sponsors
Elevate Conference Sept Booth Bentley Systems Natalie Plummer Meghan Goff Stephanie Robinson Gen Taurand
Elevate Virtual Booth The New Club Laura Du Danielle McLaughlin

40 Mentors kicked off the conference – volunteering in the Mentor Lounge that was buzzing with questions and advice on everything from engineering, product, security, AI, healthtech, non-coding roles in tech – to interviewing and career search tips.

Mentors joined from companies like Google, Airbnb, Amazon, Autodesk, Twilio, Fastly, Okta, Bayer, Alphawave Semi, Anthropic, Kohl’s, Riot Games and more. Mentors ranged from CTO to engineering managers, VPs to product managers and engineers.

elevate mentors lounge sept
elevate virtual career fair for mid-senior women in tech, 2023
If your company is looking to recruit more women this year, please don’t let them miss out on our next Conference & Career Fair sponsorship opportunity! 

We want to hear from you. The next ELEVATE Conferences are December 6th, 2023 and March 8, 2024. We also partner with companies on Girl Geek Dinners in the San Francisco Bay Area.

Please email and we’ll be in touch.

Thank you in advance!

– Angie Chang, Sukrutha Bhadouria, Amy Weicker, Amanda Beaty and the team at Girl Geek X
girl geek x elevate fall conference speakers words

September 6th, 2023 ELEVATE Conference & Career Fair was a hit!

Looking forward to December 6, 2023 and March 8th, 2024 (International Women’s Day) ELEVATE Virtual Conferences!

Girl Geek X Grammarly Lightning Talks on Engineering, Product, Machine Learning & Brand Design (Video + Transcript)

Over 100 girl geeks joined networking and lightning talks from women working in engineering, product, and design at the sold-out Grammarly Girl Geek Dinner at Grammarly’s office in downtown San Francisco, California on August 29, 2023.

Grammarly women shared lightning talks about building GrammarlyGO, Grammarly’s new contextually aware generative AI communication assistant that allows you to instantly compose, rewrite, ideate, and reply. Grammarly is hiring!

Table of Contents

  1. Welcome – Angie Chang – Founder at Girl Geek X – watch her talk or read her words

  2. Fireside Chat – Heidi Williams – Director of Engineering at Grammarly with – Charlandra Rachal – Technical Sourcer at Grammarly – watch the fireside chat or read their words

  3. Building GrammarlyGO From Zero To OneJennifer van Dam – Senior Product Manager at Grammarly – watch her talk or read her words

  4. Engineering GrammarlyGO – Bhavana Ramachandra – Machine Learning Engineer at Grammarly – watch her talk or read her words

  5. Designing GrammarlyGOSarah Jacczak – Brand Designer at Grammarly – watch her talk or read her words

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Transcript of Grammarly Girl Geek Dinner – Lightning Talks:

grammarly girl geek dinner angie chang girl geek x founder welcome august

Angie Chang: Is [this] your first Girl Geek Dinner? Wow, that’s a lot. How many of you have been to more than five Girl Geek Dinners? Yay! So good to see everyone. My name’s Angie Chang, in case you didn’t know, and you could tell by the t-shirt, I am the Girl Geek X Founder, and started Girl Geek Dinners in the Bay Area 15 years ago, so we’ve been doing events like this at hot tech startups up and down from San Francisco to San Jose, and I’m in the East Bay, so I wish there was more events over there as well. Tell your employers they need to have one of these showing off their amazing women in tech and product.

Girl Geek X founder Angie Chang welcomes the sold-out crowd to Grammarly Girl Geek Dinner on August 29, 2023 in San Francsico(Watch on YouTube)

Angie Chang: I want to say thank you so much to everyone at Grammarly for helping put this event together. They have been so amazing and supportive and they’re definitely hiring, so please talk to someone here that has Grammarly on their shirt. They’re very friendly, so I’m going to say thank you for coming and hopefully you’ve made a lot of good connections. I know I’ve seen a lot of people talking to each other and I hope you have LinkedIn with each other or Facebook or whatever people are using these days, and continue to stay in touch.

Angie Chang: A lot of us are in this industry working to keep women in tech and I think that involves all of us together, so thank you. Keep coming back to events! Keep giving each other job leads! Keep poking other girl geek to get in the car ride together to get to that event after work when we’re all tired! Thank you for coming! I hope you learn something, make a new friend, and have a good night!

Charlandra Rachal: Thanks, Angie. I’m super excited to kick things off and host this fireside chat with director Heidi Williams, who’s been very involved in building our generative AI features for enterprise. Heidi, welcome!

Heidi Williams: Hi. Thanks for having me! Great to see you all here. It’s awesome. Full crowd!

Charlandra Rachal: Yeah. For those who aren’t super familiar with Grammarly, can you give us a quick overview of our company and our product?

Heidi Williams: Sure. I like to make a joke that either people have never heard of Grammarly or they love it! I know I talked to a few folks already that love it, but for folks who aren’t familiar, we are an AI-enabled writing assistance that helps with your communication wherever you write, and I do mean everywhere. Our mission is to improve lives by improving communication. Earlier this year, we also launched our first generative AI product to help you with even more writing and communication assistance beyond just revision, but also getting into ideation and brainstorming and composition and comprehension. It’s been really fun to see the product evolve in the time that I’ve been here.

Charlandra Rachal: I hear that you just celebrated three years here, so woo woo! Three years! Can you tell us what brought you here and what really keeps you here?

grammarly girl geek dinner Charlandra Rachal Heidi Williams speakers

Grammarly Technical Sourcer Charlandra Rachal and Director of Engineering Heidi Williams welcome the audience at Grammarly Girl Geek Dinner. (Watch on YouTube)

Heidi Williams: When I was speaking about the mission, improving lives by improving communication, I do feel like I got to a point in my career, I’m a little farther along maybe than some of you, that I really wanted to work on something impactful and I feel like Grammarly more than any other place, it resonated with me that improving lives by improving communication is so real. It’s not a fake slogan because communication is what makes us uniquely human.

Heidi Williams: I was excited about the idea that we’re not just a platform to help you communicate more effectively, but also to help educate you along the way, especially thinking about things, like insensitive language or bias, there’s an opportunity to help educate people about the possible impact of their words that they may not even know is having a negative impact on someone, and so I got really inspired about the mission.

Heidi Williams: I’m also a word nerd, so that part was really fun as well. I think what keeps me here, is that everyone is so excited about the mission and the people. I think our values are amazing. We really live by our values, we hire and fire by them.

Heidi Williams: The last thing I’ll say is, we’re an amazing size company, where there’s still interesting problems to solve, but we’re small enough that people can really take the initiative if they see a problem that needs to be solved, or they want to advocate for something to change in some way, they’re really empowered to do that. I love being at that size company and our values really help us be successful doing that as well.

Charlandra Rachal: Nice. I like you mentioned initiative and impact. Do you have any sharing stories that you can share that where you seeing either yourself or someone else really make impact?

Heidi Williams: I have three examples if you’ll bear with me for a minute, but I see it all over the place, and it’s not just in the product. It’s about our organization, our culture. There’s an engineer on my team, her name is Lena, and she recognized on the product side that engineers were struggling with a certain pattern of ‘how do I reliably save settings about the individual, about their team and their organization about specific features. Then if I have all of these settings, how do I combine them and know which setting to apply at any time?’

Heidi Williams: She interviewed a bunch of engineers, realized it really was a problem for folks, and then proposed a new project called the Settings Registry, and advocated for it to be on our roadmap. It’s been exciting that she could spot an opportunity and a challenge for our developers and really advocate for that. That’s exciting!

Heidi Williams: The second one, I actually led an initiative where I noticed that I love our hiring process, but I noticed that we had one particular gap, which was that we didn’t necessarily have an interview where we asked people about their experiences. We ask about their knowledge, but we don’t ask, ‘what is the proudest thing that you ever built and tell me how it was designed and what did you learn?’

Heidi Williams: I noticed similarly that we weren’t necessarily getting the accept rates from underrepresented groups that I thought we should be getting, and advocated that this might give people an opportunity to talk about themselves, and for folks who aren’t used to bragging about themselves, that might not come out in a normal interview, but if you give them an opportunity to talk about themselves, then they can actually show off how good they are at stuff, which is exciting.

Heidi Williams: That pilot was successful, showed that we greatly increase the accept rates for folks from underrepresented groups to a really high degree, and now we’ve rolled that out as an interview across the engineering organization, so really proud of that.

Heidi Williams: The last one I’ll mention related to culture, Bhavana, who you’ll hear from later, identified an opportunity that folks were looking for mentorship inside of our women in tech group, and so she started a pilot with a few other folks to introduce an internal mentorship program for women in tech and we’re kicking that off in September.

Charlandra Rachal: I love that. Yes. I feel like the last two really spoke to me, especially being in recruiting so I love that a lot. Now, Grammarly continues to expand in its enterprise space. How do you drive value for Grammarly business with generative AI?

Heidi Williams: It was very exciting to see our generative AI product come out. A little bit of context: the part of the product that I’ve worked on is Grammarly Business, which is our B2B product for teams and organizations.

Heidi Williams: As we all know, communication is not a one person sport. There’s a team dynamic, there are team norms, there’s organizational knowledge that are part of the communication that you have at work. We looked at opportunities for how to incorporate organizational knowledge.

Heidi Williams: We have a feature called Knowledge Share that helps you define terms, definitions related links, key people, and then we can use that as part of the generative AI output to help you have something that knows something about your organization instead of maybe a more generic response.

Heidi Williams: We did things like that and then incorporated some of our Grammarly business features like style guides and brand tones, which help you speak with a consistent voice, and brand tones in particular, you can have a response from our generative AI product, and then choose ‘make it sound on brand to my company’.

Heidi Williams: That was a way that we could really make the information, both the information and the tone be tailored to your organization.

Charlandra Rachal: Nice. Well, I heard that there was some quick turnaround times. Can you tell us more about that?

Heidi Williams: It was definitely felt like this huge opportunity, this huge moment where a lot of folks are talking about generative AI and it’s an area (LLMs) we’ve been investigating for a long time and understanding what their capabilities and limitations were and whatnot, and so I think we really rallied as an engineering organization, and I think the way that we were able to turn things around quickly really came from our leadership approach, which is the idea that we really want to empower teams to make the best possible decisions on the ground.

Heidi Williams: The way to do that is to help with transparency and sharing context around ‘what are the business needs, what are the product needs, what are our customer needs, what problem are we solving for the user?’ Let me give you all of that information, all of that context. At the end of the day, if you need to choose, should this be a radio button or a dropdown or this should work this way or connect with that system, you can make that decision because you have all of that information. Really trying to be transparent and share context so that people are empowered to make decisions on the ground and not feel like they’re stuck with somebody else making decisions and kind of blocking them from things.

Charlandra Rachal: I hear you that you mentioned customer feedback. Do you have any feedback that you’re able to share with us?

Heidi Williams: Sure. You’ll hear more about it in one of the talks today. We did run a survey after launching GrammarlyGO and wanted to know how are people using the product and what’s working and what’s not working. Through that feedback, one of the themes that we heard was that ‘it didn’t sound like me’.

Heidi Williams: We started investigating – ‘how do you tailor the output to sound authentic to you?’ And it sounds, I see a lot of head nods, that resonates and what not. We invested in an area called My Voice and figuring out how to have your own voice profile and use that for all of the responses that are generated, so it’s more likely to sound like you than not and saves you an extra step for trying to even interpret what your own voice is. We can actually help you with that, so you’ll hear more about that when Jen talks about it.

Charlandra Rachal: Great. Well, I know this is one question that I know a lot of people probably want to ask but probably wouldn’t ask, but what would you say really sets us apart from our competitors?

Heidi Williams: Yeah, I was talking to someone ahead of time who asked this question, I’m like, oh, you’ll have to wait. <laughs> Great question. There’s one thing. First of all, I mentioned earlier, we work everywhere and that is one difference from some of the other products that are out there. We work in every writing surface, desktop web, and so we can be right in line for where you’re already doing your thinking, your writing your communication, so that’s certainly one.

Heidi Williams: The two I wanted to really call out, which I think are kind of reinforced by our engineering culture, is our important focus on security, trust and privacy, and also responsible AI. Because at the foundation of everything we do, we really want our customers and users to trust us with their writing and to feel like we can do things to make personalized experiences and what not, and so, what’s interesting to me, I feel like more than any engineering organization I’ve ever been at, because we are so mission-aligned, we recognize we have this huge responsibility to our users to be thoughtful about their data and their privacy, their security.

Heidi Williams: I feel like we care a lot about security maybe earlier than most engineering teams where at the very end before you ship security goes, ‘oh, not yet!’ And you’re like, ‘oh, I can’t’. The whole idea that engineers will advocate for, am I doing this right from the beginning, and wanting to make sure that’s so they’re proactive about asking for feedback about security and privacy. Or even there was a scenario where we had an idea about a feature and people are like, ‘That feels like it might invade privacy. Can we talk about that before we launch it?’

Heidi Williams: I really loved that people could bring that up and that we’re all trying to achieve the same thing, and so it’s a very fair question and let’s make sure we’re holding that to high regard.

Heidi Williams: Then on the responsible AI side, I think we’re so lucky to have an incredible team of linguists who can help us beyond what other competitors can do who don’t have a team of linguists where we can help sort of filter things like the inputs to generative AI to make sure that people are not asking for something harmful, but also that whatever they type in, they’re not getting harmful responses, which are either insensitive or inflammatory or traumatizing in some way.

Heidi Williams: I love the fact that we have the capabilities of being able to create these filters and create a safe environment for people to use these large language models, which have who-knows-what in them. Love that we are actually able to do that. We’ve also been able to build that not just through humans, but figuring out how to build automation and testing and all through the development process help you understand that you’re not going to create a feature that unintentionally create some sort of biased output or something like that, and so just tremendous examples over our long history in this of finding ways to make sure that we are building a product that is responsible and then also keeps everybody safe, secure, and all their information private as well.

Charlandra Rachal: Nice. Well that was fascinating, right, everybody? Alright, so we are going to dive deeper now to exactly how our generative AI features were built. As a heads up, we are going to ask for questions at the end and I’ll bring up all of the speakers including Heidi herself. For now, welcome Jennifer van Dam, who’s a senior product manager here!

Jennifer van Dam: Hey everyone. I’m Jennifer van Dam, product manager here at Grammarly. I’ve been here for three years and I worked on our features like emotional intelligence, tone detection, tone rewrites, inclusive language, and most recently I helped build out our generative AI product, GrammarlyGO, which I’ll be talking about today, so super excited to take you all through the journey.

grammarly girl geek dinner jennifer van dam product manager speaker

Grammarly Senior Product Manager Jennifer van Dam talks about building the generative AI product GrammarlyGO from zero to one. (Watch on YouTube)

Jennifer van Dam: First off, I want to give a huge shout out to my fellow girl geek PMs that helped build GrammarlyGO together with me. We were a team of three PMs leading multiple product efforts. Specifically, my product focus was on the UX and also on the zero to one stage, so figuring out the UX framework and the zero to one building process. That’s what I’ll dive in deeper today. First off, I wanted to start with a refresh of Grammarly before GrammarlyGO.

Jennifer van Dam: What Grammarly has been focused on for many, many years is helping make your communication more effective by proof writing and proof reading and editing your writing. Anywhere you write, let’s say you’re writing an email, a message, a Google doc, Grammarly will read the text that you have written already and make sure it’s correct and clear and delivered in a way that you want to come across. But, we have a big mission of improving lives by improving communication, so we fully were aware that this is a small part of communication that we want to help with, and we’ve had many dreams beyond proof writing and editing.

Jennifer van Dam: One big user problem we always heard, for example, was the ‘blank page problem’. For years, we’ve heard that our users really struggle with the inception stage of communication – the writing, getting those initial ideas on paper – and it was a huge productivity blocker. That’s just an example from user problems we’ve been hearing for years, and we always dreamt about solving it, and we were super excited with this recent technological leap that with generative AI – now we have the technology to solve all those user problems we always dreamt about.

Jennifer van Dam: That’s how we built GrammarlyGO. We went from proofreading and editing towards helping solve composition, brainstorming, and all these new use cases, which was really, really complex, because we went from a decade of in-depth expertise of rewriting, towards composition and brainstorming, and we had a pretty aggressive timeline as well. This was super, super challenging.

Jennifer van Dam: What made it really challenging? First of all, it was zero to one. We had no prior experience how this would land with our users and there was no data we could rely on, so we had to make a really risky decisions because we went from a proven product concept with product-market-fit towards a huge uncertainty and risk area, which it was really exciting, but super, super challenging. How can we predict how it will be received with the absence of data?

Jennifer van Dam: Essentially we really had to take on a beginner’s mindset to solve these new use cases and almost operate like a startup again to build this new product from scratch, but we’re also an established company – pretty big – and we have millions of users, 30 million daily active users that have a super high bar of our product. We were building zero to one moving fast, but also had a very high bar we wanted to meet for our users in terms of quality, responsible AI, and security that we wanted to deliver.

Jennifer van Dam: How do you solve such a huge, huge problem? What we did was, let’s just start with the earliest draft possible, and get it out – get it out to users. What we did is we created this really highly-engaged alpha community, and we built very early prototype, and we shifted and we asked for continuous feedback, and it was really, really engaged community that would give us feedback super fast and inform next iterations. We really focused on the core experience before we wanted to invest in any type of polish or any type of design polish, we made a commitment – let’s not focus on that. Let’s figure out the UX framework.

Jennifer van Dam: We had a big challenge. How do we create a UX where someone can brainstorm and compose something from scratch? What is intuitive? What will land with our users? To give you an example of the fidelity of prototypes, what we did is we started in grayscale, because we made a commitment to figure out the framework, before deeply investing into building something out, because we weren’t sure if this is the version to commit to.

Jennifer van Dam: This turned out to be a great idea because we did end up throwing away a couple of prototypes, and the third prototype was the one that we felt landed the most and that we committed to building out and refining, which was of course a huge process as well and took us a lot of time. Sarah will actually be giving a fascinating talk later about all the design and brand work that went into polishing this prototype, so I won’t go too deep into that.

Jennifer van Dam: What was really cool about this prototyping stage is, the user empathy led to innovation. We came up with things that we didn’t necessarily plan from the start. One thing we kept on hearing when we were asking for feedback on the UX and was it intuitive to compose and brainstorm?

Jennifer van Dam: A lot of the feedback we were getting was ‘it just doesn’t really sound like me’. And that made people drop off. They would compose an email or a document, but it didn’t sound like something they would write or want to use, so this was a huge risk of people dropping off and also, it wasn’t of the quality we wanted to meet. This led us to come up with the voice feature that Heidi talked about with before.

Jennifer van Dam: This is a classic example – it wasn’t on our roadmap from the start, but it really, being in tune with the user made us come up with this feature. I remember when we launched our first basic version of it, how excited everyone was, and that made us realize how important voices in generative AI – and it led us to much deeply invest in this area, so we’re keeping investing in this and also it helped actually become an important competitive differentiation.

Jennifer van Dam: To take it even further, we would also hear from users, okay, now it sounds like me, but in this situation it doesn’t sound how I want to sound, which was also a really hard problem. We heard this a lot in the email reply use case, and what we came up with is harmonizing your voice preference with your audience as well. Let’s say, I prefer to sound casual maybe 80% of the time, but I got this super formal email, it would be a little bit awkward if I replied casually there.

Jennifer van Dam: We also created a model that looks at the context of your communication and your audience, and harmonize that with your voice preference so it doesn’t diverge too much, but lands somewhere in the middle. This was an awesome, awesome project and Bhava is going to do a much deeper dive into replies after this.

Jennifer van Dam: Looking back, this was a huge product, and when I reflect back on what were the things that made it successful, I think first of all the team was really, really important when we started this project because building in a zero to one, very high ambiguity, it’s not for everyone. It can be quite chaotic.

Jennifer van Dam: We started with a very small team that was comfortable with iterations and ambiguity and okay throwing away work for the sake of learning. We intentionally kept this team very, very small until we resolved the main ambiguities, we started to scale up the team a little bit or slowly.

Jennifer van Dam: We were very intentional about the initial zero to one stage, and then scaling the team, and we had a lot of high alignment and energy because of this, because the people on the team were excited about these problems.

Jennifer van Dam: We also learned that prototypes are huge to align leadership, because it’s easy to get stuck in discussions, discussing strategy or design flows, but there’s nothing like proving it with real concepts and real user feedback and real prototypes. And then also our transparency principle really helped. We had a ton of cross-functional collaborators and of course it’s inevitable zero to one, there’s going to be all these changes and all these teams are relying on you.

Jennifer van Dam: We were super, super transparent with changes and reasoning and this really helped us creatively problem solve. In the case when there were changes, we would come together and this basically set up a place for innovation with cross-functional collaboration as well.

Jennifer van Dam: What’s next for Grammarly? At Grammarly, we believe that AI is here to augment your intelligence. That is really our product philosophy. We believe that AI is not here to take over your life or dictate you, but it’s here as a superpower, to help you communicate more effectively.

Jennifer van Dam: This is the product philosophy we’ve taken building GrammarlyGO, and this is our philosophy with all our next products and features that we’ll be launching. I can’t share too much about it, but I can share that this is the philosophy we take in building the next features that we’ll be releasing. Thank you. Alright, next up is Bhavana who’s going to be talking about the fascinating project called Quick Replies.

grammarly girl geek dinner bhavana ramachandra engineering speaker

Grammarly Machine Learning Engineer Bhavana Ramachandra talks about engineering the generative AI product GrammarlyGO at Grammarly Girl Geek Dinner. (Watch on YouTube)

Bhavana Ramachandra: Thanks Jen for that awesome overview. Jen spoke about how Grammarly expanded into the user’s writing journey, and we’re going to take a small detour into one of the features that we built, which was Quick Reply, or replying quickly to emails. My name is Bhavana, I’m an ML engineer at Grammarly. I’ve been here for about three years. I was one of the many engineering geeks on this project. There are a couple of folks here in the audience today – Jenny’s here, Yichen is here, yeah, wanted to give a shout out to the team.

Bhavana Ramachandra: Today, I’ll be really talking about foundations in motion and with respect to the quick reply feature. Jen touched upon this previously. We have invested quite a bit in understanding what our users want in terms of their writing, and we were looking at expanding into this user journey, so really we built a lot of fundamental understanding over the years that helped us accelerate into the new product areas that we wanted to go into.

Bhavana Ramachandra: Another shout out is that, the team that worked on Quick Reply, all the point people, all the cross-functional people, were women. We have an analytical linguist, computational linguist, ML engineer, senior PM, and interestingly this has not been my first project when we were all women but yeah. Four foundations deriving from two projects that I have worked on coming into this one. One was Tone. Jen mentioned she has been working on Tone as well.

Bhavana Ramachandra: I’ve been here for three years. She’s been here for three years. Tone was our first project together, so Tone was one of them as well as Recap, which is our investment in 2022 to really go beyond the writing phase and to also to the reading phase to help users read faster so that we can help them write better. With the respect to tone, this was the first version of this. We also have tone rewrites, but this one helps users identify the top three tones in their text so that they can reflect on if that’s exactly how they want to sound.

Bhavana Ramachandra: Zooming into what was the fundamental understanding we built in each of these projects. The first one I’ll cover is Tone, and the three areas we invested is product definition, quality and our AI. For product definition, some of you might be thinking like, ‘Hey, this sounds like sentiment analysis and that is a pretty well solved problem’ but really our product team tries to think about what is the user value of sentiment. If you look at user text, honestly, eight out of 10 times users sound positive. That is not helpful to know.

Bhavana Ramachandra: What the product team did was actually define 50 tones over different aspects of your writing that’s actually helpful for you to know. Do you sound optimistic? Do you sound direct? Do you sound confident? Do you sound worried? Do you sound concerned? The product team really came up with a wide range of tones. In terms of quality, we iterated quite a bit over it and during this phase we actually came up with three levels of hierarchy.

Bhavana Ramachandra: When you have 50 tones, especially if you’re building models for 50 tones, it’s a bit hard – one to get data and to make sure you’re iterating over quality of all of ’em. The way we tackle this is, we define three levels. We have the tone at the really granular level, we have the sentiment at the highest level, but we also came up with tone groups that was maybe around, eight tone groups, and that helped us identify quality at different levels. Then, we really try to nail quality in terms of what is the user values.

Bhavana Ramachandra: Now as an ML engineer, I like to see quality always improving, but is it really worth it to invest in taking one tone from 90 to 92% or is it better for us to improve on a certain tone group that is really valuable to our users?

Bhavana Ramachandra: That’s the kind of trade-off that we had to make and then we really derive over time. I also want to mention our AI is one of our biggest tenants as Heidi mentioned. In this case, this feature was one of the first few pieces to pilot our sensitivity process. The REI manager today was during this process shaping up our formal sensitivity process. We’ve always done it and I think she was making that a very formal process.

Bhavana Ramachandra: Apart from that, we also wanted to make sure that any tone suggestions we make – because we have a varying level of quality, we did want to understand – what are the sensitive cases, and what is our risk of quality with respect to sensitivity. That’s something that we understood during this project as well.

Bhavana Ramachandra: The second one is Recap, which was our comprehension project that we worked on in 2022. Here we were going beyond the writing journey into the reading journey of the user. We invested a lot in understanding the user problem. We had many, many discussions about certain areas that surprised me that I’ll get into. There were also technical challenges because now we needed to again look at the context outside of the text that you’re writing. Where is it? Where are we getting this context from? And then we had a whole new set of ML problems, which is exciting for me.

Bhavana Ramachandra: For the user problem, I wanted to touch on two things. Delight versus value. We wanted to provide summaries and so we identified emails and we wanted to provide summaries as well as to to-do items. But does it really make sense in all use cases? For example, if you have a one line email, it doesn’t make sense to summarize that.

Bhavana Ramachandra: Or, if you have a social promotional email that says, sign up now, that sounds like a task, but all of us know that’s not really a to-do item for any of us. These are the kind of gotchas that we were like, ‘oh, we have a model, but is it actually useful in all cases’ or ‘how long should a summary for a really long email be’ versus ‘a one paragraph email’ be? These are the kind of things that we iterated over quite a bit. And also understanding the context and intent of the user.

Bhavana Ramachandra: Imagine you have an email, an announcement to your entire organization. If you’re a manager versus if you are an engineer versus if you are in design, you might have different takeaways from that email. Trying to understand a bit more of what is that context and what is the intent of the user.

Bhavana Ramachandra: We also solved a lot of technical challenges. Again, shutting out like our AI is one of our biggest pillars. Privacy is also our biggest pillar. We are very, very cautious about what is it that we are asking users to share with us and are we really providing value from it? Before this we didn’t look at the context of the user because we looked at suggestions of what they were writing. Now we wanted to provide value from that, so then we had to update our privacy policy and we also had to update our client side logic to derive this context.

Bhavana Ramachandra: Coming down to the ML problems itself, like I said, there were two things that we were trying to provide – summarization – as well as – task extraction – or to-do list, but because we were talking about delight versus value, context and intent, we also invested in a couple of different areas, including signature detection, intent understanding, and email taxonomy. Trying to understand what is the context of the user – that was more of email taxonomy and intent understanding, and signature detection really helped us. When you look at emails sometimes especially short emails, if the signature is longer than the email, then the model sometimes gets tripped up.

Bhavana Ramachandra: This is true for generative AI as well because yeah, for many different reasons, sometimes models are not perfect, so it helps to help them along the way, and signature detection was one of those areas.

Bhavana Ramachandra: In all of these areas, we spend time annotating our own data sets because email is a space where data sets are not as public, so this was one where we had to understand what data sets existed, what were the things that we were trying to build. As Heidi said, we have a big internal team of analytical linguists, and they help us identify the data, identify what our guidelines are, and go get us annotations that we can build models with, and these were all the areas that we collaborated with them on.

Bhavana Ramachandra: Putting all of that together, from the Tone project, we knew tone was something that our users cared about that we wanted to bring into this feature. You’ll see it says, ‘Jason sounds caring’, but that’s not like models don’t know to think about that. That’s something we have to prompt them to think about.

Bhavana Ramachandra: All of the tone taxonomy that I spoke about, the 50 tones, that made it into the prompt as well. In terms of the recap project, we really built a reply user – like, who are the users? You might get a hundred emails, but you probably reply to 10. What are these reply use cases is something that we had built an understanding that came into this project as well. That really helped us understand quality for launch.

Bhavana Ramachandra: As Jen said, we were not trying to polish, but we were trying to aim for user value. That meant, are we comfortable with the quality for launch? We know that we’re going to iterate over it, but for launch, does this look good? It’s something we try to understand.

Bhavana Ramachandra: Then, on the client side, a lot of the logic, that we built for the earlier project, got repackaged and reused for this one as well. We were using a new protocol, we were using, so it wasn’t just copy paste, but repackage. And then our AI as always, because it’s a generative AI output, we want to be sure that any output that we’re sharing with our users does not have bias, and some high risk scenarios. That’s something as well that we made sure this feature and the output of this feature goes through.

Bhavana Ramachandra: This was one of the few features we built for launch, but it did get a couple of different shoutouts. I know WSJ called out, we had a lot of users sending us, this was awesome. I specifically wanted to, we had a segment on NBC where Courtney Naples, who is our director of language research, spoke about this and the host in fact called out the feature and mentioned how the output of GrammarlyGO sounds like him, versus OpenAI does not. And yeah, that was a really nice moment for us to see. That’s it. Next off, we have Sarah who will be talking to us through all the explorations that the brand design team did for launch as well as our product.

grammarly girl geek dinner sarah jacczak brand designer speaker

Grammarly Brand Designer Sarah Jacczak talks about designing the generative AI product GrammarlyGO at Grammarly Girl Geek Dinner. (Watch on YouTube)

Sarah Jacczak: Thanks so much, Bhavana. Hi everyone, my name is Sarah and I’m a brand designer at Grammarly. I’ve also worked here for three years and I’m so excited to share the brand design team’s work and show some of the behind the scenes process of the GrammarlyGO launch.

Sarah Jacczak: To start off, I want to intro the go-to-market design team. The team consisted of product brand designers, motion designers, content designers, brand writers, design researchers, and design operations. This was a complex launch and we were designing something completely new, and there were a lot of moving pieces and constant changes. On top of that, we needed to move fast. Having a team with a wide range of expertise, it allowed us to work quickly and collaboratively, and we were able to impact areas across product, brand, and marketing for this launch.

Sarah Jacczak: I want to give a special shout out to the brand and content designers and brand writers. I’ll be sharing some of their incredible work on the GrammarlyGO identity and campaign later on. To give a quick overview of the scope of work, the brand design team worked on in-product systems, a new brand identity that included a new logo and color palette, and a go-to-market campaign toolkit, which included guidance on how to design and write about Grammarly’s generative AI features.

Sarah Jacczak: To do this work, we had to consider how users will interact with this new experience and how we would differentiate Grammarly go from competitors. This required close collaboration with product and engineering teams as well.

Sarah Jacczak: When designing GrammarlyGO, one problem we identified early on was, we needed a way for users to access this new experience. We knew that users were familiar with clicking on the Grammarly icon to open the Assistant Panel and accept writing suggestions, but integrating GrammarlyGO features with this existing UI was not an option for the launch and it was something we would have to address in the future.

Sarah Jacczak: For the launch, we needed to keep these two experiences separate. and we decided to add a second entry point into the Grammarly widget, which would open the GrammarlyGO experience.

Sarah Jacczak: Here are some early explorations of the GrammarlyGO entry point. So on the left we tried two different button designs for the desktop app and browser extension, and we consider it a badge treatment on the desktop app, which has floating widget. The benefit here is that on desktop, the widget wouldn’t be much larger, so it wouldn’t interfere more with text fields.

Sarah Jacczak: However, the visual treatment, it felt kind of like a notification and because of its small size, we were worried it wouldn’t attract much attention. And so we moved on to another exploration. On the right is another exploration where we considered having multiple inline buttons with different icons, so there would be a new unique icon for composed reply and rewrite features, but when prototyping this design, we found that it was a little too cumbersome, and so we decided to simplify it down to one icon for all GrammarlyGO features. And this is what we launched with a single light bulb icon to open the GrammarlyGO assistant window.

Sarah Jacczak: Having one icon as the entry point gave us room to surface prompts that have unique icons. You can see on the example on the right, we have the improve it icon with the pencil, and this prompt appears when a user highlights their text and it gives them a quick and easy way to generate another version of their writing.

Sarah Jacczak: While we were designing how users would access GrammarlyGO, we’re also designing icons. We started exploring icons before we had a name, but we knew it needed to be unique and it would live next to the G icon. We explored a wide variety of approaches. Some were more literal and represented generative AI, like writing and pencils and sparkles and magic, and other explorations were more focused on abstract representations of speed and ideation. But yeah, we could have kept going and going, and this is not even all the explorations, but because of the tight timeline, we had to make a decision.

Sarah Jacczak: We went with the light bulb because we felt it was effective in conveying the new ideation capabilities of GrammarlyGO. We also saw an opportunity to design new product iconography for prompts. These icons would accompany the suggested prompts that appear based on a user’s writing.

Sarah Jacczak: Early testing showed that prompt writing is challenging, and so we prioritize these suggested prompts that are based on a user’s context, and we wanted to make this experience more visual and more delightful. Again, icon explorations range from abstract to literal, but we saw that these icons needed to convey meaning, and also support the prompt compi so we move forward with the literal direction.

Sarah Jacczak: Another discovery was that operating within the new limited color palette was challenging and it didn’t quite feel unified with the existing UI, so we looked to Grammarly’s tone detector iconography, and these emojis, they would appear in the same UI as the prompt icons, so it made sense to create a cohesive experience here.

Sarah Jacczak: We referenced the colors and styling of these emoji to create the foundation for the new prompt icons and here the prompt icons that we designed for launch. You can see they’re literal in that they depict the meaning of the prompt in a simple way, and keeping them simple also ensured that they could scale and be legible at small sizes. We also selected colors and subtle gradients that felt cohesive with the existing emoji icons. This resulted in an icon set that feels warm, friendly, and is hopefully fun to interact with.

Sarah Jacczak: We also needed to consider scalability. There would be hundreds of prompts and we wouldn’t be able to design an icon for each prompt, so we grouped them into categories. Each category has an icon, and within that category are prompts that share that icon. For example, any prompts about writing or composition, we’ll use the pen and paper icon and any prompts about ideation, we’ll use the light bulb and so on. We also identified which prompts we feel would be used frequently and created unique icons for those to add variety and more delight.

Sarah Jacczak: While some of the team was working on iconography and content design in the product, others were working on the identity and go-to-market campaign. Here are some of those explorations – a variety of logos were explored and taglines, as well as graphics for the campaign, and some visual explorations use gradient orbs while others focused on movement and transformation by using over layers of shapes and lines. And for the go-to-market campaign, we created a new tagline as well – ‘go beyond words’. It’s active and it conveys Grammarly’s ability to assist users beyond their writing.

Sarah Jacczak: We also designed a new logo that incorporates a bolder G with a circle forming the O as a nod to the classic Grammarly button. For the GrammarlyGO identity and campaign, the brand design team landed on a concept that uses overlapping shapes to convey transformation and the iterative process where one idea is built on the next. The softness of the gradients also speak to the human qualities, and they’re juxtaposed with hard edges to represent technology, and these overlapping shapes were further brought to life with animation.

Sarah Jacczak: The team also worked on a design toolkit, and this toolkit was shared across the company. The toolkit included logo, color palette, illustrations, photography, motion guidelines, and a library of product examples to be used across the campaign. A style and verbal direction guide was also created to ensure how we speak about GrammarlyGO is consistent. The brand writers provided headline examples based on themes. There was headlines about creativity, such as let your ideas take shape, headlines about productivity, such as ‘discover new ways to get things done’, and headlines about trust like ‘AI innovation with integrity at its center’.

Sarah Jacczak: This campaign was pretty large. We had a lot of requests and a lot of marketing channels to design for, but because the brand and brand writers and brand designers collaborated and built these systems and guidelines, we were able to move quickly and create consistency despite many people working on the campaign production.

Sarah Jacczak: Here are just a few examples of the work created for the campaign. The team created a series of demo videos and animated gifs that show product functionality, and these were used across marketing and PR. The team also worked on onboarding emails, landing pages, in product onboarding, blog posts, ads, and social assets,

Sarah Jacczak: To get a further sense of the scope of work, here’s some numbers from the naming and identity work. Over 500 names were considered, 188 Jira tickets were completed, over 105 taglines were explored, 45 videos explored, 39 product examples designed and animated, and over 230 logos were explored. And so, while these numbers don’t tell the full story, and we had challenges along the way, the team was able to overcome this and collaboratively design a new experience and produce a successful launch in a short amount of time. Thank you.

Grammarly girl geek dinner questions audience Charlanda Rachal Heidi Williams Bhavana Ramachandra Jennifer van Dam Sarah Jacczak panel

Charlandra Rachal: Thanks Sarah. To all the speakers that put together this incredible presentation, I learned a lot and I work here, so I hope you guys really enjoyed all of that. Let’s welcome back all of our speakers for Q&A. There is someone who is going to be in the audience with a mic, and I see it first hand already, so we will get a mic right over to you.

Audience Member: As mentioned, it was uncharted territory. I was curious how you went about ideating the first project. Was it based on existing user information you had? Was it academic papers? How’d you go about it?

Jennifer van Dam: That’s referencing my talk, so happy to talk more about it. What I mean with uncharted territory is the solution. We knew it was a problem – we’ve been hearing for years that since we started, we hear from our users, they struggle with these communication problems. What was the uncharted territory is the solution and delivering the product in a way that lends and resonate with our users.

Jennifer van Dam: The approach we decided to take is directly into the prototyping stage because we felt it was really important to connect the text and the product to the user. Let’s say, you want to compose an email, we can design and show you concepts, but we need that moment of you writing your text and seeing the output. That’s why we jumped right into the prototyping stage as our way to research the solutions and the design approach.

Charlandra Rachal: There was another hand right here…

Audience Member: Hello, my name is Kate, and probably question also to Jennifer because it was on one of your slides. When talking about prototyping, you were speaking about more empathy and I took a screenshot. Let me see how it looked like there.

Audience Member: ‘Deeper user empathy.’ Can you please elaborate a little bit more on that, how it worked? How did you do it while you were still prototyping, please?

Jennifer van Dam: Yeah, deep user empathy. What I really meant with that was to understand and dive into the types of things our users are trying to achieve. What are the types of use cases that, here’s a prototype, did you use it for rewrites? Did you use it for emails? What were those things?

Jennifer van Dam: We did so many sessions talking to people and getting their feedback to get empathy and then of course we had questions, but then the feedback we got was, ‘oh, but it didn’t sound like me’. This is what I mean by deep user empathy is really getting into the mindset of empathizing what is working and what is missing. That really helped us inform iterations and changes and new features or scrapping features.

Audience Member: Thank you. I would assume that the launch of ChatGPT definitely affected Grammarly. What would be the key learnings? What were the key learnings for you as product leaders from getting the LLMS viral and thanks for the presentations. My name is Maria.

Heidi Williams: One of the things which I think was interesting – we’ve been doing AI for a long time, we’ve used a lot of different technologies, whether it’s rule-based or machine learning, or all sorts of different technologies, exploring LLMs on our own.

Heidi Williams: The biggest thing about ChatGPT that ended up, was actually the discussion in the world about how to use AI as an augmentation tool. Before, you would have to convince people AI was okay and trusted and then all of a sudden overnight everyone’s like, ‘of course you trust it. Look at this’.

Heidi Williams: Now all of a sudden we don’t have to waste time talking about should you use AI? Now it’s about how can we be a trusted partner on how best to use AI and help you be more effective and help you succeed in your job or in your life. It has changed the conversation of ‘should I’ to ‘how should I’, and that’s been interesting and amazing that we can now focus on just solving real problems as opposed to convincing people they have a problem that AI can help with.

Audience Member: Thank you so much. First of all, want to say huge shout out for Girls Geek and Grammarly for putting together. Thank you. That’s a great event. I’m a huge fan of Grammarly. Go now. It finally sounds like Shakespeare in my emails and not like a broken machine. I want to ask you this question. I think Heidi and Bhavana, that’s questions for you. You said that one of the feedback was, ‘it doesn’t sound like me’.

Audience Member: Are you using LLMs to train data which your users are input? And if so, how do you also prevent some data security in terms of, for example, I’m putting something in my email as a product manager about revenue or about some specific of the product, which haven’t been on the market, and I’m always a little bit worried where this data is coming from. Yeah, I think it’s a good question. One is LLMS for make me sounds like it’s me. And the second one is the data privacy. Thank you.

Heidi Williams: I can talk about at least part of it and then if you have things to add as well. Because Grammarly has been around for so long, and that we are a trusted source, we were able to negotiate a really amazing contract with our LLM provider, which means that they don’t train on any data that we send to them. Not everybody could negotiate that, but because we’re Grammarly and we’ve been around so long and have such a big user base, we were able to do that.

Heidi Williams: I feel like that was a huge thing that is very differentiated from just using whatever’s on the market is that they’re not training on any of the data that we send. From that perspective of the data privacy, but if you want to talk more about the my voice and about how we do that and how do we then, if either of you want to add…

Bhavana Ramachandra: I’m going to pass to Jen.

Jennifer van Dam: Could you repeat the question?

Audience Member: Sure. How you make, if we’re not sending data to any LLMs, how you make it sound more like me, for example, GrammarlyGO, always suggesting me to be more assertive, which I think I’m already too much and then I’m like, no, no, no, let’s make it more positive. Yeah, how this happened, how it sounds more like me is like data being trained.

Jennifer van Dam: We look at context and communication patterns, so it doesn’t necessarily train on your data per se, but on the patterns of your communication and the context. That’s how we understand your voice profile across.

Bhavana Ramachandra: To add to that – we understand what your tones you prefer or you use are, but we don’t actually pass that on to LLMs. We had our tone detectors since 2019. We’ve been telling users how they sound for a bit now. We use that information to really update the writing rather than train the LLMs itself with your data.

Charlandra Rachal: I feel like I’ve been neglecting this side over here, so right here in the front.

Audience Member: More of a quick technical question. Do you list all of the tones that you detect for and measure somewhere publicly, or is that behind closed doors?

Jennifer van Dam: We have a homepage that lists a lot, but not all, of our tones. We feel it’s too competitive to reveal 50 plus. But yeah, you definitely can find information about a lot of our tones that we support with tone detection.

Bhavana Ramachandra: Maybe this is a challenge. Can you write enough with Grammarly to find all of them?

Audience Member: The tables have turned. I wanted to direct a question to the second speaker after Jennifer van Dam. Yes, you, um, why does my voice sound like this? Ah ha ha I see what you did there. The question I have more specifically was, when tone is being cited as a suggestion, when you write a sentence and it connotes that, ‘oh, your tone is serious and neutral’, and when you add a word or two and it changes the tone entirely, I’m curious, what quantitative scales do you use behind the scenes to make those on the spot judgements? You mentioned your team had a lot of linguists on it, and I was hoping you could expand on that because that has been an object of curiosity of mine for a while, possibly. Thank you.

Bhavana Ramachandra: Yeah, I’m just going to Jen about… Yeah, I think for, in terms of how do we decide, in fact, when we started looking at rewriting for tone, I think our initial exploration had just neutral and new tone versus in certain cases we were actually able to provide three levels, friend, friendlier, friendliest, but really depended on how much data we’d have. If you can actually, you can be neutral, but you can’t be more neutral. It really depended on the tone and how much data we have.

Bhavana Ramachandra: I think this is a part where our linguist really helped us really dive deep into this and look into each of the tone that we have, get data for each one of these tones. I think for our tone retype explorations, we started with the tones where we had most understanding and first started with two levels and then moved on to three.

Audience Member: Thank you. Hi, my name is Leanne and big fan of Grammarly. My question for whoever thinks that they’re best equipped to answer this is, could you tell us a bit more about how Grammarly is fighting bias, and what are some of those solutions that are currently in place? And maybe thinking about, in the future roadmap?

Bhavana Ramachandra: Yeah. Our AI is one of our biggest tenants, as Heidi said, and we’ve always invested in this area. The couple of things that we have done are actually very, very public. We have blog posts about how we look at pronouns or bias in gender bias in data, and how do we make sure our generative AI suggestions, how do we measure that and how do we prevent that as something that we do? And as Heidi said, this is part of the process.

Bhavana Ramachandra: This is not something that you think about at the end of the day, you plan for this. You plan to have a sensitivity analysis right from the get-go. The other part of this, we’ve published a couple of different papers this year. In fact, in the REI space in, I want to say ACL. Okay, thank you Dana. You’ll actually find a lot of public information. I don’t want to pretend that I know more than I do in this area. I am getting onboarded though. Definitely, we have blogs and papers out there that talk about what are the solutions we have implemented.

Heidi Williams: Maybe just one thing to add to that is part of it is actually cultivating a good data set because you could imagine that you just take, I think we’ve seen this with LLMs as well, you just take the words out there and you might see a bias of a gender bias around when you’re referring to a male, they might be more associated with certain words in the general public than a female. Then you would imagine that percentage wise, it might suggest like, oh, if you’re talking about a man, you must be referring to this, et cetera.

Heidi Williams: We’ve done a good job of cultivating our data sets to help ensure that the data sets themselves are not biased, and that’s a huge aspect of it is just making sure that we’re not having any gender weighting as one example, or it could be racial, whatever it is. There’s just making sure that you have a data set that’s representative and it’s not going to sort of skew things in one direction or another.

Bhavana Ramachandra: Do you want to add to that?

Jennifer van Dam: I wanted to add to that a little bit about how much we care about this investment. We also, besides of course all the deep investments in the modeling, we also have inclusive language suggestions for end users that help basically eliminate gender bias in your language while writing or talking to your coworkers or your team.

Jennifer van Dam:This is an area I also worked on and it’s a really great part of our product. For example, maybe you’re writing, ‘the businessmen are wearing suits’, we’ll underline ‘businessmen’ and we’ll ask if you’re writing to an audience that you want to be inclusive of everyone, maybe replace it with ‘business people’ rather than ‘businessmen’. We also tackled this from the end user standpoint and helping them communicate more inclusively and eliminate bias where they’d like.

Audience Member: <inaudible>

Charlandra Rachal: For those who didn’t hear, the question was, how much do we focus on educating, when people continue to make mistakes in their writing?

Jennifer van Dam: The inclusive language product, I encourage you all to check out because education was a huge part of the product UI and the way we wanted to position it. We always want to be educational rather than forcing you because at the end of the day, the user is in control. You have agency. That’s what we always believe in. We also realized we have to explain why are we saying ‘consider replacing businessmen with business people?’ How do we explain? Because some people don’t realize that it’s so ingrained, you just type it, you don’t really stand still.

Jennifer van Dam: Another example, like whitelist, blacklist, that suggestion, a lot of users didn’t understand why we were suggesting to replace blacklist with blocklist, so we actually focused our UI around education. Why is blacklist/whitelist is perpetuating certain stereotypes, so consider replacing it – and that was a real aha moment because when you say that in a training, it’s different than when in real life you’re writing a text and seeing it and applying it, so it’s actually been really powerful in educating people

Audience Member: To Jennifer and Bhavana. Earlier you mentioned deploying LLM models, initially, were you were skeptical how receptive the users would be and how they would perform.

Audience Member:Can you talk about your A/B testing strategies? Did you roll out to a part of your audience, I mean part of the user base first, and then started gradually increasing, rolling out the new features too, especially the generative AI features? Could you talk about your A/B testing strategy and how did you scale it to the whole user base? And after you employed gen-AI features and these new features that you earlier talked about, how did it impact the revenue subscription revenue and the user base?

Heidi Williams: Start?

Bhavana Ramachandra: I think Jen covered maybe some of the A/B tests. I’ll let her, I can maybe speak to the launch plans, the alpha testing. I can cover that a bit. Especially for projects like Tone where we did iterate over quality quite a bit, we would try to identify one. We had internal annotations. Every time we improve our quality, we do more internal annotations to understand how much of a bump is it? And once we have a fair understanding, we run experiments with Gen AI, we had to take a slightly different process.

Bhavana Ramachandra:As Jen said, we had more alpha testing with users, really deep conversations in terms of understanding what is a useful generative AI LLM feature because we’ve had rewrite features in terms of generative AI for the longest time, but what’s a useful composed feature? What’s a useful quick reply feature? All of that was not really A/B testing. We were building understanding in this case. That was a lot more alpha testing.

Bhavana Ramachandra: Then for the launch plan itself, we have 30 million users, we have five different surfaces, we have extension, we have desktop app, we have an editor, we have our website, and we’re in many different countries, so this to me firstly was the most impressive one, because we had to do a geo launch across many different clients that all have different release cycles, so all of them had to be in sync because we wanted feature parity. We started with certain countries to make sure, one, we can handle the traffic. Two, all the features are looking or performing as they should.

Jennifer van Dam: The difference with how we approach modeling quality, it depends on the maturity of the track. In the zero to one stage we do a lot of offline quality evaluations and make sure it meets our quality bar and the metrics. We don’t necessarily test out multiple models yet, but in the iteration stage we do. One example where we’ve A/B tested our improve it rewrite, which in one click will improve your text, and there was a lot of experimentation we did there with tone behind it and conciseness and what lands the best with improving my text with one click. Typically, we focus a lot on offline quality evaluation of our models, and then in the iteration stages we do a lot of A/B modeling.

Audience Member: After new features, did you see any bump in the overall user base?

Heidi Williams: I can’t talk about specific numbers, but I think obviously there was a lot of excitement and interest in this area. I think we did see that there was new interest, and then also just seeing interest and engagement from our existing users using the product maybe in a different pattern than they had been before as well. It definitely feels like there have been changes, but I can’t speak about specific numbers.

Audience Member: What’s your tech stack typically for the whole GrammarlyGO is hosted on?

Heidi Williams: The tech stack? A lot of different parts of it… it’s hard to answer with a really quick answer. Our particular LLMM provider is Azure OpenAI. And then there’s a variety of tech stacks above that – different things that we’re using for the linguistic side of things, and then there’s Java, there’s Closure, there’s all sorts of different technology stacks and then we run on AWS otherwise.

Charlandra Rachal: Thank you. Then I only have time. Oh, oh, oh. I was coming for you. I know you had your hand out. If you still wanted to answer, we would definitely break the mic over your way. The last I have one time for one more question. Alright, Nancy

Audience Member: Behind.

Charlandra Rachal: She’s coming. Yeah, she’s coming.

Audience Member: Oh hi. Alright. I’ve used Grammarly for a really long time and this may be more of a product manager question because I’m also, I can write circles around everybody, so I don’t really need GrammarlyGO. I’m actually wondering, I’m thinking about, the roadmap further down for advanced writers, people like me who write. What’s coming?

Audience Member: Because I will say now, I use Claude a lot just to be like, Hey Claude, this is what I wrote. What do you think? And then Claude will say, that’s really good or not or whatever. I’m just wondering it’s GrammarlyGO moving in that direction for people who don’t really need help getting stuff on paper or on screen.

Bhavana Ramachandra: These are the comprehension projects that I was talking about. They’re all about trying to understand what the user is reading or what the user has written. For example, tone is something, even if it’s not correction, if it’s not editorial, you still might want to understand how your tone is coming across, especially in cross-cultural communication.

Bhavana Ramachandra: That’s something that’s helpful and in general as well, especially for long writing, we’ve gotten a lot of feedback about, I think this is one area that we were investing in – how we, so we show top three tones and let’s say people use Grammarly to write books or their fictional books, and does it make sense to show top three tones? Then they want a different – so this is the kind of evolution of the features that we see.

Bhavana Ramachandra: Comprehension is one area. In our generative AI in GrammarlyGO, if you actually open it up in a document, we provide a lot of prompts around understanding the gaps in your document, identifying what are your main points. All of these are just comprehension. This is just not how to improve your writing. Rather like this is what’s there in your document. You can review it based on a couple of different dimensions.

Audience Member: I should use Claude and Grammarly. Yes.

Bhavana Ramachandra: That’s the answer.

Charlandra Rachal: Yes. Alright, I wanted to say thank you so much to all of the speakers here and all you wonderful guests. I’m going to give a shameless plug if you didn’t already see, I’m in recruiting and we are hiring! Definitely talk to us, talk to me. I know we’re going to send a link out as well.

Charlandra Rachal: I believe there are more refreshments in the back and everyone is welcome to kind of hang out, chat, network. If you have more questions, I feel like we got through a lot of them without telling all of our secrets, but feel free to pull them aside and ask more questions. I hope you have a great night. Thanks again for coming out.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!


Mentorship Lounge – ELEVATE (September 6, 2023)

Mentors ELEVATE Sept eng mgmt eng IC product career development interviewing leadership networks career mgmt healthtech cyberai interviewing

Table #1 – Engineering Management – Mentors:

Table Topics: Engineering Leadership, Negotiation, Transitioning to a Manager Role, 1:1s, Career Management, Career Transitions, Management & Leadership, Promotion, Breadth vs Depth, Networking, Managing One’s Career

Elevate Mentor Table Engineering Management Dominique Simoneau Ritchie Madhuparna Datta Megha Krishnamurthy Nono Guimbi

Eng Mgmt Mentors: Dominique Simoneau-Ritchie (CTO, Affinity), Madhuparna Datta (Application Engineer Architect, Cadence Design Systems), Megha Krishnamurthy (Senior Engineering Manager, Adobe), Nono Guimbi (Engineering Manager, Airbnb)

Table #2 – Eng Growth As Individual Contributor (IC) – Mentors:

Table Topics: Making Technology Platform Switches, Mobile Platform, Growing On IC Career Track, A/B Testing & Experimentation, Front End Engineering (Angular / React), How To Tech Lead Efficiently, Handling Conflict, How To Work Cross Functionally, Backend, Web Development, Neurodivergence, Self Care

Elevate Mentor Table Eng Growth As IC Cheryl Aranha Devin Nicholson Gulbaniz Ahmadova Jenna Cooke

Eng Grow As IC Mentors: Cheryl Aranha (Principal Software Engineer, Intuit), Devin Nicholson (Senior Full Stack Engineer, BILL), Gulbaniz Ahmadova (Software Engineer, Per Scholas), Jenna Cooke (Senior Developer, Referoo)

Table #3 – Product Management – Mentors:

Table Topics: Product Leadership, Transition to Product Management, Time Management, Stakeholder Management, Product Management, Career Strategy, Career Planning, Interview Preparation, Positioning for Success, Executive Presence, Career Transitions, Layoffs

Elevate Mentor Table Product Management Angela Yee Chinmayee Rathi Sheenam Maheshwari Tracy Taylor

Product Management Mentors: Angela Yee (Product & Leadership Coach, Consultant), Chinmayee Rathi (Senior Product Manager, Box), Sheenam Maheshwari (Senior Product Manager, Google), Tracy Taylor (Head of Product, Shelf)

Table #4 – Non-Coding Roles In Tech – Mentors:

Table Topics: Legal & Tech, Returning To Higher Education, Mental Health, DEI, Ops, Emotional Regulation, Communication Skills, Startups, Product Design, Design Careers With Non-Design Background, Latinas in Tech, Sales, Marketing, Community, Customer

Elevate Mentor Table Non Coding Roles in Tech Jenny Jennings Lucy Jia Chen Olivia Ouyang Paola Johnson

Non-Coding Roles in Tech Mentors: Jenny Jennings (Commercial Counsel, Twilio), Lucy Chen (Head of Education & Fulfillment, Curious Cardinals), Olivia Ouyang (Product Designer, Finix), Paola Johnson (Director, Community & Customer Advocacy, ThoughtSpot)

Table #5 – Career Development / Promotion – Mentors:

Table Topics: Navigating Workplace Environments, Asking For Promotions, Technical Topics, Career Pivots, Negotiation, Building Your Brand, Elevating The Narrative, Storytelling, Career Trajectory, Negotiating Salary / Promotions, Career Progression, Startup Growth / Sales

Elevate Mentor Table Career Development Promotion Allison Colyer Karin Williams Neha Srivastava Paula Bejarano

Career Development / Promotion Mentors: Allison Colyer (Software Engineer, Top Hat), Karin Williams (Director, Risk Strategy, BILL), Neha Srivastava (Senior Software Engineer, Attentive), Paula Bejarano (Director, Business Development – Autonomous Vehicles Navistar)

Table #6 – Leadership / Building Good Networks – Mentors:

Table Topics: Project / Program / Portfolio Management, Building Good Networks, Glass Ceilings / Cliffs, Building Trust With Your Manager, Career Planning, Promotion, Networking, Career Transitions, Leadership, Executive Presence, Office Politics, Continuous Education, Supporting Teammates, Inclusivity, Embracing Technology

Elevate Mentor Table Leadership Building Good Networks Gayathri Kamath Karen Ko Madhavi Bhasin Nahla El Helbawi

Leadership / Building Good Networks Mentors: Gayathri Kamath (Staff Program Manager, Fastly), Karen Ko (Managing Director, WEST Diversity & Inclusion), Madhavi Basin (VP, Diversity, Inclusion & Belonging, Okta), Nahla El Helbawi (Asst. Director Web Academic Content Management, American University in Cairo)

Table #7 – Managing Your Career – Mentors:

Table Topics: Communications, Early Career Exploration, Taking Chances, Decision Making, E-Commerce, Digital Media, Networking / Growing Your Career, Identifying Mentors & Sponsors, Tips & Traits For a Good Leader / Manager, Career Development, Building Skills, Management & Leadership, How To Overcome Blockers.

Elevate Mentor Table Managing Your Career Caitlin Anderson Eiman Hassan Mackenzie Hartmann Radha Ranganathan

Managing Your Career Mentors: Caitlin Anderson (Senior Manager, Internal Communications, Autodesk), Eiman Hassan (VP, Program Operations, Alphawave Semi), Mackenzie Hartmann (Manager, Campaign & Creative Management, Amazon Ads), Radha Ranganathan (Engineering Manager, Human Interest)

Table #8 – Healthtech / Biotech / Medtech – Mentors:

Table Topics: Navigating Career Transitions, Overcoming Imposter Syndrome, Startups, Clinical Trials, Sports, Nutrition, Medical Devices, Healthcare, Science, Medtech, Navigating Politics, Presentation Skills, Communications, Influence, Business Partnership, Financial Acumen, Business Acumen, HealthTech, Product Management, Tips for International Students, Biomedical Engineering & Research

Elevate Mentor Table HealthTech BioTech MedTech Abigail Stack Emily Hu Lydia Wheeler Madhumita Srikanth

HealthTech / Biotech / Medtech Mentors: Abigail Stack (Genotyping Innovation & Technology Lead, Bayer), Emily Hu (Senior Director of Clinical Trials, Unilever), Lydia Wheeler (America Commercial CFO, Illumina), Madhumita Srikanth (Quality Engineer Product Management Liason, Iridex)

Table #9 – Cybersecurity & AI – Mentors:

Table Topics: Career Pivots, Cybersecurity, Career Paths, Management, Owning & Using Your Diversity, How To Toot Your Own Horn, Networking, Landing a Dream Job, Troubleshooting, Startup Life, Stepping Into Leadership Roles

Elevate Mentor Table Cybersecurity AI Dipti Shiralkar Irene Murray Jennifer Pisansky Saskia Hoffmann

Cybersecurity & AI Mentors: Dipti Shiralkar (Principal Software Engineer, Okta), Irene Murray (VP, Sales Engineering, KnowBe4), Jennifer Pisansky (Recruiter, Anthropic), Saskia Hoffmann (CEO & Founder, Stealth Cybersecurity Startup)

Table #10 – Career Search / Interviewing – Mentors:

Elevate Mentor Table Career Search Interviewing Anran Li Juliana Almeida Meighan Agosta Nic Amos

Table Topics: Career Development, Technical Interviewing, Resume & Job Application, Web Services & Backend Engineering, Game Development, Individual Contributor vs. Management Track, Finding a Mentor, Career Switching, Career Transitions, Interviewing, Edtech, Tech, UX, UX Research, Career Search Tips (AI Tools)

Career Search / Interviewing Mentors: Anran Li (Engineering Manager, Riot Games), Juliana Almeida (Software Engineer), Meighan Agosta (Career Coach / UX Researcher), Nic Amos (Product Manager Leader, Kohl’s)

Ready to meet your mentors?

The virtual Mentorship Lounge will be open for one hour during ELEVATE, 8AM – 9AM PDT, on September 6th, 2023. The 10 tables will each be open to up to 30 participants. Attendees can hop between tables freely throughout the hour, so you’ll have the opportunity to meet as many of our 40 Mentors as you’d like! (Camera on or off, your call!)

What are you waiting for? Register today, it’s FREE and 100% virtual!

Plus network with fellow attendees, meet with recruiters & hiring managers in the virtual recruiting booths, attend over a dozen tech & career-focused lightning talks from girl geeks working at companies like Amazon, Microsoft, Guild, Google, Lighthouse Labs, and more! Get your FREE pass today!

Meet Bentley Systems staff & recruiters on Wednesday, September 6th, 2023 – 12pm-1pm PDT at their virtual booth at ELEVATE Conference & Career Fair!

Bentley Systems

Tune in to the livestream on September 6th at 11:40am-11:50am PDT for company welcome & introductions of Bentley staff. It’s FREE to RSVP & attend – REGISTER HERE & join us online.

Meet The New Club staff & recruiters on Wednesday, September 6th, 2023 – 12pm-1pm PDT at their virtual booth at ELEVATE Conference & Career Fair!

The New Club ELEVATE Conference and Career Fair Sept Girl Geek X

Tune in to the livestream on September 6th at 11:50am-12:00pm PDT for company welcome & introductions of The New Club staff. It’s FREE to RSVP & attend – REGISTER HERE & join us online.

elevate june speakers sponsors

Best of ELEVATE Sessions – From Career Learning Circles to Fighting Fatigue and Burnout – and Grokking your Technical Interview!

elevate june speakers

Our highly-anticipated Girl Geek X: ELEVATE Conference and Career Fair on June 7, 2023 hosted over ONE THOUSAND mid-to-senior women in tech around the world online for inspiration, connection, and learning. 

Thank you to our inspiring speakers & sponsors for helping make ELEVATE conference an incredible experience — Check out featured jobs – they are actively hiring for REMOTE (FLEXIBLE) JOBS – please check them out, and help a girl geek find her next role in tech!

Here are the most popular talks from June’s ELEVATE 2023! You can watch (or re-watch) them at the links below:

  1. Launch Your Career Learning Circle (Morning Keynote) – Margarita Akterskaia, Senior Software Engineer at Roblox, provides practical guidance on creating & maintaining a successful career learning circle.
  2. Fighting Fatigue and Burnout as Employee Resource Group Leadership – Emily Garcia, Head of Pixel Supply & Demand at Google, & Janice Litvin, Author of Banish Burnout Toolkit, discuss strategies for sustainable ERGs, motivating & rewarding volunteers, & generally preventing burnout.
  3. Grokking Your Technical Interview – Neha Srivastava, Senior Software Engineer at Attentive, shares crucial insights on what hiring managers are really looking for in the technical interview process.
  4. How To Get a No-Code Role – Amulya Vishwanath, Head of Developer Relations, Emerging Markets at Nvidia, explores today’s landscape of job functions, from non-technical to no-code roles in tech.
  5. The Most Important Product You’ll Ever Work On: You! – Cindy Deng, Leadership Coach at Pacific Blue Leadership, explains why having a vision, considering your strategy, embracing your unique value, & executing with focus – means you can & will be your enthusiastic advocate.
  6. Beyond the Algorithm: the Human Element in Developing Trustworthy AI – Yunwen Tu, Senior UX Designer at Vianai, & Sanchika Gupta, Data Scientist at Vianai, share perspectives on designing trustworthy artificial intelligence.
  7. AR/VR in Education: Immersive Technologies, Limitless Learning – San Robinson, Mobile UI Engineer at CrowdStrike, on the potential for immersive technologies (AR/VR/XR) for developers to create with Unity & Unreal Engine.

elevate virtual career fair for mid-senior women in tech, 2023
If your company is looking to recruit more women this year, please don’t let them miss out on our next Career Fair sponsorship opportunity! We want to hear from you. Please email

Thank you in advance!

– Angie Chang, Sukrutha Bhadouria, Amy Weicker, and 
the team at Girl Geek X 

Planning Your Girl Geek Dinner As A Startup Host

mosaicml girl geek dinner

Ignite your startup’s potential: Unleash the power of hosting a Girl Geek Dinner for unrivaled recruitment and talent branding. Discover the extraordinary reasons and indispensable roadmap for planning a triumphant soirée that will revolutionize your startup’s trajectory. 

Startups are squarely positioned to benefit from hosting a Girl Geek Dinner for recruiting and talent brand purposes. Here’s why and how startups can plan a successful Girl Geek Dinner.

Driving the partnership: MosaicML Girl Geek Dinner executive sponsor Julie Choi (Chief Community & Marketing Officer) wanted to build her startup’s employer brand for recruiting diverse engineers and research scientists.

From champion to speaker: She had helped plan and spoke at multiple Girl Geek X events while she was heading up AI Product & Research Marketing at Intel Corporation, where she saw first-hand the excitement among speakers and employees who attended the events, and the positive impacts of the partnership on retention and recruitment of high-quality, diverse female candidates.

Challenge: Naturally, when Julie joined MosaicML, she reached out to partner with Girl Geek X again. Despite being small by employee count (the 62-person MosaicML was acquired by Databricks for $1.3 billion this week), startups can establish their talent brand by leveraging and recruiting a diverse speaker roster from leading companies in the field (e.g. Meta AI, Salesforce Research, OpenAI, AWS AI).

Results: Over 120 girl geeks enjoyed networking and AI talks at the sold-out MosaicML Girl Geek Dinner featuring women speaking about AI/ML at leading companies.

Pro-tip: Recruit expert speakers from other companies! Julie and her speakers discussed topics including efficient machine learning training with MosaicML, reinforcement learning, ML-based drug discovery with AtomNet, evaluating recommendation robustness with RGRecSys, turning generative models into products at OpenAI, and much more.

Here’s what MosaicML Girl Geek Dinner 2022 announced for an incredible speaker roster:

julie shin choi tiffany williams banu nagasundarum ai

#1 Julie Choi, MosaicML Chief of Growth
 Executive welcome – watch or read

#2 – Laura Florescu, MosaicML AI Researcher 
Making ML Training Faster, Algorithmically – watch or read

#3 – Amy Zhang, Meta AI Research Scientist
Reinforcement Learning: A Career Journey – watch or read

#4 – Tiffany Williams, Atomwise Staff Software Engineer
Addressing Challenges in Drug Discovery – watch or read

#5 – Shelby Heinecke, Salesforce Research Senior Research Scientist
Evaluating Recommendation System Robustness – watch or read

#6 – Angela Jiang, OpenAI Product Manager
Turning Generative Models From Research Into Products – watch or read

shelby heinecke angela jiang lamya alaoui ai

#7 – Banu Nagasundaram, Amazon Web Services Machine Learning Product Leader
Seeking the Bigger Picture – watch or read

#8 – Lamya Alaoui, Hala Systems Chief People Officer
10 Lessons Learned from Building High Performance Diverse Teams – watch or read

By investing the time and effort, together we can create an inclusive environment that fosters diversity and enriches our events. It takes commitment to values and mindfulness to embrace this challenge and commit ourselves to doing the necessary work to recruit a diverse range of speakers. Between the networks of the executive sponsor Julie Choi and the Girl Geek X team, we invited speakers who are industry experts and provided thank you gifts to the speakers. The speakers found it to be a good event for them professionally, from sharing their expertise onstage (recorded for YouTube) to networking with fellow industry experts.

Recruiting set up tables, swag, and signage for the sponsoring company to talk to interested attendees. Most Girl Geek Dinners feature a slew of talks and speakers who inspire and motivate attendees to talk to recruiting afterward. At a recent OpenAI Girl Geek Dinner, a speaker spoke about how:

“10 years ago, my dream was to get into artificial intelligence, but I didn’t know how to do it. I was a software engineer… so attending OpenAI Girl Geek Dinner in 2019 and meeting a recruiter introduced me to Residency Program. I ended up applying full-time and I got three offers [to work at OpenAI]. (applause) I wasn’t trying to brag, but thank you. This is more to encourage you.

Alethea Power, OpenAI Member of Technical Staff

The AI specialist is the new “it” girl in tech, writes a Vox reporter on the AI boom.

At a 2021 Girl Geek X event, a speaker from OpenAI talked about prompt design and engineering for GPT-3 – almost two years before ChatGPT became commonly mentioned in New York Times articles about the raise of AI.

By partnering with Girl Geek X, startups like OpenAI and MosaicML bring pioneering AI/ML technology and diverse women technologists closer together, highlighting how their technologies may become the next invaluable skill set – and perhaps even employ more girl geeks in AI/ML tech!

girl geek x pixies eml

For more inspiring women in tech:

10 Girl Geeks Making History & Changing The World

frances haugen sheila lirio marcelo mira murati shanea leven

We spotlight 10 girl geeks this week with valuable insights on mentorship, leadership, engineering and so much more – in celebration of International Women in Engineering Day! Watch their sessions from Girl Geek Dinners at the video links below:

#1 – Frances Haugen, Former Product Manager, Facebook – speaking on Product Management and Gender (video) – at GitHub Girl Geek Dinner:

“You don’t have to accept bad behavior. Vote with your feet.”

“You should find things that you love, and you should go do them. If you can dream it, you can build it. I’ve been in environments where it seemed there were insurmountable odds, yet we went out and did it. The world isn’t fair, but you belong in it.”

“What you do today matters to the women who come after you. I wouldn’t be here today if Marissa Mayer hadn’t been in my management chain [at Google].”

frances haugen vote with your feet girl geek dinner github girl geek x quote

#2 – Sheila Lirio Marcelo, Founder, – speaking (video) – at Girl Geek Dinner:

“Being Asian, we had designated professions. There were supposed to be the doctor, the dentist, the engineer, the lawyer, but God forbid no one should ever become an entrepreneur.

care girl geek dinner sheila lirio marcelo

I was working at a technology company, but using the Yellow Pages to look for care. Something really didn’t add up…”

If women and men worked equally, from the McKinsey study, the worldwide GDP would grow by $28 trillion or 26% by 2025. Apparently, that’s the size of the combined US and China GDP, if they were just equal. The single biggest obstacle to women’s equal workforce participation across the globe is balancing work and family responsibilities

#3 – Mira Murati, Chief Technology Officer, OpenAI – speaking (video) – at OpenAI Girl Geek Dinner:

There is definitely a lot of hype, but there is also a ton of technological advancement that’s happening.”

“My background is in mechanical engineering but most of my work has been dedicated to practical applications of technology.  Every day we wake up to headlines. We wonder what this is going to do to our minds and to our societies, our workplaces and healthcare.”

Even politicians and cultural commentators are aware of what’s happening with AI to some extent, and politicians like this, to the extent that there’s a lot of nations out there that have published their AI strategies.”

mira murati openai girl geek dinner quote IG

#4 – Charlotte Yarkoni, President, Commerce and Ecosystems, Microsoft – speaking (video) – at Microsoft Girl Geek Dinner:

I was part of an inaugural program at the time called Electrical Engineering or Computer Science, or EECS as it was known.”

charlotte yarkoni microsoft girl geek dinner IG quote

“Kicking it old school.. that was my education, if you will, and my real foray into tech. The challenge is, though, it comes with a responsibility. At Microsoft, GitHub, and LinkedIn, we spend a lot of time on that. It’s not just about innovating, it’s about innovating with purpose, and making sure that you’re leaving the world in a better place than you found it before you introduced your solutions. It’s those unintended consequences that you have to be very thoughtful about.”

#5 – Citlali Solano Leonce, Director of Engineering, Palo Alto Networks – speaking (video) – at Palo Alto Networks Girl Geek Dinner:

“I’m hoping that tomorrow is going to be a little safer than today, so that the world that I leave to my kids and my legacy is much better than what I’m living right now.”

“Back in the day there were no cell phones, no tablets, no flat TV screens… Now, we all have our lives in the digital world. How many of you do your banking online? How many of you do video gaming or your kids do video gaming? We have a big responsibility, everything is interconnected. How do we prevent the bad guys from getting that? I personally love working here because I identify with our mission of securing our digital way of life.”

citlalli solano leonce palo alto networks girl geek dinner IG quote

#6 – Mariana Tessel, Chief Technology Officer, Intuit – speaking (video) – at Intuit Girl Geek Dinner:

“One of the things that I’ve learned in my [career] path that your network is one of the most important things that are going to help you in your career.”

marianna tessel intuit girl geek dinner IG quote

“Sometimes you find your network in unexpected places.”

“To give you a few ideas, the people you know then later on they will go places, or you need something, or they need something and then you have that connection to really make a dent for them, for the companies, for you, for your career, for your company or sometimes even just getting advice when you need it or sometimes it’s finding that next job when you need it, whatever that is.”

#7 – Claire Hough, Chief Technology Officer, Carbon Health – speaking (video) – at SquareTrade Girl Geek Dinner:

“One of the reasons why I stay there is for all of you to know that you can be Head of Engineering anywhere you want, right?

“I have to say I’ve been around the block a few times but young women these days impress me every day. There were a lot of opportunities for me to get out of engineering. People offered me other jobs like product, GM, or whatever. But because tech is such an unfriendly place for women, I didn’t want me to add to the number of women getting out.”

Be resilient, stay in it, and add value. That’s my story of why I’m so old but I’m still in it.”

claire hough cto IG quote squaretrade girl geek dinner

#8 – Neha Narkhede, Co-Founder, Confluent & Co-Creator of Apache Kafka – speaking (video) at Confluent Girl Geek Dinner:

Developing a sisterhood will take us far in seeing the change we want to see in the industry.

neha narkhede open source confluent girl geek dinner IG quote

“I was lucky enough to be on a team that got a chance to create a very popular distributed system called Apache Kafka. We open sourced it, it went viral. I sourced a business opportunity around Confluent, pitched it. Fortunately, they agreed to start this company with me.”

“Most of my career has been about introducing this new category of software called Kafka and event streaming into the world.”

#9 – Shanea Leven, CEO & Co-Founder, CodeSee – speaking (video) – at CodeSee Girl Geek Dinner:

“We need to support more women in developer tools. They’re basically force multipliers because you enable developers to do something better to enable their end users.”

“I am fortunate to get to ride this trend, to build a developer tool and help people learn how large scale codebases work, so that we can spend more time building. There are a bunch of startups popping up to help you understand and build features faster and help you understand in a very different way than traditionally you’ve had to understand a codebase. As new types of people, women, underrepresented people, get into engineering, our dev tools need to evolve with them.”

shanea leven codesee girl geek dinner dev tools

#10 – Alethea Power, Member of Technical Staff at OpenAI – speaking (video) – at OpenAI Girl Geek Dinner:

“This talk is called ‘If At First You Don’t Succeed, Try Try Again’.

Screenshot at .. PM

“About 10 years ago, my dream was to get into artificial intelligence, but I didn’t know how to do it.”

“I was a software engineer and site reliability engineer, so attending OpenAI Girl Geek Dinner in 2019 and meeting a recruiter introduced me to Residency Program. I ended up applying full-time and I got three offers here. Thank you. I wasn’t trying to brag, but thank you. This is more to encourage you.”

openai girl geek dinner Alethea Power

How to plan a Girl Geek Dinner

girl geek x pixies eml

For more inspiring women in tech:

ELEVATE 2023 Career Fair Kickoff – Employer Intro – Vannevar Labs (Video + Transcript)

Breanna Carodine (People Operations Specialist at Vannevar Labs), Caitlin Stangland (Senior Talent Acquisition Specialist at Vannevar Labs) and Ann Zeng (Software Engineer at Vannevar Labs) talk about the company’s culture, values, and mission, as well as the opportunities for growth and development within the organization.

VANNEVAR LABS combines top software engineering talent with decades of mission experience to get state of the art technology to the people that keep us safe. Vannevar Decrypt is a foreign text workflow platform built for national security.


Check out open jobs at Vannevar Labs!


Angie Chang: We are here today to kick off our career fair session with Vannevar Labs, and I want to introduce Breanna.

Breanna Carodine: Hi everybody, super happy to be here and to talk to you guys all. My name is Breanna Carodine. I work for Vannevar Labs as the people operations specialist, and that just means I basically work with a lot of onboarding stuff and employee experience and a couple of other things like that. I’ve worked here since December of last year and it’s been such a great company so far.

Breanna Carodine: We’re growing so fast and we’re doing a lot of really great things. Some of my favorite parts about the company is the way that we value certain things. Transparency is really important to us. We want to make sure that everyone is aware of everything that’s going on behind the scenes. We’re not afraid to share some of our trials and tribulations with each other but also celebrate our successes.

Breanna Carodine: One of the things that we really love is that we are a team of Jedi, meaning that all of the people who work here are really good at what they do, and that means that we can accomplish so much more. And everyone has very similar mindsets around being user and mission focused on our work.

Breanna Carodine: Everyone comes together with the same mission, being really great at what they do so we accomplish so much. Also, a value that I think is really important as that we put your wellbeing first. Something I tell every new hire who meets me and everyone within the company, “You cannot do your best work if you’re not your best selves.” And we truly, truly believe that here.

Breanna Carodine: We hope that everybody who comes into this company understands that we have time that you can take off. We have mental health benefits so that you can always be working at your best selves because we truly, truly do care about that.

Breanna Carodine: We are also a remote-first environment so that you can create the environment around you the way that it’s going to be best for you to do your work. I know that’s really great for me because sometimes I don’t need a lot of people around to get a lot of work done. Sometimes, I remove myself from this space and go to a coffee shop and work in a space that I can have all that energy, so that’s really awesome.

Breanna Carodine: Just a little bit about Vannevar and our mission, our basic spiel is that we bring together multidisciplinary groups of people with a wide range of experience and over 40 years of military experiences, engineers from some of the top tech companies and startups, and a passion for delivering mission-critical tools to support public servants on the front lines of the country’s most important national security problem.

Breanna Carodine: A lot of the work that we do, we work right alongside with the Department of Defense and we do some really critical and important work for the country. So to get into a little bit more of those positions and even more details, I’m going to pass it over to Caitlin.

Caitlin Stangland: All right. Hi everyone, so excited to be here today. My name’s Caitlin, I’m based in Northern California, I am a senior recruiter here working on technical and non-technical positions. I’ve been with Vannevar now since January. It’s gone really quick, it goes fast here. And some of the open roles that we currently have, I’ll just give high level overview.

Caitlin Stangland: Currently, we have our mission team and mission roles. Those are broken up into groups, mission success and mission development. Mission success, think of it as customer success within an organization. So most profiles we’re looking for are people with military backgrounds and then have worked within a customer success team at a tech company. So they’re used to working and communicating with clients.

Caitlin Stangland: These positions, our employees on the mission success team are communicating with our customers within the Department of Defense. And there is frequent travel involved with these roles. Mission development, think of it as growth and sales. Usually, we’re looking for someone with military background, and active clearance, and have that sales background and mentality. We also have various engineering roles open.

Caitlin Stangland: We offer internships. Currently, we have deployment lead internship openings for this year. We’re actively recruiting on those roles. We also currently have an opening for a data researcher position. This role’s actually very highly confidential so I’m not able to say too much about what this position does. Then we also are currently looking to build out our finance department. We have a controller opening right now who would own all of the accounting function at Vannevar and would build out that team.

Caitlin Stangland: Those are currently high level overview of our current openings. You can see them all posted on our site and on LinkedIn. Usually at Vannevar, our interview process, we like to really keep it to the same process across all positions at the company. Always starts with a recruiter screen, then a hiring manager interview dependent on the role.

Caitlin Stangland: For mission roles, we usually like to include a decom, think of it as like a conversation prep document that we would send you prior to meeting with the hiring manager so you have a focus of that interview. From there, there’s usually homework or a technical assessment. And then the final part is a top grade interview that’s usually one hour long where someone at the company just gets really, it’s just a lot of questions about your background and goes kind of role by role. So we like to just streamline it and keep it as quick as possible.

Caitlin Stangland: And overall, like Breanna was mentioning, we have awesome benefits. The company culture I love, the startup environment, Vannevar, we’re fast-paced, it’s very collaborative, we’re growing like crazy. But first and foremost, I feel like everyone just genuinely really likes each other.

Caitlin Stangland: We do a really good job of hiring nice people. Every team, no matter who you’re working with, they just do a great job of collaborating with one another. When I joined Vannevar, I did not have a defense tech background, so this was new to me and there was a learning curve there. And I never felt like no question was off limits, no question’s a bad question here.

Caitlin Stangland: Everyone’s open to educating and just giving all the information that you need to start your position at Vannevar. And we have great benefits, competitive pay, equity, the flexible working environment, remote working environment. We do have the flexible PTO policy.

Caitlin Stangland: Mental health like you were mentioning earlier, we definitely prioritize that so we do have a mental health benefit. And overall, I just feel like we’ve done such a good job of creating such an amazing culture. And I’m excited to talk to more of you later. All right, and I’ll kick it off to Anne.

Ann Zeng: Thanks, Caitlin. Hey everyone, my name’s Ann, my pronouns are she, they, I’m a software engineer here at Vannevar Labs and I’ve been here for a little over a year now. Also, my internet has been going on and off so I hope I don’t cut out. Well, I’ll do my best.

Ann Zeng: I currently work on the team that focuses on collection and ingestion of the foreign media that powers the Decrypt platform. Like Breanna said, we’re focusing on solutions in the national security space. And one of our biggest products is called Decrypt.

vannevar labs intro at elevate

Ann Zeng: Let me talk everyone a little bit about day-to-day life as an engineer. We’re at about 30 to 35 people on the engineering department among a couple of different teams. I wouldn’t say that the team boundaries are super solid. Somebody described it as a semi-permeable membrane. There’s lots of projects that are across teams. We really welcome people to advocate for themselves in terms of what they want to work on, what they want to learn, and to the extent that there’s room to have people work on things slightly outside of their job description. That’s definitely still there. I mean, we are a startup. No one’s going to say, “Hey, we don’t want you working on more stuff. Why wouldn’t we want people to work on more stuff?”

Ann Zeng: In terms of the tech that we use, it kind of depends on the team, but I would say broadly Python, Node, React, TypeScript, Postgres, OpenSearch. Our stuff is deployed in AWS so yeah, if you have any experience in any of those or are you interested in any of them, feel free to come by and say hi.

Ann Zeng: Because we are remote first. We’re pretty understanding about people’s time zones. There’s no hard start or stop times for working hours, like if someone messages me but it’s like 7:00 AM my time, nobody’s like, “Why aren’t you answering my messages?” For required meetings, at least on the engineering side, we’re pretty light on those, sprint rituals, standup retro planning refinement, one-on-ones with your manager. Those are usually the required ones.

Ann Zeng: We really encourage people to sit in on other meetings. So there’s an engineering department wide round table, there’s a company-wide demo space for engineers to show off like, “These are the things that we’re working on.” I love to lurk in different Slack channels just to see what’s going on. We really encourage people to get involved or to just spread transparency about the progress of work, that kind of stuff. Like Caitlin and Breanna said, we’re trying to solve problems in the national defense space and we want to do it with some full people. I’m really excited to meet everyone. Please feel free to come by to the booths. Thanks.

Angie Chang: Thank you so much for these great introduction to you all and the roles and the teams. We’re going to meet you in your booth in about 10 minutes. And now, we’re going to move on to our next session. Thank you so much, Vannevar ladies!

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

“Beyond the Algorithm: the Human Element in Developing Trustworthy AI”: Yunwen Tu, Senior UX Designer & Sanchika Gupta, Data Scientist at Vianai (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Sukrutha Bhadouria: Sanchika Gupta and Tutu. The talk’s title is “Beyond the Algorithm: the Human Element in Developing Trustworthy AI”. Welcome, ladies.

Yunwen Tu: Thank you. Thanks everyone for joining the session today. Today we will share our thoughts and learning about building trustworthy AI. First of all, we will share a little bit more about ourselves, ourselves. My name is Yunwen, go by Tutu. I’m a user experience designer. I enjoy using design as an approach to learn and solve problems for people in both digital and physical way.

Sanchika Gupta: Hey, I’m Sanchika. I’m a data scientist with experience in the field of technology, cybersecurity, and human-centered AI. In a former life, I was a computer science professor, though while we are building the human-centered trustworthy AI and have been working with each other. I’m generally curious about this thought of what is the human involvement in building the trustworthy AI.

Sanchika Gupta: There are the three topics that we are going to discuss in this talk today. The uncertainty and concerns of AI development. How do we build and maintain trust relationship in AI and the unique value of human expertise in the age of AI.

Yunwen Tu: What’s your reason, conversation, or search about AI? For me, I chatted with my designer friends, and we talk about how AI can streamline some of our work, and is there anything can be replaced by it so that we can do things faster, such as making icons or some simple websites.

Yunwen Tu: We also talk about the new design trends that brought by the new development of AI. Since the breakthrough of the large development in large language model, more people have exposed to this technology and learn about the possibility how AI can be applied to their work. Lately, I have even heard users, they share that they have the desire to add this and that function using AI in their product after their self-education.

Yunwen Tu: In general, I got a sense that many of us are concerned and uncertain about the impact that AI can bring to us or our society. We start wondering, has the time finally come, is finally replacing us now? As we already see AI in is applied in many fields such as healthcare, automobile and et cetera. I ask my co-speaker Sanchika, where do you feel AI has already made a big impact and what kind of role is AI playing?

Screenshot at .. AM

Sanchika Gupta: Automation of AI may lead to replacement of certain human roles. However, AI also presents us with newer opportunities and creates our jobs easier. Let me talk about certain examples where I feel AI has been present around for so, so many decades now, and it already feels as a partner. First example that I would like to talk about is natural language processing.

Sanchika Gupta: AI has significantly improved natural language processing capabilities. There are virtual assistants like Google Assistant, Amazon Alexa, which utilize the AI algorithm to understand and respond to voice commands. Now, it may seem a little trivial to talk about these because they have become a part of our daily lives, and I personally use them on a daily basis to like set up reminders, ask about the weather, ask it to provide directions for a destination while in the car, making our daily lives more convenient and efficient.

Sanchika Gupta: Another example I would like to give is of natural language translation. There are platforms like Google Translate, which leverage AI algorithms to provide real time translations between different languages. If you go to a country where they don’t speak your language, you can still communicate effectively. I have done it myself so many times. With this, I want to say that AI may automate tasks that require basic skills while human can focus on higher level responsibilities, harnessing creativity and imagination.

Sanchika Gupta: The next question is how can we have more access to the system so that we can use AI as a partner? Generally speaking, education and awareness are crucial in fostering trust in ai. Trust is very essential for building a reliable, transparent, and available use of system. AI has been present in various forms for many decades now., even during my university studies, I delved into the neural networks topic and AI’s potential to replace job was already circulating at that time.

Sanchika Gupta: However, the conversation around trustworthy AI only gained prominence with the emergence of large language models, instances of AI generated hallucinations where the system just make stuff up, started gaining attention while getting a recommendation for an unwanted TV show may have minimal impact. A recent incident in which an AI system made a judgment on a legal case without the attorney’s verification highlighted the potential consequence. These recent repercussions have brought the issue of trustworthiness to the forefront, causing it to enter the collective consciousness of all of us.

Sanchika Gupta: Let me throw another example on you and you tell me which would you prefer. If we were to compare a trust in an AI driven car accident versus a human driven car accident, what would you choose the second time? My opinion as humans, we tend to trust other humans more than AI. How can we bridge the gap?

Sanchika Gupta: I believe that by focusing on AI literacy, upskilling collaboration, and ethical considerations, us individuals can also be empowered to embrace AI as a tool to enhance our skills, productivity and relevance in the job market. Now, I would like to ask Tutu, how do you as a designer build and maintain trust relationship in ai?

Yunwen Tu: As a designer, I started this journey by understanding the technology, especially why AI failed and why people don’t trust AI. The major distrust I learned in AI is the lack of transparency. We haven’t considered enough that trustworthiness is a high priority in building that.

Yunwen Tu: Many AI models feel like a giant black box sitting between the input and output. We don’t, we don’t know how it works and it just did it work. When handling very mundane tasks such as grammar correction, language translation, it’s great when they’re magically done by machines. But when it comes to like riskier cases with bigger impacts such as loan approval, it’s impossible to rely on a black box like this, which you don’t even know if can understand 10% of your problem. There are also potential ethical biases in a model that needs to be monitored closely.

Yunwen Tu: We help user increase the transparency, the observability and the visibility to make the process and the model more interpretable and explainable in their context work. That’s also baked in our design principles.

Yunwen Tu: And as part of our design principles, we also use design thinking method that to work with our users to understand what does trust mean to them, and discover how AI can solve their business problems in a trustworthy way. For here, I would like to give you two examples that we use the user interviews and other thinking methods to solve problems for our users.

Yunwen Tu: The first example that we are working with an insurance company to reduce their business loss. Through many rounds of discovery interviews with underwriters and their managers, we actually found their primary challenge is not about finding the best algorithm to analyze their internal data.

Yunwen Tu : Instead, they want to understand better how the events happened in the past 10 decades that had impact their current business performance. In the meanwhile, we feel everything moves way faster now. For our underwriters, they need to quickly catch up with all the new updates such as the regulation change, the new settlements of lawsuits in their professional area. In the end, we build a tool that use natural language processing to help our user to connect the dots and find the needle in the ocean of internet data, which is the result we would not expect if we didn’t spend that much time to talk with our users.

Yunwen Tu: The second example I would like to share is related to our ops platform. And as part of my UX research, I regularly chat with different users such as data scientists, business ops, and et cetera.

Yunwen Tu: I found the expectations from our data scientists on monitoring AI models are very different from other general business users. They’re not looking for a no code or a fully automated experience. Instead, their philosophy is not to trust the data, not to trust the model until they have seen enough evidence to take action. It’s very crucial for us to deliver those insights clearly and efficiently.

Yunwen Tu: From our users, I actually learned trust is not purely top performance, not the best performing model. Trust means making informative decisions after peeling off the complexity and the root causes. Now Sanchika, what does trust mean to you as a data scientist and how have you built trust in your practice?

Sanchika Gupta: Demystifying the AI systems and ensuring reliability helps human use them with confidence. There are certain visible limitations of AI, for example, drift observability, root cause analysis, bias and ethical use are important to establish trust. So let me explain with an example how I establish trust in AI.

Sanchika Gupta: We as data scientists do not tend to trust our model or data. Instead, try to gather enough evidence around it and then be able to trust it. Let me talk you through that process.

Sanchika Gupta: Let’s consider a case of an AI driven customer service chat bot used by an e-commerce company. The AI chat bot is deployed to handle customer inquiries and support requests. Over time, they noticed that there is a decrease in the customer satisfaction scores and there is an increase in the unresolved issues.

Sanchika Gupta: The first step at this point, which I as a data scientist would like to do is to check on the model performance, let’s say, model performance evaluation, reveal that there is a decline in the chatbot’s accuracy and performance compared to previous months indicating a potential issue. Then we can begin closely monitoring the chatbot’s interactions and collect data on input queries, chatbot responses and user feedback. Now by looking at all of this data, closely plotting it, we might be able to identify certain patterns or anomalies in the chat bot’s behavior. During all of this observability and analysis, let’s say that we were able to identify a set of queries that consistently receive incorrect or nonsensical response from the chat bot. Now these queries stand out as potential outliers as they significantly deviate from the expected behavior. Then the next step could be to, let’s say, look at drift analysis on the chat bots performance.

Sanchika Gupta: We can compare key metricses like customer satisfaction scores, response accuracy, and resolution rates over different time periods. During this analysis, we notice there is a significant decline in performance starting around the same time as there was an update in chatbot’s knowledge base. Based on all these findings, we start the root cause analysis and discover that the update to the chatbot’s knowledge base introduced some incorrect or incomplete information resulting in chatbots diminished performance.

Sanchika Gupta: Through this example, we observe how model performance analysis, observability, outlier detection and drift analysis collectively contribute to the identification of the root cause leading to targeted corrective actions for enhancing the chatbots performance. This provides a glimpse into the case that I mentioned above showcasing the methods employed to established trust in an AI system. This also demonstrates the importance of human involvement in analyzing and improving the AI system, reinforcing the notion that despite all of its capabilities, AI cannot fully replace human judgment and decision making.

Screenshot at .. AM

Sanchika Gupta: Now this leads to my third question that we would like to discuss here. What is the unique value of human expertise in this age of AI? Now again, creativity, imagination, and diverse opinions are very unique to humans. If let’s say there were to be a discussion, we humans can participate in a discussion and arrive at different conclusions in the same situation

Sanchika Gupta: At the same time, AI lacks the ability to participate in a discussion as an equal lacking both opinions or any standing in human conversations. Now let me quote another example here. New neural networks founder Professor Jeffrey Hinton, six years back in 2016 said, we won’t need radiologists to analyze scans and image perceptional hams can do all the scanning and diagnosis by themselves. Six years have gone by and we are nowhere nearby. It is not because of compute power or resources because I feel that compute power and resources have only been growing in the last couple of years.

Sanchika Gupta: What I believe is that AI can only solve very well-defined problems. What happens when it is posed with ill-defined problems? That is where human ingenuity comes in. All AI attempts to do is to recreate memory and computation capability of human brain. But what makes human a human is not just being able to solve the task, but be able to synthesize the complexity of this world and make decisions on the basis of that. Now, at this point, I would like to ask Tutu to give her thought process around this topic.

Yunwen Tu: Thank you, Sanchika. Those are great takeaways and what you just shared also, remind me again the user interview process in our design method. We do that to understand user’s journey and the pain points, and then present a personal story that summarize our learnings and the synthesis. Sometimes I feel AI is like an abstract persona that is summarized in a way with a well-designed cover and well-defined title. However, when we are doing the interview, the persona story, it’s not about creating an abstract figure, but to emphasize with our users’ needs, their feelings and the the reason why they’re making those decisions. There are all, and also this process are all done through our communication and synthesis in person. But AI does not learn new insights as we do in those contexts. They also don’t understand the complexity of the world like us.

Yunwen Tu: For example, when I read the news, the debates on the news, when I also work with different people, design for different users, I always feel so, and this comes from our unique experience, our desire and the belief that makes us like diverse, unexpected. And sometimes we argue we also have conflicting ideas, but this also made the world and humans like very unique that AI cannot replace with. That covers all we want to share today. Thanks for staying with us. And if we have a little bit more time, please feel free to ask us question now or reach out to us afterward. Thank you all.

Sukrutha Bhadouria: Thank you. Thank you ladies. Yes I would definitely encourage everyone to reach out to you both on and ask their questions on LinkedIn. Let’s keep the conversations going and I really encourage everybody to rewatch this content, share this content as much as possible with everybody. We really appreciate the time that you’ve taken out Sanchika and Tutu. Thank you everyone for attending. Bye everyone.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

“Exposing Gaps: Cybersecurity Workforce and Education”: Rahmira Rufus, CEO at AWT Solutions (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Angie Chang: With us today, we have Rahmira Rufus. She is a next-generation enterprise security architect, a university professor, academic and industry scientist. Welcome, Rahmira.

Rahmira Rufus: Thank you. Thank you very much, Angie. Hello everyone. Very honored to be here to share with all of the insights, all of the wonderful things that’s happening at ELEVATE this session. Like Angie said, my name is Dr. Rahmira Rufus. I’ve run a gamut of things that I’ve done in the cyberspace – lot of technical work with the elements that she spoke about, contracting areas, purely and academic. This element focuses more on a lot of the outreach work that I like to do that I’m not really able to dive into within, you know, places of employment and things of that nature. That’s what I’m gonna talk to you about today. I thought this would be a really good topic to bring up at a Career Fair, as folks are trying to find their way and navigate through the employment process, excuse me, workforce and academia and that whole gambit, that pipeline.

Rahmira Rufus: I’m gonna dive right in. Like I said, the title is Exposing Gaps in Cybersecurity Workforce and Education. I like to say that this particular work or slide deck is stemming from some work that had a different direction. This is more of a tangent of that work. I’ll start by addressing why are we even doing this analysis? Originally we wanted to find out what was happening within the cybersecurity paradigm as far as that pipeline for supply and demand. The workforce for the talent and things of that nature. What we discovered is that this is not a unique problem. This actually occurs in other industries. One in particular biomedical things of that nature, where you have very specialized kind of criteria for resourcing talent and staffing needs.

Screenshot at .. PM

Rahmira Rufus: What we discovered here that’s the reason why we’re doing a gap analysis. A gap analysis is a very simplistic term. We’re trying to see what are those kind of hindrances, are those kind of blockers for where a particular environment or situation is at an actual level of performance? How do you get to a desired level? How that relates here through this series that I use when I work with my clients, how that relates here is that we’re trying to see what is this huge disparity against folks trying to either jump into the cyber field, maneuver within their career path, or transition to higher levels, or, even in some ways, try to make an entire pivot in another direction. What we discovered in this work was that there are some elements that are not being taken into consideration that makes the process a little more convoluted than it needs to be. That’s pretty much what we’re addressed. If there’s anything someone needs me to go back, just let me know. but other than that, I’m gonna dive right into it.

Rahmira Rufus: What we started was, we looked at a generalized problem,. What’s happening within the cybersecurity workforce and basically feeling that void of the workforce, of the talent that’s out there and the talent that’s more importantly needed. We started at a global scale, and the diagram to the left, the far left, is what we collected as data, globally. Every single place on the planet is not represented, but we try to get a proper representation of the skillset and those particular numbers that we would thought were relevant for the sample population that we were looking at.

Rahmira Rufus: We saw a disparity when in the US and wanted to focus there. Now, I do want to say the data that was collected for this very huge gap that shows for China, <laugh> and India, there is some emphasis there. However, that’s not a part of the scope of this talk, but that is something that as time progresses, we would like to see if we could journey into that gap.

Rahmira Rufus: We have some assumptions, but, you know, since we’re scientists, we’re not allowed to do that. We actually have to follow a process. Right now we’re in pretty early qualitative work with that study. If I dive over to the area that we’re looking at, we then wanted to focus in the United States. In my next slide, I want to show where we started seeing some disparities. I’m taking those two slides over the far right and they’re on the far right on this slide. We had looked at basically a percentage in a kind of you empirical actual value for each one of the data points for the job openings up against, compared against the employed workforce, right? The diagram on the right looks like it kind of flows in a normal fashion.

Rahmira Rufus: The problem is that the diagram in the middle, we saw little disparity around 2021, moving into this year. It didn’t quite match, it wasn’t like a huge disparity, but that triangle with that slant at the bottom, that that variation was really, really off. We said, okay, look, let’s dive into more granular elements. And we started looking at for state, and once again, every state ends represented here, but this is a sample population that we think would be a best representation. We noticed was by state that matched or was similar, kind of congruent to the, the diagram on the far right, but we were still concerned about that percentage. Why did we get a percentage that did not look like the two diagrams that basically are on the far left and the far li right of the middle one?

Rahmira Rufus: That’s where we started diving into what we considered to be this problem. And the reason why this is important for this particular forum is because in the next few slides, you’re gonna start to see some of the reasons why when folks are looking to streamline and understand exactly what they want to do in cyber, it’s a little more complex than than you actually thought from us just taking that tally that you saw on the previous slide, a very general approach to trying to solve or answer this question.

Rahmira Rufus: What we discovered was that approach was a little off because we didn’t look at the true nature or components that were involved with the supply and this demand. Now if we jump to this slide, we’re talking about a simple supply and demand problem, right? We then associated demand with the knowledge, the education that’s requested, as you see in the chart in the top, and then we looked at entry-level jobs and then against paired up against their requested education. If you notice in each one of these charts, it seems like folks in the bachelor’s degree area, they’re winning, right? For these particular cyber skills. Then, you have some luck in the kind of the sub, those are pretty much a lot of the associate degrees, junior college, but also a lot of the certifications and licensing that people engage in when they either have done a lot of higher education / learning, or they could be adult learners, it could be all kinds of different reasons, but they don’t necessarily go through a traditional degree program.

Rahmira Rufus: Then you see the graduate level of folks having that kind of smaller percentage of kind of matching a little bit of the sub bas. Now, what is funny, thank you. What is kind of funny here is that in actuality, because you know, I’m a teacher / professor as well in my background, I notice that’s not true to what’s happening here. Students, uh, adult learners, uh, folks in the field trying to figure out where they are, where their next step is gonna be, what’s their next direction, they’re not having the success that’s being represented in these charts. What we had then decided, okay, since we’re talking about supply and demand, well now let’s look at the supply side. Now let’s look at the workforce. What we had discovered was that this was a much more complicated problem than we thought, than someone just saying, Hey, I would like to get into the cyber field and I’m just gonna protect, I’m just gonna pick this particular area and I should be fine.

Rahmira Rufus: I’m just gonna grow here in years. Now an example that I will give, and not to shed any kind of negative light or anything on this particular field, but if we look at fields like psychology, most folks know this is something that’s been told by everybody in that field. Psychology, psychiatry. If you are not pursuing the master’s or PhD level, you’re gonna have a difficult time finding the type of placement that you thought you would being that breakthrough psychologist or psychiatrist. That’s something that’s a normal situation in that field. From what I’ve told, it’s been worked on over the years, but that’s usually something that’s been accepted in this space. That’s not necessarily the case. There are absolutely so many options and so many variations of what you can do, not only in cyber, but if you go down into its larger focus, like a computer science or pure network admin, or you name it, this becomes a little convoluted.

Rahmira Rufus: Now, I’m being very, very nice with this slide. Thank you. I’m being very, very nice with this slide because the next slide I’ll show is actually a very large data set that we had to work through so that I could bring you such a pretty visual right here as far as just some of the things that you have to take into consideration when you’re trying to think about what’s going to be your cybersecurity career path journey.

Rahmira Rufus: One thing really quick, I want to note if folks are paying attention right here to this graph over here, notice how low security intelligence is, I can give you a little hint of that. Remember, this is a trend, a growing field. Right now, the reason why this number is so small, that is actually being developed as a track.

Rahmira Rufus: All of the data analysis, all of the business intelligence, all of the different elements out there are now being able to be fueled and tunneled into a path that’s going to be security intelligence. Just letting you all know, look for that on horizon. However, these other tracks, they had experienced the same thing previously. That is another added complexity to this field, as things change, as things morph, as we’re talking about the next generation of computing, how do you know where to go, where you’re at, what you need to be? was able to break down some of this in some of these charts, but, I think everyone can kinda understand what’s going on here. Right here, you’re talking about, we’re, we’re usually using like as the little bit of a control, right?

Rahmira Rufus: The job opening, say, nationwide. Then you have to look at the entry-level jobs versus the knowledge that you would need. Then also you’d have to look at the skills and knowledge that would be required for particular sets. And then on top of that, what type of certifications and licensing would I need to perfectly be where I wanna be or where it is I’m trying to go see just that quick. On the next slide, as I said, this is just to give you an example. We have so many repositories that we pulled this information from, and I was only able to get 25 in these tiny little visuals on this screen. That is not something we want you to do. What does this mean to all of you? All right, here’s a little in the second slide you saw that I was gonna talk about dilemma, and then I’m gonna jump into the, what I think is a quick solution.

Rahmira Rufus: Here’s a method that you could help you without having to go through all of that granularity that we discovered in this process. One, be proactive, right here is a diagram of a chart, right? That’s all of the data that you see before, and I’m going to have this recorded, basically provided for you in the next two slides is from platforms like cyber, ISC squared, the, their platform, their education focus of the nice framework, things of that nature. You can go and get all of this data, get all of this information, and you can get metrics about these particular fields, right? Right here is a combined chart, a visual where you should go out there instead of waiting to go to an employer or waiting to figure out what’s out there or talk to your professor or whatever, kind of overindulgent process that you normally do.

Rahmira Rufus: You go out and be proactive and find out what these, what these particular roles and these skillsets are, and start diagramming them and them in a skillset similar to this slide. Now, to break it down even more, and I said to DIY – do it yourself – break them down into different skill sets that you find to be requested skills. And like I said, and these particular prac platforms, you can go to the go there to find this information. Right here, this is basically a broken down version of the slide you just saw. And you can pick particular skills and find out where you measure right now, right? And what is it that you’re gonna need to either increase that proficiency or, you know, whatever it is that you’re trying to kind of get at, right?

Rahmira Rufus: Be realistic with yourself, okay? Because this is about improving you and making yourself a much more viable product as you move forward. Use elements like that to set yourself up for success. Some real quick wins, as I said – I can tell you about – is do your own analysis. Like I said throughout this slide, you don’t have to dive deep like we did with the tangent that we found as a new area of research for the disparity in the path between the cybersecurity workforce and its talent and resourcing, but at least try to leverage different areas, like I said,, these different platforms.

Rahmira Rufus: On top of that, when you actually capture all of these roles, develop proficiencies with analytic service or, data, data analysis… You can leverage different visualizations, power bi, you know, build your chops in those areas or create diagrams or metrics like these. I will let us say, since this is a Career Fair, these types of visualizations go very, very well in front of potential employers. Department heads, You know, folks that are trying to move up within an organization and even for folks that have their own endeavors, you know, clients like to be able to see the skillset and, proficiency of your talent, of your personnel, and being able to provide these types of metrics really, really set the right type of parameters for what you want to show as one, being proactive and kind of owning your area.

Rahmira Rufus: Some of the other things do self-assessments and skills inventory, slightly like the one that I just showed you in the previous two slides. Try to keep a skills inventory. If these things ever come up, you can tell people, “Hey, I’m at a level three or six at this particular coding element” (or this particular project management software, whatever it is). Gauge where you are and know where you’re headed, right? Or where you’re trying to go. Put your skills to the test battle, test your elements, okay. And things of that nature. I’m sorry, Angie, are you gonna say something?

Angie Chang: I was gonna say thank you so much for this talk. We’re at time, but we will say that you are on LinkedIn and people can connect with you there. Thank you so much, Rahmira.

Rahmira Rufus: Thank you. I hope everyone enjoy my talk and enjoy the career fair. Best of luck and be the best You.

Angie Chang: Thank you.

Rahmira Rufus: Thank you. Bye-bye.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!