Mode Girl Geek Dinner & Lightning Talks: “Limitless” (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Meeting people is fun and easy at Mode Girl Geek Dinner in San Francisco.

Meeting fellow girl geeks is fun and easy at Mode Girl Geek Dinner in San Francisco’s Design District.  Erica Kawamoto Hsu / Girl Geek X

Transcript of Mode Girl Geek Dinner – Lightning Talks:

Heather Rivers: Oh my gosh, there are so many of you here. This is very exciting. Welcome to our very first Girl Geek Dinner, I am Heather. I am Mode’s CTO. Let’s see if we can get the technology to work. Yup, there’s me. Yeah. When I started at Mode, I didn’t have gray hair. This is dyed so long ago. So yeah, really, really excited to be hosting. You may have noticed that the theme for tonight is limitless. That can mean a lot of different things in different contexts.

Heather Rivers: Let’s just do a quick poll, why do you think we chose limitless? This is room of self-selected geeks. I am also a geek. Who here thought we meant the SQL LIMIT keyword. Anyone? Okay, not too many. Yeah, we got some Mode employees, definitely. Okay, raise your hand if you thought we meant the Bradley Cooper media franchise? Yup, okay. Yes, you’re all correct. We meant both of those. We also meant it in a third way.

Heather Rivers: So in 2008, at the Democratic National Convention, Michelle Obama famously said, “The only limit to the height of your achievement is the reach of your dreams and your willingness to work for them.” You’re about to see seven incredible women who fully embody this quote every day. I work with them, so I can say that.

Heather Rivers: Nobody joins a startup because it’s easy. Some startup people in the audience? Yeah. Any of you join startups because it was easy? No? Cool. Yeah, me either. Or obvious, or because success is guaranteed? No, not seeing a lot of yeses there. So, you join a startup because your dreams are high and because you’re willing to work for them. That’s what all of… Oh, it’s lo-res, sorry. Enhance. Enhance. No, enhance. Okay. Technology.

Heather Rivers: That’s what all of these women have done, along with the rest of the team day by day. They took a chance on the startup, they dreamed big, they worked hard, and as a result, they’ve set both themselves and this company on an incredible growth trajectory.

Heather Rivers: So in the six years that I’ve been in Mode, again, the hair. I’ve seen it go from a pre-seed proof of concept, in a super crowded market, by the way, to a simple but promising little app with a few customers, to a real product with traction and revenue, to a leader among data science platforms.

Heather Rivers: And it’s been really exciting to watch us win power users among data scientists and analysts, but we’re not done. There’s still so much more we can do. We don’t have to limit ourselves to just serving data science teams.

Heather Rivers: And that’s why just a couple weeks ago, we launched the latest step change in Mode’s trajectory. We call it Helix. So Helix is an instant responsive data engine that lets not just data scientists, but anyone, run analysis on huge data stats, up to 10 gigabytes at a time. All in the browser, and all without writing a single line of code.

Heather Rivers: Helix lets you explore your data without limits, SQL or otherwise. And I can’t be 100% sure, but I’m pretty sure that’s what Michelle Obama was talking about in her talk. Don’t try to look that up, that’s not verified.

Heather Rivers: So, in one way or another, everyone you’re about to hear from played a huge part in building, launching, and supporting Helix. So let’s give them a huge round of applause.

Kaitlin Hart speaking

Senior Enterprise Account Executive Kaitlin Hart gives a talk on “Sales is Life, The Rest is Just Details” at Mode Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Kaitlin Hart: I’m super excited to be here, this is my first Girl Geek. I’d like to kick this off with a really quick question. Who here is in sales, can you raise your hands? Okay. All right. You saw the topic title there, I see that.

Kaitlin Hart: I want to start by exploring why I believe everyone in this room should raise their hands, but first I want to start with a little confession. I’ve been field testing this talk for years, usually in ride shares, as weird as that might sound, but it happened just yesterday. So it’s still very relevant.

Kaitlin Hart: And it happens when people ask, “What do you do?” And I tell them I’m in sales. What happens next is this crash between perception and reality that I get to explore for whatever the duration of the ride might be. Because culturally, we perceive sales to look like this. Or maybe this.

Kaitlin Hart: And we associate salespeople with the traits of being aggressive, competitive, maybe selfish and untrustworthy. And apparently, as men with slicked back hair.

Kaitlin Hart: So while it’s nice to know I’m not any of those things, it’s kind of sad for me to hear because none of these things reflect what I love about sales. In reality, sales to me is much more like this. Two strangers trying to devise a plan. A parent listening to their child’s problems. Or two robots trying to form a relationship.

Kaitlin Hart: I know exactly what you’re thinking right now. These looks like everyday interactions, except the robot part, bummer. And you’re absolutely right. There’s a ton of research on how sales and life are intertwined. Daniel Pink is one of those folks, you don’t have to take my word for it. He wrote a book on it, and he said, “If you spend time persuading, influencing, or convincing others, you’re in sales.”

Kaitlin Hart: So, regardless of what business unit you’re in, you might be a PM, you might be in Dev, you might be in marketing, doesn’t matter. Because about half of your job is still spent on sales related activities.

Kaitlin Hart: So, are you reconsidering yet keeping your hand up? The point here is that sales is just life. You don’t need a special degree. We don’t need to learn any special language. And forget about it being your job title, it doesn’t even need to be in your job description. That’s how ingrained it is into your everyday life.

Kaitlin Hart: Basically, you’re all in sales, and now hopefully you know it, congrats. Comp checks are going to be at the end, [inaudible]. But we’re not going to end there. Because the details are also really important. And what really separates us is how we spend our time and focus. I spend my time focusing on developing interactions and trying to make them more effective. You probably spend your time on something else. And maybe, until one slide ago, you didn’t even know you were in sales. So that’s okay, I’ll give you a pass.

Kaitlin Hart: In the meantime, I’d like to help you get up to speed by sharing some specific skills, aka details, that we know lead to success and growth over time. And I’m not just saying this, we have data to back it up. It’s called revenue. So let’s just dive in.

Kaitlin Hart: The first one is being curious, and this one’s super close to my heart because I was born curious. Over time when I started my career, I realized this was much more of a skill than it was a trait. Because when you approach conversations with a genuine curiosity, people feel that. And when you learn, or when you ask questions that are based on understanding them, and then you listen to their responses instead of thinking about your responses, there’s this feeling of trust that’s built in your conversations.

Kaitlin Hart: And then to take it a step further, you’re going to replace judgment with curiosity wherever possible, and you’re going to avoid assumptions by, again, being curious instead of diving into your assumptions. And this is both for your career and your personal lives. Knowing nothing about someone, this is how you build a relationship rooted in respect right out of the gate. If I don’t know you, but I ask you questions that are thoughtful, and I ask and I listen to understand as opposed to respond, that’s the start to a very fruitful relationship. And then you practice this over time and you see as it grows in other areas of your life.

Kaitlin Hart: Another thing people in sales love, plans. We have account plans, territory plans, comp plans. Name it, we probably have a plan for it. But what we know is that there’s no such thing as a perfect plan. So instead, I like to take the approach of being prepared. Because when you think of being prepared, you can think of it as an outline as opposed to a filled out plan to perfection.

Kaitlin Hart: So, as you outline what it is that you want to achieve and you think about your desired outcomes, think about the how. And then you adjust by collaborating with others, being flexible to changes as they might come, and over time you learn. It’s definitely okay to fail here. That’s part of the learning process. And over time, you’ll naturally learn what leads to more successful plans and you’ll be able to grow from there.

Kaitlin Hart: And then third, we have storytelling. Anyone here read Sapiens? Or listened to the audiobook, that counts. Okay, cool. So basically, Yuval says that stories are the reasons humans rule the world. And he even says that society was built by stories. So look at politics, religion, societal norms. And so without stories, we’d be living in a very different world today.

Kaitlin Hart: But the reason stories are powerful is because they tap into emotion or imagination. Data and facts simply can’t do that. But you don’t have to take my word for it, I have a couple examples for you.

Kaitlin Hart: Here’s an ad that uses facts. Okay, this is a shoe that is breathable and supportive. How does it make you feel? Let’s compare. Here’s an ad that uses a story. Note that there’s no features, there’s not even a product clearly defined here. They’re 100% relying on storytelling and the feeling this imagery plus words are making you feel.

Kaitlin Hart: Maya Angelou actually said it best. It’s not about what you feel, I think I’m missing a slide here, that’s okay. It’s not about what people say that you remember, you remember what people make you feel. And when you think about telling a story, then you should think about how it is that you want to make someone feel. Because there’s a lot of power there.

Kaitlin Hart: And so in order to do that, you just apply this really simple framework. Know your audience, have a clear point, and use either emotion or imagination to deliver a why that connects with the audience. Then you practice. And then you field test, I hear ride shares are great for that. And over time, you’ll discover how to deliver your own powerful stories.

Kaitlin Hart: Ultimately, my hope is that you adopt curiosity, preparedness, and storytelling, and then you develop them over time, both separately and together, to unearth your own limitless opportunities. And selfishly, maybe next time when someone asks if you’re in sales, you’ll raise your hands. Thank you.

Heather Rivers: All right, next up we have Senior Product Designer Sam Novak.

Sam Novak speaking

Senior Product Designer Sam Novak gives a talk on “R, Rice Chex, and Re-usable Frameworks” at Mode Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Sam Novak: Hi everyone. I’m excited to be here today. I’m going to be talking to you about rice Chex, R, and reusable design frameworks. Here’s a photo of me with lots of pixels, but to really set the stage for this talk, let’s go with something a bit more historically relevant. There we go. Let me paint the scene. But be warned, you might need to prepare yourselves for a bit of nostalgia.

Sam Novak: The year is 1996. Pokemon has just been introduced to the world. Independence Day is the largest grossing film. The Macarena is a song and dance beloved by all. And I am almost seven years old. My favorite food is, and I quote, “white rice with butter.” But when I’m not eating buttery carbs, you can find me playing video games on my Windows 95. But alas, this story is not about me. This story is about cereal. Chex cereal.

Sam Novak: In fact, I was merely one of about six million children to fall in love with one of the most ingenious computer game strategies in all of history. That’s right, I’m talking about Chex Quest. For those not familiar, Chex Quest was the largest single mass replication of a CD-Rom ever. 6,000,000 free video games delivered as prizes in boxes of Chex cereal. How did they do it?

Sam Novak: The team was six people. The budget, $500K. The deadline, six months. They were tasked with the objective of creating an educational video game with the ultimate goal of reinvigorating the Chex cereal brand. So they set off to invent a video game from the ground up, to teach users about Chex cereal that kids would want to play. In six months, no big deal.

Sam Novak: The original game concept involved navigating around a cornfield with a flashlight, looking for ghosts. But despite their efforts, the game was just not landing with children. Until leadership came to the team and said, “Look, you’ve got 24 hours to come up with a better idea.” Enter Doom. For context, Doom is a first person shooter game that had been released three years prior. The style of gameplay was really landing with kids, and even today it is still often cited as one of the greatest video games in history.

Sam Novak: Now you may have heard this phrase. “Good artists copy, great artists steal.” Well, the team did just that. They relicensed the Doom engine to build Chex Quest. Now, the Doom team was pumped. They actually thought this was a really creative use of the engine. And the Chex team was happy. It was Doom with a facelift. The gameplay was largely unchanged, and this decision sped up the decision making process tremendously. They were now able to focus on creative ways to make the game nonviolent by redesigning the weapons, and by having the main character, yes a piece of Chex cereal, save the world by sending aliens back to their home planet.

Sam Novak: Everything started to come together. Finally it was time to release it to the world. All six million copies sold out in 6 weeks. Chex cereal sales went up 248%. It received major awards for advertising effectiveness and promotional achievements, and despite a bit of initial heat from video game critics, it developed this huge cult following really quickly. All in all, the project was a hit.

Sam Novak: The thing I love about this story is that the team had no pride or fear around leveraging existing technology. And reusing a style of gameplay that was already resonating with children. And as a result, they ended up creating something pretty inventive and magical. By applying this huge limitation, the results became that much more limitless. “Good artists copy, great artists steal.”

Sam Novak: But what does any of this have to do with modern product design? Well, there seems to be this never-ending debate in design that if you merely copy what others are doing around you, you will never truly innovate. And yet, here lies Chex Quest, one of the most innovative promotional strategies of the 90s. So how can we reconcile these points? How can we take this success story and apply it to modern software development?

Sam Novak: After all, relicensing a video game engine isn’t exactly the same thing as copying an interface, and stealing the user experience workflow. But what if it was? What if we weren’t afraid to get up here and talk about our justifications for stealing, when it led to great design schemes?

Sam Novak: So I’m going to use one more recent example from [inaudible]. The introduction of the [inaudible] notebook interface. I’ll justify stealing from two angles. First, you need to have the right intent. And second, you need a goal of building user trust. Are you ready?

Sam Novak: The timeline for the R project was three months, which was a super aggressive deadline. And the goal was to add support for R, a statistical programming language, in addition to Python while fitting in as many design improvements to the interface and experience as we could. Make no mistake though, this was a redesign of our notebooks. A redesign that would involve a fair amount of stealing.

Sam Novak: So my first justification for stealing is having the right intent. What do I mean by that? Well, you could have argued that our goal was to simply add support for R to our existing UI, but in reality RStudio in IDE was far more popular than writing R in a notebook interface among our user base. So in the same way the Chex team looked to Doom, we stepped back and asked ourselves, why do people love RStudio so much, and how can we recreate some of that passion in Mode? So we asked. Not what features do you like, but what makes RStudio a great experience for you? We documented ideas that were resonating and time and time again, in-app documentation came up as being particularly valuable. So we built in-app documentation. It didn’t matter that our interface wasn’t the same as RStudio, or that they had built documentation first. Adding documentation was just a clear user improvement. Now, the intent here was not to check a box. It was to help both Python and R writers learn about having to leave the context of our notebooks.

Sam Novak: My second justification for stealing is building user trust. Predictability and dependability are two of the largest foundations of building trust with your user base. Now, our old UI resembled a notebook, yes. But it didn’t look or work much like Jupiter Notebooks, the most widely adopted notebook interface. And as a result, the switching costs and the cognitive load, the mental energy required to learn our notebooks increased. It felt different, it looked different, and that difference didn’t necessarily lead to immediate user trust.

Sam Novak: Now, imagine trying to get a ride at the airport in a hurry, switching over from Uber to Lyft and having to learn an entirely new paradigm. But you don’t need to do that, it’s extremely easy to jump between the two. The point I want to make here is that there are workflows and patterns out there that are understood, that are resonating with users. You should have really strong reasoning to completely reinvent something new. Significant change will almost always increase the cognitive overhead required for users to adapt a product.

Sam Novak: What I’m not saying here is that there are never good reasons for introducing newer, better ways of doing things because of course there are. What I’m talking about instead is avoiding an NIH, an acronym that stands for “not invented here” syndrome. Don’t be afraid to reference design patterns that are working well just because you yourself didn’t design them. So, we re-skinned our interface to make it more trustworthy. Better accessibility, better usability, and frankly a familiarity you should come to expect after using other notebook products.

Sam Novak: So in closing, I would challenge you to keep these two justifications in mind when you’re looking to steal. First, don’t just steal for the sake of stealing. Your aim is not to win a feature [inaudible] contest or skip the design process altogether. The goal is to recognize great ideas and innovate on them. Your intent should be to learn and improve. And second, know when to steal. Borrow when it helps to build user trust. By creating something that feels familiar, dependable, and predictable, you reduce both cognitive load and switching cost to your platform.

Sam Novak: And finally for the sake of innovation, I’d like to make one slight improvement to Pablo’s phrase. Good artists copy, great artists steal, but the best artists eat Chex cereal. Thanks, y’all.

Heather Rivers: I will immediately apply the lessons I just learned and steal the mic to introduce our People Operations Partner, Josee Smith.

Josee Smith speaking

People Operations Partner Josee Smith gives a talk on “How To Ruin Your Team’s Effectiveness in 5 Easy Steps: A Guide To Eliminating Psychological Safety” at Mode Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Josee Smith: Hi everyone. My name is Josee. As Heather said, I’m the People Operations Partner. Today I’m going to talk a little bit about psychological safety. So here at Mode, I spend my days empowering our managers and building out people programs. In that work with my managers, I bet you all can guess the number one question that I get from our managers here at Mode. How can I make my team less effective?

Josee Smith: Okay, quick story. So, before coming to Mode, I worked as a paralegal at a law firm. And as part of this role as a paralegal, we would have performance reviews every six months with the HR manager. So she would go around to all the different attorneys and ask for feedback on our work, then we’d go over it during the performance review.

Josee Smith: So in the session, she proceeded to tell me about a mistake I had made about four months previously on a project I wasn’t working on anymore. I had some follow-up questions for her, such as was I still making the mistake, did other attorneys think I was making this mistake, or more specific details about the mistake. And she gave me nothing. She had no additional information. And I was really frustrated because here was this person sitting there telling me about this problem, this mistake I had made, but not giving me any information to adjust it or feel like I was being set up for success. I lost a lot of motivation in my work because I felt like they weren’t trying to help me be better at my job.

Josee Smith: So this brings us back to this question. But clearly, this is not what we’re going to talk about today, because no one wants to be less effective or less successful. But I can guarantee that there are companies out there doing things to make their teams less effective.

Josee Smith: So just some ideas of what this can look like. Asking employees for feedback, and then doing nothing with that feedback. Who here has experienced that before? Okay. Making big changes, and then not informing employees affected by those changes. Who’s experienced that? A few more people. My personal favorite. Inconsistent, vague feedback. Anyone, anyone? I think we should all, should all put our hands up. Because I think this is something, it’s a serious problem. A lot of us have gone through some of these things. I’ve experienced a lot of these things, including at the law firm, and I’m no longer at those companies because these actions not only make teams less effective and less successful, but they’ve been shown to drive away diverse talent.

Josee Smith: We’re a values driven team here at Mode, and underlying a lot of those values, there’s this idea of psychological safety. So, for those who haven’t taken a psych 101 course or if you don’t work in HR and think about this all the time. Psychological safety is created when team members feel comfortable taking risks and being vulnerable with each other. Here at Mode, we also see it being created when team members feel comfortable bringing their whole selves to work. Of course, in a way that is respectful of their teammates.

Josee Smith: A climate of psychological safety makes it easier for people to speak up and share their different thoughts and perspectives. And not feeling comfortable sharing your thoughts, or not feeling safe in that environment to speak up, can be a powerful barrier to collaboration and good decision-making. Psychological safety is particularly important in regards to underrepresented groups as a lack of the safety can lead to the kind of undermining behaviors that can drive these groups out of tech, such as feeling excluded from meetings or social events, feeling talked over, or feeling like your thoughts and perspectives aren’t being heard.

Josee Smith: A lot of research has been done on this topic, including a 2015 report from Google summarizing their findings from a two year study of their highest performing teams. And so I’d like to go over some of those traits. At a high level, successful, psychologically safe teams foster curiosity. So just encouraging teammates to study topics outside the scope of their role.

Josee Smith: Taking and encouraging risks. Skydiving, that is me up there. It doesn’t always have to look like skydiving, of course. It can be starting a new project that might fail for the sake of learning from it.

Josee Smith: Promoting respect throughout your company and your team and being thoughtful about how teammates talk about each other. And it also looks like encouraging candid conversations, such as managers asking employees for feedback and then actually doing something with it.

Josee Smith: So, as I mentioned, we’re a values driven team here and I see psychological safety being created through some of those values. I’d like to focus in on one specific value that has been instrumental to my success here at Mode. Honest words, kindly delivered. So I’ve been at Mode for about two and a half years. In that time, I’ve had the same manager, her name is Bailey. Maybe you’ve talked to her tonight. And one of the many great… Obviously one of the many things I can count on from her is consistent, constructive feedback. I know that as soon as I make a mistake, but also as soon as I’m doing really well, she’ll tell me about it because it’s important for her, it’s important to her to make sure I understand how my performance is doing. And that makes me really happy. My performance is not a secret to be talked about every six months.

Josee Smith: Okay, so, you might be sitting there and thinking, “Well great, Josee, that’s excellent for Mode. So happy for you that you found this place, but how do I practice it? How can I go about creating a more psychologically safe team?” Don’t worry, I have some tips. Here at Mode, we make it a habit of appreciating when someone is vulnerable. It can be hard to express yourself and take risks, especially if you don’t know how it’ll be received. So, when someone speaks up in a meeting when they’re normally silent, or if someone says they’re nervous about a project or presentation and then they go in and then absolutely crush it, give them some kudos. Let them know that you appreciate their efforts and you’re proud of them for stepping outside of their comfort zone.

Josee Smith: So I learned this next tip from Heather, actually. She’s somewhere. Oh, there she is. Be mindful about meetings. Not everyone likes to speak up during meetings, nor should they have to. So pay attention during meetings to who is and isn’t speaking, what is and isn’t being said, and encourage your teammates to expand on their thoughts. Consider sending a follow-up message after the meeting summarizing your thoughts, and asking your teammates to chime in with their opinions. You might unearth a perspective that didn’t come through during the meeting, but could be vital to the task at hand.

Josee Smith: So in my experience, the number one way to create a psychologically safe environment is to change your mindset around failure. To some, failure is the worst possible outcome and something to be avoided at all costs. In a psychologically safe environment however, failure can instead be viewed as a stop on the road to success or as something to learn from. So, when considering how failure plays out in your own work, don’t view it as something to be avoid, just the worst thing that could possibly happen. But instead, think about how it can be something to learn from or how it can get you one step closer to the right solution to a tricky problem. Sometimes, you have to fail to get there.

Josee Smith: I encourage all of you to think about how psychological safety plays out in your current workplace. Do you think you could bring up the topic or these ideas with your manager? If you don’t feel like you could bring this up because you think your manager won’t listen, or you worry they’ll think you’re a low performer for caring about this subject, think about how, what kind of workplace you’re going to thrive in, and what role psychological safety will play for you, like I did at the law firm. And, if you feel like you’re not getting anywhere, that your manager isn’t listening, lucky for you, Mode is hiring. Thank you.

Heather Rivers: We have one last talk before a quick break, and this is from Back-end Engineering Manager Max Edmands.

Max Edmands speaking

Backend Engineering Manager Max Edmands gives a talk on “Constructing Feedback Loops for Fun and Profit” at Mode Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Max Edmands: Hi. This is me, kind of. It’s a really pixelated version of me. This is an animation of someone pouring milk for a cappuccino, but this talk is not about being a barista. This is actually a talk about feedback loops. So, I want to start by breaking this term down a little bit.

Max Edmands: Feedback generally means conversations between humans. Someone noticing something about what you’re doing and giving you an opinion about it. Totally true, but feedback is actually more than that. Here is an example of extremely valuable feedback. As the barista’s pouring, they can see and feel the results of what they’re doing. Are they keeping the espresso crema intact? Is the milk the right ratio of liquid to foam? Is the pattern on the top of the milk glass the pattern they wanted to create? Are they filling the glass to top? Is everything the right temperature?

Max Edmands: All of this can be generalized into three attributes for great feedback. One, great feedback is high bandwidth. Lots of information coming in as quickly as possible. There’s the weight of the cup and the milk jug. There’s the temperature of the liquid in the cup. There’s the pattern that the milk is making on the surface. There’s the sound that it makes when it’s pouring. There’s so much there.

Max Edmands: Two, great feedback is relevant. There’s very little distraction here. Everything is signal and there’s no noise. You’re seeing it and you’re holding it, and all of the senses you’re getting are relevant.

Max Edmands: Three, great feedback is timely. It’s actually all in real time. The barista can change the angle of the cup and immediately they see a change in the surface area of the crema and the resulting change in the milk pattern they’re creating. So that’s feedback.

Max Edmands: Now let’s talk about feedback loops. A feedback loop is when you can take the feedback you got and try to use it again, or sorry, use it to try again. But this time, a little bit more effectively. And then use the results to get more information and then do it over and over again.

Max Edmands: So, there are a lot of great examples of feedback loops in video games. This animation is from a game called Celeste. Definitely recommend this game, by the way, it’s super great. Every feedback loop follows five steps.

Max Edmands: So step one, identify a goal. In this case, the goal is get the strawberry and bring it to the top left hand, right hand corner of the screen. Two, take concrete actions toward that goal. So jump on the block and ride it to the other end, and then try and jump off of it onto the platform and, oh no, falling into the spikes. So three, step three, evaluate your feedback. Ask yourself questions like, what did I learn just there? In this case, really clear information, if you jump in that way, then you’re probably going to fall onto the spikes and that’s bad.

Max Edmands: So then four, adjust your approach and try again. Maybe this time let’s try a dash jump when we’re in the air so we jump a little bit higher, so we can get onto that platform. And then see what we can do when we’re up there. And it works. Cool. Do it over and over again until we reach the goal. But now we’re on the platform, we have to figure out what to do next. So now, probably, we’re going to start a new feedback loop with a slightly different challenge. Great. So that’s games.

Max Edmands: But what about stuff you’re probably doing every day? Here’s an at-work example for those of you who write code. Test-driven development. So, here we have two sides of the screen. One of the sides has test results, and the other has the code. We’re adjusting the code in order to make the test suspect something that isn’t true yet. Then, we’re adjusting the code to make the test pass again, and then repeat. We know that something’s, I guess, needing to be fixed when stuff is showing up in red, and we know that stuff needs to be made to fail once stuff is showing up in green. It’s a very clear set of what do I do next.

Max Edmands: There’s another, smaller loop going on there too. As we’re editing the code, the editor is underlining certain things in red to let us know that the syntax isn’t quite right. The moment that we finish typing a line, or the moment we fix the syntax error, the red goes away to let us know it’s fixed. And then repeat.

Max Edmands: Working together with other humans is another great way to create an immediate feedback loop. I think this is a super cool photo. Two early programmers, collaborating on one of the world’s first computers. Early pair programming. I’m not 100% sure what they’re doing here. In my imagination, Esther is holding a specification that says what patch cords need to be connected to which ports. She’s reading the list out to Gloria, giving her time to connect or verify each one, and probably doing a visual check too just to make sure. Esther’s also got a bundle of extra cords ready for when they move onto the next one.

Max Edmands: So together, they’re able to keep track of where they are and move from one step to the next. They’re much more likely to notice and correct mistakes early. They’re somewhat less likely to get distracted, since they’re both concentrating on the same thing at the same time. And they’re way more likely to come up with new approaches or make adjustments to their process as they go.

Max Edmands: Which brings me back to conversational feedback. Retrospectives, one on ones, coffee walks. Words are an incredible way to fit lots of information into a really small space. Setting up regular places to have more of those conversations between either teams or between individuals, you and your manager, you and a peer, gives you way more opportunities to get and give feedback.

Max Edmands: So, how do you go from no loop to a feedback loop? Well first, we have to define the goal. Let’s say I want to draw an owl. So, now we need to figure out how we’re going to do it. I already have a process for drawing an owl. It’s something along the lines of flail along the page with a pen for a while and use a lot of white out. Eventually we got somewhere interesting. If I put it on a timeline, it might look something like this.

Max Edmands: So, next up is we identify specific decision points that’ll get us there. In this case, every time that I’ve scribbled on the page a little bit, I take a step back to figure out where to start adding the next round of details. But which details specifically should I add? This is the perfect place to start getting feedback. So, what feedback would be good here? Feedback could be comparing it against another implementation of the goal and figuring out what tweaks to make. It could also be user testing. Show your picture to another human, ask them what they think. It could also be, try to sell it and see if people will buy it. Would you buy this owl?

Max Edmands: Then iterate. Keep thinking of ways to increase the number of decision points and increase the quality of the feedback you’re getting at each point. Warning though, make sure that the additional process you’re adding is worth the cost you’re paying for it. Too much process is busy work. Nobody likes busy work. Too little process is confusion, doing the wrong thing. There’s a really fine line between the two, and staying in that balance itself is actually pretty tricky. Which is why I recommend, build feedback loops out of the quality of your feedback loops.

Max Edmands: It sounds like a joke, but I’m being completely sincere. The best way to figure out if you’re balancing cost versus benefit of process is to think about the process in exactly the same way that you’re thinking about the thing that you’re doing. Be continuously learning if there’s too much or too little, and be continuously adjusting as you go.

Christin Price speaking

Senior Manager, Business Strategy & Operations Christin Price gives a talk on “Ops, Table for 1” at Mode Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Christin Price: Hey everybody, thanks for coming out tonight. I am Christin Price, and I work in the finance and operations department at Mode. Tomorrow actually happens to be my two year anniversary. So, I joined Mode shortly after we raised our series B and I was our first in-house finance hire. And that sounded extremely cool to me.

Christin Price: So at the time, Mode was experiencing some of the typical growing pains you might see at a series B startup. For example, we’d grown out of our office space. At the same time, I was going through some of my own growing pains. I was getting whiplash from how quickly my job title kept changing. I went from leading an annual planning cycle to doing a deep dive audit on a revenue number to prepare for a series C, and I even inherited a sales ops function.

Christin Price: As these demands kept mounting, I felt like I always needed more time or more people to get anything done. Everything felt like a fire, and I didn’t feel like I was getting to do my best work. As this persisted across multiple months, I began to wonder if I’d made a mistake. My career trajectory felt like it was getting buried under the number of tasks it took just to keep the lights on.

Christin Price: Historically, I may have taken this as a sign that Mode wasn’t invested in me. And a mentor challenged me on this line of thinking. She asked me if I knew what the difference was between a fixed mindset and a growth mindset. A fixed mindset was believing that the situation was permanent, and that I had no power to influence or mold it into something different than what it was. Whereas, a growth mindset was to develop a true love of learning and believe that the best learning opportunities are presented through challenges.

Christin Price: Here are some examples of fixed versus growth mindset. Wow, that feedback really hurts my feelings. I’ve been working on this for months and clearly I’m not valued here. Versus, that’s an interesting perspective this person brings. I wonder if I incorporate that feedback into my work how it will change my work product. Or, I’ve never gotten along with this person and we just shouldn’t work together. Versus, these are this person’s strengths and these are my strengths. I’m really interested on iterating on them together to figure out how we can best work together.

Christin Price: So tonight I’d like to share a framework with you that I use to develop a growth mindset while also ensuring my career trajectory doesn’t get buried beneath the day to day. First off, do I have an executive sponsor? A mentor is someone we rely on and learn from their experiences to shape our own viewpoint. A sponsor is someone who will fight for us behind closed doors. I encourage you to ask your direct manager to be this for you. Ask them what would it take for you to have zero hesitation fighting for me?

Christin Price: The second question I ask is, am I soliciting continuous feedback? Y’all, feedback is exceptionally hard. Sometimes I feel like I’m walking to the edge of a cliff and asking someone else to push me off. But, with time and practice, I’ve gotten quite comfortable being uncomfortable. The best way to solicit feedback is to make a verbal contract with everyone you work with. Say, “I’d like to solicit ongoing feedback. Are you able to do this?” And then as you work together on projects, check in frequently, and I’m talking a couple times a week, and say, “Hey. What do you think is going well and what could I be doing better?”

Christin Price: Am I advocating for myself? As a society and especially as women, I feel like we’re pre-conditioned to believe that hard work in and of itself pays off. And I haven’t found this to be particularly true. Now that I’m comfortable being uncomfortable, I practice stepping outside my lane. I ask to be in the room.

Christin Price: Last month, there was a strategy meeting about how we hit our revenue number for the remainder of the year. It was 8 pm on a Tuesday night and I was asked to put together a model for the meeting the next day. I did so and I got my boss up to speed on it, and then I thought, “I have a valuable contribution here and I’m an expert on the subject.” I asked to be in the room. Not only did I join the meeting, I ended up leading it and one of our co-founders chased me out of the room with follow-up questions. It ended up being one of my most productive meetings in my two years at Mode.

Christin Price: Asking for public recognition. This past spring, I did a reboot on our commissions policy for our customer success function. And it took a lot of hard work, and the head of that team thanked me, privately, for the work I’d done. I asked him if he’d stand up at our Thursday all hands meeting and give me that recognition publicly. Not only did he agree to this, he thanked me for asking him. These small asks will increase your exposure to others within the organization, and also increase your level of influence.

Christin Price: Building multi-threaded relationships. This is actually a sales strategy. Imagine you’re working a deal, and your single point of contact leaves the company. It makes that inherently risky. Similarly, by building multi-threaded relationships with all different people at all different levels and in all different departments of your company, it ensures that there’s no single point of failure. Our CEO left on maternity leave earlier this year. If I relied exclusively on him to give me a voice within Mode, I would’ve been starting from scratch. Instead, I had many strong relationships to lean on during that time.

Christin Price: Don’t try to be everything to everyone. I had an epiphany about a year ago. I have always considered myself a direct person who establishes clear boundaries, but reflecting on my time, I’d realized I was trying to prove my worth by being a yes woman. Telling people I need more time, or that a project isn’t high priority, and then subsequently not doing all of the late work necessary to find that project a home is a really good practice. Others respect my ability to prioritize, and more importantly, I have the energy to bring the intellectual and emotional intelligence to the work that does fall within my purview.

Christin Price: Am I giving myself room and grace to make mistakes? I had a pretty serious miscommunication with a senior leader at Mode. Instead of accepting that I burned that bridge and beating myself up over it, I decided to apply a growth mindset. I apologized, I collected feedback, and I incorporated that feedback to rebuild our relationship. Today, I can gladly say we have a great working relationship. And furthermore, I don’t regret that mistake because of how much learning I got out of it.

Christin Price: So yes, this framework is a work in progress and yes, it takes serious energy to execute on it every day. And no, by no means have I mastered it. But I choose to apply a growth mindset and believe that with time and practice, I can continue to improve. It will become second nature. And I do truly believe there is something to be learned from every situation, especially the tough ones. And every day, I see the dividends of this practice.

Christin Price: Today, I am no longer on an island, as other people have joined the department. I was promoted to be a people manager, and I even have two open recs to continue growing the team. So, if you are a pace setter with a growth mindset who is hungry to learn and step out of your lane, Mode rewards that. And come find me after, because I want you on my team.

Christin Price: So today, I’m glad to say my relationship with Mode is mutually beneficial. It’s both give and take, and my career trajectory continues to crystallize. And with my growth mindset, I see limitless opportunity.

Heather Rivers: Next up, we have Senior Product Manager Nishi Patel.

Nishi Patel speaking

Senior PM Nishi Patel gives a talk on “Limitless Success: Influencing without Authority” at Mode Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Nishi Patel: All right, hi everyone. As Heather mentioned, my name is Nishi and I’m a PM here at Mode. I’m going to start off with a little story. So, it’s 2013, I had just landed my first PM gig after many interviews and I was so excited. I was bright eyed, bushy tailed, had done all of the reading I needed to do, and walked in on my first day. Within the first few weeks, I had been given the perfect project for our team. I’d been working with the design team, the engineering team, we’d done customer research, we’d built a bunch of prototypes, done testing. And we thought we had the ideal solution to bring our company’s first app to life.

Nishi Patel: So, in the following weeks we were going to have our first planning meeting. The CEO was going to be there, there was going to be a lot of stakeholders, some people that I hadn’t really even talked to yet. But I felt really confident in our solution and I was ready for it to be applauded and praised, and just feeling really good about it. What actually happened was that the CEO, amongst others, questioned every single point that I brought up. He pretty much shot down every single one of our ideas, and he couldn’t really connect the dots between how what we were doing and what we were proposing was going to get us to our revenue goals in three months, which is what was really on his mind.

Nishi Patel: I left, feeling pretty defeated. I went home that night, I remember, and was just circling all of the thoughts in my head and thinking what could I have done better, I thought I did everything I was supposed to do. But what I didn’t realize at that time was that our solution was actually pretty spot on to what we would end up building in a few months. They just didn’t resonate with the audience, and fell flat in that meeting that we had.

Nishi Patel: So what could I have done better? In one answer, instead of trying to explain a bunch of tactics around how we were going to build a solution, I could have used influence. So bringing us back to the point, why is influence important?

Nishi Patel: A lot of us here today are in tech or at startups, or maybe both. And we find ourselves working more and more cross functionally. On top of that, orgs are getting flatter, and so there’s a better chance that we may or may not have direct reports to help us ease into influence.

Nishi Patel: Daniel Pink, which apparently is popular amongst our group of speakers, I’m going to bring him up again. He said that we spend about half of our time at work trying to persuade others to part with resources. Resources in this context can mean time, someone’s ownership, someone’s decision making, or maybe even money. So, if it’s something that we all need to do, what are some ways to get there?

Nishi Patel: I’m going to talk through a few tactics that have worked for me, which is by no means exhaustive, but a few that I’ve had a great experience with. And also I’m going to talk a little bit about why sometimes we fail, and things that we can do to combat those failures.

Nishi Patel: So here’s one of the first influence tactics. Know your audience. I’m sure we’ve all heard this, but it’s something that’s really easy to glaze over when you’re really excited about something. What are the things that they care about? What are the things that get in the way of them doing their job? What are the things that keep them up at night? What are the things happening in their day to day that maybe affect them that you don’t even realize? I think most importantly out of all of this is really understanding how what they want, their incentives, and their motivations, can really align what you’re trying to bring to the table.

Nishi Patel: So going back to my story from earlier. I could have been much better at influencing and getting my message across if I was to understand better who was going to be in that room. I could have socialized the idea beforehand, and probably learned that the revenue goals were huge for our company and I could have better framed my story, to better connect the dots between why our solution was going to get us there and make our users happy.

Nishi Patel: Next, build trust and be vulnerable. This is easy to say, but pretty hard to do. I think the things that have worked best for me are just showing that I care and empathizing with the people that I’m talking to. And most importantly, being vulnerable. Definitely scary at first, but once you learn to put yourself out there, you can really show everyone that you’re talking to that you’re human. Consistently showing up is an amazing way to build trust and show that you care, because people can clearly see it.

Nishi Patel: So something I could have done in that situation is instead of just walking into that room with this really great presentation and this really great solution, or so I thought, I could’ve built a relationship with some of the people that were going to be in that room and really gotten their trust prior to entering and presenting.

Nishi Patel: And lastly, be clear about what you’re proposing. Be clear about how it impacts them, what you potentially need from them, or from my quote earlier, what resources they need to part with, and how it could positively benefit both their day to day and make their lives easier, and benefit the company. And also, stay true to you. If you’re not convinced about what you’re saying, they’re not going to either.

Nishi Patel: All right, so this is all good and great, and you might have even seen some of these, heard some of them, I know I have an inbox full of blog posts and newsletters that I could probably find even more tactics. But, sometimes we have every intention of doing all of these things and we prepare, and our message just falls flat. And we have to ask ourselves, why? So for me, the reality is we get in our own heads. I know in that situation, I was thinking, why would the CEO believe me? What if I fail? I’m new, why would that person even want to believe what I’m saying?

Nishi Patel: And a lot of this is fear of failure, and a lot of this is imposter syndrome. It’s a vicious loop. We don’t want to fail, so we don’t put ourselves out there, and we don’t put ourselves out there so we can’t even set ourselves up to succeed or even to fail. So at this point, we’re kind of just stagnant and we’re not doing anything at all. So if we get in our own way, how can we get out of it?

Nishi Patel: Here are a few things that I’ve come up with. Socializing your ideas. Pressure test your idea, and share it with others. This is a really simple way to start small, especially if you’re not this comfortable with everyone you’re going to be with in that room. And it’s a great way to get a signal of the things that are on people’s minds and how people feel about things. It’s also a great way to get advanced feedback, make sure no one is hearing it for the first time, and also to learn the opposing viewpoints that can help you in advance to shape your message when you walk into that room.

Nishi Patel: At Mode, we have a culture that’s pretty open and we have lunches. And so we all try to eat lunch together, and that’s a great way to have some of these casual conversations. Or, we also do a lot of coffee walks, and this has been a great way for me to kind of understand what’s going on around the company.

Nishi Patel: Secondly, observe and adapt to what works. So Sam talked earlier about not necessarily needing to reinvent the wheel if something works. So if there’s someone that you look up to, or someone in your company that you see that’s really good at influencing or maybe even in your life, build that into how you influence people. Analyze and pick up the things that work and put that into your message. Like Sam mentioned earlier… Sorry, not Sam.

Nishi Patel: Another story with Sam is earlier this year, I was actually doing a talk where I had to really incorporate the audience and really influence them with the case study that I was presenting. And there was a lot of things that I had observed with her when we practiced with each other that I was able to incorporate into my own talk.

Nishi Patel: And lastly, as a PM, I have to put in a shameless plug for learning and iterating. So in true product fashion, learn what works, learn what doesn’t, and iterate on this. And apply this thinking so that you can continuously improve.

Nishi Patel: So if there’s a couple big takeaways, it’s these. Find what works for you, and know that not all of these tactics are equal. There’s not one good formula and perfect formula to use, but the more you put yourself out there, the more you can try and figure out what works for you. And for those times where maybe the message doesn’t land, or you don’t influence the way you want, that’s totally okay. We’re all human, and the one thing that we have control over is that we can always and forever learn and iterate. Thanks.

Heather Rivers: All right, we have one final talk by our director of back-end engineering, Ushashi Chakraborty.

Ushashi Chakraborty speaking

Director of Backend Application Engineering Ushashi Chakraborty gives a talk on “Limitless Growth: Practicing Inclusion in Performance Reviews” at Mode Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Ushashi Chakraborty: Thank you. Hello everyone, Ushashi. I’m going to be talking about limitless growth, who doesn’t want that? Practicing inclusion in performance reviews.

Ushashi Chakraborty: So for those of you who are people managers, hopefully this talk is going to help you practice performing reviews in such a way that they are inclusive and they incentivize engineers with different kinds of stress. For those of you that are not people managers, hopefully this talk sparks an idea that helps you to ask for things from your manager when you’re sitting across them, and being deliberate in performance review.

Ushashi Chakraborty: This talk is going to have three takeaways. Let’s begin with the first one. That’s not a takeaway, that’s me. That’s the takeaway. Every engineer is different. Think about that for a second. Think about yourself, then think about your peers, various engineers that you have worked with. Seniors, juniors. Think about their strengths. Think about their ways of working. You’ll find that every engineer is different from each other even if they have largely to similar strength areas, for example, both of them are good at JavaScript. Even there, you will find nuanced differences.

Ushashi Chakraborty: Very recently, I was at the Grace Hopper conference, and there was a lot of chatter about bringing more underrepresented folks into computer science. It is great that we are at a place where we are encouraging everyone, everyone that is interested to come to this industry. Yet, don’t you think it’s absurd that we still think that people that are good at math and science, or people that are coming from traditional computer science backgrounds are the only ones that can make it well in this industry?

Ushashi Chakraborty: That is a flawed ideology. And if you take that flawed ideology, you are going to have biases. And if you start building a performance framework, you’re going to end up having a flawed performance framework.

Ushashi Chakraborty: Takeaway two. Don’t look at only one type of data. Data is great, but if you look at only one type of data, you will end up incentivizing engineers with one kind of strength. And if you incentivize engineers with one kind of strength, ultimately you will be left with an org where the engineers can solve only one kind of problem. And we don’t want that.

Ushashi Chakraborty: Most reviews that I have been part of have focused heavily on delivery. Things like code reviews, the number of code reviews that you have given. Number of comments, code quality are often given a lot of emphasis. I understand, because there’s a very easy metric to attach to these skills. And they are important skills to have. But sometimes, a different value that you add to an org, for example mentoring and interim. Or perhaps, sitting with your coworker and helping them debug a problem. Or perhaps writing a blog post for your eng blog.

Ushashi Chakraborty: We have to find ways to incentivize those skills, because all of these skills are important to excel as a software engineer. As an engineer myself, I have had reviews where those four skills are put together in one group, one bucket. And these skills are different from each other. And hence, those skills need to be talked about.

Ushashi Chakraborty: Takeaway number three. While a review conversation walks you through how your past performance has been, it is incomplete without a conversation about your future growth. How many of you here have gotten a performance review that scored you as does not meet, or meets, or exceeds? You’re familiar with that framework, right?

Ushashi Chakraborty: Now, something that has happened to me in the past is that I would get a great review that has been, a couple of times, in the past where I have gotten an exceeds. And I would be sitting there across from the manager waiting for the promotion to happen, very excited. Only there would be no promotion, there would be no talk about it at all. And I would be too uncomfortable at that point to ask for it, or ask why I didn’t get it.

Ushashi Chakraborty: When I look back at my career today, I can understand why I did not get it. Even though I was doing very well for my role at that time, I still had gaps for the next level. And hence, while the meets, not meets, exceeds framework is great, and its giving you context about how you are doing, that context is not complete unless you know how far you are from the next step. And hence, managers need to have that conversation when they’re giving you your performance reviews.

Ushashi Chakraborty: So now that we have learned about those three takeaways, let’s talk about how we do engineering performance reviews at Mode. We have adapted heavily from a framework built by Medium called Snowflake. It’s open source, you can check it out. And we rely a lot on robust conversations from managers to employees about their performance, as well as future growth. And we also take into account the inclusivity, such that engineers with different kinds of scripts are able to thrive.

Ushashi Chakraborty: The framework has four main tracks: build, execute, support, and strengthen. We will look at our favorite engineer’s performance review last quarter. Yeah, we are not embarrassed to say we have a favorite engineer. That’s our favorite engineer, Marshawn.

Ushashi Chakraborty: So let’s look at Marshawn’s performance review. So right now, Marshawn has not yet gotten a review, I’m going to review Marshawn very soon. First, let me explain the framework. So, on the right hand side you see a flake. We will start coloring that flake up as Marshawn gets some points. On the bottom, don’t worry if you can’t see, or if you can’t read what they say. The colors depict the different tracks I talked about. The building track, the executing track, the strengthening track, and the supporting track. We’re not going to get into the details of those tracks, but each track has about three to four skills.

Ushashi Chakraborty: Building is all about your code. Executing is everything that you do to get that code to production, for example in project management, communication. Supporting is the skills that you need to be supportive of your team, for example, their well-being. And strengthening is about building community inside and outside, for example, evangelism, recruit, those kind of things.

Ushashi Chakraborty: So each of these skills go from zero to five, and your manager evaluates you on those. At present, Marshawn is at zero and is an Engineer I, and total points zero. We’ll be walking you through two different scenarios of two different personas of Marshawn, and see how Marshawn plays out in these.

Ushashi Chakraborty: In the first persona, Marshawn is now having some depth of skills, or some really good skills around building and executing. You see those colors pop up, those are like getting two and three numbers in those skills. You see now, Marshawn’s title has changed to Engineer II, and Marshawn has 18 total points. So here, Marshawn is getting incentivized because of their deeper skills in building and executing, and they have shown depth in a portion of the flake.

Ushashi Chakraborty: Let’s talk about a different persona. Marshawn as a different engineer. This flake looks different. In this flake, Marshawn has a different kind of skill set. Once again, Marshawn is now in Engineer II with 18 points, but is a more holistic skillset that encompasses larger breadth of the flake. So perhaps lesser on the executing and building side, but still there, decent amount of skills. But they’re also having skills on the supporting and strengthening side of things.

Ushashi Chakraborty: Now let’s go back to those takeaways that I talked about. First, every engineer is different. So we see those personas. Those are real life engineers that we perhaps work with, having those skills. And now we are learning how to incentivize all kinds of skills while also having a way of our framework where we can provide feedback for the other kind of skills that they need to grow or hone.

Ushashi Chakraborty: The next thing, don’t look at only one type of data. Had we only focused on [inaudible] focusing on the blue and green that is the building and executing skills, and then Marshawn in persona two wouldn’t have been as successful.

Ushashi Chakraborty: Takeaway three. The framework should force the conversation about future growth. So in this little block here, the points to the next level which is 18, we see that Marshawn has 18 more points to get to the next level. So while your manager will be having a conversation with you as to how many points you are at today and how did that add up, they’ll also be having a conversation with you about how far you are from the next level and what you need to do to get there. And build this strategy with you to help you progressively get to there.

Ushashi Chakraborty: In conclusion, [inaudible] whichever side you’re sitting on during the performance review conversation, it is a challenging space to be in. I get it. And handling it with inclusivity is going to help you build an org that has all kinds of engineers that can thrive there and have professional growth that is wide, that is limitless. Thank you so much.

Heather Rivers: That was our last amazing lightning talk for the evening, but the party’s not over. Feel free to hang out here until 9:00. If you’re interested in talking to anybody about Mode, we have these green shirts, or if you’re interested in learning more about our product we have a little demo booth over there, very cool. And finally, we’re hiring in all departments, so feel free to ask any of us about our open roles. And yeah, let’s just hear one last huge round of applause for all the amazing speakers.

Mode girl geeks attending

A warm round of applause for all of the speakers at Mode Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X


Our mission-aligned Girl Geek X partners are hiring!

 

Intuit Girl Geek Dinner: “Powering Prosperity for Small Businesses” (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

 

Transcript from Intuit Girl Geek Dinner – Lightning Talks:

Angie Chang: Thank you to everyone for coming out tonight to another Girl Geek dinner at Intuit. I know we were here five years ago and had a terrific time and this time it’s going to be even better. In case you missed it, there are amazing demo stations with engineers and product managers speaking and giving talks about what they’re working on – and donuts. So, they’re out there and thank you so much to the folks at Intuit for hosting us again. This is an amazing welcome. The campus is even more beautiful than I remembered and thank you so much for bringing these amazing women together.

Gretchen DeKnikker: Hi, I’m Gretchen. In addition to doing these almost every week, which if this is your first event, raise your hand. Okay, so now you know that we do these almost every week, so you’ll be on our mailing list, come check out more of them. We also have a podcast, we’ve done like 20 episodes now and we would love some feedback because we’re thinking about what we want to do for the next season.

Gretchen DeKnikker: So, mentorship and career transitions and switching job function and the definition of intersectionality, like all sorts of stuff. So definitely go check that out and rate it and please give us your honest feedback because that’s the only way we’re going to get better. Then we also have … We’re going to do our virtual conference, our all day online conference. I don’t know how many of you guys have come the last two years but it’ll be just before International Women’s Day on March 6th.

Gretchen DeKnikker: So keep an eye out if you think you might want to have your company participate or if you might want to be a speaker, then start thinking about those topic ideas and then we’ll let you know when we’re sort of ready for all of that. If this looks like a good time, you could do it at your company too. So, just email us or grab us. Our information’s everywhere on the web and we’d love to talk to you about it. So, thank you.

Tracy Stone: Thank you. So welcome to Intuit. We are so thrilled to have all of you here tonight and thank you to Angie and the Girl Geek team for partnering with us on this wonderful event. For those of you who we haven’t had a chance to meet, my name is Tracy Stone and I lead the Tech Women Intuit initiative here at Intuit, and that initiative is sponsored out of our CTO’s organization as an initiative to attract and recruit, retain and advance women in technical roles.

Tracy Stone: So an event like this is so wonderful for us to partner with Girl Geek and to be able to bring the community together as we build connections and empower our women in technology. For those of you that aren’t familiar with Intuit, I hope you got a chance, as Angie said, we had some amazing demos from our technologists of some of the technology that we’re developing as we are in our mission.

Tracy Stone: Our mission at Intuit is to power prosperity around the world. We are a global financial platform company, makers of QuickBooks, TurboTax, Mint, and so I hope tonight you’ll get a chance to learn more about Intuit, about our products and the technologies. In addition, we featured some of our small business customers today, so I hope you got a chance to interact with them, get some of the swag to take home, some cool stuff.

Tracy Stone: So I hope you got a bag and got to take home some of those, some swag and the treats from our small business customers. So tonight we have amazing program in store for you. We’re going to start with a fireside chat with our CTO, and then we have some of our technologists and leaders across the company going to share some of their lightning talks with all of you. In the middle of all that, we’ll offer raffle prizes throughout the evening.

Tracy Stone: So, we’ll go ahead and get started. I want to introduce our first talk, which is a fireside chat with our CTO, Marianna Tessel. Olga Braylovskiy will be leading the chat with Mariana. Olga leads HR for our technology teams and partners closely with Mariana and all of her organizations on talent related items. Mariana is our CTO and as our CTO she oversees technology strategy and leads our product engineering, data science, information technology and information security teams worldwide.

Tracy Stone: She joined Intuit in 2017 to lead product development in our small business and self-employed group. Before Intuit, she was an executive vice president of strategic development at Docker and she’s held engineering leadership roles at VMware, Ariba, And General Magic. We’re so thrilled to have both Olga and Marianna with us tonight.

Olga Braylovskiy: All right, welcome everyone. What a turn out, incredible. Lots of power in this room. So, let’s start, Mariana, with you sharing a little bit of your career journey and how did you get you this amazing role of being CTO at Intuit?

 

Marianna Tessel speaking

Intuit girl geeks: CTO Marianna Tessel shares her career journey with VP Olga Braylovskiy at Intuit Girl Geek Dinner in Mountain View, California.  Erica Kawamoto Hsu / Girl Geek X

 

Marianna Tessel: Wow, well, thank you. First of all, thank you everybody for coming in today. So wonderful to see so many geek women, I love it. So, a little bit about my journey. I’m an engineer in my background, so I studied engineering and then I worked as an engineer for quite a long time. I actually started, I’m from Israel. Anybody from Israel here? Whoo, hi, shalom.

Marianna Tessel: So, I started in the Israeli military as an engineer. So I worked there for some time and then I came here to the US after that and I worked in a company that was trying to do devices like iPhones and we weren’t successful, but there was a documentary on the company, it’s called General Magic. You should look it up. It basically says that almost any device today, you can trace back to the roots of that company because the people that worked on iPhone and on Android both came from General Magic.

Marianna Tessel: So that was actually a real pleasure to work with an amazing set of people and doing something so like envelope pushy. Is that a thing? Yeah, pushing the envelope and then just thought I’ll use it in a different way, why not? Then actually I worked in other companies. I worked at Ariba, et cetera, but actually at General Magic I became a manager. I didn’t expect that, they had a need for a manager and they came to me and I said, what me? And then I said, actually, I really like it and I started my leadership career then, and from there on I was a leader in multiple companies.

Marianna Tessel: Then, just before actually coming to Intuit, I spent a long time in infrastructure. So that means that VMware and Docker. VMware by the way, I learned way too much, more than I ever wanted to know about storage, network compute and whatnot, really down in the guts of systems. Then when I came to Docker, that was amazing because that was like a start-up and we were changing the world and that was a really great experience.

Marianna Tessel: But I decided to join Intuit and to your question, Olga, I’m trying to answer your question. To your question, how I became a CTO, I think that I spent a long time as actually an engineer and then I became a leader and kind of grew with different roles, and because I had really different roles, actually I think I had like different perspectives and sometimes it’s luck, sometimes I grabbed the luck when I had it and here I am and I’m really happy about that.

Olga Braylovskiy: Luck is largely about preparation and clearly you had a good one. You should also mention how passionate you are about people and technical talent, which leads us to another question. Intuit is a really special company, a little plug-in for Intuit and we have amazing culture and we have very unique engineering culture that I know you’re super passionate about. So what’s unique about our engineering culture?

Marianna Tessel: Yeah, I think Intuit is really known for its culture and I knew it before coming to Intuit, but I think what’s really unique about Intuit and for engineers here is that you actually get to work on things that are really meaningful for lives of people, and that is a great feeling. You heard our mission is to power prosperity around the world and you saw some of our customers here, and when you work on something that you feel fundamentally touches the lives and people and really help them, that is really, really powerful.

Marianna Tessel: At Intuit, we are very good about looking at it this way and not just building awesome technology, which I’m really passionate about, but also thinking about how it helps our customers. So I think that intersection of really working on great technology and being able to, across the stack, really exercise your craft as an engineer, then working on a mission that is meaningful and then as a company having this culture where it’s really welcoming and nice, and just kind of not that very cut throat, et cetera.

Marianna Tessel: So, it’s kind of unique in its culture of how it’s a very welcoming of people and I think that’s why we actually have also actually high percentage of women relatively because we’re a very welcoming culture where you can feel like you can come in and be yourself, a lot more than I’ve seen in other companies.

Olga Braylovskiy: Awesome, so we use the word awesome a lot and that applies to our culture.

Marianna Tessel: That’s awesome.

Olga Braylovskiy: You touched on amazing, kind of use of amazing technology and being part of our culture. What are some cool and interesting uses of technology, especially more modern tech really as we try to fulfill on our mission of powering prosperity around the world?

Marianna Tessel: Right, and as all you guys you know, we declared our strategy to fulfill that mission to be an AI driven expert platform and what that means … And by the way, this is one of the things about Intuit that I’ve learned. You are very clear about our mission, our strategy, our values. In the beginning, I was like, wow, that’s really heavy, but I actually learned to really, really appreciate it and the clarity that it brings.

Marianna Tessel: So, I really appreciate it now. But our strategy is to be an AI driven expert platform, and what that means is actually it’s a combination of technology and being a platform, both in terms of how we build product, as well as interacting with other entities and actually people because what we have also coming to our platform are real people experts and bless ya, accountants, et cetera. So we are allowing, not just we’re building a great platform but we’re allowing people to be very, very productive on our products.

Marianna Tessel: The AI part is the one that I’m recently very excited about because this is where we really use innovation and a lot of kind of industry buzzwords and applying them to customers to again, like in a real life changing way. We actually we’re … One of the things and nice things we’ve done, we actually defined AI for ourselves, and we said, when we talk about AI, what we mean is machine learning, knowledge engineering and natural language processing.

Marianna Tessel: Now, you’ve probably heard about machine learning and natural language processing, but just to give you a bit of a taste of knowledge engineering, it’s actually about taking rules and relationships and turning them into code. Where it becomes very interesting for us is as you know, we have a leading tax product and other products that have to do with compliance, and what it enabled us to do is really encode compliance in a super efficient way. So that’s kind of one of the things we’re super excited about.

Marianna Tessel: So here is like, on the surface, a problem that could be like sounding to an engineer, slightly boring like compliance and you go ahead and you end up with a technology that’s actually really amazing and completely revolutionizing that field. So, that’s some examples that I’m excited on.

Olga Braylovskiy: Awesome, so if you reflect back to earlier in your career, what is something that maybe a true that you held at the time that you no longer hold true? Like you kind of reevaluated, your perspective shifted.

Marianna Tessel: How early? Like yesterday or?

Olga Braylovskiy: When we’re saying earlier, think back maybe 10, 15 years and I know that our perspective shifts all the time. Something that would be useful especially to this group of geeks who aspire to be the CTO.

Marianna Tessel: Right, I actually … There’s a lot of things that I change my perspective about, but let me kind of touch on a few that came to mind when you asked. The first one is how much do I want to plan my career or not? So, early on I was like this, I would say, like a leaf in the wind that I was like, oh, whatever that takes me. I’m like that sounds interesting, that sounds interesting and I didn’t really plan my career.

Marianna Tessel: I was like, oh, whatever is the next thing, if it sounds good, I’ll just go with it. What I realized at some point is that I need to have a little bit more direction to my career and not necessarily that I have to decide that I want to be like a CTO or whatever, but just kind of think about how the combination of my experiences is actually adding up and where am I going is not just like, oh the cool people. It’s also like, hey, what does it mean? How is it all adding up as a path?

Marianna Tessel: So that will be like one thing that at some point I was like, I’ve started to think … So I’m not a huge planner of my career, but I will be thoughtful about, you know what, I don’t think I’m learning anything here or I don’t think this is like … Doesn’t sound like the traditional step, but I think I’m going to learn a lot, so I want to go and do that. Like an example would be at VMware and it would be at Docker.

Marianna Tessel: I did end up doing a lot of business development while still having an engineering role and I’ve learned a ton from it. That’s something that early on I would be like, no, I’m an engineer, don’t talk to me about anything else. Another thing that I changed my mind about is around leadership. Early in the leadership … And maybe again because that’s kind of a little of what was expected from me, at least I felt, is like I was really focused on the people leadership side.

Marianna Tessel: This is an area that I’m gravitating to anyways. So I would really think myself as a leader of people and I just focused on that as in my leadership, and what I’ve noticed is that I actually really need to continue to develop my craft and I also need to be a technical leader, not just kind of a people leader. So that’s another thing that I changed my mind about how I lead and now I really focus on making sure that I follow technology, I understand technology, I understand the craft to a really big depth, not just focusing on leading people and that’s just like its more fun, but it’s also like, I feel like I add a lot more value to companies when I do it this way. So, I can also talk about other things about learning to be more assertive and things like that, but at a high level, those are a couple of examples.

Olga Braylovskiy: Awesome, last question. I think we have time for one more question. This is super powerful event. We’re all here to learn, share ideas, network. What’s your perspective on leveraging this type of event to the fullest? And just advice on how to make it most effective as a contributing factor to developing relationships and career.

Marianna Tessel: One of the things that actually I’ve learned in, again in my path that your network is one of the most important things that are going to help you in your career. Sometimes you find it in an unexpected places, so just to give you a few ideas around it. When you think about network is like the people you know then later on they will go places, or you need something, or they need something and then you have that connection to really make a dent for them, for the companies, for you, for your career, for your company or sometimes even just getting advice when you need it or sometimes it’s finding that next job when you need it, whatever that is.

Marianna Tessel: So developing a network, if you take one thing away today to at least kind of from what I’m have to offer, developing a network is something super important and I will really focus on that. Then the … What I will say that it’s really important to develop a network, it cannot be like this give and take. You can’t say like, oh I have a bunch of people that when I need them I’m just going to call them, is you need to think about it.

Marianna Tessel: You want to give your network more than what you are taking and you want to really develop great relationship and really, really think about it. Not just like something that you, a tool but something that you’re really caring about. So again, I always focus on am I giving more to my network than I’m taking, because that’s what I think is like a best set up. Of course you don’t want to have people that just always take, take, take from you and will never … There when you need them.

Marianna Tessel: That’s not good part of your network, and then how you develop your network is through events like that. You meet people, you exchange maybe information, then you can follow up. Maybe you have like, you decide to have a coffee or et cetera, but invest time in that. When you think about your day and when you think about your week or your month or whatever, make sure you allocate time to develop your network and to again, make sure you meet with people, you continue getting advice, you continue offering advice, and remember the golden rule, you give more than you take. So anyway, that will be my advice.

Olga Braylovskiy: Awesome, thank you so much for all the insights, Marianna. One and only Marianna Tessel.

Marianna Tessel: Thank you, Olga.

Olga Braylovskiy: You’re welcome. We like hearts in Intuit. That’s true though.

Tracy Stone: You can put those there. Thank you, Marianna and Olga, we appreciate you being here and your wonderful words of advice. Okay, so as promised we’ll do our first raffle and our raffles tonight are some goodies from our small business customers. So, our first raffle is from the Basik Candle company up in South San Francisco, and so I need to draw a ticket. Everybody have their tickets ready? You didn’t get a ticket?

Audience Member: I didn’t get to.

Tracy Stone: Okay, so ready? It’s 691156. Right here? No worries, there you go. From Basik Candle-

Audience: Is this right, 69?

Tracy Stone: Yeah, you are winner, congratulations. Thank you. Okay, so next up we’re going to hear from Rajashree. So Rajashree leads our engineering team for Intuit’s external developer platform and third party app experiences, enabling an ecosystem of thousands of applications that connect to Intuit’s QuickBooks platform. She’s passionate about building purpose-driven engineering teams with a customer first thinking and an inclusive culture. Before Intuit, Rajashree held various engineering roles at PayPal. Thank you, Rajashree.

 

Rajashree Pimpalkhare speaking

Director of Product Development Rajashree Pimpalkhare gives a talk on “Building Solutions with 3rd-Party Developers to Serve the Needs of Small Businesses” at Intuit Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

 

Rajashree Pimpalkhare: Thank you. First of all, let me say this is amazing to see all of you guys here and then secondly, how do I get to follow that? That’s really not fair, but I’ll do my best. So, I lead engineering for Intuit developer group and my focus and my team’s focus is empowering developers that want to work with our customers, deliver solutions to our customers that work with QuickBooks, which is our main product for small businesses.

Rajashree Pimpalkhare: I always like to start with customers. Actually, it’s the first thing I learned when I joined Intuit, that the customer is always in the house, and for us customers are three main type of customers, small businesses, self-employed and consumers. The key thing about all of these personas is they have their financial lives mingle between their business world and their personal world. So most small businesses will have bank accounts that they keep moving money from their personal account to their business account and so on.

Rajashree Pimpalkhare: So we always think of all of those customers in one breaststroke while we may provide different products for them. Then at the bottom you see our partners, so accountants, developers, financial institutions, mega platforms like the Amazons and Googles of the world, educational institutes and governments. These are all the partners that we work with. So, at Intuit we really believe that we want to power prosperity of our customers and we want to bring our partners along, and one big constituent in this part among the partners, are the third party developers.

Rajashree Pimpalkhare: So this is a little bit of geeking out. When you think of third party developers working with QuickBooks, they want to connect their applications to QuickBooks in one of three different ways. Before I go into the specific ways, let me put your mind to who are these developers, what are these apps and what is QuickBooks?

Rajashree Pimpalkhare: So, QuickBooks is our product that does accounting for small businesses. We offer them a payment solution. We offer them payroll solutions, we offer them capital, so we take it off of their financial lives. But these same customers think of like a food truck that you might have seen when you entered building 20. The food truck has things that they need to order. They need a POS system that they need to run credit cards through. They might have some discounts or coupons that they want to give out and those go through social networks and so on, so they use many different apps.

Rajashree Pimpalkhare: A typical small business uses anywhere from four to 15 apps to run their business and it is really critical that their data from these apps flow seamlessly into QuickBooks. Because at the end of the day, this is where they go to understand whether they’re making money, they’re losing money, or how are they going to pay for their dinner the end of the week, right. So there are three ways in which the platform enables us to or enable the third party developer to connect their app with QuickBooks.

Rajashree Pimpalkhare: The first one is data connections. So, this is what most of our developers use. So an eCommerce app, or a POS app, or any kind of other app that’s doing financial transactions for the developer can write that data into QuickBooks. So that can be reconciled into our books if it’s inventory for a product based business. Somebody is selling something on Shopify, the inventory can reconcile with QuickBooks.

Rajashree Pimpalkhare: If it’s a set of customers and a general contractor that’s doing 10 different jobs or a plumber that’s doing 10 different jobs for 10 different customers, those customer names can all come in here and they can understand where they spend money versus where they invoice their customers. So that data flowing in and out. So everything works together is the primary use case that our platform enables. The second use case is for certain types of experiences.

Rajashree Pimpalkhare: When small businesses are into QuickBooks and they’re trying to run their business, they don’t want to have to go and open up another app. So this might be something like you are in QuickBooks and you want to pay your bills. Now, we don’t offer bill pay as our own capability, but we do power it through a partner. If you are a small business in the UK and using QuickBooks, you can actually run payroll through a product that’s run, that’s developed by a different company but the experience is seamless and the platform underneath enables that. Then the third piece is we want to be where our customers are.

Rajashree Pimpalkhare: So, if you are in Google and if you are responding to some emails from your customers, you are able to invoice your customers right from there, from Gmail and that invoicing is powered by Intuit. So the three kind of ways of allowing integrations or ways for third parties to integrate with us, are data connections, so everything works together.

Rajashree Pimpalkhare: Powered by partners, so seamless experiences for our customers where they don’t know where our experience starts and ends and the partner experience starts. The third piece is powering through partners and this is really critical when we want to actually go and serve customers where they are. So what do we do with this? Why is this so important? So we are the small business platform of choice in the United States and increasingly in the global countries where we are.

Rajashree Pimpalkhare: The reason that’s true is because we have over four million customers that use QuickBooks today and for all of those customers, they have needs beyond what QuickBooks provides. So small businesses say, I need additional tools beyond QuickBooks to help me run my business and I would rather get it from Intuit because I trust Intuit and Intuit knows what really is needed for my business to run, and for my business to work, and for me to be profitable.

Rajashree Pimpalkhare: Then the developers need access to customers because if you are a small developer creating a niche app for a specific segment of small businesses, it is incredibly difficult to get them to know you and get them to pick you. There are so many different channels, but if they came in through, we have an App Store that the QuickBooks customers can view. If these developers put their apps in the App Store, they have access to all of their customers and they can put forth a value proposition and reach them.

Rajashree Pimpalkhare: So the developers come to us because they need access to customers to build a business that can grow and be profitable and we sit in the middle of it being able to really power that network effect. So it’s actually really, really gratifying because we work for developers and we work for customers and it’s great to be able to connect the two. A lot of build-outs. So I love this slide because it tells you a little bit about what kind of … So it tells you multiple things. First it tells you look at how big our ecosystem is.

Rajashree Pimpalkhare: All of these apps that show up here are connected to QuickBooks in one of the three different ways that we looked at. The second thing it tells you is look at how difficult it is for small businesses to really know what they want and what they need and the QuickBooks at the center is where we really take our role really seriously. We want to recommend the right apps to our customers. We want to make sure they’re successful with those. We want to make sure they know exactly how they interact with our products and so on. So this is just to tell you where we are today.

Rajashree Pimpalkhare: So I joined Intuit five years ago. I think Tracy said I came from PayPal and I came from a world where it was all about money, and all about revenue, and all about the money that came into the pocket of the company. I love PayPal as well, by the way, but at Intuit it’s just a little bit different, right? The purpose is just a little bit higher and the passion you can bring to the table is just that much bigger. In the last five years we have gone from about 700,000 QuickBooks customers to over four million.

Rajashree Pimpalkhare: I think the number is 4.2 million today. 40 apps that were on the App Store to more than 700 and one in five customers of QuickBooks today use at least one app, and we are continuing to work on it and continuing to grow it. So just super proud to be associated with the Intuit developer group to be working here and we are the global trusted platform of choice. So, with that…

Tracy Stone: Thank you.

Rajashree Pimpalkhare: I assume you’re not taking questions now, but I love to talk to you about it after.

Tracy Stone: We would love for you to engage with our speakers after. We’ll have some time at the end of the session. So thank you, Rajashree. Now we’d like to invite Nhung up to our virtual stage up here, I guess. Nhung, you want to advance the slide. Nhung is the director of data science for our QuickBooks ecosystem and customer success data science teams. She leads the strategy and execution of applied machine learning programs that span the range of marketing product, forecasting, and other strategic capabilities.

Tracy Stone: Her applied machine learning teams build new to the world products and services backed by artificial intelligence to serve Intuit’s customers across the whole customer life cycle. So I’m excited to hear from Nhung. So let’s give Nhung a warm welcome.

 

Nhung Ho speaking

Director of Data Science Nhung Ho gives a talk on “Using Machine Learning to Solve Small Businesses’ Problems” at Intuit Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

 

Nhung Ho: All righty, can you hear me? All right, so couple things as ground rules. I speak really quickly and I tend to pace. If I walk into these chairs, feel free to laugh at me. All right, so just a quick introduction about myself. I’m actually a former astrophysicist turned data scientist and one of the things I’m super passionate about is actually using math and code to solve real world problems. That’s why I escaped from looking at the stars, actually looking at people and turning my gaze down.

Nhung Ho: The other thing is I also love food and so if you’re traveling to any city and want restaurant recommendations, I have a running list. I made a Google maps, I pin everything and I have a list of all the dishes that I liked. So I’m happy to share that with anyone. All right, so I’m here to talk to you about how my team uses machine learning to solve small business problems because the business of running your business is actually really hard and it all starts with starting your business, right?

Nhung Ho: You would think that like Michael Scott in The Office, when he says, I declare bankruptcy, right? You can say, I declare a business and then you have a business. But actually that’s not true and this is just the first six of 25 steps that you have to take to even start a business, right? There are a lot of questions that you have to answer before you even get your first customer through the door. Once you start doing that, then you say, okay, I expect that I’m going to open a clothing store.

Nhung Ho: I’m going to be able to dress women, make them happy, make them look good and everything is going to be awesome, dandy. I’m going to go home feeling great, but what you do is you actually go home to a mountain of paperwork, right? I know this really intimately because I have 10 siblings and five of them own small businesses and every day they go home, the stacks and stacks of papers, stacks and stacks of receipts.

Nhung Ho: It is incredibly hard to manage. So, while during the day you’re satisfying your customers, you’re doing what you love, you go home at night and you’re dealing with paperwork because you actually have to run your business and make sure that you have enough cash flow to continue serving your customers. Just a simplified view of some of the things you need to do as a small business, you need to track your expenses.

Nhung Ho: You’re going to go buy supplies, you’re going to go buy inventory, and you’re going to have a stack of receipts and invoices that you have to keep track of. If you drive for a living, you have to keep track of how much you use your car because that ends up being tax deductible, right? You want to be able to track your income so now you have bank statements to take care of. If you invoice people, you have to keep track of that as well. Then, finally at the end of the year, you need to make sure you’re tax compliant, right?

Nhung Ho: So if you’re a sole proprietor, you need to know about this thing called a schedule C and you then you need an EIN. So there’s a whole host of things that you need to be able to do and know and manage as a small business owner that I think a lot of us who don’t own small businesses don’t realize. So where do I come in, right? Why do you need an astrophysicist here at Intuit for solving these problems?

Nhung Ho: It’s because a lot of these things that are really rote, and boring, and tedious, you can actually make much easier using machine learning. So I’m going to go through two examples. The first one is on receipt tracking. So I mentioned that a lot of small businesses go out and buy a bunch of things. They need to be able to track their receipts. They need to be able to take those receipts and actually transcribe every single piece of information into an accounting system. In this case, in QuickBooks, right?

Nhung Ho: So if you as a human, you have to go through and scan this and say, where did I buy this from? What is the address? When did I buy it? What are all the things that I bought? What is the individual prices? And then decompose that one receipt into individual lines in there. Think about how much time that takes and how tedious it is.

Nhung Ho: So what that ends up doing is actually causing you to say, it’s not worth it, I’m just not going to do this and then you end up leaving money on the table because you could have deducted these. So if you can imagine, what my team did was we actually married computer vision and natural language processing. You can utilize an OCR system to go through and actually pick out every single character. You can figure out exactly where the bounding boxes are for each of these fields and then begin to lift that information out.

Nhung Ho: You can then use a deep learning system to say, okay, I know that when I see Aroma Cafe, that’s a vendor, and when I see that number format, it’s actually a date, right? We can use the latest deep learning technology, scan through the system that’s already been OCR, pull out that information and then make sense of it. Then finally put that into your accounting system and you’re done.

Nhung Ho: This system is 10 times faster than what a human can do and we can go and grind through thousands of receipts for you as long as you send us the image. That’s kind of the power of what machine learning can do to help a small business succeed. So the next example I want to talk about is mileage tracking. As I mentioned earlier, how many of you here know that if you use a personal vehicle for business purposes, every single mile that you drive is deductible at 54.5 cents per mile?

Nhung Ho: Actually, quite a bit. Actually, I definitely did not know that when I started this. But if you are a real estate agent and you’re driving between showings, that could be hundreds of miles per day. That’s a lot of money that you can deduct on your taxes. We actually have a product called QuickBooks Self-Employed, that makes this really easy for you. You turn on our automated mileage tracking service, we will say, okay, we know that Nhung traveled from point A to point B on this day and this is the distance that she drives, which is super awesome, except we don’t actually know exactly what the purpose was, right?

Nhung Ho: Because again, you need to be able to say, this trip is a business trip versus a personal trip. If I drive to the grocery store, I can’t deduct that on my taxes and if the IRS finds out, it’s not going to be good times for anybody. So you have to go in and say, is this a personal trip? Is this a business trip? And if it’s a business trip, why did you actually use your car for that purpose, right? So it’s multiple pieces of work.

Nhung Ho: So to give you a real example that I have scrubbed a lot of personal information from is, meet Claire. Claire works at City Hall during the day and she actually teaches piano on the side. These are all of the trips that Claire took. You can see that some of them are for business purposes and some of them are for personal, but there’s no real pattern that jumps out here. In an average month, she takes 200 trips and you can imagine that if Claire, like I am, there’s a huge procrastinator, at the end of the month, she’s got to answer basically 800 questions.

Nhung Ho: What was this trip? When did I take this trip? Where was I going and what was the purpose of this trip? Is it for because I’m driving to teach piano or is it because I’m driving to City Hall for my job? We don’t make that distinguish. We don’t distinguish that, you do it for us, but I can build a machine learning system that can do this in less than a second for Claire and that’s exactly what my team did. We utilized frequent pattern mining and we essentially automatically learn very personalized and highly individualized rules per user.

Nhung Ho: So we can group all of the trips on the left side that she took exactly to the same destination, show that to her and say, we think that this is a personal trip and if it ends up being a personal trip or a business trip, here is a deduction that you get. For our users, they take 50 million trips per year. Now you can imagine building a system like this that is scalable, extensible, and is ready to use for any new user who comes in.

Nhung Ho: So what I hope I showed you was how machine learning can actually make the business of doing business much easier, right? And it’s not … I can tell you it wasn’t obvious to me when I started here, why we need a data scientist to work on accounting. Isn’t it solved? It’s so old, it’s so boring. But I can tell you some of the most boring, mundane things are the areas that are the most exciting because those are the areas that have not seen innovation in a really long time and that’s where you can go in and make a difference. So thank you for letting me share those with you.

Tracy Stone: Thank you, Nhung. I don’t think you tripped over the chairs either, but that might be my job then, huh? Okay, one more raffle now. So get your tickets out and this time our raffle prize is from Origaudio in the set of very cool noise canceling headphones. You like this? Yeah, all right. Let’s see if I can pull your raffle card. Okay, it is 691056. Anybody? 691056. Here you go, I believe you. Okay, wonderful. All right, so next we have Kristina Fox, and Kristina Fox is our staff iOS engineer working on the QuickBooks Self-Employed iOS app.

Tracy Stone: She has spoken internationally at over 25 meet-ups and conferences, including Grace Hopper, iOS Dev UK. Were some people at Grace Hopper a few weeks ago? iOS Dev UK, WeAreDevelopers World Congress and many others on topics ranging from Apple watch development to diversity inclusion. So thank you Kristina for being here and let’s give her a warm welcome.

 

Kristina Fox speaking

Staff iOS Engineer Kristina Fox gives a talk on “Using Mobile Capabilities to Save Time and Money for the Self-Employed” at Intuit Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

 

Kristina Fox: Thank you everyone. Okay, so we’re going to be talking about using mobile capabilities to save time and money for the self employed, and this is really a topic that’s near and dear to my heart because, well I’m an iOS engineer and it’s basically one of the most versatile platforms out there. Okay, is there anyone here that doesn’t have a phone on them right now?

Kristina Fox: Exactly, so that’s why this is one of those platforms that’s literally always on you and it can make such a huge difference in your lives than it already has, considering the fact that none of us really travel without phones anymore. Okay, so let’s get started. Just to give you a bit of context, I work on QuickBooks Self-Employed, it was actually the app that Nhung talked about earlier, and for those of you who aren’t familiar with it, it is an app that helps freelancers and contractors help manage their business finances.

Kristina Fox: So she talked a little bit about the mileage tracking. We also do transaction and expense reporting and also invoicing. So there’s lots of really cool stuff that our app does to help all of those self-proprietors out there, the sole proprietors out there. If you think about it, the types of people that we help are people like this freelance photographer out there or maybe even the Lyft or Uber driver that took you here today.

Kristina Fox: So lots of really interesting industries that we can also help out there. Okay, so the next thing that I basically want to dive into is specifically a project that I worked on and developed called Receipt Capture. This is this is one of these most probably my favorite project that I’ve ever worked on here, so it’s really fun for me to share this with you. Okay, so let’s take a step back and say, oh, we decided to go shopping and this is just to set some context about why receipts are so important to our self employed people, right?

Kristina Fox: So we decided to go shopping and we found a really amazing dress. It looked awesome on you in the dressing room and you decided to buy it and bring it home. But once you got home, you really … It ended up changing colors on you. You didn’t really know what color that dress was. Was it white and gold? Was it blue and black? You’re not really sure anymore. So, well, okay, you decided the blue and black doesn’t really work for you and so you want to return it and what exactly do you need usually to return things to a store?

Kristina Fox: Receipts, exactly. So receipts are really important to us as proofs of purchase as consumers, but for our small businesses, they’re really important to help prove that the things that we’re buying are actually business purchases and so this is really important, especially if the IRS comes to audit at some point, you need to be able to prove that the things that you purchased at the end of the day are actually for your business.

Kristina Fox: So, let’s talk about the basic Receipt Capture experience and so I direct your attention all the way to the left side. Usually you’ll hook up QuickBooks Self-Employed to your bank account and you’ll see the list of transactions coming through from your credit card or your checking account. From there, if you have a receipt for Starbucks for example, you’ll want to tap on Starbucks and then that will pull up a detailed view of your transaction.

Kristina Fox: From there you can tap, attach receipt and then you’ll be able to either take a photo or choose from your camera roll if you decided to take that photo earlier. That’s fine, pretty simple experience. You’re really just attaching a photo to a transaction, but one thing that’s kind of interesting about this is that if you’ve ever gone into your checking account or your credit card account and you just bought something, you’ll notice that there might be a pending transaction, right? It means that it hadn’t exactly cleared yet because maybe the restaurant doesn’t have your final total or your credit card is still being authorized.

Kristina Fox: So it can take generally about one to two days for these pending transactions to clear and unfortunately until that time we can’t really attach any receipts to it because it’s not a real transaction at that point. One to two days might not sound like very much time, but really anything can happen in these receipts in that one or two days, anything at all.

Kristina Fox: So what do you do? What do you do when you lose that receipt? Well, hopefully with our new enhanced Receipt Capture experience you won’t have to find out. So I’m going to do a quick demo of the new enhanced Receipt Capture experience that I ended up building for QuickBooks Self-Employed and this is going to be fun to juggle. So hopping in, I’m going to go ahead and end the show and I bring up my phone screen here. So this is the QuickBooks Self-Employed app. I’m trying to make this bigger.

Kristina Fox: You can go into the transactions tab where you can actually snap a receipt. So if you remember before we had to go and tap into an individual transaction to take a picture, well now you don’t really have to do that anymore. So you go into here, we’ll tap snap receipt and then we bring up this new camera view that we have, and so what’s cool about this is it actually does the cropping for you. So I’m shaking a little bit too much right now.

Kristina Fox: Demos are always nerve wracking. Okay, so we’ll steady. Okay, so now we’re able to get a cropped version of our receipt photo here and if you ever wanted to mess around with it, say it didn’t quite crop correctly, you could even go in, it takes you back to that original photo. So then you can manipulate those crop points or you can even rotate that image. So, there’s lots of really cool image enhancements that you can do here, and then from here all you have to do is hit at the bottom, use this photo and it disappears off. It gets uploaded to our service in the background and that’s it.

Kristina Fox: That’s all you have to do now. So this is our new enhanced Receipt Capture experience. So what exactly is happening here? Well, in the background we start off with QuickBooks Self-Employed. From there it actually gets uploaded to our document service. So this is where it actually stores that receipt image for you and then from there it goes to our data extraction service.

Kristina Fox: So, it’s something that Nhung was alluding to earlier where we can actually go and run optical character recognition on that receipt itself and then pull out the data that we really care about. So that’s stuff like, again, the vendor at the top of the receipt, we’re looking for the date, the total, and the credit card number too. So we can pull out all of that receipt data for you, and then we do automatic matching.

Kristina Fox: So now we go back into your list of transactions. Instead of actually having to go in and find that Starbucks transaction again, this algorithm actually takes that receipt and automatically looks at the data that’s coming in from that transaction and it attaches it for you. So it’s literally you take a picture and then you just forget about it and it’s already in your account, all done. Yeah, it’s literally a life changing event.

Kristina Fox: So what are some of the mobile capabilities of this? Well, obviously we’re using the camera to take a photo. We have a lot of touch … We have the touch screen capability where we can actually manipulate the photo if we need to, and we’re also running a lot of image processing algorithms in the background in order to make sure that the photo is usable. So you saw that it was telling me to hold steady.

Kristina Fox: In some cases, if there’s not enough light, it will tell me, oh, you need to add some more light to the background’s too dark, things like that. So there’s a lot of really cool things that are running in the background, just doing image processing there. It’s also a very on the go capability, like as you can see, a lot of the people we’re supporting are Lyft drivers. They’re a freelance photographers, they’re always constantly on the go, they’re gig economists and so they need to be able to have this type of capability wherever they’re going.

Kristina Fox: Of course, on one last thing, if you’re using the iOS platform, you might be familiar with Siri shortcuts, and so if I hop out again and on my phone. Okay, so let’s tell Siri, snap this receipt. It hops directly into that camera. So now the user doesn’t even have to go in and launch QuickBooks Self-Employed. You can just add a Siri shortcut with your own custom phrase and then it will go into the exact view that you need.

Kristina Fox: So we’re really taking advantage of all those platform capabilities here too, and here’s a look at what that looks like. So you can add this custom phrase to Siri and then it will do whatever you tell it to. So this is the one of the cool Siri shortcuts capabilities that we have. That’s it for me. Thank you so much.

Tracy Stone: Thank you, Kristina, and a live demo was so awesome. All right, next up, we have Cassie Divine and Cassie’s the senior vice president of the Intuit QuickBooks online platform where she leads the business units responsible for small business, self-employed, accountant, and the Intuit developer group. She is especially passionate about driving diversity and inclusion in technology with an eye on fostering a favorable work environment for women.

Tracy Stone: Cassie’s been awarded the Silicon Valley Business Journal Women of Influence award and has twice received the Intuit CEO Leadership award. A passionate small business owner herself, she also has an Etsy shop where she sells her DIY kids’ Halloween costumes, might come in handy. So let’s welcome Cassie Divine.

 

Cassie Divine speaking

SVP of QuickBooks Online Platform Cassie Divine gives a talk on “Owning Your Career: Reinventing Yourself to Create Impact at Scale” at Intuit Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

 

Cassie Divine: Thank you, Tracy. That glorious introduction just made my impostor syndrome in this room even worse. I’m so grateful to be here and crowned a Girl Geek with this amazing lineup of my Intuit colleagues. My topic, my lightning talk isn’t a technical one. It’s how to hack your own career and for all of you who know you’ve been working, there is so much conventional wisdom about what you’re supposed to do to move up, and advance, and create an enriching career.

Cassie Divine: By all of those conventional rules, I have done nothing right, and yet I find myself with a big fancy job, and title, and opportunity to make a huge impact. So tonight I want to tell you five quick stories. The stories are unique to me and they’re my crazy stories, but I think the lessons I’ve learned that I’ll share are applicable to lots of you, I hope. So I hope you hear something you already knew and needed to hear or something you could learn.

Cassie Divine: So story one, I started at Intuit 12 years ago and I joined as a senior manager and I had a plan to get promoted in two years. I had talked to the boss I came to work for. He had seemed to think that that was a reasonable expectation and I was at a point in my career where I was convinced that I was better than a lot of people and I needed to go. It was a sure thing that was going to happen. Of course in this story, it doesn’t happen that way and I was devastated.

Cassie Divine: I was devastated because I didn’t get this title that I wanted and the wisdom from my boss was just to work hard, to work even harder and to sort of ride the time and that it would happen in one to two years. I found myself being frustrated with the idea that I was waiting to get a label from someone else and in a moment I decided to create a label for myself. I’d been thinking about starting an Etsy store, a side gig, for a long time and I decided that this would be my moment and screw not being the title I’d wanted, I would be the CEO of my own thing.

Cassie Divine: And I kept working, but the work changed because so much of my creativity, and hustle, and anxiety, instead of worrying about some label, it went into something that was for me, and it was really enriching for me, and I actually probably stopped staying as late and stopped doing some of the things you do when you’re searching for a title and just started focusing on impact at work.

Cassie Divine: Because I had this creative outlet and don’t you know I got promoted six months later, and the lesson for me though was you define you, nobody else gets to do that. Titles can just be labels and we don’t like them when they’re bad, and I worry we give them too much power when we think they’re good and a lot of it is about the impact and the journey. Story two, I was coming back from my maternity leave and I got a lot of wisdom from women and men, which was, the plan was just to come back and show that I was the same person.

Cassie Divine: I could just, nothing had changed and I could work the same way and I remember thinking like, I don’t think I can even do that, everything has changed. I don’t want to be here the same hours. My priorities have shifted and I had never cashed in any of my street credibility or what I had had. I had never asked for anything and I decided that this might be the moment that I would, and I did my research, I talked to a lot of people who had organized creative arrangements.

Cassie Divine: A lot of people gave me advice that it was absolutely the worst time in my career to take a step back and there are a lot of studies that say women are going to earn less. I just decided to go for it and I asked to take 80% of my salary and have every Friday to be with my little girl. But I made a deal with my boss, which was, I will deliver the same impact, it just is going to look different. I did that arrangement for two years. It was so awesome and I got promoted after I came back full time.

Cassie Divine: I learned in it to ask for what you want. The worst thing anybody can say is no, but in asking for what you want, to be clear about what you’re going to deliver so people know what they get when they say yes. Third story, I was 15 years into my career, seven years at Intuit, and I had just been promoted to vice president and conventional wisdom is like, this is your career, this is what you’re going to do, this is your path. You build on it.

Cassie Divine: I had been unhappy for a long time and just going through the motions, and I loved the company, I loved the company’s impact, but I wasn’t finding it was meaningful to me, and I started working on my network. I agree with what Marianna said and I had always given to my network and this was a point I asked, I worked my network on what would someone take a bet on me to do that would be different from my job and I got my big break and it came with a demotion.

Cassie Divine: It came with giving back a team, which I had been led to believe that it was all about creating this big kingdom and you grow in giving … You grow and getting this big team and I went to being an individual contributor and people inside and outside the company thought it was a little crazy because it just seemed like it was too late to make a change and that it on paper looked a little insane to a lot of people. But I was so excited about it to get to have an impact on moving into a product business when I actually moved into the QuickBooks Self-Employed business.

Cassie Divine: I would move into a role in BizOps and I was excited because the person I was going to work for saw something in me that I didn’t even know I was capable of doing, but both of us were willing to take that bet. 18 months later after getting that job, I got the biggest job I’d ever gotten. I almost went for something bigger than what I had stepped down from, and it came when I wasn’t focusing on trying to get promoted. I was just trying to focus on the biggest impact. But the lesson that I learned in this is bet on yourself, but that has two parts.

Cassie Divine: It has the courage that you make the bet and it’s also about finding somebody who will make that bet with you, and my advice to all of you would be seek someone, your boss or mentor who sees what you’re capable of doing and suspends disbelief and isn’t just obsessed with what you’ve been able to do on paper already.

Cassie Divine: Fourth story, I was now in my big job leading this product business and it was really fun and I knew the rap on me is I had taken over from someone who was really successful. It was sort of like my was seen as the person who was just helping keep it up and running, and there was a job similar to mine that had been vacant for three to four months and it was bigger, and it was a lot harder, and it had a different profile and I wasn’t a candidate for it.

Cassie Divine: In fact, I was on the interview committee to go and search for the person who would be the right person and based on every discussion of what we thought we were looking for, wisdom said I had no business to do this. As we talked to the team and as we kept interviewing people, I started thinking, I think I could do this job and I actually think I could start to show that it’s not just about keeping the easy thing running, but I could show that I could work on something hard and lead a turnaround.

Cassie Divine: But I didn’t necessarily have the courage at the time to raise my hand to just take it, and so I raised my hand to help with it. I had said, what if I help in a capacity as we continue to search for this person and I will make it better. This team is in need, I have a great leadership team in Self-employed. They’ll step up and do more and I’ll play this dual role and six months later we decided that I actually was the right person for the job and I stepped into the role.

Cassie Divine: But more importantly, I showed that I was willing to take on the hard things. I was willing to do a lot more and as a result, I got my big job today, which is far bigger. What I learned in that is it’s okay to be your own hype woman and raise your hand and sometimes you’re the best person qualified to say you might be able to go do a job and be considered for it.

Cassie Divine: What I’d offer for you is, if you’re worried about it or if they’re worried about it, one of the best ways to get that shot is in a volunteer or rotational capacity because there’s nothing that is a downside to that except you do a lot more work in the meantime, but it shows you and it shows them that it’s a great opportunity. Last story isn’t a story, it’s something that has weaved through all of my stories, which is conventional wisdom is that career success is all about you and you answer this question, what do I need to do to get ahead and how can I show that I’m, I’m being successful?

Cassie Divine: What I found is career success is actually about making everybody else successful and investing in your teams, and your peers, and your boss, and your customers. One of my favorite books, I encourage all of you to read it is by Adam Grant. It’s called Give and Take and it is all about this idea that investing in other’s success is what creates the most success for us. Now, it feels counterintuitive when you think about what it feels like to give someone credit, God forbid it’s someone mansplaining something to you. It’s hard sometimes because it feels random.

Cassie Divine: Maybe you don’t know them or you haven’t … It’s not sort of about this give and take, but do it anyway. Real leadership is about creating impact and it is more easy to do. It is more fun to do with others, and as you do that, I promise it comes back to you. You become the person that everybody wants to work with and it has its ups and downs and I agree with Marianna, you have to worry about the people who are just taking and not giving back to that, but I encourage you to do it.

Cassie Divine: So the lesson I’ve learned there is what I’d call, what Adam Grant says is givers take all, and invest in others and raise others up and make that a part of your leadership brand. It will unlock things that are just amazing for you. To wrap up, start … I like starting with a plan. I like starting with a path. I like the point Marianna made about the things that are important. I would encourage you to be open to the idea that it can change dramatically and I would think a lot more about moving forward than moving up. Hack your own career.

Cassie Divine: You define you. Ask for what you want with accountability, bet on yourself and find the person who’s going to bet on you, raise your hand to take on work people didn’t know, but you did, that you’re capable of all while investing in others’ success. I’m so excited to find out what all of you will go and do and the impact that you create in your own careers. Thank you.

Tracy Stone: Okay, so with that, that ends our formal program tonight. I want to thank you all for coming. I want to thank all of our speakers. So these are some amazing women that work at Intuit doing some really cool work and leading teams around the work we’re doing to power prosperity for our 50 million customers and growing. You can just spend some time and interact with them as long as they’re able to stay.

Tracy Stone: Ask them questions; those questions that you had that you didn’t get a chance to ask. There are drinks out there. You can go out on the patio and network for a little while longer. I hope you all had a chance to connect with everybody and walked away with some few, those nuggets of wisdom that you’ll take as you go on to do amazing things, as Cassie said. So thank you all.

 

Cindy Osmon speaking

Distinguished Engineer Cindy Osmon demos “Smart Mirror” at Intuit Girl Geek Dinner. Erica Kawamoto Hsu / Girl Geek X

 

Nirmala Ranganathan speaking

Principal Product Manager Nirmala Ranganathan demos “Shield” at Intuit Girl Geek Dinner. Erica Kawamoto Hsu / Girl Geek X

 

Yi Ng speaking

Principal Product Manager Yi Ng and Senior Software Engineer Regina Garcia demo “QuickBooks for New Users” at Intuit Girl Geek Dinner. Erica Kawamoto Hsu / Girl Geek X

 

Intuit Tech Women at Intuit Girl Geek Dinner group of women in tech

Thanks to all the Tech Women @ Intuit for hosting us warmly at Intuit’s Mountain View headquarters for our Intuit Girl Geek Dinner! Erica Kawamoto Hsu / Girl Geek X


Our mission-aligned Girl Geek X partners are hiring!

Scale Your Career with Open Source: Girl Geek X Confluent Talks & Panel (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Angie Chang and Sukrutha Bhaduoria speak

Girl Geek X team: Angie Chang and Sukrutha Bhadouria welcome the sold-out crowd to Confluent Girl Geek Dinner in San Francisco, California.  Erica Kawamoto Hsu / Girl Geek X

Transcript from Confluent Girl Geek Dinner – Lightning Talks:

Angie Chang: Hi, everybody, thank you for coming out tonight on a Sunday night. This is our first Girl Geek dinner on a Sunday night after over 10 years of hosting almost weekly Girl Geek dinners. My name is Angie Chang, founder of Girl Geek X. I wanted to say thank you for coming out on a weekend. It’s really great to see everyone’s faces here at Confluent in San Francisco, to meet everyone, and also really excited to introduce Sukrutha, my co-organizer at Girl Geek X, who is six weeks into her maternity leave. So she has the littlest Girl Geek now.

Sukrutha Bhadouria: Hi, everyone. Welcome. Like Angie said, so nice to see such a huge crowd on a Sunday. I honestly can’t tell the difference anymore between a weekend and a weekday. But thanks for reminding me it’s a Sunday. But hey, I really wanted to explain, we always do this, we ask how many of you, is it your first time at a Girl Geek dinner? So do raise your hands. Wow. What’s been amazing in the last, I don’t know, a little over 12 months is that that number’s been increasing and increasing. And that’s been great because we want more and more people to join our community.

Sukrutha Bhadouria: Why we do this is we want to elevate more women in tech, in various roles in tech, and each dinner and each event is sponsored by a different company. And these companies are kind enough to host all of you in their space and they provide you with great content through their talks. We also use this content in our podcasts because Angie and I used to do these long drives to Girl Geek dinners all across the Bay. And we started to talk about what else we should do besides dinners. And now in the last 11 years, we’ve evolved beyond dinners to podcasts and virtual conferences as well. So we’ve had two virtual conferences so far, and we want to make it annual. Do check out our podcasts. And we want to know if you have any other ideas for what you’d like the content to be, please share it with us. Do share on social media tonight. Know we have a lot of great speakers tonight. So do share what you’re learning tonight with the #girlgeekxconfluent. I can’t speak full sentences anymore. But that’s all I had to say. I don’t want to take any more time. Thank you again to Confluent for making this happen. Thanks.

Dani Traphagen speaking

Senior Systems Engineer Dani Traphagen emcees Confluent Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Dani Traphagen: All right, what a beautiful crowd. There’s so many of you here and we absolutely love to see that. So welcome, everyone. We are really happy to have you here for dinner in our special San Francisco office. This is actually a satellite office to our home down in Palo Alto, and we’ll actually be moving soon next Wednesday to a new home in Mountain View. So we’re absolutely thrilled to have this stellar company of Girl Geek Dinner, dinners here at Confluent tonight, and I’ll bet you’re wondering what we do here at Confluent. So I’ll have a couple words about that. I actually luckily consult people about that in the subject of my day to day life.

Dani Traphagen: So my name is Dani Traphagen, and I am a senior systems engineer here at Confluent. What I do on my day to day is I work with account executives, specifically in the sales organization. So I technically consult large organizations anywhere, basically above $1 billion in revenue on how to leverage our technology. So that’s what I do. I really like my job. I love working with large companies on how to leverage our infrastructure and working specifically in the software realm on how to use real time software specifically.

Dani Traphagen: So this is my third company doing this kind of work. I have a background in database technology. And this is my third open source project and working in an enterprise on that. I actually ended up transitioning from bioengineering, though, specifically, into a career in tech after college, and this was many years ago. I will not tell you how many years ago. And I heavily leveraged events exactly like this to end up making that transition. So I really believe in them and the power of them, networking with people, making key mentoring relationships, and learning from role models, like some of the ones that you’ll hear from tonight, and kind of how full circle things are here, which is super bizarre. One of the men here tonight, Peter Feria, was at one of the events that I went to. He’s one of our videographers. Tim Berglund, who if you know anything about the Confluent’s ecosystem itself, and who’s kind of who in that world, you’ll see a lot of his videos online. He is one of our developer relationship folks. And he was my first boss at a company called DataStax.

Dani Traphagen: So it’s kind of crazy how full circle things go. So I really encourage you to meet, to network, to speak with people. And to just kind of learn more about all the things that you could possibly do. So now, a word about kind of what we do in this very building that you’re in right now, to just kind of bring things to a real visceral meaning. So Confluent provides enterprises exceptional expertise and tooling around the open source project Apache Kafka, and Apache Kafka as a fundamental way of moving event driven data from different sources within an organization to other interested parties within that same organization.

Dani Traphagen: So the way that I like to think about it is pretty much like the true heartbeat of your data pipeline. And it has become the central nervous system of many organizations, those specific organizations that I consult. With the Confluent stack, businesses can support streaming data use cases and optimize their insights and user experiences for many of their mission critical applications. So these are applications that are essential to their day to day operations. It has become an industry standard for the modern enterprise.

Dani Traphagen: Apache Kafka is an extremely robust technology, and it was co-created by tonight’s speaker, and pardon me, I should probably use my mic here. It was co-created by tonight’s speaker, Neha Narkhede, who is also Confluent’s co-founder and Chief Product Officer. It was inspired back during her time at LinkedIn in an effort to help manage the massive scaling efforts, along with her fellow Confluent co-founders. Neha has been an exceptional role model for so many women, including myself, and she has shown me in a sea of Bills and Elons and Steves, that something more is so possible in this world. And that has left a truly indelible mark in my path. So please join me in giving her a sincere and warm welcome.

Neha Narkhede: Thank you, Dani, for a very warm welcome and welcome Girl Geeks to Confluent’s very first Girl Geek dinner. It’s been such a long time since I first spoke at a Girl Geek event. This was about seven or eight years ago when I was an engineer at LinkedIn. Today, I’m so humbled to be hosting one and be here in front of all of you. I hope that you learn something new from this event. I hope that you make new introductions, and thank you all for taking the time to be here.

Neha Narkhede speaking

Confluent co-founder Neha Narkhede talks about starting and scaling the billion-dollar infrastructure startup at Confluent Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Neha Narkhede: So let me start off by telling you a little bit about myself. I was born and brought up in India. I learned computers at the age of eight, mostly to play video games and draw on MS Paint. So while I didn’t learn programming while learning to write, like all the whiz kid stories that you might have heard in the Valley, it did interest me enough to take up computer science. So I moved to the U.S. to get my masters in Georgia Tech. After that, I took a job in a big company, Oracle, mostly to find a safe path into an H1B visa. This was during the 2008 crash. Pretty soon, I realized the power of the open source community to accelerate my growth and learn new things. So I specifically applied to a company that had a real investment in open source communities, LinkedIn.

Neha Narkhede: I taught myself distributed systems on the job. I was lucky enough to be on a team that got a chance to create a very popular distributed system called Apache Kafka. We open sourced it, it went viral. I sourced a business opportunity around Confluent, pitched it to my teammates. Fortunately enough for me, they agreed to start this company with me. This was five years ago. Today, we’re more than 900 people worldwide and growing very quickly. Over time, I’ve worn many hats. I started off as an engineer, and then I ran engineering teams, and I transitioned to product, so quite a few changes. That was a little bit about my technical journey. For fun, I travel, I go scuba diving, and I engage in a fair bit amount of retail therapy.

Neha Narkhede: So most of my career has been about introducing this new category of software called Kafka and event streaming into the world. So to tell you a little bit about why we started this, we were facing a pretty unique challenge at LinkedIn. And the challenge was that our users could use our product and they were using it 24 hours a day, in a very real time fashion. But all the software that LinkedIn had could only get access to all that data and studied enough to produce more patterns and produce better products, maybe a couple times a week. So that was pretty slow. We wanted to take that all the way down to real time experiences. And so this meant processing billions of events a day in real time. There was nothing out there that did that. So we started Apache Kafka to solve this very problem to process lots of events in real time. And to basically give all of LinkedIn software access to all of its data at a millisecond level.

Neha Narkhede: We thought that this couldn’t have just been LinkedIn’s problems, so we open sourced it and we were right. Pretty soon, in the early days, Silicon Valley companies, all the top tech brands that you can think of, adopted Apache Kafka. After that, it entered the enterprise. And today, we know that about 60% of Fortune 100 companies depend on Kafka as a foundational technology platform. And any company that starts off as a digital one, they ingest all their data is in Kafka from day one.

Neha Narkhede: So anytime this sort of an adoption happens for infrastructure software, there’s a lot more to it than a good product. There’s usually a paradigm shift that drives such a change. And 10 years ago, that paradigm shift was that every company was not only becoming a software company, but it was literally getting turned into software. So what do I mean by that? You don’t call a cab company anymore, you go to your app, and the entire ride is managed entirely in software. You don’t go to an ATM Teller, the whole transaction happens online. So entire parts of businesses and businesses themselves are being replaced by software. So that’s the entire sort of business paradigm shift. But that’s leading to a lot of technology paradigm shifts.

Neha Narkhede: So the rise of the public cloud for developer velocity, rise of machine learning to use data and software to make better business decisions, mobile first customer experiences, and last but not the least, event streaming, because all of these trends, if you look at them, they all need access to data in real time. And event streaming is sort of the underlying paradigm that ties all of these things together. So not only was Kafka, of course, a great product, all of these changes were happening at the same time over the last 10 years that led to that massive adoption curve that I showed you.

Neha Narkhede: Event streaming is disrupting entire industries. To give you an idea, this is what Kafka users and Confluent customers are doing with Apache Kafka. Your ride sharing apps are powering your ETA feature and surge pricing using Kafka. Your bank is doing your credit card fraud detection using Kafka. Practically every retail company is doing real time inventory management using Kafka behind the scenes and your Netflix movie recommendations are also powered by Kafka. This sort of a widespread adoption of Kafka was possible, largely because of a large and thriving open source community. That was sort of the impetus behind Kafka’s adoption. I just want to say that the same open source community can act as a real catalyst for your own career growth. This is what it did for me, and it can get broad reach. You can learn from a pretty large community of people. You can diversify learning. You can be part of actually multiple communities at the same time versus one particular company.

Neha Narkhede: Large foundational technologies today start off as open source. So if you’re in the community, you’re part of a paradigm shift in and of itself. And I think that kind of impact is pretty large, because whatever you work on gets adopted across many businesses versus one particular one. You get to learn quite a lot. So this is the theme for today, I thought I’d mention. So that was a little bit about the what, in my journey, I did want to spend a few minutes about what it felt like, my experience, my career has felt a little bit like this, an obstacle race of sorts, and not all of those obstacles were technical in nature. And in fact, many times I’ve had to work 2X harder than my male counterparts to get the same thing.

Neha Narkhede: And while that might sound a little stressful and unfair, I want to share some perspective that my brother shared with me. He’s a many time Ironman finisher. He says that if you have to swim a mile in the ocean, and you train to swim a mile in the pool and expect it to feel similar, you’re going to be disappointed. It’s the currents that you need to prepare for. So that keeps me going quite a bit. In the moment, it feels like an obstacle race. But when I zoom out, and I look back at the last 10 years, I’ve started realizing that it feels like crossing a chasm. And I like to call this the credibility chasm. This is a phenomenon that I’ve observed where underrepresented minorities early on in their careers, they get marginalized, doubted, have to work much harder than everybody else to prove themselves over and over again, until you finally cross and make it somewhere on the other side, where sort of the opposite happens, you get noticed pretty easily, you get celebrated pretty widely for your achievements. While I have not crossed this chasm, what keeps me going are two things, long term thinking and a lot of grit, judgment to make decisions sort of not optimized towards the short term objective, but towards some long term goal and the stubborn persistence to just keep going.

Neha Narkhede: I believe this grit is rooted at an early childhood value, that many of you who grew up in middle class urban India will identify with. This is what we now know as the growth mindset. My parents sort of instilled this value in me that if you were open to learning and worked very, very hard, that you can actually learn anything you want to and you can be whoever you wanted to. And that sort of has stuck with me the value of education and hard work. How many of you in the audience know what this picture is about? Blurt it out.

Neha Narkhede: Yeah, this is the ISRO project managers or scientists. ISRO is India Space Research Organization. This picture was taken when they were celebrating a successful Mars mission. They put a satellite in Mars’ orbit, and they made an attempt at probably one tenth, the cost of any other mission in the world that has done that. This picture went viral when it was published. When a young girl in India looks at this picture, I think she believes with conviction that she can be one of these scientists when she grows up. And I had the privilege to be inspired by a lot of role models, even though not these particular ones. And I can say that role models are a primary driver, I think of a lasting change. And I get very excited when I look at that picture. But not just role models, but I think, the one last thing I want to leave you with is developing a real sisterhood will take us very far in seeing the change we want to see in the industry.

Neha Narkhede: What do I mean by that? Little gestures go a very long way. Pull a fellow woman aside who you think is screwing up, give her direct feedback, all the guys I know do that very often and it helps a long way, stop a conversation in a meeting to hear her out, vouch for each other very loudly in calibration discussions, and give credit if possible and very frequently publicly, these are all the little things we can do to sort of see the change we want to see in the industry. So that is sort of something I want to leave you with. With that, I’m going to conclude this very short talk and now we can move toward the next part of the segment. All right.

Dani Traphagen: All right, and for this next part of the process tonight, we’re actually going to have a panel session with Angie and Neha. So I’ll leave them to it.

Angie Chang: Awesome. So I have some prepared questions to ask you. Thank you for that presentation. So people might know Kafka from its creation at LinkedIn. And for those who don’t know what it is, can you briefly summarize what it is and how it’s evolved as a technology?

Neha Narkhede: Yeah, so Kafka is a highly scalable pub-sub messaging system. That’s how it started. What it does is it sits at the heart of your company’s data center. It connects up all the applications and all the data systems so that they can share data in real time and process data in real time to power all the things that you saw, real time customer experiences. Over time, we added functionality to Kafka that made a lot of sense. So the a-ha moment in Kafka is that it was not only scalable, which no other messaging system was, but it could remember, it could store data, so you can rewind and reprocess data. And that’s what caused its success in the world. Over time, we added related functionality, we added connectors so you can move data from all the other systems in a plug and play manner. We added stream processing so you can do sort of SQL on top of Kafka like maps and joins and aggregates.

Neha Narkhede: So this sort of combined functionality of pub-sub, connectors, and stream processing is what is now called an event streaming platform. So Kafka has evolved from a pub-sub system into an event streaming platform.

Angie Chang: Awesome. Yeah, I’m getting really familiar with event streaming platform as a category now. Let’s talk about your career. You started as an engineer, and then became an engineering manager, startup founder, and now you’re running product. Can you share something from your playbook with everyone here?

Neha Narkhede: Playbook? So yeah, that’s a lot of changes into it. So my playbook is I do a couple of things when I have to deal with a lot of change, like the first thing is I do believe, I firmly do believe in the growth mindset. So when I encounter something new, I’m fairly sure that if I spend enough time on it, that I can learn the ropes of it. The second thing I do is sort of this crazy knowledge gathering. So I read every book on the new subject, I reach out to experts, and I set up time and ask them questions. I just sort of like to learn a new area before I jump into it. And the third thing I do is reflection. I sort of sit down and try to calibrate myself on how I’m doing in that new area. And I talk to a couple of my close champions to sort of get their view on the subject, and then just keep iterating from there on. So that’s sort of my, I don’t know if it’s playbook, but I do that very often.

Angie Chang: It sounds good. There aren’t too many infrastructure unicorn companies that were started by women. I think we could only think of one, Diane Greene of VMware, and now adding to that list Confluent and Neha as a co-founder. And we hear women starting consumer companies, and they’re on the cover of magazines, but they’re often consumer. And before the infrastructure startups, we don’t see any women starting B2B infrastructure companies. So what is it about you, Neha, that’s different in that you can do this? And how can we get more women in infrastructure to start companies?

Neha Narkhede: Yeah, it’s one of my pet peeves is that there aren’t a lot of us starting, not only just infrastructure, but B2B companies. I think there’s there’s some luck and a lot of hard work. But if I were to hypothesize on why that is, I think there are a few things maybe. The first is that it’s a very male dominated field to begin with. And so when you don’t see people that look like you as founders of B2B companies, and when you know that starting a company is probably like five or 10 years of very hard work, then you may not be encouraged to take that very first step.

Neha Narkhede: So that’s probably one reason. The second one is, and I think I can only hypothesize is, I think there’s some perception that women may be uniquely qualified to start consumer companies or marketplace companies, because you have a better understanding of the end consumer. And while I’m really happy about the rise of consumer unicorns, I think that’s the same reason that women are successful with consumer companies, this is the same reason they will be successful with B2B companies is we’re smart, capable people. But I think that will change over a period of time. 2019 is probably the first year when we saw so many unicorn companies that were started by women. That’s like the first step in the change. I think we need a couple more women starting B2B companies. I will say that starting B2B companies is much more of a playbook than starting consumer companies. Predicting company behavior is a lot easier than predicting consumer behavior, so if you’re thinking of starting a company, I can tell you that a B2B company will be easier to start and grow. And I think we just need to see a couple more successful examples to tip that.

Angie Chang: Definitely, I think, so I would definitely like to say, people that we know on a first name basis, Mark, Larry, whatnot, now we add Neha to that. So tell your friends. We need more role models out there. So thank you for hosting us tonight. We talk a lot about women in tech in general. So let’s focus on the leadership aspect. Based on your own experience, what’s the greatest barrier for getting women into leadership positions?

Neha Narkhede: Wow, I wish there was just one barrier. That way, it would be much easier to cross. I’ll say a few things. I think we hear a lot about the imposter syndrome, and I can tell you that not just women, but men face it too. I think the reason it’s so much more magnified for minorities is because this sort of external feedback loop is much more skeptical than the usual, “Go man, you can just kill it, and you can do this.” So it’s a lot harder, but I can tell you what it does for leadership is it can sort of not encourage you to take risks, because if you think about what leadership is, you’re there to take a few calculated risks, and then lead successful execution of that.

Neha Narkhede: And so, if you can think about it, just sort of take the first leap. The second thing is, there’s a ton of unconscious bias, and just sort of you experience it, as you grow in your career. And the impact it has is women are evaluated on experience and men are evaluated on potential. So the same thing that you think you deserve and which you do, you get it later down the line. And for that, I would say that ask for that thing until you hear a very loud and clear no. Hearing a no never killed anybody, and it will only help you in that journey, and that’s what I’ve done. But I think those are a couple of really big barriers, I would say. There are a lot of upsides too. So when you’re growing in your career as a minority, it’s much easier for you to get noticed and so it’s much easier for you to recruit good mentors. There are people in the Valley who want to help women in tech and want to help minorities in tech, and they will gladly allow you to sort of reach out and give you the time.

Angie Chang: Great. The theme for tonight’s Confluent Girl Geek dinner is open source for career paths. And what are some things that Girl Geeks can do to leverage the open source community for their technical growth and learning?

Neha Narkhede: All right? Well, let me start off by saying that I don’t want to recommend open source as sort of a silver bullet for your career growth. But I will say that any outcome is driven by a series of choices you make. So in so far that open source is one of the choices that are presented to you, I would say think very seriously and probably even take the leap. The reason is that you get a lot of broad reach, you can learn a lot of things, but you also don’t need to invest all your time. So there are many ways to get involved, you can get involved in community discussions and critique designs or you can submit newbie patches or you can take up a full time job and get paid to work in open source.

Neha Narkhede: I think the best thing about the community is evangelism. So if you try out a new software, write a blog post about your user experience or if you want to critique the design, write a blog post and explain what you thought was good and bad about the design. I can tell you that Confluent has done a lot of successful recruiting by reaching out to people who wrote blog posts, and not just good ones, the critiques also. And so you will get noticed and obviously learn a lot along the way when you write about something.

Angie Chang: That’s really good advice, to write a blog post. All right, let’s get down to the nitty gritty for the technical in the audience. Kafka is known for its scalability. So where there is a continuous flow of streaming events, what’s the operational challenge in navigating new software versions, especially if something is backward incompatible? And how do you ensure the high quality of service?

Neha Narkhede: Lots and lots of things to say here. I think this would be a a segment of its own. But I’ll say two important things. I think really paying attention to the public APIs and contracts of your particular system is really, really critical. And especially so for infrastructure software, because a lot of applications depend on it and you have to be clearly careful about compatibility.

Neha Narkhede: Something the Kafka community did to be careful about this is a discipline we call Kafka improvement proposals. So we took a leaf out of the Python community playbook, and we introduced this discipline, because over time, you don’t get time to review code patches. Everyone gets busy. So this is a discipline where we ask people to write a wiki on the public API and contract chain so that the community can pay closer attention to version compatibility, and also the user experience, so that goes a long way.

Neha Narkhede: The second thing, because you asked for quality of service, I think being able to operate that software as a service in a company or as a fully managed service goes a really long way. I think there’s one investment I can say about high quality of service, it’s operating it as a fully managed service. So something people may not know about Kafka is Kafka, its first claim to fame was it just worked right out of the box. And that happened because we always deployed the version of Kafka into internal LinkedIn systems and it’s sort of baked in production for some time, before we even released the version into the open source community. So the community always got a well tested baked in version. That wouldn’t have been possible if we couldn’t deploy it internally at LinkedIn. So it goes a pretty long way.

Angie Chang: Great. And final question, what is one thing that surprised you, something that you believed in earlier in your career, and isn’t true today?

Neha Narkhede: Fun. So I moved here to the Valley from a different country. So the impression that I had about the Valley, which is pretty well known is that that’s where the American Dream gets made, that it’s a very meritocratic environment, as long as you’re smart, and you can work hard that you can win. That is mostly true except for underrepresented minorities. It’s a little bit harder than that. So I was surprised about that. I came in with big, optimistic eyes, and I was a little taken aback by all the challenges.

Neha Narkhede: The other thing that surprised me, I think, is the tech industry’s appetite for failure. And that one is really important. There is a ton of opportunity, no matter whether you succeed or fail, and that was from my upbringing, that was sort of a surprise for me as well, whether you succeed or fail, there’s always going to be a good opportunity waiting for you. So I would say, definitely go ahead and take that leap. There’s probably something else that is good waiting for you, no matter whether you succeed or fail. And so take those risks.

Angie Chang: That’s great advice. Thank you so much for this fireside chat. And I think we will now hand the stage over to the Confluent Girl Geeks.

Neha Narkhede: Thank you for having me, Angie.

Angie Chang: Thank you.

Dani Traphagen: All right. Well, that was fantastic. I don’t know about all of you but my favorite part was about getting more of us into infrastructure, and kind of developing the B2B experience and more diverse voices there. I think that Neha outlined an excellent roadmap of how to start a business, so I hope you were taking notes. I think all of those questions were fantastic on how to do that and the play by play of it. So next up, we’re going to have Bret Scofield. And she’s going to give a lightning talk on her experience here at Confluent.

Bret Scofield speaking

UX Researcher Bret Scofield speaking at Confluent Girl Geek Dinner.   Erica Kawamoto Hsu / Girl Geek X

Bret Scofield: Thanks, Dani. All right, so I’m Bret Scofield. I do UX research at Confluent. And I wanted to start off with a little bit of background about me. So the theme throughout my career has been building things from scratch. So in undergrad, I built metal sculptures, did a lot of welding, all that sort of stuff. When I graduated, I transitioned into digital products and I worked as a product designer for quite a while. And then in the past few startups that I’ve been at, the big theme has been building a discipline and a team from scratch.

Bret Scofield: And so that’s what I want to talk to you all about tonight. So I wanted to talk through, this is a brief listicle, there’s a lot of stuff from Twitter, I love Twitter so tweet me. But the big thing here is five things that you need to learn or that you need before you build something like UX research or any sort of discipline from scratch.

Bret Scofield: So the first thing, sorry, the first thing is defining UX research. And because I think a lot of people in here have familiar with engineering, et cetera, and maybe you haven’t worked with a UX researcher before, so essentially, UX research is narrowing the gap between these two groups of people. So I think there’s always one set of people who are building a product, and then another set of people who are using that product. And the people who are using that product are doing all these amazing things with it. They’re talking about it, they’re doing unexpected things, et cetera, and they’re all these amazing insights.

Bret Scofield: And so the goal of UX research is really to narrow that gap between the people who are building the product and the people who are using it. It’s just sharing those insights with the people who are building things. And they will make better products if they know and can empathize with the people who are using it. So there you go.

Bret Scofield: So the first thing that I think you need to build something like UX research is fertile ground to build. And so when I came to Confluent, the concept of UX research existed. It wasn’t formalized, or anything, but a lot of product managers, designers, et cetera, were speaking with customers about designs, about ideas that we had, et cetera. And they knew that this was super important to do and to get feedback from our customers, from our users, all that sort of stuff.

Bret Scofield: And so when I came in, the work that I did to establish UX research is really just taking what had already existed, formalizing it, adding a bit more rigor, putting it on a regular schedule, that sort of thing. And with this tweet, I don’t necessarily think that Jay, our CEO, knows what I do, but I hope that Neha does. And so I hope that UX research can continue to grow.

Bret Scofield: The second thing that you need is really three people. And so the first person that you need is an unconditional believer. And so I think a lot of us, as underrepresented minorities, you’re going to have tough days when you’re trying to establish things. And so like with UX research, a lot of times there’s days when people are like, “What are you doing? What’s the value of this? I’ve never done UX research before. I’ve never heard of this thing.” And so you need someone who’s always in your corner, who’s always believing in you, and is willing to talk through those tough days.

Bret Scofield: The other two people that I think you need are a sponsor and a mentor. And so a sponsor is someone generally in your organization who can connect you to the right projects that have good visibility, high impact, all that sort of stuff. And then the last person is a mentor. So the best mentors that I have have been outside of my organization, and I think that’s really necessary. They don’t have to be outside of your organization, but they should be outside of your management chain. And I think that’s necessary because you really want that unbiased feedback. You don’t want people who are incentivized to have you act a certain way or do certain things.

Bret Scofield: So yeah, I think that mentor should be outside. And ideally, they’ve been in tech or your industry for longer than you have. And I think that’s super important, because essentially, they’ve seen the same situation happen 15, 20 times, and you get to leverage that knowledge. You don’t have to go through 15 or 20 like I want to smack my head against the wall, you get to leapfrog.

Bret Scofield: Then the third thing is just enough knowledge. So a lot of times, I think that as a UX researcher, we think that we have to inhabit and totally become the people that we’re studying. And so with enterprise software, these people are sys admins. There’s just no way I’m ever going to become a sys admin, a lot of them were born in the command line, all that sort of stuff. However, it’s super important that I do a certain amount of research and learn this is what the command line is. I need to be able to tell what people are doing in there, what their intentions are, et cetera. But I don’t need to know every single thing. And so it’s very important in the past experience I’ve had to learn but to draw the line and not totally become a sys admin.

Bret Scofield: The fourth thing is balancing strategic and tactical work. So I think the impulse when you’re starting something new is to right away be like, “How can I provide value? Let me do this tactical stuff that’s going to provide value and insights to the team. They can take action on it right away, that sort of thing.” And I want to encourage you to do that, of course, but to also balance it out with strategic work. And by strategic work, in the context of UX research, one of the things that we do that’s useful for the next six months to two years, et cetera, is personas and journey mapping. So deeply understanding our people, deeply understanding where are they touching the product, how are they feeling at each of those points, et cetera.

Bret Scofield: Whereas the tactical work is with an individual team. We’re focused on what can they take away from this research and immediately put into practice. So yes, so I think the fourth thing is really that mix of short term. This is valuable right away, and the long term, this can be valuable for a longer horizon.

Bret Scofield: And then the last message, I think, is to get popular first and then get selective. So in the beginning, people aren’t going to know what UX researcher is, they’re going to come to you with all this kind of science fair sort of ideas. And that’s awesome. You should say yes, to all of them, do all of them. And then as you’re doing good work, as you’re finding insights, all that sort of stuff, your reputation and your legend is going to grow. And then you can start getting selective about those projects and take the really high impact ones, that sort of thing.

Bret Scofield: So those are the five tips if you’re building UX research or any new discipline within your organization. The other thing here is that, the next thing that I’m looking at is people to join the team. We anticipate hiring in the next quarter or so. So please, if you’re interested in being the second UX researcher at Confluent, come chat with me. And I also want to hand over to Liz Bennett from our software engineering side.

Liz Bennett speaking

Software Engineer Liz Bennett talks about her epiphany about being a hedgehog in a workplace at Confluent Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Liz Bennett: Thanks, Bret. Okay, right. I’m Liz Bennett. I’m a software engineer at Confluent. So I’ll start with a little bit about myself. I went to school at Oberlin College, and I studied music and computer science. And after graduating, I went to LinkedIn. And I was on the newsfeed team there. And I really enjoyed being on the newsfeed team. But it didn’t take long for the itch to join a startup to get to me. So I went to Loggly, which is a cloud based logs as a service company. And that was really great. I learned a lot about streaming infrastructure. I got to work with Kafka a lot and Elasticsearch.

Liz Bennett: But after a few years, I wanted to expand my skill set. And so I joined the data platform team at Stitch Fix. And the data platform team is the team that builds all of the infrastructure and tools for the data scientists at Stitch Fix. And for those of you who might not know, Stitch Fix has an absolutely gargantuan data department. There were I think, 100 data scientists there when I joined. So I really got a chance to level up my big data skills. And I also built all of their logging infrastructure and their data integration infrastructure from scratch. But after about three years, I went looking for another change. And as of six days ago, I’m now at Confluent.

Liz Bennett: Neha asked me if I would speak at this. And my first thought was, “Yes, I would love to.” My second thought was, “What the heck am I going to talk about?” And at this time, I was between jobs. I had just left Stitch Fix, I was waiting to join Confluent. And the only thing I could think about was this job change I had just done. It was really actually quite a difficult experience. It was really painful, much more so than any of my other job changes.

Liz Bennett: So I wanted to just tell my story, and I hope that it might be useful for some of you out there, now or in the future. Okay, so why did I join Confluent? I could also frame this as why did I leave Stitch Fix, because that was kind of the real crux of what was going on. Every other job change I’d had before, I knew what I wanted, I knew what kind of opportunity I was looking for. It was much easier. This time, the one thing I knew was that something didn’t feel right at Stitch Fix. It didn’t feel like the right fit. It took so long to really put my finger on it. I was completely blindsided by it when I joined.

Liz Bennett: And I waited three years and it never felt right. It never got better. So I did a lot of soul searching and I came to a few realizations and I realized that the team I was on and the role I had was fundamentally mismatched with who I am as a person. So everyone, I’m a hedgehog. Has anybody heard the parable of the fox and the hedgehog? Anybody? So I heard this recently on the Hidden Brain podcast. And as soon as I heard it, it seemed to explain a lot of things for me. So the parable comes from this quote by the ancient Greek poet Archilochus. And the quote is, “The fox knows many things but the hedgehog knows one big thing.”

Liz Bennett: And over the years, this has been interpreted to be like there are two kinds of people in this world, foxes and hedgehogs kind of thing. And some psychologists have even used this as a way to describe two kinds of cognitive styles and people. There’re foxes who draw on a wide variety of experiences, and they use many different strategies to solve problems. They’re comfortable with nuance and even contradictions. Hedgehogs, on the other hand, they see the world through the lens of one unifying idea. They love to think in terms of big pictures.

Liz Bennett: And as soon as I heard this, I was like, “I’m a hedgehog.” I told my best friend about it, too. And she’s like, “Yep, you’re a hedgehog.” And I also knew, all of my teammates at Stitch Fix were all foxes. My manager was a fox, my manager’s manager was a fox, not in a good way. No, just kidding. And so I thought, “Okay, is that what’s going on here? Should I just go find another data platform team somewhere else, hoping that there’s going to be more hedgehogs there?” And in the end, I decided no, that’s not what’s going on. What’s happening is the team I’m on is just a better fit for foxes. As a hedgehog, I need to find an entirely different kind of team. So I kept thinking this through, and I came up with something that ultimately really illuminated this problem for me. And it was a really useful device when I was trying to explain to my friends and my colleagues, and especially my manager, why I was leaving Stitch Fix and joining Confluent.

Liz Bennett: So I like to call it the product platform spectrum. What is the product platform spectrum? It is the spectrum of teams that exists within a technology company, that span product, customer facing teams on one end, all the way down to internal, low level infrastructure teams on the other end. And depending on where you are on the spectrum, your role is going to feel really different. So at the top of the spectrum, you have your product teams, these teams are very close to the customer, they’re generally the source of revenue for the company. You’re really close to the company mission, there tends to be a lot of separation of roles and separation of expertise, like they’ll be UX researchers on product teams.

Liz Bennett: Supporting the product teams, there’s usually platform teams, and companies invest in platform teams because hiring somebody on a platform team is like hiring somebody on all of your product teams. That makes them all more effective and more productive. Platform teams, though, are further away from the customer. They tend to wear more hats, I think. There’s less specialized roles. I think they tend to own more surface area. They have to own more technologies and services. Underneath the platform team, in some companies, this will vary, but very often, there’ll be infrastructure teams. And these teams own the very bottom layer of data systems and services. They’re like the bedrock that the whole rest of the company sits on top of.

Liz Bennett: And these teams are great because their work is leveraged across the whole company. They’re also the furthest away from customers. And there’s no platform team supporting them. So they kind of have to write their own tools. They do a lot of dog fooding. They can be very autonomous, though, they, they set their own strategies, they do their own research, and they’re masters of their own destiny. So at Stitch Fix, I was on the very bottom layer of infrastructure. I built all of the Kafka infrastructure, I was doing that. And where I really wanted to be, though, was at the very top of the product spectrum. That was where I had been my last couple of roles.

Liz Bennett: Since I’m a hedgehog, given my hedgehog nature, I thought it would be much more comfortable in that role again, where I could really focus on deepening my skills as a software engineer, and also be really close to the company mission. So how can I get from the bottom of the spectrum to the very top? Could I get there while I was at Stitch Fix? My answer basically was no. The product teams at Stitch Fix were mostly full stack Ruby on Rails engineers, and I didn’t want to completely retool myself just to get to the top of the product spectrum.

Liz Bennett: So at that point, I realized, okay, I need to make a change. I need to leave Stitch Fix. But where? What do I do? Well, smeared across this whole spectrum is our B2B vendors. So there’ll be platform as a service companies selling products to platform–or product teams. There’s infrastructure as a service companies selling products to platform teams. So all I really actually needed to do was go from here, at the very bottom of the spectrum, took one little step into the B2B space. And suddenly, I was going to be at the very top of the product spectrum again.

Liz Bennett: So I mean, the one thing is I left a consumer business and went to a B2B business, but I was in the B2B business at Loggly, so I felt pretty confident that that was going to be fine. So at this point, I realized I needed to, I knew where I needed to go, I knew what sector I needed to go to. So then the last question is why Confluent? Why did I pick Confluent?

Liz Bennett: Well, I had been working with event infrastructure for the last three years at Stitch Fix, and I had become absolutely obsessed with this mission, Confluent’s mission. And being a hedgehog, I just wanted to go deeper. I wanted to keep doing it. And I wanted to focus on that. And I realized, what could be more satisfying than going from building event infrastructure for one company at Stitch Fix to going to Confluent where I could build it for the whole entire world? So that’s my story. I hope that it’s useful for some of you out there. Transitioning jobs is really tough. I don’t think people talk about it nearly as much and give it as much credit for how hard it is. So if anybody wants to talk more afterwards, I’m super happy. Connect with me on LinkedIn. And thanks, everybody, for coming. All right, so I’ll hand it off to Priya.

Priya Shivakumar speaking

Senior Director of Product Priya Shivakumar talks about her career jungle gym at Confluent Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Priya Shivakumar: Hello, everyone. I hope everyone’s having a great time tonight. I certainly am. It’s a pleasure to be in the company of all of you. So my talk is going to be a little bit about my career path, some learnings that I’ve had along the way, and how that’s come to apply to what I do at Confluent. All of us are looking to grow in different ways. And so the paths we take sort of reflect that. But for me, the common theme throughout has been to continuously broaden my perspective and keep learning along the way. That’s kind of the key decision driver for me.

Priya Shivakumar: And so my career path has looked something like this. I’ve used this format, instead of the format that my colleagues and friends have used before me, for two reasons. One, because it would actually age me and the second because it just wouldn’t fit on one slide. So this is kind of the path that I took. Growing up, early on, I developed a passion for engineering. My dad’s an electrical engineer, and he encouraged my brother and I to sort of take things apart to learn how they work. And I remember him and I taking apart quite a few VCRs and a few transformers actually in his station to get to the magnets inside. And those magnets were coveted positions. So I naturally gravitated to engineering for my undergrad. And from there, my career has spanned three key disciplines: engineering, product, and consulting. And I’ll talk to you a little bit about each one of those.

Priya Shivakumar: So in engineering, it was about building the product, how do you build a product. I enjoyed all aspects of problem solving, logic, it was a natural fit. Success was mostly individual in nature. I could have been successful without having interacted with another soul, potentially, or at least having a little bit of interaction maybe.

Priya Shivakumar: But I did not get to see how my code was being used. What was the impact it was having, who are my customers. And so that’s the reason why I moved out of engineering. As I stepped into IT consulting as an engagement manager at BearingPoint first, and then later into product management, the focus shifted over to customers, stakeholders, clients. There were a lot of competing priorities and a dearth of resources, and that’s the name of the game. And that required–success now meant being able to influence people, align teams, kind of create common goals and create common objectives. And that’s a very difficult and hard skill to acquire.

Priya Shivakumar: And what little I know of it, I will attribute to my consulting days. An example comes to mind, there was a post merger integration project. One large company had acquired another large company. And as a result of that, the system and the processes we were putting in place would result in the elimination of 30 to 40 jobs. And the data that we needed to build the system had to come from these very same people. So you can imagine how painful and difficult it was. And this particular example actually falls on the extreme end, but most consulting projects have some element of tension or friction in them. Think about it. You’re an outsider, you’re trying to advise somebody how they should do their job, nobody likes to be told that, one. Two, they may have some kind of perception that they may lose some control. They don’t like that. And they may also kind of think that their domains are going to shrink, or there may be a job loss in the future.

Priya Shivakumar: And so all of these things create for some very delicate waters that you need to navigate and kind of balance. So I think that was a core skill but it’s still a learning. It’s never, I mean, I wouldn’t say I’ve completely mastered that. But that’s something that I picked up a little bit in consulting. I would like to share another key thing that kind of happened during this time. So when I was at BearingPoint, after a couple of years in IT consulting, the work got repetitive. And like most of you here, I have a healthy paranoia about stagnancy. I wasn’t learning. And I went to my MD, my managing director. And I told her that I really wanted to move into strategy, from IT consulting to strategy consulting, and BearingPoint had both of those practices. She was supportive, she was actually well intentioned, and she sort of grew me within my role, but I had tech expertise, and I was a billable resource. So it did not make business sense for her to move me to the strategy consulting practice within BearingPoint.

Priya Shivakumar: So I realized early on that that was not going to happen. But I had to stay put for two more years. And that’s because I was trying to get my green card. I was pregnant with my first child. And the key here is that that’s okay. Right? That’s okay. There will be times in your life when other priorities take over. There’ll be times when you have obstacles that cannot be overcome, things that are outside of your control, like immigration things. So in those instances, it’s okay to set your own pace. Take your time, wait, rather, bide your time. And when the time’s right, get up and get going again. Just know what it is you want and what makes you happy. Go after that, though. So in my case, I wrote my GMAT in my ninth month of pregnancy, finished out my B-School essays in my maternity leave, got my green card, and I was out of there.

Priya Shivakumar: So the next thing I did was I went into, post B-School, I joined LEK Consulting. It’s a niche strategy consulting firm. And the reason I joined that was because I primarily wanted to work only in strategy, in the broad discipline that is management consulting. So it was a bit of an insane choice to make. I call it insane because it required me to work 60 to 80 hours a week. And the work itself was intense. We were advising veterans in an industry, typically the C suite, about what they should do to grow their business. It required you to get up to speed on their industry within a short period of time, do the research that was needed to draw insights from data, model out the market size trends, things like that, and then advise them about what they need to do to kind of grow the company by X percent.

Priya Shivakumar: So that was intense. And my husband was traveling on a weekly basis. And I had a two year old to take care of, two and a half, three year old to take care of. So there was a fundamental thing that I did, which not only helped me survive, but succeed in that role. I still wanted that role. I still went and got that role. But the fundamental thing that I did was hiring the right child care, and people say this all the time, but I cannot enunciate that enough. I applied the same rigor that I would to my job to finding child care.

Priya Shivakumar: So to me, the criteria that I defined for that was that I needed an au pair who would be with us 24/7. She had to be educated so that my child would learn from her and it would be easy for the whole interaction for the family. Also, I preferred someone who would have taken on responsibility early on in their life, so that she could independently run the place. And so the au pair was Aleja. She was 24 years old from Colombia. She had a law degree. She had taken care of two siblings while growing up and while her parents were working hard on their small business.

Priya Shivakumar: She came home. She just seamlessly became part of our family and completely ran my household, enabling me to focus on my work. So there will be inflection points in your career. And during those times, you have to get the support that you need. Do not skimp on that. I’ve seen too many people make that mistake. And it just results in burnout and a lot of not very good things. So I would highly encourage you to do that.

Priya Shivakumar: One other–then the time came to become a partner at LEK, it gave me pause. It was a very lucrative path and one that was a sure path, actually. But the reason I paused and decided to leave consulting, first was because I realized becoming a partner meant greater focus on sales, and lesser focus on problem solving and casework, which is what I truly enjoyed. The second part was that as part of an advisory or a think tank, that’s what you do. You advise, and you walk away, right? There is an innate satisfaction in seeing things come to fruition, things that you build, whether it’s the product that you build that launches, or a strategy that you can come up with that is activated, and you see that work in the market. And I really missed that aspect of it. And lastly, the industries we were working on weren’t super exciting to me and I was always passionate about high tech. And that’s why I moved back into VMware, and now to Confluent.

Priya Shivakumar: So putting it all together, the engineer in me loves the innovation we’re driving at Confluent. We are fundamentally changing Kafka in ways to make it ready for the cloud. The market is nascent, there are no clear answers, this data is limited. And this is where I really lean on my strategy consulting frameworks to answer questions like how should we price cloud, what segments to go after, what features are important by which segment?

Priya Shivakumar: I think there’s significant competition in the market but I do believe that we are uniquely positioned to really make Confluent’s mission successful. So putting it all together, I would say my execution from my engineering days and strategy from consulting and product thinking from VMware, enable me to drive this key initiative at Confluent, which is to grow the cloud business. Thanks so much. Dani, over to you.

Dani Traphagen: Thank you so much, Priya. All right, so everyone, I hope that that was really informative and gave you some food for thought. And now let’s actually have some food. But before we do that, just a couple of quick announcements. The other thing that we’re doing, as well, is providing a Women in Tech events at Confluent’s headquarters down in Mountain View in November. So if you’re interested in that, there’s sign up sheets also out at the front. And we’re going to do a quick Q&A before we start to network and have some food as well. So I’m going to invite everybody up for that right now.

Liz Bennett and Neha Narkhede speaking

Confluent girl geeks: Software Engineer Liz Bennett speaking on a panel with Neha Narkhede at Confluent Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Liz Bennett: Okay, who’s our first victim?

Jiang: Hi, I’m Jiang, and I am particularly interested in the theme of this talk, because it’s talking about how open resource open opportunities for your career. I’m just wondering, how do you all kind of assess opportunities in your career? For myself, I kind of felt like sometimes it’s really hard to find the good opportunities. And I’m particularly interested how people looking for opportunities and how they consider those are the good opportunities.

Liz Bennett: Well, such a tough question. I think for me, I’ve usually, the best opportunities I’ve found are the ones where there’s the biggest vacuum, I guess, like there’s the biggest need for people who have your skills or experience or something that you want to learn. And you just go and find those vacuums, and you fill them as fast and as well as you can. I think that would be my short answer. Yeah.

Priya Shivakumar: I think that’s a great question. I want to add to that a little bit. I think, you go through the interview process, right? It’s really important to understand the culture of the place during that process as you meet people. One of the things at Confluent that I absolutely fell in love with was this smart but humble…requirement, almost. And that was very apparent throughout my interviews. I had eight separate interviews, and each person sort of embodied that requirement, I would say, and then I also look for how many women are at that company. And how many women are at the top. It indicates a certain thing, and it should not be… it is an important thing. Those are some of the things I look for, among other things.

Dani Traphagen: Next question?

Audience Member: Hi, I have a question regarding, I’m somebody who came from large companies and worked in the large company environment. And in that, you’re reporting to a manager, who reports into another manager, who reports into director or whatever. So you get a lot of hierarchy. So I recently switched from that large company environment. And now I’m at a startup where I reported to the CEO. So I’m curious, how do you manage that dynamic of this isn’t just my manager, but it is my manager, but they’re also here and not here. So any insights you have on how to define that relationship, how to set the tone for that relationship?

Neha Narkhede: I could probably add a little bit. So because you talked a little bit about the big company to start up transition, depending on the stage of your startup, early days or early years are all about survival, and that’s what the CEO is responsible for. So likely, they do not have a lot of time to tackle the day to day issues that a manager’s supposed to tackle as much as they would like to. Just the practicalities of running a start up don’t allow for that time. And so I would suggest sort of look for mentorship elsewhere, if you are running into those kind of problems, but really ask the person like, “What is your biggest problem?”.

Neha Narkhede: And that was sort of my way of working at LinkedIn is nobody really wanted to work on this Kafka problem. It was sort of just something that my co-founder, Jay was dealing with, and I sort of asked him, “What’s the biggest problem on your plate?”, and he was like, “Well, there’s this Kafka thing, but no one really wants to work on it.”, because it was sort of a mess of a situation at LinkedIn that we had to clean up using Kafka.

Neha Narkhede: So I think the what I learned from that is, if you work on the biggest problem their business is facing, and the CEO is likely to know that biggest problem, you’re quickly going to become a go to resource and you’re quickly going to learn quite a lot that would then position you for other opportunities in the company. So that’s sort of the way to look at it is expectation management is not going to have a lot of time for all the day to day problems as well as asking for what’s the biggest problem on their plate that you can take off.

Jenia: Hi. Thank you for your talks. Jenia, a founder of a B2B startup with its first paying customers. So I wanted to ask you all with the limited resources that companies have, especially in the beginning, how to make customers happy? What are the secrets like hacks?

Neha Narkhede: Well, I can add a little bit. So early days, and I imagine you’re probably talking about pre product market fit. And so I think pre product market fit is a lot more of an art than a science. I think Priya talked a lot about managing data to draw insights that happens later in the life of a startup. Early days is all about landing your first 10 customers. So it’s incredibly important to not worry too much about over fitting the problem, because first time customers are going to ask for the world, but it’s really, really important that you land them successfully, because then you know which are the next 100 customers that you want. So that’s probably a really important thing. Life is going to be very hard when you satisfy all the problems of your first 10 customers. You should just go in expecting that to be the case. That’s very expected. But landing your first 10 customers is probably your your biggest and most important problem in that phase.

Mike: Hi. My name is Mike. I have a question for you, Neha. I want to know what is your best practice or solutions that worked to receive feedback at the company and from your employees. I mean, there is so much to read in different books about recommended ways to receive feedback from employees. But having worked for a couple of companies, I see it being quite difficult for people at C level positions, specifically, to receive feedback from engineers. I want to know what are the things that work for you, like when was the last time that a really junior engineer could openly and honestly share with you feedback, how you receive that. I would appreciate your thoughts on that.

Neha Narkhede: That’s a great question. So something someone said to me reminded me of this when you asked this question, and he said that, Neha, you got to be careful in this stage of your company, because fat fingers cannot make small changes. And what he was really trying to say is, the more your company grows, and now we’re 900 people and more, is it’s going to be harder and harder for you to get that feedback.

Neha Narkhede: I think a couple things have helped us at Confluent to get that feedback, and I couldn’t deny that it’s getting a little bit harder, the first thing is setting the tone of the culture from the very early days. So when we started the company, all the founders, we encouraged a lot of open dialogue, a lot of open sort of pushback. You can actually get up and challenge the founders on their ideas, or even the CEO on many occasions, all the engineers could do that and they felt comfortable doing that. So when new people join the company, they could see that debate happening on an open Slack channel. So everybody could see how the people are dealing with it and we encouraged that sort of debate quite a bit.

Neha Narkhede: I think that sort of has helped us quite a bit. The second thing is anonymous sort of feedback channels, doing surveys in the company that sort of scales when you get to a certain size. And then what has helped me in particular is I have these friends who are sort of in different parts of the organization, and they’re at the beginner sort of medium levels. So they collect feedback, and they sort of bring it back to me, and they’re sort of my champions in their processes. There are some engineers who are going to bring sort of gossip or chat that’s happening at the lunch table. And I have other sort of champions sitting here that have gotten me feedback on what I should be careful about. All of that sort of really helps. You got to make sure that you have some of those champions sprinkled around in your organization, who could actually give you the second degree feedback, because people are not going to come and give you that feedback directly as you grow your organization.

Denise Hummel: Hi. I’m Denise Hummel, and I’m the founder and CEO of a technology enabled diversity and inclusion firm. And I’m way older than like 90% of you and it’s blowing my mind because I’m, generally speaking, the mentor of trying to move women through middle management to senior leadership. And I look at you guys, and you are an inspiration to me. So my first firm was a consulting firm that I scaled to 65 countries and sold to Ernst and Young, and became a senior partner there leading culture, inclusion, and innovation. And I thought, wow, this is just the story of the century. I was a single mom who raised two kids on my own while I was building this company. And then when I got there, I felt unable to navigate the nuance of standing out and fitting in, and everything that I had known to be the core of who I was and why I was successful as an entrepreneur, which is basically never take no for an answer and just keep forging ahead, was actually the bane of my existence, because I was considered to be too aggressive.

Denise Hummel: So here I am now. I have left to start this new firm, which basically is technology, using AI and nudge messaging to bring inclusive leadership to leaders in real time, which is super exciting. And I have to pitch for VC and I’m still running into the same issues that I was running into before, which is that as a woman founder, I have to be this assertive, take no prisoners person in order to convince VC that I have the stick-to-itiveness to get this done. But when I do, then I’m the aggressive woman, who isn’t like the quintessential female persona that they’re all looking for. So that’s a really long background question. But the actual question itself is, do you have any feedback on what we can do as women to walk that line to have that nuance between standing out and fitting in and being assertive enough to make it but not so assertive that we are the aggressive ones that no one wants to do business with?

Bret Scofield: I guess my initial thought on that is, because I think in a lot of situations, especially in enterprise and dealing with a lot of customers who are aggressive men, and I’ve tried being aggressive, assertive, et cetera. And it doesn’t feel right to me, it just doesn’t feel like Bret. And I think that’s fine because I think that I can be me and still get the message across and still be successful, and all that sort of stuff. And it feels better, it doesn’t feel like I have to be super pushy, and that sort of thing. So yeah, that’s where I come out on that because I think that you can try and be this person that you’re not, and it’s ultimately not going to come through as much as being you. So are there other thoughts on this?

Neha Narkhede speaking

Chief  Product Officer Neha Narkhede speaks on panel at Confluent Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Neha Narkhede: I can add a little bit. I’m going to channel an RBG quote on that, which is it pays to be deaf sometimes. And I say that because you got to keep going. You got to pitch your startup and it’s an extremely arduous opportunity. You don’t want to get bogged down by all this feedback, because it turns out that in order to start a company, you have to be ambitious and aggressive, and very, very persistent. So I wouldn’t worry a lot about the perception. There’s going to be feedback, I’ve gotten a lot of this feedback, “You’re too ambitious, you’re too aggressive.”, and I’m saying, “Well, thank you. I’m going to try to work on how that doesn’t come across sometimes.” But it’s absolutely necessary to sort of put your blinders on during a certain stage and just keep on going. Because you do not want to stop in your journey, because of a lot of this increased skepticism from the outside. So I’m going to just say, keep going.

Karen: Hi, I’m Karen, super lucky to ask the last question. First, thank you all for the talk. Totally loved it. This question is for Liz and Neha. I am a software engineer and I observed in my company, and maybe what a lot of companies, being an engineer, a female engineer, as you go up, you see less and less senior women engineers, actually have the data from my company. So I can’t share, but it’s like at some point, there’s a huge drop. Beyond that, you just don’t see women anymore. And in the industry, we definitely see less women architects or women CTOs as compared to other roles. So one thing I would like to know, is first, Neha, I looked at your LinkedIn. So I see you have a good career growth at LinkedIn. So at that time, what pushed you through getting to be a more senior software engineer? Same for Liz, I know you want to keep growing, being really focused on this one area. So what is your view of this problem? Yeah, I think that’s it.

Liz Bennett: Yeah, it’s definitely true. It’s kind of eerie how there’s so few women the further up the stack you get. I think, for me, I have always thought of myself as just a person. I don’t see myself, I don’t often think of myself as like a female engineer. I almost actively avoid thinking about how I’m the only woman in the room, and after years of doing that, it just kind of stopped occurring to me when it happened, and it just became a normal thing. And I think the less I think about that, the more I can just focus on being an engineer and focus on doing what I love doing, which is technical work.

Liz Bennett: I really love doing it. And I think people see that, and they see that you love it. And they see that you’re competent. I do have to go out of my way sometimes to advocate for myself. When I do it intentionally, and when I’m doing it, I’m conscious that I’m doing it. And it helps a lot when you say, “Hey, I built all of the streaming infrastructure at Stitch Fix.” People are like, “Oh, okay, she knows what she’s talking about.” You have to actually do that. There’s one thing that I think a lot about, well, I used to think about, but not so much anymore, but it’s kind of like the competency chasm that you’re talking about. I think for a lot of women, they’re seen as incompetent until they prove themselves to be competent. And for men, it’s the opposite. They’re seen as competent until they prove themselves to be incompetent.

Neha Narkhede: Sometimes.

Liz Bennett: So I think that I’m like, Okay, I have to go through this process, I have to prove myself. And I haven’t had too many problems with it. But it is something that I’ve come to learn over the years.

Neha Narkhede: I’ll add one more thing to that is, when you’re on this technology ladder, there’s going to be a point where you sort of feel like you’re running out of options. And that’s when you try to fall back to this management option, which is sort of a parallel option. And a lot of us take that because at some point, you get tired of advocating for yourself or pushing for that new opportunity. If it is the right choice you want to make, then you should take it, but if not, I would recommend, ask for things explicitly, ask for that new opportunity that, the same way Priya asked for this new opportunity on the strategy side, ask for things until you hear a clear no, because you never know where there is an opportunity where someone like you might be a good fit, but people are not quite thinking about it actively. You don’t want to wait until that sort of thing walks up to you. You want to go aggressively vouch for it and not be scared to hear a no.

Dani Traphagen: Cool. All right, thank you so much, ladies. Okay, so networking, deserts, maybe an added La Croix for the road. Whatever you’d like to do next, I hope that that was really useful for you. I know it was for me, I learned I’m a hedgehog. An informative night the whole way around. And yeah, if you have any questions for any of us, feel free to come up. The Confluent careers page is a fantastic place to check out. I hope you will. Feel free to check out our LinkedIn pages and and just go ahead and connect with us. If you have any further questions, we’re really happy to give you any advice that we can or help in any way and just actually have some friends in our community. So anything we can do. Thanks again.

Neha Narkhede: Thank you.


Our mission-aligned Girl Geek X partners are hiring!

Neha Narkhede and Sarah Allen

Confluent founder Neha Narkhede and Bridge Foundry founder Sarah Allen meeting at Confluent Girl Geek Dinner in 2019.  Erica Kawamoto Hsu / Girl Geek X

Why changing the face of “superstar developer” matters

Neha Narkhede began her career as a software engineer, working at Oracle and LinkedIn. She was a co-creator of Apache Kafka, a popular open-source stream-processing software platform that was created at LinkedIn. She spoke on a panel Girl Geek Dinner while she was still in engineering there. She saw a big opportunity with Kafka and convinced her fellow Kafka co-creators to start Confluent as a B2B infrastructure company in 2014 – Kafka’s event streaming is used by 60% of Fortune 100 companies today.

With only 2% of venture capital going to women entrepreneurs, Neha beat the odds and demonstrated that it’s possible to thrive as a technical leader. She served five years as the company’s Chief Technology Officer, and recently became Chief Product Officer to continue growing the brand. Confluent’s founders recently raised Series D venture funding for the company at a valuation of $2.5 billion, and they employ over 900 people.

Silicon Valley needs more Nehas! Read more.

Girl Geek X Microsoft Lightning Talks & Panel (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Angie Chang speaking

Girl Geek X Welcome: Angie Chang kicks off a sold-out Microsoft Girl Geek Dinner at Microsoft Reactor in San Francisco, California.  Erica Kawamoto Hsu / Girl Geek X

Transcript of Microsoft Girl Geek Dinner – Lightning Talks & Panel:

Angie Chang: So hi, everyone. My name is Angie Chang and I’m the founder of Girl Geek X. I want to thank you so much for coming out tonight to the Microsoft Reactor. I’m super excited to see everyone here and to introduce you to all of Microsoft’s girl geeks, to see this amazing art and tech demos. Who here signed up for a demo? I saw a lot of people interested in demos and getting tours, so I’m really excited that you are able to do that. Thank you once again to Microsoft and to all the people who helped plan this night.

Angie Chang: How many of you this is your first Girl Geek Dinner? Wow. And how many of you consider yourself like a regular at Girl Geek Dinners? Thank you so much for coming back again and again. We do this almost every week, going to different tech companies, meeting the girl geeks, and we hope you tune into our podcast. We have a regular podcast on topics from internet security, to emotional security, to management, to working in the Silicon Valley. So please tune in on iTunes or Spotify. We also have a very active social media. So if you follow us at Girl Geek X, you can also tweet and share with Girl Geek X Microsoft tonight and we will retweet and reshare.

Angie Chang: Now I would like to introduce our first presenter. Her name is Kaitlyn Hova and she is the co-owner of Hova Labs, where they have designed and produced the Hovalin, which is a 3D printed violin. Kaitlyn.

Kaitlyn Hova: Thank you so much for having me. This is wonderful. So my name is Kaitlyn Hova. I currently work at Join and I also co-own a company called Hova Labs, where we like to make a bunch of weird projects. It’s kind of like one of those like, “If I had time, why wouldn’t I make this?” kind of companies. So it’s just me and my husband and the biggest thing that we really wanted to do was to find a way to convey what synesthesia was like in real time. Who here knows what synesthesia is? Yeah, it’s not very many people. It’s all right. So synesthesia is a neurological phenomenon in which two senses are inherently crossed, causing sensations from one sense to lead to an automatic but also involuntary experience in another. A version of this is called chromesthesia, which is when people can physically see sounds.

Kaitlyn Hova: I didn’t know this was in any way unusual until I was around 21 years old when I was in my final music theory course and our professor just mentioned, “Isn’t it crazy? That some people can see sounds?” Yeah, I ended up dropping my music degree and going into neuroscience, because that’s way more interesting, right?

Kaitlyn Hova: So, ever since then, I’ve been trying to find a way to display what synesthesia was like, because when you’re discussing it with people, it tends to end up going into the more like psychedelic conversation, and it’s not really. So, how to display it? I play violin, so we thought, “Wouldn’t it be wonderful if there was a violin that we could light up with the colors that I see in real time?” This didn’t exist, so of course you have to go to the drawing board, and the first thing on our list was, “What if we had a clear violin and we just put LEDs in that?” We couldn’t find a clear violin and if we could, it was probably too expensive.

Kaitlyn Hova: So, ended up deciding like, “Well, how hard would it be to 3D print one?” It took a year and a half to figure out how not to make a violin and then to figure out how to. I think we went through about like 30 or 40 iterations because you end up getting really desperate and saying like, “Well, what is the violin anyway?” because it’s really hard to make this. It started out as a stick with strings and then kind of grew from there.

Kaitlyn Hova: So now, here it is. Once we got our first prototype, we ended up deciding that this violin on its own, LEDs aside, was a really great product, so why not release it open source for people to 3D print their own music programs? We’re still seeing a trend in schools where music is systematically underfunded, while these same schools are getting STEM grants, so why not? Seems like a connection there. Thank you.

Kaitlyn Hova violin playing synthesia

Violinist Kaitlyn Hova plays a few songs at Microsoft Girl Geek Dinner.   Erica Kawamoto Hsu / Girl Geek X

Emily Hove: Let’s hear it for Kaitlyn. Kaitlyn, thank you so much.

Kaitlyn Hova: Thank you.

Emily Hove: This is fantastic. What a great way to start off such an inspirational evening.

Kaitlyn Hova: Thanks.

Emily Hove: So thank you very much.

Kaitlyn Hova: Cheers.

Emily Hove speaking

Program Manager Emily Hove welcomes the Girl Geek X community to Microsoft Reactors around the world, from San Francisco to London!  Erica Kawamoto Hsu / Girl Geek X

Emily Hove: Welcome, everybody. Welcome to the San Francisco Microsoft Reactor and the Girl Geek Dinner.

Kaitlyn Hova: Thank you, Chloe.

Emily Hove: My name is Emily Hove. I’m part of the global Microsoft Reactor program and we have a lot of synergies between Girl Geek and the Microsoft Reactors. Similar to the way Girl Geek inspires and connects women in technology, our Reactors are all about being community hubs and everything that is related to developers and startups, giving developers and startups the tools where they can learn, connect, and build. So, we hope you all find a night that is inspiring and where you’re able to connect and build today.

Emily Hove: If you’re interested in a little bit more about the Reactor program, we’ve got some cards around the room and they talk about some of the fantastic upcoming workshops and meetups that we have. So we’d love to encourage you to check out our calendar of events and invite you all to attend. With that, I’d like to bring up Chloe Condon, who will be our MC for the evening, and help introduce some of the inspiring people and inspiring women in technology that we have for you tonight. So Chloe, cloud developer advocate extraordinaire.

Chloe Condon: Hello. Thank you so much for coming. This is theater in the round. So I’m just going to keep walking in a circle like I’m giving a very serious keynote so you all don’t see my back. Thank you so much for coming tonight. We are so excited to have you here at the Reactor. Who’s first time at the Reactor, this event? Incredible. That is so exciting. I hope we see you here a lot more. If you want to participate in one of the Fake Boyfriend workshops that I put on here, you can build a button to get you out of awkward social situations, come see me after. We are doing those all the time here. They’re so much fun. Also ask me about my smart badge. This is a little scrolling LED badge that we’re probably going to do a workshop for pretty soon, as well. So come see me after if you’re interested at all in learning about those events and we’ll get you signed up for them.

Chloe Condon: I’m going to tell a little story before I introduce our first guest. I am so, so excited to be your MC tonight. I actually met Angie because I went to Hackbright. Do we have any Hackbright or bootcamp grads in the audience? No. Amazing. So, Angie spoke at my bootcamp and told us all about Girl Geek Dinner and I thought, “That sounds so cool. I would love to go to one someday.” So it’s literally a dream come true to be here with all of you today. This is my first Girl Geek Dinner ever, and I get to be your MC.

Chloe Condon: So, I’m so excited to introduce our first speaker tonight. She is incredible. Please, please show everybody how cool your dress is when you come up here, or I’ll be very upset. I would like to introduce Kitty who is going to tell us all about the incredible technology and fashion that she uses to make things like the amazing dress that I’m sure she’s about to tell you about. So Kitty, come on up. All right.

Kitty Yeung Microsoft Girl Geek Dinner

Microsoft Garage Manager Kitty Yeung gives a talk on “Hacking at the Microsoft Garage” at Microsoft Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Kitty Yeung: Hi, everybody. Good evening. Thank you so much Chloe for introducing me. In fact, I’m not going to talk about my dress. That’s for the demo later. I’m going to talk about actually what’s behind that, all the innovation work that we’ve been doing at Microsoft. So, I’m the manager of The Garage at Microsoft. How many of you have heard of The Garage before? Some of you, some of you I’ve met actually.

Kitty Yeung: So, this is a program that drives the innovation, drives a culture of innovation and experimentation. How do we do that? We say, “Doers not talkers.” We actually get our hands dirty. When we think about something, we act on it. These are the culture pillars for Microsoft. To a lot of us when we first see them, they saw just words, but how do we actually implement these and achieve this? We have all kinds of programs and mechanism to drive innovation in Microsoft. Hacking, we have global sites, we have internship programs, experimental outlet is how we ship projects out, and we have intrapreneurs program, and we do storytelling. So I’m going to go into each of these.

Kitty Yeung: The hacking at Microsoft has become the culture. We actually organize the world’s largest global hackathon at Microsoft, and The Garage is the organization that organizes it. Guess how many people attended this year? Globally, there were 27,000 people attending our hackathon, and everyone was excitedly bringing their great ideas to the hackathon and forming teams all around the world. Whether or not you know them, whether or not you’re from the same org, same teams, you can put your skills together and build something that you feel passionate about. We had thousands of projects every year submitted to the hackathon, and The Garage helps people not only have these ideas submitted, we help them grow their ideas into prototypes, and we help them ship.

Kitty Yeung: Satya is a big supporter of our hackathon. He walks in the tent and look at the projects. He said last year, “Bigger ideas, more customers.” So, we can hack on anything we want. So it could be small things. It could be something that we use every day. It could be something that has real impact in the society, we can really help our customers achieve their industry scale ideas. So we also work with our customers and we bring our customer come here to hack.

Kitty Yeung: The experimental outlet, we also call it a ship channel. So this is a mechanism for us to get those ideas in but also provide them with the business model, idea building, how to enter the market, and we help our employees ship those projects out. So if you go to The Garage website, you will see about 100 projects that’s already in the market, and we feature our employees who came up with those good ideas. You can see all the teams on the website, everyone who put their part time together to really achieve something. So, we also have very big projects that we collaborated with industry partners and customers.

Kitty Yeung: Intrapreneurs program is kind of a internal startup program. It involves these ideas, these teams, hackathon teams, to actually pitch their ideas to the leaders and get support. So some of these projects can grow into a feature of an existing Microsoft product, or sometimes they become a product of Microsoft.

Kitty Yeung: We also run our internship program very differently. If you are familiar with traditional internships, usually students come in and they work under one manager in a big team working on a small part of a big project. Instead, our interns come in as a team and inside a team usually we hire like 30 students per site. Silicon Valley just started our first pilot program, so we only had one team, but we have six really, really good students. Usually we’ll have teams of six to eight, and they have developers, usually a PM, and a designer, forming a complete skill set. Then business teams at Microsoft pitch their ideas to our interns and the interns pick which one they like to do, and they drive it like a startup in the company for 12 weeks. Then they can deliver the projects back to the team, or even better, we can ship it directly into the market. It’s a very, very competitive and rewarding program. So if you’re undergrad, think about applying to that internship program at The Garage.

Kitty Yeung: We also engage with storytelling, those ideas, those projects got shipped out. We tell a story, we have a PR team, and you will see a lot of news articles about Microsoft innovation. Pay attention next time when you read an article like that if they mention The Garage.

Kitty Yeung: The global sites is also our feature. We have seven global locations right now for The Garage, and we are expanding. Each location has our own ecosystem, and also, each location has our facility. We have maker spaces, we have technologies that we provide to our employees. They can do prototyping, they can bring their ideas to share with their colleagues. We do startup pitching. We do show and tell and workshops to educate our people and also give them a platform to achieve their collaborations.

Kitty Yeung: So these are the seven sites worldwide. We’re in Silicon Valley and we are now called The Garage Bay Area. And as you can imagine, we have a unique ecosystem of a lot of startups, a lot of big companies and universities. So we work with all of these people in the ecosystem and we collaborate to really build projects that can impact the world. So, as I mentioned, we work with our employees and engage with all of our business teams inside Microsoft, and we work with customers. We bring them to work on projects and hack with us.

Kitty Yeung: Here are some numbers. You can see that we have very global and diverse team, but we actually only have 20 people worldwide. So, the 20 people drive all of those activities that I just mentioned. 27,000 hackers this year is an updated number. Last year, behind that 27, there was 23,000. You can see that it’s growing every year. It’s only going to get bigger. 76 countries participate and we’ve held more than 100 interns already. With the most competitive schools around our local areas. You can find more than 100 projects that’s in the market and on the global website. 19 of them became actual Microsoft products and lots of social media posts, lots of news articles about Microsoft innovation. So, make sure you follow us on the social media.

Kitty Yeung: Some of the Bay Area’s specific projects. Seeing AI, we build a lot of projects that help the people with needs, people who have disabilities. Seeing AI is a project that we shipped a few years ago that help blind people see through technology. So you can hold a phone, the camera will detect what’s in front of you and also read it out, interpret. It can also detect facial expressions and people’s age. So it gives blind people information about their surroundings.

Kitty Yeung: Sketch 360 is a project we just shipped last year, is by an artist inside Microsoft, Michael Scherotter. He had an idea of, “Why don’t we sketch 360 pictures directly?” So, we can build like a full environmental canvas and you can draw anything you want. You can also put that into VR or AR to visualize it. We also last year shipped some apps. Spend is by MileIQ team. So, lots of local projects. We’re just going through our hackathon projects this year.

Kitty Yeung: So personally, that’s why I’m also here to do a demo. I’ve build some of the projects in The Garage to satisfy personal ambitions of anyone in Microsoft can use The Garage as a resource to build their communities, can build their projects. So I have built a lot of wearable technologies. I’m doing a demo right there. We have these different dresses with different sensors and AI, machine learning functionality, and robotic dresses that I can show you later on. But I also have a passion for quantum computing because of my physics background. I’m a physicist, actually. So, I see the need to build a community of people learning about quantum. So this is a study group that I founded in Bay Area, teaching people how quantum computing works, including physics, maths, the hardware, and software, and any employee with good ideas, they can do this. So we have a lot of employees who wanted to do, say AR tech community, they can come to The Garage and do that. Or they have passion for IOT, they can come to The Garage and do that. So, these are just some examples.

Kitty Yeung: So since Girls Geek is also sort of about career, I think this will be my last slide to show you something about your aspiration. This is a guide. So see where you are in this chart of Ikigai and see where you are and figure out what would you like to be. I think for me, I can feel Ikigai in Microsoft because I’m doing something I love, something the world needs, and something I can be paid for that’s important, and something I’m good at. So, if you can get to that sweet spot, that should be your goal. Also, think about how you’re aligned to the global goals. That’s what I can do. I highlighted some of the goals that I could do in the company as well as through my personal projects. I think I would love to expand this and I think this will be a good guide for everyone, how we can do more impactful work for the world. Thank you.

Chloe Condon: Okay. Wait. You cannot leave the stage without sharing this dress. I’m going to make you model it. It is so incredible. So, do you want to say a little bit about it first?

Kitty Yeung: Okay. This is one of my designs, among the other ones I brought. All of these prints are my own paintings. This is a painting of Saturn and I wanted to simulate Saturn on the dress. How do I do that? Because Saturn has a ring, so why don’t I make a ring that when I rotate it will show Saturn. It also has an angle detector. There’s an accelerometer in here. So if it achieves a certain angle it will light up like the stars.

Chloe Condon: Amazing, amazing.

Kitty Yeung: Thank you.

Chloe Condon: Thank you so much. When you wear such a fabulous dress, we should have had a catwalk. I’m so sorry everyone. Amazing. Thank you so much, Kitty. I really, really love that and I loved that final slide. I took pictures of it so I can look at it later and map out my own plan. I am so excited to introduce our next guest that is going to tell us all about machine learning. Priyanka, come on up to the stage. I have a little … do you need a clicker? Amazing. Here you go.

Priyanka Gariba speaking

Head of TPM for AI Priyanka Gariba gives a talk on “Leading a large scale and complex machine learning program at LinkedIn” at Microsoft Girl Geek Dinner.  Erica Kawamoto Hsu

Priyanka Gariba: Hi, everyone. First off, I’m not showing off anything as cool as what the other women did, but I also want to say this is my first time here at Girl Geek Dinner and I think this is amazing. Look at the energy, like room full of women. How many times in a day do we get to see that, or even a month, right? So thank you for having me. My name is Priyanka Gariba and I lead Artificial Intelligence Technical Program Management group at LinkedIn. My talk for today is going to be how we are scaling machine learning at LinkedIn. We are one of the large and complex program that has been funded by our engineering group.

Priyanka Gariba: So, I’ve structured my talk into four different areas. I’ll give a quick introduction on LinkedIn and some of the products that are really powered very heavily by machine learning. I will then get into the problem statement of what we are trying to do in order to scale machine learning. Then talk a little bit about our technology, and then wrap it up with sure, we can scale with building a solution and with technology, but there’s also an aspect of people, and so how do we scale that, and what is LinkedIn doing about it? Okay. All right. With that, let’s get started with the vision and mission for LinkedIn.

Priyanka Gariba: Our vision is to create economic opportunity for every single member in the global workforce. Our mission is, the way we are going to realize it is of course by connecting world’s professional to make them more productive. Let’s take an example of this room itself, right? So many cool things that were shown up, so many cool people, so many cool women that we spoke to. Just imagine if we were connected to one another, there’s so much value we can bring in each other’s life, and LinkedIn can help us do that. So, how are we trying to realize our vision and our mission is through some of our products.

Priyanka Gariba: I’m hoping and I think everyone here is at least having a profile on LinkedIn, and if you’re not connected to the cool women here in the room, I encourage that before you leave, definitely connect with one another. But some of the products that really help us do that is People You May Know. This is a product line that really helps us build our connections. It understands, there is a recommendation system that runs behind it, there is machine learning models that run behind it, very heavily AI powered, and it really allows us to know who are the people, like minded people, that we need to be connected to, and the value we can bring in each other’s life by just having that connection.

Priyanka Gariba: Then of course there is Feed. Everybody who goes on LinkedIn as a platform is going to see Feed as the first product. Jobs is another product, which is very heavily powered by machine learning behind it. Why am I talking about all these products? AI at LinkedIn is like oxygen, and one thing that all these products have in common is AI. With that, what that means is we know that machine learning is everywhere. It’s powering every single product line that we build, it’s helping us bring the best experiences to all our members across the board. So, because of that one reason, we know that what we need to do is we need to enable more people to do machine learning at LinkedIn.

Priyanka Gariba: So, there are two pieces to my talk. One, which I think I’ll dive into more than the second one, is going to be technology. There’s one way we can scale technology, is by building a solution. How do we enable our machine learning engineers to really build and deploy models faster so that the experiences that they can bring to all the members is at a faster rate. The second one is by scaling people.

Priyanka Gariba: So, to tap into the exact problem that we are trying to solve, let’s look at our machine learning development life cycle. It’s as simple as any software development life cycle, right? Basically a machine learning engineer has an idea, there’s something you want to solve for, what is the first couple of things that they would do? They’ll think about what are the machine learning features that are available to them? How do you crank up all these features together? Try and test it in an offline model, train with some datasets, and once you value it and feel comfortable that this is something good, the next big piece is going to be actually serving it in production and then seeing results through AB testing and all of that.

Priyanka Gariba: I’m not going to dive too much into this. This really just is an extension of that life cycle. Basically you start with an idea and then there are different functions along the way. There is a product management, there’s dev, and the way we really make decisions on product is very heavily powered by our AB testing platform. We make ramp decisions only based on that. Once we see the results, only then do we believe that that is a model that we want to ramp further to our members.

Priyanka Gariba: Why talk about all of this? Why talk about the life cycle, right? If all these products are being built at LinkedIn and if so many people are doing it and all the teams are doing this, what that means is every single team is doing and deploying models in a very different way. There are many, many technologies, they are all on different stacks, it’s not standardized across the board, and one thing we encourage at LinkedIn is for people to move around within teams. So today if you want to work on a Feed team, tomorrow you want to work on a Job Recommendation team, how do you do that? Your stack is different. Half the days are going to be spent in just ramping up.

Priyanka Gariba: So, we introduced something called as Productive Machine Learning. Really our goal is to enable end to end experience of machine development life cycle to be more robust, reliable, and consistent, and standardized. The experience we are looking for is for an ML engineer, all you have to worry about is come up with an idea, and then there is everything else is opaque for you. There is a big box and you don’t have to worry on how you move from one phase to the other. Ideation to machine learning features to training to scoring to serving it in the introduction. You don’t have to worry about this and how are we going to do that.

Priyanka Gariba: So, we’ve put together this program, it’s to give you context, this is a really large scale program, about 6,200 engineers across the board working on it, different geolocations. The way we are structuring it is by talking about three different phases.

Priyanka Gariba: Model creation, going back to that life cycle that you saw, everything from ideation to training and evaluating your model comes under model creation. So we have multiple components that blend into that. Then the next piece for us is deployment. Once you believe that your model is really good and ready for serving, you deploy it in production. The third piece, this is not really a phase, but something that cuts across, is making sure your quality is accurate. Meaning features that you used for your offline training are very similar to what you see in online. So online, offline consistency.

Priyanka Gariba: So, I just wanted to, because I had 10 minutes, I just wanted to give you a flavor of this big undertaking that we are doing at LinkedIn and also give you a little bit of flavor of how we are structured. Typically, every time we build something, we follow a traditional model. You have a leader, you have multiple managers, you have engineers, and you come up with a goal on a project and everyone works together. This one, we wanted to do something different. What we did is, let’s bring every single person in LinkedIn who is really passionate about solving this problem.

Priyanka Gariba: So put together what’s your team, we had everyone across the board, in different geolocations too. There is someone who will be infrastructure heavy. There is someone who is a machine learning engineer who can help us really give us inputs when we are building the solution that it’s really going to work for them. Then there’s product managers, CPMs, engineers, across the board, but it’s really all of these coming together, forgetting the boundaries of management, realizing that there is one goal that we have, is to get an end to end machine learning life cycle ready, was the key thing for us. I already mentioned that, team of teams, we’re geolocated. That is also one reason why we wanted to do that, is we wanted engineers across the board because if we were solving a problem just for headquarters, which is in Mountain View, we will not be solving for everyone at LinkedIn.

Priyanka Gariba: Then of course with any product that you build in any company, there is a big piece of adoption. So, for us, the strategy that we have used is that let’s, the three big phases that we spoke about, let’s build small components underneath it and let’s allow every product team to pick up a component and adopt that depending on what their pain point is. So, for example, if a Feed team is really struggling with how do you train a model, then what we wanted to offer them is pick up that component and get adopted on that. Once you buy the idea, then slowly and gradually navigate into the adoption of the other components too. This helped both ways. This helped us get real early feedback from our customers and users, and then it also allowed us to load balance. So we could develop things while something was already being tested and we were getting that iteration loop from our users.

Priyanka Gariba: So, I spoke about the technology, and I spoke about the solution. The second thing that LinkedIn is doing, and I’m just giving a very high level preview of this, is in order for us to democratize AI or to make it readily available and to enable more engineers to do that, there’s a program that LinkedIn’s kicked off, it’s called AI Academy. There are three different types of courseworks of program, AI 100, 200, 300. As you graduate from one to the other, really the intensity of the techniques and machine learning increases. So AI 100 is really just getting a flavor of what AI is, what machine learning is, and get you familiarized with it. And then 200 you start understanding how do you build a model, and three is when you actually build your own model and put it in production. I can talk all about this and I’m happy to talk about it later on, but this is just a preview, and there’s a lot of blogs and things that we’ve already put on LinkedIn.

Priyanka Gariba: This is another blog for Productive Machine Learning for those of you who are interested in reading more about it, and I’ll share my slides as well. That’s it. Just a quick flavor. I had 10 minutes, so I thought at least I’ll come up here and talk to you and give you a flavor of what we are doing to democratize machine learning at LinkedIn. But happy to, I don’t know if I have time for questions, but I can take questions later on as well. Thank you.

Priyanka Gariba: Okay. I can take a question or two if … After. Okay. All right. Sure.

Chloe Condon: Thank you so much. All right. So, next up, I will take that from you. Next up we have a very special treat, but before I introduce our very special guest, I’m going to show you my favorite LinkedIn feature. How many people have added someone on LinkedIn tonight? Okay. Well now you’re going to add more people. So, if you go to your LinkedIn app in the very top in the search bar, there is a barcode, a scanning barcode, and if you click on that, instead of having to type out the person’s name and awkwardly ask for spelling, you can just scan their barcode tonight. So you can share that secret tip that I learned recently from someone else at a meet up that I now pass onto you to make spelling people’s names less awkward. So definitely scan everyone’s badge here tonight. My best advice always in tech is to meet as many people as you can, and tell your story and share their stories while you’re here tonight with all these amazing people.

Chloe Condon: I am going to welcome our very, very special guest for tonight, Charlotte. Come on down. We are so excited to welcome Charlotte Yarkoni to the SF Reactor. Here you go.

Charlotte Yarkoni speaking

Corporate Vice President, Cloud + AI Division, Charlotte Yarkoni gives a warm welcome at Microsoft Girl Geek Dinner.  Erica Kawamoto Hsu

Charlotte Yarkoni: Thank you. I need to start out and tell you guys, I’m sick. I really, really apologize for my voice. I’ve been told I don’t look as bad as I sound, so I thought it’d still be okay to show up, but hopefully you’ll manage to go with me this evening. It was important for me to come. So again, I hope you can work with me on the sound quality. But my problem is as I’m watching everybody on stage, I wanted one of these mics so I can put it down, cough, and anywhere I go I’m going to … somebody’s in my blast radius. So, if I come over here and stand by the post, please don’t be offended.

Charlotte Yarkoni: Anyways, good to be here tonight. Thank you guys all for coming. I thought what I would do is first share with you a little bit about my journey of being a woman in tech and what that’s meant to me in my career. I do need a clicker. My telepathic PowerPoint clicking slides are not on today due to the head cold. So, I actually go talk a lot to universities. I go to some high schools. I love talking to young girls about STEM, but I always kind of have to ground in. Let me tell you what tech looked like when I was in middle school and high school.

Charlotte Yarkoni: This was it, by the way. There were no smartphones, there were no tablets, there were no laptops. I remember when Asteroids came out and me and my brothers thought it was amazing. Right? So that’s kind of where we were. Then this was our social network. There was no Twitter, there was no WeChat, there was no Snapchat. It was pretty much a bonfire in somebody’s field when their parents were out of town in the town I grew up in. So, that’s kind of where I come from.

Charlotte Yarkoni: I actually, I grew up in South Carolina. I was super fortunate to get a scholarship to come to UC Berkeley. I’m pretty sure I’m the only person from South Carolina to ever go to Berkeley. I was actually part of an inaugural program at the time called Electrical Engineering or Computer Science, or EECS as it was known. This is what code looked like when I was coding. Has anybody ever written in Lisp? Anyone? Did anyone? Yeah. Kicking it old school. All right. So, that was sort of my education, if you will, and my real foray into tech.

Charlotte Yarkoni: Then, I got out of college and started working and figuring out how to use technology as an applied science, not just in an academic sense, and this was kind of the world I was in. Actually cell phones came out and yes, that’s what they looked like for those of you that weren’t born then, because I know there’s a few of you here. Windows 95 was all the rage, right? You remember that? Then we get to today and it’s just a very, very different world.

Charlotte Yarkoni: One of the things that I love about technology is the fact that it has actually opened up all of our worlds, in so many ways that we can have so much more impact. We can instantly connect to people that we could never connect to 30, 40, 50 years ago. I’m not that old, I’m just framing my comments. But you think about that and it’s not just connecting to those people, it’s the access to information that you also have immediately at your fingertips. It’s amazing. It’s amazing that what you can harness with that kind of resources at your fingertips.

Charlotte Yarkoni: The challenge is, though, it comes with a responsibility, and I will tell you, at Microsoft, and GitHub, and LinkedIn, we spend a lot of time on that. In fact, it’s not just about innovating, it’s about innovating with purpose, and really making sure that you’re actually leaving the world in a better place than you found it before you introduced your solutions. So it’s those unintended consequences that you have to be very thoughtful about. As we continue to get more and more technology at our disposal, how do we use it for good? That kind of brings me to really, what’s my role.

Charlotte Yarkoni: Today in my role is, at Microsoft, I run a group called Commerce and Ecosystems. You can tell I’m not a marketing person, so there you go. But I’m really here. I focus on answering three questions. The first is, how do people actually discover who we are and what we do in our products and services? And Microsoft’s a very big company, it’s a global landscape. We offer lots of different products and services across our portfolio, but there are a lot of ecosystems and communities that actually don’t know who we are and what we do.

Charlotte Yarkoni: Five years ago it was a lot about open source, and I remember I actually went to … I started at Microsoft about three years ago and I went to an open source conference. By the way, I grew up in open source, so my background actually started out in Unix and moved to Linux. I never wrote a piece of code in .NET. Would probably look and feel a little bit like Lisp to me, honestly, if I tried to do it now. So when I came to Microsoft, I went to a familiar conference, and people were like, “Why are you here, man? Azure doesn’t run Linux.” I’m like, “What are you talking about? Yeah, it does.” People need to know, right? So we had to go fix that.

Charlotte Yarkoni: Second thing I focus on is after you discover us, how do you engage with us in a way that’s meaningful to you? And most of that is online. People don’t always want to have to go somewhere to learn how to do something. They will now have to sign up for a week long course, right? Necessarily to know how to build a solution using the technology that they have. So we spend a lot of time and energy focused on that and what’s the set of tooling or resources that we can offer.

Charlotte Yarkoni: Then the final point is, how do we just get easier to do business with our customers and partners? That’s where the commerce piece comes in and it’s all about what are some of the new business models we need to create to actually, how do we run all those capabilities across all our products and all our channels today? So there is a good bit of engineering that comes in each one of these aspects, but there’s also a lot of business work that I have to focus on. And again, it comes with that overarching layer of responsibility, is to how do we think about continuing to make progress in a positive way so we can have a positive impact on the communities we serve.

Charlotte Yarkoni: So that’s kind of who I am, and I think what we’re going to do at this stage is a little bit of like an AMA, and I’m really hoping you guys don’t ask me too many questions because the more I talk I think the worse I sound, but I will try to answer everything for sure. I was going to have Chloe join me, and I was going to have Shaloo Garg join me. So, just as a reminder of both, Chloe and Shaloo are part of my team and they’re part of the drive discovery effort. So I’ll let you guys, you guys will talk a little bit more about yourselves, I’m sure, but I’m going to turn it over to our master of ceremonies. Kick us off. Do you want that mic or you want–

Chloe Condon: Sure. Mics all round here.

Charlotte Yarkoni: This one may be contaminated.

Chloe Condon: All right. I wouldn’t want to catch the virus, the Charlotte virus. Amazing. So, I figure we’ll have a seat. Have a seat wherever. We had a bunch of people submit questions earlier in our fishbowl, thank you so much for all of the questions that we got earlier. So, what I figured I would do is we would start with an introduction with Shaloo. Would you like to tell everyone who you are, what you do?

Shaloo Garg, Chloe Condon, Charlotte Yarkoni

Microsoft girl geeks: Senior Cloud Developer Advocate Chloe Condon, Corporate Vice President for Cloud + AI Charlotte Yarkoni, and Managing Director of Silicon Valley’s Microsoft for Startups Shaloo Garg answer audience questions with candor at Microsoft Girl Geek Dinner.  Erica Kawamoto Hsu

Shaloo Garg: Yeah. Absolutely. Firstly, thank you guys so much for coming here today. It means a lot. My name is Shaloo Garg and I lead the startup business growth for Silicon Valley for Microsoft, and entire California as well. It’s an exciting space to be in, and part of Charlotte’s team and part of what we do is not only engage with founders and CTOs and CIOs here of startups, but also drive meaningful partnerships, which is … this is Silicon Valley, there are a lot of partners here, how do we work with them to drive awareness of how Microsoft can help entrepreneurs there? So good to be here.

Chloe Condon: Amazing. Thank you so much. I have these randomly selected questions here.

Shaloo Garg: Those are a lot of questions.

Chloe Condon: It’s a lot of questions. I don’t know if we’re going to get through all of them. We may do kind of a rapid inside the actor’s studio type of lightning round at the end here. But I love this first one. I chose this one first and this is for Charlotte. It says, “What’s it like being an executive at one of the top companies? Do you have a life?” Great phrasing, whoever wrote this.

Charlotte Yarkoni: I’d like to think I have a life. Yes, I do have a life. I have two children, both girls, one–

Chloe Condon: Great. Are they coding already?

Charlotte Yarkoni: One is 23, just graduated. She went to Reed College, and by the way, back to Berkeley, I thought when I went to Berkeley from South Carolina, I was an enlightened liberal. And when I dropped my daughter off at Reed College, I felt like I was the most conservative person on the planet. I was a little worried about my life choices at that point. But she graduated there in linguistics and she actually is starting school this week, getting her master’s at University of Washington.

Charlotte Yarkoni: She would be very offended if I called her a developer or an engineer, yet she spends a lot of time writing programs and are doing statistical analysis on languages because she focuses on Russian, Japanese, Spanish language and language heritage.

Chloe Condon: Wow.

Charlotte Yarkoni: So, that’s my oldest. My youngest is 13, and a prolific gamer and developer. Python is her language of choice. She has lots of opinions about every other language.

Chloe Condon: As she should.

Charlotte Yarkoni: It kind of takes me longer these days to set up an environment for her to code in than it does for her to whip out a new game that she’s thinking about. So, I’m pretty sure she’s going to end up somewhere in the engineer community as a professional at one point. I also have three horses. I ride. I grew up three day eventing, for those of you who know what that is. Now that I’m older and have kids, I wondered what my parents were thinking when they let me do that. But I still ride and I still compete. Then I do my day job.

Chloe Condon: That is a fun fact.

Charlotte Yarkoni: I think the thing about today’s technology is, the good and the bad is it allows you to be accessible all the time. So, you can actually, you have to know how to be at the right place at the right time, which is usually the conflict that occurs, but you are able to go do what you need to do personally and do things professionally as you go. So that’s something I’m really, I feel privileged by who I work for in the industry I’m in and the technologies that we’ll be bringing for all the working moms out there.

Chloe Condon: Wow. That’s actually a great segue into the next question, which I’ll direct to Shaloo first, which is, how do you relax and unwind? Like with how long and tough your day jobs are, how do you get to chill?

Shaloo Garg: So, best is tennis. I love playing tennis and that’s how I unwind, and when I go out and play tennis, I try not to take my cell phone with me or my kids. So I have a 13-year-old daughter too, and a nine-year-old son who quite a handful.

Charlotte Yarkoni: Do you have any Serena moments on the court?

Shaloo Garg: I do. But that’s how I unwind, which is just completely unplug, just a moment of Zen and just go out there and hit it.

Chloe Condon: I’m very similar. I craft. I like to do like things with my hands and not look at a screen and just build something fun, like a costume or something that lights up. And you’re riding horses.

Charlotte Yarkoni: Yeah, but I could not build a costume. So, we each have our strengths.

Chloe Condon: Hit me up for Halloween. We’ll get you guys–

Charlotte Yarkoni: I’m going to hit you up for Halloween. Okay.

Chloe Condon: This one says, “What would be your advice for your past self coming straight out of college?” I love that question.

Charlotte Yarkoni: Who you asking?

Chloe Condon: Anyone can jump in. Yeah.

Shaloo Garg: I think coming out of college, I wish I was more aware of getting a coach or a mentor, which I was not aware. And during my career I sort of looked upon women leaders and requested them to be mentors and coaches. So what I try to do now is go out and coach and mentor women or young girls myself. So, I realize that they may be in the same situation as I was in, which is, “Hey, I can ask a woman leader to say, ‘Would you mind spending 30 minutes with me?'” But they don’t ask. Right? So I preemptively do that in schools, colleges here in Silicon Valley. Actually right up our Market Street office, that’s another office of ours, every month, I host open office hours for young women who are out there, budding entrepreneurs. It doesn’t have to do anything with Microsoft. So, as soon as you walk in the door, it doesn’t have to be, “Hey, you have to sign up to work with us,” but it’s just coaching, and I love it. So, wish I had that, but a part of me is just giving back, just making sure that someone out there is benefiting.

Chloe Condon: Yeah, that’s great advice. Charlotte.

Charlotte Yarkoni: I think, for me, one of the things that it’s taken me a long time to appreciate and I really, I encourage everybody to have some thought about this for their own journey, both personally and professionally, resilience is such an important thing. When I look back on my career, I feel, again, very privileged to have worked in all the places and spaces that I have. But the successes I had weren’t one success right after the other. It was a success built off of quite frankly, a mountain of failures and trials to get there. It was about taking those learnings and applying and getting better. I think a lot of what we do as an industry is about solving a problem, solving an opportunity, and getting better as we go, and iterating, and it’s really hard to do that as a person.

Charlotte Yarkoni: I’m going to go out on a limb and assume all you people here are somewhat overachievers. So every time that you have a failure, you want to prosecute the failure and you want to prosecute yourself, and that’s okay as long as you make it a constructive thing and learn from it, and the older you get and the more experienced you get, the more you start to really embrace and almost be proud of those failures for what they taught you, because you wouldn’t be wherever you are without it. That’s just a fact. I don’t know that I appreciated that in my younger age. I was certainly an overachiever and thought I knew a lot more than I knew at the time. I know that’s shocking, but it’s true. But as I went through my career, it was a process for me to understand how to really get value in the mistakes, how to really give value in the failures, and use them to move forward.

Charlotte Yarkoni: I just would encourage everybody, get out there and try. That’s step one and step two, is make sure you learn and embrace the mistakes, right? And it is about that of resilience that will just make you so much of a better person whatever you decide to do, however you decide to do it.

Chloe Condon: My advice would be, I don’t think I knew right when I graduated what I wanted to do with the rest of my life. I wish I had taken a little time to travel or maybe to explore different industries and fields that maybe I wanted to dip my toe in. Because I think what the wonderful thing about working in tech is you don’t have to commit to doing the same thing for your entire life. You can always change and learn a completely new technology or … There was a tweet that I think I retweeted this morning, which was, “Your job that you have in five years may not even exist. So try not to plan out your life too strategically,” and I think that’s really wonderful advice because technology is growing at a rapid rate and we may be working for something we don’t even know exists yet. The new, I don’t know, a new iPhone. Who knows?

Chloe Condon: Great. Next question that I have is, I love this one, “What’s the best book you’ve read this year?” Does anyone have one? I know mine. I can go first while people think.

Shaloo Garg: Go, go for it.

Chloe Condon: I read a book. Oh no, you go first because I want to make sure I get her name right, the author’s name right.

Shaloo Garg: So I think the life-changing moment for me was the book that I read by Eckhart Tolle. It’s called The Power of Now, and it teaches you a lot about what Charlotte talked about, failure. It also teaches you how to stay engaged but not attached, which is you’re really passionate about something that you’re doing. Keep that passion, but don’t get so emotionally sucked into it that you break down. So it also teaches you mindfulness and awareness. And then how to be an A player, which is you’re mindful, you’re aware of what you’re doing, but guess what? You got to go and get it. So I thought that was completely life-changing for me because I learned quite a bit in terms of just being strong, being very passionate about what I do, but not emotional, and then just chasing it, chasing the ball and just chasing the heck out of it.

Charlotte Yarkoni: Mine’s an oldie but a goodie, because my youngest was doing a book report on this one, the Life of Pi.

Chloe Condon: That’s a good one.

Charlotte Yarkoni: I just loved that. I haven’t read it in many years and so she brought it home and I brought out my copy so we could read it together. It is just an amazing book.

Chloe Condon: That is on my list. You said yours was The Power of Now?

Shaloo Garg: Power of Now.

Chloe Condon: Okay. Write that one down, everyone. I recently read Just the Funny Parts by Nell Scovell, she’s a female comedy writer, and I found … it’s an autobiographical piece. She used to write for Saturday Night Live, David Letterman, and it’s a completely male dominated field. It was the first time I had read about an industry other than tech that was similarly structured and formatted and it talked about, she’s a comedy writer, so it comes from this place of empathy and humor, and I would highly recommend it. She helped write Sheryl Sandberg’s book. She also wrote a lot of Obama’s jokes, I found out in that book. So, a lot of the things that made us chuckle from Obama came from her.

Chloe Condon: So, next one is, “Who has influenced you most in your life and why?”

Charlotte Yarkoni: That one’s actually really hard. I will tell you both my parents passed away in the last year. They were quite older. I’m the youngest of a large family. Pretty sure I was an accident, so, it’s okay. But you spend a lot of time reflecting on your nuclear family when those kinds of things happen, and they happen inevitably to everyone. So I definitely think my parents had a large influence on my life. I think my teachers had a large influence on my life. I’m the proud product of the public education system of South Carolina, which I think at the time I was growing up was like 49th in the country. But I went from there to UC Berkeley, which was an amazing school. And I had some amazing teachers to help me learn how to learn, is what I got from that.

Charlotte Yarkoni: I’ve been super fortunate to have some great mentors and what I would call guidance counselors throughout my career, that I still do lunch with and dinners with and catch up with. So, I feel like I’ve had a lot of influences and I do think for the last 20 plus years, though, my kids have probably taught me more humility and patience and resilience and all the other virtues we speak so highly of. They’ve probably been the biggest forcing function in my life in recent years.

Chloe Condon: What about the horses?

Charlotte Yarkoni: The horses are my sanity. I will tell you, we moved to Australia for a couple of years and I couldn’t take my horses with me and I was, my husband will tell you, I was a miserable person for the time I was gone.

Chloe Condon: I’m picturing you writing postcards back to your horses at home.

Charlotte Yarkoni: I came home. I came home every two months to see them.

Chloe Condon: Aww. How about you, Shaloo?

Shaloo Garg: So, parents, but I think my mom. So I lost my parents at a very young age. I remember when thinking back growing up, so I was born in India, but I grew up in Middle East, and I grew up in a community where there was lot of domestic violence and girls were not allowed to go to school. And so there were a lot of changes that were happening around me. In fact, while growing up, I went to 14 different schools between elementary, middle, and high school. So you can imagine moving from Saudi Arabia to Iraq, to Kuwait during the war zone time. But I remember going through all this, my mom always taught me and my sister is that, if there’s ever a problem in life and there is a simpler solution, and there is a hard solution, guess what? Pick the hardest one, because it’s going to make you go through that process, whereas a simpler one, you’re just going to take it and just sit with it and you’re not going to learn anything. So I do look back and I think that she’s had an amazing influence on me.

Shaloo Garg: And as Charlotte said, my kids, I keep learning from them every single day. They teach me so many things in terms of if I get upset about something, they’ll just say, “Hey mom, just relax. This is just a small thing, just move on.” I think that’s how I keep learning more and more. And of course, amazing coaches and mentors and some really amazing female leaders who I look upon to.

Chloe Condon: I would have to agree. My mother passed away when I was 16, but she was a costume designer, graphic designer, creative arts person, and I try to bring my creative arts training and background into all the technology that I do and create. So I think that was probably the biggest influence on me, would have to be my mom as well.

Chloe Condon: What is the biggest challenge we are facing in tech currently? A tough one.

Charlotte Yarkoni: I actually think our biggest challenge as a society is climate change. I think technology can be a solution for that. So, that’s an indirect answer to a direct question, but I would say that is the thing that I would love to see all of us, I don’t care what you’re doing, where you’re working, but to start having serious thoughts about how we can go reverse decades of adverse effect on the planet. It helps everybody, and I do think the real accelerants are going to lie not just in changing our behavior and our consumption, but also in having technology help us. I don’t think we’ve really gone there yet as a society at large. So for me, it’s something I’m kind of anxious to push along however I can in whatever small way that I can. I think that’s how I think about it.

Charlotte Yarkoni: With technology, you have things like quantum, which is just amazing. The beauty of working somewhere like Microsoft is we are spending a ton of research and we have really crazy people, crazy smart people working on this, and every now and then if I have to go give a talk and I need to give my five minutes of quantum computing update for the cloud, I always ask, “Are there any theoretical physicists in the audience? Because if there are, I’m not going to do this because you know way more than me,” kind of thing.

Chloe Condon: Come on up.

Charlotte Yarkoni: But it’s amazing, and in essence you take what sits in a data center the size of a football field today and you can run it in what’s in the size of a refrigerator in your house. But, the cooling you need to do that is extraordinarily more than the power we’re consuming today, and the impact that will have, by the way, if it’s not done right, either we’re not producing it correctly and/or we’re not cooling it correctly, can have a devastating effect. So how do we think about things like that, these new trends with this aspect of sustainability around the climate, I think is super important. So I apologize, I kind of rambled on that answer, but I actually think this one’s a really important one.

Chloe Condon: I agree. I actually met someone at Open Source Summit recently who works on our IOT team here at Microsoft in Redmond, and his job on the IOT team is to help offset our carbon emissions from our server center. So I thought, “That’s such an important, important way for us to help make the environment a better place with Microsoft.” So, yeah.

Charlotte Yarkoni: Absolutely, and the lady who runs our data centers, her name is Noelle, she’s a peer of mine. I love her dearly. She’s just an amazing woman. She actually grew up as a chemical engineer.

Chloe Condon: Wow.

Charlotte Yarkoni: A lot of her time on how do we run our data centers is spent in areas that you and I wouldn’t know how to go solve, because it is about how do you think about power? How do you think about new sources like geothermal and things like that. I think it’s great. I think it’s great we’re thinking that way, but we got to do more.

Chloe Condon: Yeah.

Shaloo Garg: I think the biggest challenge is the knowledge or the lack of awareness behind power of technology. So, I often see this, I keep bringing up edtech as a very common example, and in fact, here in the Valley, edtech is right now the hottest topic in the social impact circle. I can guarantee you, when I throw the word school out here and I ask you to just close your eyes and think of, tell me what you think of. You’re going to think of a building. You’re going to think of kids running, a blackboard, and a teacher. But that’s not what education is only. Education can be a seven-year-old girl sitting in Uganda who’s not allowed to go to school, but she can sit at home and do schooling at home using an iPad, right? Just because she’s a girl, she’s not allowed to go to school.

Shaloo Garg: That is the power of technology, and it kills me every single day when I read about places like Somalia and Syria, and so many other places, where easily companies, and Microsoft does amazing job, that’s one thing I’m really proud to be, which is be part of this company. We do amazing work globally in enabling this. I think we need to continue to talk about the power of technology, which we do in our jobs and outside our jobs, but we need more and more people to go out there and coach people and say, “Hey guys, education is just not about textbooks. It can be digital education powered by technology.” I think that to me is the biggest challenge right now, which is lack of awareness.

Chloe Condon: Yeah, accessibility and access to that is so important.

Charlotte Yarkoni: Can I interrupt this broadcast? Do we have any recruiters in the audience? Because I think we have our newest recruit. She did an awesome walk-in by the way.

Chloe Condon: Love the pants. Great pants. This is a very fun question. What emoji do you use most often?

Charlotte Yarkoni: I don’t use them correctly, as my children … I always send them stuff–

Chloe Condon: It’s the horse one, right?

Charlotte Yarkoni: … and they’re like, “Why did you send me this? Do you know what this means?” I’m like, “No. No.”

Chloe Condon: I think that’s part of your job as a mom, right?

Charlotte Yarkoni: Well, I have gotten in this habit of sending random ones just to freak my kids out.

Chloe Condon: Love it.

Charlotte Yarkoni: I usually am pretty clean at work with the okay and the goofball face, and the smiley face, but it cracks me up because we were just having this discussion the other day, because I sent something that apparently I shouldn’t have sent as a parent.

Chloe Condon: It’s like a secret hidden emoji language.

Charlotte Yarkoni: It really is.

Chloe Condon: Yeah.

Charlotte Yarkoni: And you, what do you use?

Chloe Condon: I would say it’s a tie between the sobbing emoji and the laugh crying emoji, because I don’t have any other two emotions other than those two extremes. There’s no in between for me. I’m either hysterically laughing or hysterically crying.

Charlotte Yarkoni: What do you use, Shaloo?

Shaloo Garg: Smile and laughter, and that’s it. For the kids, with the kids, I’ll just use hearts, and sometimes my daughter says, “Mom, just stop using those… You’re embarrassing me, mom.”

Chloe Condon: Yeah. What are the most important decisions you face every day? Or what is the most important decision you face every day?

Shaloo Garg: How to make founders successful, and especially in a market like this. I just love it. It’s an upstream market, constantly challenging ourselves. What else can we do? What else can we do in this market? I absolutely love it. It is challenging. It’s extremely challenging.

Chloe Condon: It’s a huge question.

Shaloo Garg: It’s a huge question. I’ve been with the company for eight months and when I joined initially, I was a bit nervous. I was like, “Great, I’m so excited about this job,” and when I went out there, talked to founders, everyone was like, everyone gave me a standard response, “Well, yeah, okay.” But now slowly and slowly we’ve started building it as part of the narrative that we haven’t only the meetings, which is how do we help the founders, and if we switched that, our jobs become much more easier, which is, “I’m here to help you and this is how I can help you.” So I think that to me is absolutely the most fun part.

Chloe Condon: Yeah.

Charlotte Yarkoni: By the way, as part of my team, that’s a great answer for these little startups. I think my job is really making the set of decisions that best serve our customers, our partners, best serve the team. It’s always a balance, right? We have so much we’ve got to get done. We love innovating, we love getting new capabilities out there, making sure that we’re doing that with the right sense of urgency and the right balance for the teams delivering them. Most of my day, in any one of my teams that I look at, is just making the right calls to make sure that we’re doing right by the community, as both our community that’s working on it and the communities we’re trying to serve.

Chloe Condon: Yeah. I would say for me it’s how to get people excited to learn, and what is going to get them having fun. Because I think we work all day, we work like an eight-hour plus day sometimes in front of machines using technology, and what are fun creative ways to get people excited about that and to build really cool, amazing things together that can solve these big questions and problems like the environment and getting accessibility to folks who don’t have the access to this technology. So, it’s always fun to enable that power to people.

Chloe Condon: How much time do we have? Do we want to do maybe one or two more questions? One more question. Okay, cool. Let’s see. I think this is a really good … Actually, I would love to end with your advice to all of our amazing women in this audience, and men in the audience. What would be your advice to someone who’s looking to move up in their career and have a successful career as a person in tech?

Charlotte Yarkoni: I think being you is the most important part. Whatever that means, right? Just be your most authentic self. It’s a hard thing to do. It’s a hard thing in our industry. It’s a hard thing in super competitive environments like here in San Francisco. Seattle is very similar in that regard. I have found people get the most reward and have the most success when they’re actually themselves, whatever that means. I also think being the authentic you will not just make you better, it will actually make whatever team you’re on better. It will make whatever company you’re at better, it will make whatever product or service you’re working on better. Just be you and be proud to be you.

Chloe Condon: I love that.

Shaloo Garg: So, I would say do what you’re passionate about because when you’re passionate, you bring your best. Do not be afraid to take risk, and I know this sounds like a cliche, but really challenge yourself. If there is a risk, if you want to do something and it looks very risky, just go ahead and do it. Maximum, you’re going to fail, but you’ll learn something from it. If you come out victorious, that’s great. Then the last thing I would say is just trust yourself and just believe in your instinct that you’re doing good for the business, you’re doing good for the company, you’re also doing good for those startups or customers or whoever your stakeholders are, and just go chase it. If you keep it straight and if you keep what I call the compass straight, there’s going to be lots of amazing learning in the process.

Chloe Condon: My advice is actually a great segue into our mingling and happy hour section. Mine would be to talk to as many people as you can in this industry. If you have the opportunity to get coffee with someone you really idolize or a mentor, or someone who’s doing what you want to be doing in this industry, having conversations, I think, is so wonderful and you are all about to use that LinkedIn feature that I just taught you, and meet some really amazing people. So make connections and network and yeah, have the most amazing time.

Chloe Condon: I want to thank both of our…

Shaloo Garg: Thank you.

Chloe Condon: … panelists today. Round of applause for Shaloo and Charlotte.

Charlotte Yarkoni: Thank you for hosting.

Chloe Condon: Of course. Thank you to to Kitty. Thank you to Priyanka. Thank you to everyone, to Kaitlyn who’s not here, but oh my gosh, that amazing, amazing musical performance we had to start off the evening. Please, enjoy yourselves. I think we still have some beverages and snacks here, so have a wonderful time. Make sure you get some swag and stickers and we will be around to chat. All right. Thanks everyone.

Microsoft girl geeks, Microsoft Reactor fun

Microsoft girl geeks and allies: Thank you to all the Redmond, San Francisco and Silicon Valley teams who worked together to make this happen!   Erica Kawamoto Hsu / Girl Geek X

Kitty Yeung Microsoft Girl Geek Dinner

Microsoft Garage Manager Kitty Yeung is a creative technologist with a skirt that lights up when she spins.  Erica Kawamoto Hsu

girl geek experiencing Microsoft mix reality

Principal Program Manager Lead Jane Fang and SF Academy Head of Marketing Jo Ryall demo “Mix Reality” to a girl geek  at Microsoft Girl Geek Dinner.   Erica Kawamoto Hsu / Girl Geek X


Our mission-aligned Girl Geek X partners are hiring!

Girl Geek X OpenAI Lightning Talks and Panel (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Gretchen DeKnikker, Sukrutha Bhadouria

Girl Geek X team: Gretchen DeKnikker and Sukrutha Bhadouria kick off the evening with a warm welcome to the sold-out crowd to OpenAI Girl Geek Dinner in San Francisco, California.   Erica Kawamoto Hsu / Girl Geek X

Transcript of OpenAI Girl Geek Dinner – Lightning Talks & Panel:

Gretchen DeKnikker: All right, everybody, thank you so much for coming tonight. Welcome to OpenAI. I’m Gretchen with Girl Geek. How many people it’s your first Girl Geek? All right, okay. Lots of returning. Thank you for coming. We do these almost every week, probably like three out of four weeks a month. Up and down the peninsula, into the South Bay or everywhere. We also have a podcast that you could check out. Please check it out, find it, rate it, review it. Give us your most honest feedback because we’re really trying to make it as awesome as possible for you guys. All right.

Sukrutha Bhadouria: Hi, I’m Sukrutha. Welcome, like Gretchen said, Angie’s not here but there’s usually the three of us up here. Tonight, please tweet, share on social media, use the hashtag GirlGeekXOpenAI. I also, like Gretchen, want to echo that we love feedback, so any way you have anything that you want to share with us. Someone talked about our podcast episodes today. If there’s any specific topics you want to hear, either at a Girl Geek Dinner or on our podcast, do share that with us. Either you can find us tonight or you can email us. Our website is girlgeek.io and all our contact information’s on there. Thank you all. I don’t want to keep you all waiting because we have amazing speakers lined up from OpenAI, so.

Sukrutha Bhadouria: Oh, one more quick thing. We’re opening up sponsorship for 2020 so if your company has not sponsored a Girl Geek dinner before or has and wants to do another one, definitely now’s the time to sign up because we fill up pretty fast. We don’t want to do too many in one month. Like Gretchen said, we do one every week so definitely would love to see a more diverse set of companies–continue to see that like we did this year. Thank you, all. Oh, and over to Ashley.

Ashley Pilipiszyn speaking

Technical Director Ashley Pilipiszyn emcees OpenAI Girl Geek Dinner.   Erica Kawamoto Hsu / Girl Geek X

Ashley Pilipiszyn: All right, thank you.

Sukrutha Bhadouria: Thanks.

Ashley Pilipiszyn: All right. Hi, everybody.

Audience: Hi.

Ashley Pilipiszyn: Oh, awesome. I love when people respond back. I’m Ashley and welcome to the first ever Girl Geek Dinner at OpenAI. We have a … Whoo! Yeah.

Ashley Pilipiszyn: We have a great evening planned for you and so excited to see so many new faces in the crowd but before we get started, quick poll. How many of you currently work in AI machine learning? Show of hands. All right, awesome. How many of you are interested in learning more about AI machine learning? Everybody’s hands should be up. All right. Awesome. We’re all on the right place.

Ashley Pilipiszyn: Before we kick things off, I’d like to give just a brief introduction to OpenAI and what we’re all about. OpenAI is an AI research lab of about 100 employees, many of whom you’re going to get to meet this evening. Definitely, come talk to me. Love meeting you. We’ve got many of other folks here, and our mission is to ensure that safe, artificial general intelligence benefits all of humanity.

Ashley Pilipiszyn: To that effect, last year we created the OpenAI Charter. The charter is our set of guiding principles as we enact this mission and serves as our own internal system of checks and balances to hold ourselves accountable. In terms of how we organize our research, we have three main buckets. We have AI capabilities, what AI systems can do. We have AI safety, so ensuring that these systems are aligned with human values. We have AI policy, so ensuring proper governance of these systems.

Ashley Pilipiszyn: We recognize that today’s current AI systems do not reflect all of humanity and we aim to address this issue by increasing the diversity of contributors to these systems. Our hope is that with tonight’s event, we’re taking a step in the right direction by connecting with all of you. With that, I would like to invite our first speaker to the stage, Brooke Chan. Please help me welcome Brooke.

Brooke Chan speaking

Software Engineer Brooke Chan from the Dota team gives a talk on reinforment learning and machine learning at OpenAI Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Brooke Chan: Yeah. Hello. Is this what I’m using? Cool. I’m Brooke Chan. I was a software engineer on the Dota 2 team here at OpenAI for the past two years. Today, I’m going to talk a little bit about our project, as well as my own personal journey throughout the course of the project.

Brooke Chan: We’re going to actually start at the end. On April 13th, we hosted the OpenAI Five Finals where we beat the TI8 world champions OG at Dota 2 in back-to-back games on stage. TI stands for The International, which is a major tournament put on by Valve each year with a prize pool upwards of $30 million. You can think of it like the Super Bowl but for Dota.

Brooke Chan: There have been previous achievement/milestones of superhuman AI in both video games and games in general, such as chess and Go, but this was the first AI to beat the world champions at an eSports game. Additionally, as a slightly self-serving update, OG also won the world championship this year at TI9 just a few weeks ago.

Brooke Chan: Finals wasn’t actually our first unveiling. We started the project back in January of 2018 and by June of 2018, we started playing versus human teams. Leading up to finals, we played progressively stronger and stronger teams, both in public and in private. Then most recently, right before finals, we actually lost on stage to a professional team at TI8, which was the tournament that OG later went on to win.

Brooke Chan: Let’s go back to the basics for a minute and talk about what is reinforcement learning. Essentially, you can think of it as learning through trial and error. I personally like to compare it to dog training so that I can show off pictures of my dog. Let’s say that you want to teach a dog how to sit, you would say sit and just wait for the dog to sit, which is kind of a natural behavior because you’re holding a treat up over their head so they would sit their butt down and then you would give them that treat as a reward.

Brooke Chan: This is considered capturing the behavior. You’re making an association between your command, the action and the reward. It’s pretty straightforward for simple behaviors like sit but if you want to teach something more complicated, such as like rolling over, you would essentially be waiting forever because your dog isn’t just going to roll over because it doesn’t really understand that is something humans enjoy dogs doing.

Brooke Chan: In order to kind of teach them this, you instead reward progress in the trajectory of the goal behavior. For example, you reward them for laying down and then they kind of like lean over a little bit. You reward them for that. This is considered to be shaping rewards. You’re like teaching them to explore that direction in order to achieve ultimately your goal behavior.

Brooke Chan: Dota itself is a pretty complicated game. We can’t just reward it by purely on winning the game because that would be relatively slow so we applied this technique of shaped rewards in order to teach the AI to play the game. We rewarded it for things like gold and kills and objectives, et cetera. Going more into this, what is Dota?

Brooke Chan: Dota is a MOBA game which stands for multiplayer online battle arena. It’s a little bit of a mouthful. It’s a game that was developed by Valve and it has an average of 500,000 people playing at any given time. It’s made up of two teams of five and they play on opposite sides of the map and each player controls what’s considered a hero who has a unique set of abilities.

Brooke Chan: Everyone starts off equally weak at the beginning of the game, which means that they’re low levels and they don’t have a lot of gold and the goal is that over the course of a 30 to 60-minute game, they earn gold and become stronger and eventually, you destroy your opponent’s base. You earn gold and experience across the map through things like small fights or like picking people off, killing your enemy, taking objectives, things like that. Overall, there’s a lot of strategy to the game and a lot of different ways to approach it.

Brooke Chan: Why did we pick Dota? MOBAs in general are considered to be one of the more complex video games and out of that genre, Dota is considered the most complex. Starting off, the games tend to be pretty lengthy, especially in terms of how RL problems typically are, which means that strategy tends to be hard with a pretty delayed payoff. You might rotate into a particular lane in order to take an objective that you might not be able to take until a minute or a minute and a half later. It’s something that’s kind of like hard to associate your actions with the direct rewards that you end up getting from them.

Brooke Chan: Additionally, as opposed to games like Go and chess, Dota has partial information to it, which means that you only get vision around you and your allies. You don’t have a full state of the game. You don’t know where your enemies are and this leads to more realistic decision-making, similar to our world where you can’t like see behind walls. You can’t see beyond what your actual vision gives you.

Brooke Chan: Then, finally, it has both a large action and observation space. It’s not necessarily solvable just by considering all the possibilities. There’s about 1,000 actions that you can take at any given moment and the state you’re getting back has the value size of about 20,000. To put it in perspective, on average, your game of chess takes about 40 moves and Go takes about 150 moves and Dota is around 20,000 moves. That means that the entire duration of a game of chess really wouldn’t even get you out of the base in Dota.

Brooke Chan: This is a graph of our training process. On the left, you have workers that all play the game simultaneously. I know it’s not super readable but it’s not really important for this. Each game that they’re playing in the top left consists of two agents where an agent is considered like a snapshot of the training. The rollout workers are dedicated to these games and the eval workers who are on the bottom left are dedicated to testing games in between these different agents.

Brooke Chan: All the agents at the beginning of the training start off random. They’re basically picking their actions randomly, wandering around the map doing really awfully and not actually getting any reward. The machine in green is what’s called the optimizer so it parses in all of these rollout worker games and figures out how to update what we call the parameters which you can consider to be the core of its decision-making. It then passes these parameters back into the rollout workers and that’s how you create these continually improving agents.

Brooke Chan: What we do then is we take all of these agents and we play them against all the other agents in about 15,000 games in order to get a ranking. Each agent gets assigned a true skill, which is basically a score calculated on its win-loss records against all the other agents. Overall, in both training and evaluation, we’re really not exposing it to any kind of human play. The upside of this is that we’re not influencing the process. We know that they’re not just emulating humans and we’re not capping them out at a certain point or adding a ceiling on it based on the way that humans play.

Brooke Chan: The downside of that is that it’s incredibly slow. For the final bot that we had play against OG we calculated that it had about 45,000 years of training that went into it. Towards the end of training, it was consuming about approximately 250 years of experience per day. All of which we can really do because it’s in simulation and we can do it both asynchronously and sped up.

Brooke Chan: The first time they do get exposed to human play is during human evaluations. They don’t actually learn during any of these games because we are taking an agent, which is a snapshot and frozen in time and it’s not part of the training process. We started off playing against our internal team and our internal team was very much not impressive. I have us listed as 2K MMR, which is extremely generous. MMR means matchmaking rating which is a score that Valve assigns to the ranked play. It’s very similar to true skill. 2K is very low.

Brooke Chan: We were really quickly surpassed. We then moved on to contract teams who were around like 4K-6K MMR and they played each week and were able to give us feedback. Then in the rare opportunities, we got to play against professional teams and players. Overall, our team knew surprisingly little about Dota. I think there are about four people on our team who had ever played Dota before and that’s still true post-project, that no one really plays Dota.

Brooke Chan: This leads us to our very surprising discovery that complicated games are really complicated and we dug ourselves into this hole. We wanted a really complicated game and we definitely got one. Since the system was learning in a completely different way than humans, it became really hard to interpret what it was actually trying to do and not knowing what it was trying to do mean we didn’t know if it was doing well, if it was doing poorly, if it was doing the right thing. This really became a problem that we faced throughout the lifetime of our project.

Brooke Chan: Having learned this, there was no way to really ask it what it was thinking. We had metrics and we could surface like stats from our games but we were always leveraging our own intuition in order to interpret what decisions it was making. On the flip side, we also had human players that we could ask, but it turned out it was sometimes tough to get feedback from human players.

Brooke Chan: Dota itself is a really competitive game, which means that its players are very competitive. We got a lot of feedback immediately following games, which would be very biased or lean negatively. I can’t even count the number of times that a human team would lose maybe like, “Oh, this bot is terrible” and I was like, “Well, you lost. How is it terrible? What is bad about it?” This would create this back and forth that led to this ultimate question of is it bad or is it just different? Because, historically, humans have been the source on how to play this game. They make up the pro scene, they make up the high skill players. They are always the ones that you are going to learn from. The bots would make a move and the humans say it was different and not how the pros play and therefore, it’s bad. We always had to take the human interpretation with this kind of grain of salt.

Brooke Chan: I want to elaborate a little bit more about the differences because it goes just beyond the format of how they learn. This game in general is designed to help humans understand the game. It has like tooltips, ability descriptions, item descriptions, et cetera. As an example, here’s a frozen frame of a hero named Rana who’s the one with the bright green bar in the bottom left. She has an ability that makes you go invisible and humans understand what being invisible means. It means people can’t see you.

Brooke Chan: On the right, what we see is where we have like what the AI sees and it’s considered their observation space, it’s our input from the game. We as engineers and researchers know that this particular value is telling you whether or not you’re invisible. When we hit this ability, you can see that she gets like this little glow to her which indicates that she’s invisible and people understand that. The AI uses this ability and sees that the flag that we marked as invisible goes from 0 to 1 but they don’t see the label for that and they don’t really even understand what being invisible means.

Brooke Chan: To be honest, learning invisibility is not something trivial. If you’re walking down the street and all of a sudden, you were invisible, it’s a little bit hard to tell that anything actually changed. If you’ve ever seen Sixth Sense, maybe there’s some kind of concept there, but additionally, at the same time, all these other numbers around it are also changing due to the fact that there’s a lot of things happening on the map at once.

Brooke Chan: Associating that invisibility flag, changing directly to you, activating the ability is actually quite difficult. That’s something that’s easy for a human to do because you expect it to happen. Not to say that humans have it very easy, the AI has advantages too. The AI doesn’t have human emotions like greed or frustration and they’re always playing at their absolute 100% best. They’re also programmatically unselfish which is something that we did. We created this hyper parameter called team spirit which basically says that you share your rewards with your buddy. If you get 10 gold or your buddy gets 10 gold, it’s totally interchangeable. Theoretically, in a team game, that should be the same case for humans but inherently, it’s not. People at its core are going to play selfishly. They want to be the carrier. They want to be winning the game for the team.

Brooke Chan: All these things are going to influence pretty much every decision and every behavior. One pretty good example we have of this is called buybacks. Buybacks is a mechanic where when you die in the game, you can pay money in order to immediately come back to life and get back on the map. When we first enabled the AI to do this, there was a lot of criticism that we got. People were saying, “Oh, that’s really bad. They shouldn’t be wasting all their money” because the bots would always buy back pretty much immediately.

Brooke Chan: Over time, we continue doing this behavior and people kept saying, “Oh, that’s bad. You should fix it.” We’re like, “Well, that’s what they want to do.” Eventually, people started seeing it as an advantage to what we had, as an advantage to our play style because we were able to control the map. We were able to get back there very quickly and we were able to then force more fights and more objectives from it.

Brooke Chan: As a second self-serving anecdote, at TI9, there were way more buybacks way earlier and some people pointed this out and maybe drew conclusions that it was about us but I’m not actually personally going to make any statement. But it is one example of the potential to really push this game forward.

Brooke Chan: This is why it was difficult to have human players give direct feedback on what was broken or why because they had spent years perfecting the shared understanding of the game that is just like inherently different than what the bots thought. As one of the few people that played Dota and was familiar with the game and the scene, in the time leading up to finals, this became my full-time job. I learned to interpret the bot and how it was progressing and I kind of lived in this layer between the Dota community and ML.

Brooke Chan: It became my job to figure out what was most critical or missing or different about our playstyle and then how to convert that into changes that we could shape the behavior of our bot. Naturally, being in this layer, I also fell into designing and executing all of our events and communication of our research to the public and the Dota community.

Brooke Chan: In designing our messaging, I had the second unsurprising discovery that understanding our project was a critical piece to being excited about our results. We could easily say, “Hey, we taught this bot to learn Dota” and people would say, “So what? I learned to play Dota too. What’s the big deal?” Inherently, it’s like the project is hard to explain because in order to understand it and be as excited as we were, you had to get through both the RL layer which is complicated, and the Dota layer which is also complicated.

Brooke Chan: Through planning our events, I realized this was something we didn’t really have a lot of practice on. This was the first time that we had a lot of eyes on us belonging to people with not a lot of understanding of reinforcement learning and AI. They really just wanted to know more. A lot of our content was aimed at people that came in with the context and people that were already in the field.

Brooke Chan: This led me to take the opportunity to do a rotation for six months on the communications team actually working under Ashley. I wanted to be part of giving people resources to understand our projects. My responsibilities are now managing upcoming releases and translating our technical results to the public. For me, this is a pretty new and big step. I’ve been an engineer for about 10 years now and that was always what I loved doing and what I wanted to do. But experience on this team and growing into a role that didn’t really exist at the time allowed me to tackle other sorts of problems and because that’s what we are as engineers at the core, we want to be problem solvers.

Brooke Chan: That’s kind of my takeaway and it might seem fairly obvious but sometimes deviating from your path and taking risks let you discover new problems to work on. They do say that growth tends to be at the inverse of comfort so that means that the more you push yourself out of your comfort zone and what you’re used to, the more you give yourself opportunities for new challenges and discovering new skills. Thank you.

Lilian Weng

Research Scientist Lilian Weng on the Robotics team gives a talk on how her team uses reinforcement learning to learn dexterous in-hand manipulation policies at OpenAI Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Lilian Weng: Awesome. Cool. Today, I’m going to talk about some research projects with that at OpenAI robotics team. One big picture problem at our robotics team is to develop the algorithm to power general-purpose robots. If you think about how we humans are living this world, we can cook, we lift to move stuff, we add some more items with different tools. We fully utilize our body and especially our hands to do a variety of tasks. To some extent, we are general-purpose robots, okay?

Lilian Weng: That’s, we apply the same standard to our definition of such a thing. A general-purpose robot should be able to interact with a very complicated environment of the real world and able to manipulate all kinds of objects around it. However, unfortunately, most consumer-oriented robots nowadays are either just toys or very experimental or focus on specific functionalities and they are robots like factory arms or medical robots. They can interact with the environment and operating tools but they’re really operated by humans so human controls every move or they just play back a pre-programmed trajectory. They don’t really understand the environments and they cannot move autonomously.

Lilian Weng: In our projects, we’re taking a small step towards this goal and in this we try to teach a human-like robot hand to do in-hand manipulation by moving the objects. This is a six-phase block with OpenAI letters on it, move that to a target orientation. We believe this is an important problem because a human-like robot hand, it’s a universal effort. Imagine we can control that really well, we can potentially automate a lot of tasks that are currently done by human. Unfortunately, not a lot of progress have been made on human-like robot hand due to the complexity of such a system.

Lilian Weng: Why it is hard? Okay. First of all, the system has very high dimensionalities. For example, in our robot, which is as you can see this cool illustration. Shadow dexterity hand, it has 24 joints and 20 actuators. The task is especially hard because during the manipulation, a lot of observations are occluded and they can be noisy. For example, your sensor reading can be wrong and your sensor reading can be blocked by the object itself. Moreover, it’s virtually impossible to simulate your physical world 100% correctly.

Lilian Weng: Our approach for tackling this problem is to use reinforcement learning. We believe it is a great approach for learning how to control robots given that we have seen great progress and great success in many applications by reinforcement learning. You heard about OpenAI Five, the story of point AlphaGo and it will be very exciting to see how reinforcement learning can not only interact with this virtual world but also have an impact on our physical reality.

Lilian Weng: There is one big drawback of reinforcement learning model. In general, today, most of the models are not data efficient. You need a lot of training sample in order to get a good model trained. One potential solution is you build a robot farm. You just collect all the data in parallels with hundreds of thousands of robots but imagine just given how fragile a robot can be. It is very expensive to build and maintain. If you think of another problem, a new problem, or you want to work with new robots, it’s very hard to change. Furthermore, your data can get invalidated very quickly due to small changes in your robot status.

Lilian Weng: As that, we decided to take the sim2real approach, that is you train your model every single simulation but deploy that on physical robots. Here shows how we control the hand simulation. The hand is moving the object to a target orientation. The target is shown on the right so whenever the hand achieved the goal, we just sample a new goal. It just keeps on doing that and we cap the number of success at 50.

Lilian Weng: This is our physical setup. Everything is mounted in this giant metal cage. It’s like this big. The hand is mounted in the middle. It’s surrounded with a motion caption system. It’s actually the system that people use for filming special effects films, like the actor has dots on their bodies, kind of similar. This system tracks the five fingertip positions in the 3D space. We also have three high-resolution cameras for capturing images as input to our vision model. Our vision model predicts positional orientation of the block. However, our proposal sim2real approach might fail dramatically because there are a lot of model difference between simulation and reality. If your model all refer to the simulation, it can perform super poorly, the real robots.

Lilian Weng: In order to overcome this problem, we decided to take … we use reinforcement learning, okay. We train everything simulations so that we can generate technically, theoretically infinite amount of data. In order to overcome the sim2real difference, we use domain randomization.

Lilian Weng: Domain randomization refer to an idea of randomized different elements in simulation so that your policy can be exposed to a variety of scenarios and learn how to adapt. Eventually, we expand the policy to able to adapt to the physical reality.

Lilian Weng: Back in … This idea is relative news. I think they first proposed it in 2016. The researchers try to train a model to control drone like fly across furnitures or the indoor scenarios. They randomized the colors and texture of the walls and furnitures and without seeing any real-world images, they show that it performs pretty well in reality.

Lilian Weng: At OpenAI, we use the same approach to train a better model to protect the position orientation of the objects. As you can see some of the randomization looks totally unrealistic but somehow it worked very well when we feed the model with real images. Later, we also showed that you can randomize all the physical dynamics in simulations and this robot trained with domain randomization worked much better than the one without.

Lilian Weng: Let’s see the results. Okay. I’m going to click the … You really struggle a little bit at the first goal. Yes, okay. The ding indicates one success. This video will keep on going until goal 50 so it’s very, very long but I personally found it very soothing to look at it. I love it.

Lilian Weng: I guess that’s enough. This is our full setup of the training so in the box A, we generate a large number of environments in parallels in which we randomize the physical dynamics and the visual appearance. Based on those, we train two models independently. One is a policy model which takes in the fingertip position and object pose and the goal and output, a desired joint position of the hand so that we can control the hand. Another model is the vision that takes in three images from different camera angles and output the position orientation of the object.

Lilian Weng: When we deploy this thing into the real world, we combine the vision prediction based on the real images together with a fingertip position tracked by the motion capture system and feed that into our policy control model and output action so that then we just send it to the real robot and everything starts moving just like the movie shown. When we train our policy control model, we’ve randomized all kinds of physical parameter in the simulator such as masses, friction coefficient, motor gain, damping factor, as well as noise on the action, on observation. For a revision model, we randomized camera position, lighting, material, texture, colors, blah, blah, blah, and it just worked out.

Lilian Weng: For our model’s architecture, I’ll just go very quickly here. The policy, it’s a pretty simple recurrent unit. Has one layer of really connective layer and the LSTM. The vision model is a straightforward, multi-camera setup. All the three cameras share this RestNet stack and followed by a spatial softmax.

Lilian Weng: Our training framework is distributed and synchronized PBO, proximal policy optimization model. It’s actually the same framework used for training OpenAI Five. Our setup allowed us to generate about two years simulated experience per hour, which corresponds to 17,000 physical robots, so the gigantic robot factory and simulation is awesome.

Lilian Weng: When we deploy our model in reality, we noticed a couple of strategies learned by the robot like finger pivoting, sliding, finger gaiting. Those were also commonly used by human and interestingly, we never explicitly give it words or encouraged those strategies. They would just emerge autonomously.

Lilian Weng: Let’s see some numbers. In order to compare different versions of models, we deployed the models on the real robots and count how many successes the policy can get up to 50 before it dropped the block or time out. We first tried to deploy a model without randomization at all. It got a perfect performance in simulation but look, you can see it’s zero success median. Super bad on the real robot.

Lilian Weng: Then we’re adding domain randomization. The policy becomes much better because 13 success medians, maximum 50. Then we used RGB cameras in our vision model to track the objects. The performance only dropped slightly, still very good. The last one, I think this one’s very interesting because I just mentioned that our policies are recurrent units so like LSTM, it has internal memories.

Lilian Weng: Well, interesting, see how important this memory is so we replaced this LSTM policy with a FIFO or NAS and deployed that on robot and the performance dropped a lot, which indicates that memory play an important role in the sim2real transfers. Potentially, the policy might be using the memory and try to learn how to adapt.

Lilian Weng: However, training in randomized environments does come with a cost. Here we plot the number of success in simulation as a function of simulated experiencing measured in year. If you don’t apply randomization at all, the model can learn to achieve 40 success with about three years simulated experience but in order to get to same number like 40 success in a fully randomized environment took 100 years.

Lilian Weng: Okay, to quick summary. We’ve shown that this approach, reinforcement learning plus training simulation plus domain randomization worked on the real robot and we would like to push it forward. Thank you so much. Next one is Christine.

Christine Payne speaking

Research Scientist Christine Payne on the Music Generation team gives a talk on how MuseNet pushes the boundaries of AI creativity, both as an independent composer, and as a collaboration tool with human artists.  Erica Kawamoto Hsu / Girl Geek X

Christine Payne: Thank you. Let’s see. Thank you. It’s really great to see all of you here. After this talk, we’re going to take a short break and I’m looking forward to hopefully getting to talk to a lot of you at that point. I’ve also been especially asked to announce that there are donuts in the corner and so please help us out eating those.

Christine Payne: If you’ve been following the progress of deep learning in the past couple years, you’ve probably noticed that language generation has gotten much, much better, noticeably better in the last couple of years. But as a classical pianist, I wondered, can we take the same progress? Can we apply instead to music generation.

Christine Payne: Okay, I’m not Mira. Sorry. Hang on. One moment, I think we’re on the wrong slide deck. All right, sorry about that. Okay, trying again. Talking about music generation. You can imagine different ways of generating music and one way might be to do a programmatic approach where you say like, “Okay, I know that drums are going to be a certain pattern. Harmonies usually follow a certain pattern.” You can imagine writing rules like that but there’s whole areas of music that you wouldn’t be able to capture with that. There’s a lot of creativity, a lot of nuance, the sort of things that you really want a neural net to be able to capture.

Christine Payne: I thought I would dive right in by playing a few examples of MuseNet, which is this neural net that’s been trained on this problem of music generation. This first one is MuseNet trying to imitate Beethoven and a violin piano sonata.

Christine Payne: It goes on for a while but I’ll cut it off there. What I’m really trying to go with in this generation process is trying to get long-term structure so both the nuance and the intricacies of the pieces but also something that stays coherent over a long period of time. This is the same model but instead trying to imitate jazz.

Christine Payne: Okay, and I’ll cut this one off too. As you maybe could tell from those samples, I am more interested in the problem of composing the pieces themselves, so sort of where the notes should be and less in the actual quality of the solemnness and the timbre. I’ve been using a format that’s called MIDI which is an event-based system of writing music. It’s a lot like how you would write down notes in a music score. Like this note turns on at this moment in time played by this instrument maybe at this volume but you don’t know like this amazing cellist actually made it sound this way so I’m throwing out all of that kind of information.

Christine Payne: But the advantage of throwing that out is then you can get this longer-term structure. To build this sort of dataset, it involves a little bit of begging for data. I’ve had a bunch of people like BitMidi and ClassicalArchives were nice enough to just send me their collections and then a little bit of scraping and also MAESTRO’s Google Magenta’s dataset and then also a bunch of scraping online sets.

Christine Payne: The architecture itself, here I’m drawing really heavily from the way we do language modeling and so we use a specific kind of neural net that’s called a transformer architecture. The advantage of this architecture is that it’s specifically good at doing long-term structure so you’re able to look back not only at things that have happened in the recent past but really, you can look back like what happened in the music a minute ago or something like that, which is not possible with most other architectures.

Christine Payne: In the language world, I’d like to think of this, the model itself is trained on the task of what word is going to come next. It might initially see just like a question mark so it knows it’s supposed to start something. In English, we know like maybe it’s the or she or how or some like that. There’s some good guesses and there’s some like really bad guesses. If we know now the first word is hello then we’ve kind of narrowed down what we expect our next guesses should be. It might be how, it might be my, it’s probably not going to be cat. Maybe it could be cat. I don’t know.

Christine Payne: At this point, we’re getting pretty sure–like a trained model should actually be pretty sure that there should be a good 90% chance the next word is name and now it should be like really 100% sure or like 99.5% sure or whatever that the next word is going to be is. Then here we hit kind of an interesting branching point where there are tons of good answers so lots of names could be great answers here, lots of things could also be really bad answers so we don’t expect to see like some random verbs, some random … There are lots of things that we think would be bad choices but we get a point here to branch in good directions.

Christine Payne: The idea is once you have a model that’s really good at this, you can then turn it into a generator by sampling from the model according to those probabilities. The nice thing is you get the coherent structure. When you get a moment like this, you know like I have to choose … In music, it’s usually like I have to choose this rhythm, I have to choose … like if I choose the wrong note, it’s just going to sound bad, things like that. But then there are also a lot of points like this where the music can just go in fun and interesting different directions.

Christine Payne: But of course, now we have the problem of how do you translate words, how do you translate this kind of music into a sequence of words that the model can do. The system that I’m using is very similar to how MIDI itself works. I have a series of tokens that the model will always we see. Initially, it’ll always see the composer or the band or whoever wrote the piece. It’ll always see what instrument to expect in the piece or what set of instruments.

Christine Payne: Here, it sees the start token because it’s at the start of this particular piece and a tempo. Then as the piece begins, we have a symbol that this C and that C each turn on with a certain volume and then we have a token that says to wait a certain amount of time. Then as it moves forward, the volume zero means that first note just turned off and the G means the next note turns on. I think we have to wait and similarly, here the G turns off, the E turns on and we wait. You can just progress through the whole set of music like this.

Christine Payne: In addition to this token by token thing, I’m helping the model out a little bit by giving it a sense of the time that’s going on. I’m also giving it an extra embedding that says everything that happens in this purple line happens in the same amount of time or at the same moment in time. Everything in blue is going to get a different embedding that’s a little bit forward in time and so forth.

Christine Payne: The nice thing about an embedding or a system like this is that it’s pretty dense but also really expressive. This is the first page of a Chopin Ballade that is like actually encapsulates how the pianist played it, the volumes, the nuances, the timings, everything like that.

Christine Payne: The model is going to see that sequence of numbers like that. Like that first 1444 I think means it must mean Chopin and the next one probably means piano and the next one means start, that sort of thing. The first layer for the model, what it has to do is it needs to translate that number into a vector of numbers and then it can sort of learn a good vector that’ll represent so it’ll get a sense of like this is what it means to be Chopin or this is what it means to be like a C on a piano.

Christine Payne: The nice thing you can do once … The model will learn. Like initially it starts out with a totally random sense so it has no idea what those numbers should be but in the course of training, it’ll learn better versions of that. What you can do is you can start to map out what it’s learned for these embeddings. For example, this is what it’s learned for a piano scale like all the notes on a piano and it’s come to learn that like all of these As are kind of similar, that the notes relate to each other. This is like moving up on a piano. It’s hard to tell here but it’s learned little nuances like up a major third is closer than like up a tritone or stuff like that. Like actually really interesting musical stuff.

Christine Payne: Along with the same thing, given the fact that I’m always giving it this genre token and then the instrument token, you can look at the sort of embeddings it’s learned for the genres itself. Here, the embedding it’s learned for all these French composers. Ends up being pretty similar. I actually like that Ravel wrote like in the style of Spanish pieces and then there’s the Spanish composer that’s connected to him so like it makes a lot of good sense musically. Similarly, like over in the jazz domain, a lot of the ones. I think there are a couple of random ones that made no sense at all. I can’t remember now off the top of my head. It’s like Lady Gaga was connected to Wagner or something like but mostly, it made a lot of great sense.

Christine Payne: The other kind of fun thing you can do once you have the style tokens is you can try mismatching them. You can try things like literally taking 0.5 of the embedding for Mozart plus 0.5 of the embedding of jazz and just like adding them together and seeing what happens or in this case what I’m doing is I’m giving it the token for Bon Jovi, instruments for bands, but then I’m giving it the first six notes of a Chopin Nocturne. Then the model just has to generate as best it can at that point.

Christine Payne: You’ll hear at the start of this, it’s very much how the Chopin Nocturne itself sounds. I’ve cut off the very, very beginning of it but you’ll hear–so that left-hand pattern is going to be like straight out of Chopin and then well, you’ll see what happens.

Christine Payne: Sorry, it’s so soft but it gets very Bon Jovi at this point, the band kicks in. I always loved it like Chopin looks a little shocked but I really love that it manages to keep the left-hand pattern of the Nocturne going even though it’s like now thinks it’s in this pop sort of style.

Christine Payne: The other thing I’ve been interested in this project is in how musicians and everyone can use generators like this. If you go to our OpenAI blog you can actually play with the model itself. We’ve created, along with Justin and Eric and Nick, a sort of prototype tool of how you might co-compose pieces using this model. What you can do is you can specify the style and the instruments, how long a segment you want the model to generate and you hit start and the model will come back with four different suggestions of like how you might begin a piece in this style. You go through and you pick your favorite one, you hit the arrow again to keep generating and the model will come up with four new different ways. You can continue on this way as long as you want.

Christine Payne: What I find kind of fun about this is you’re actually really … like it feels like I’m composing but not at a note by note level and so I was really interested in how humans will be able to, and musicians will be able to guide composing this way. Just kind of wrapping up, I thought I would play an example of … This is one guy who took both GPT-2 to write the lyrics, which I guess is hence the Covered in Cold Feet and then MuseNet to do the music. It’s a full song but I’ll just play the beginning of it that he then recorded himself.

Christine Payne: (singing)

Christine Payne: Visit the page to hear the whole song but it’s been really fun to see those versions. The song, I ended up singing it the entire day. It gets really catchy but it’s been really fun to see musicians start to use it. People have used it to finish composing symphonies or to write full pieces, that sort of thing.

Christine Payne: In closing, I just wanted to share I’ve gone through this crazy path of two years ago being a classical pianist to now doing AI research here and I just wanted to … I didn’t know that Rachel was going to be right here. Give a shout out to fast.ai. She’s the fast.ai celebrity here but yeah. This has been my path, been doing it. These are the two courses I particularly love, fast.ai and deeplearning.ai and then I also went through OpenAI’s Scholars program and then the Fellows Program. Now I’m working here full-time, but happy to talk to anybody here if they’re interested in this sort of thing.

Christine Payne: The kind of fun thing about AI is that there’s so much that’s still wide open and it’s really helpful to come from different backgrounds where you bring a … It’s amazing how if you bring a new perspective or a new insight, there are a lot of things that are still just wide open that you can figure out how to do. I encourage anyone to come and check it out. We’ll have a concert. Thank you.

Mira Murati speaking

RL Team Manager Mira Murati gives a talk about reinformatiion learning and industry trends at OpenAI Girl Geek Dinner.   Erica Kawamoto Hsu / Girl Geek X 

Mira Murati: Hey, everyone, I’m Mira Murati and I’ll talk a little bit about the advancements in reinforcement learning from the lens of our research team here at OpenAI. Maybe I’ll kick things off by just telling you a bit about my background and how I ended up here.

Mira Murati: My background is in mechanical engineering but most of my work has been dedicated to practical applications of technology. Here at OpenAI, I work on Hardware Strategy and partnerships as well as managing our Reinforcement Learning research team alongside John Schulman, who is our lead researcher. I also manage our Safe Reinforcement Learning team.

Mira Murati: Before coming to OpenAI, I was leading the product and engineering teams at Leap Motion, which is a company that’s focused on the issue of human machine interface. The challenge with the human machine interface, as you know, is that we’ve been enslaved to our keyboard and mouse for 30 years, basically. Leap Motion was trying to change that by increasing the bandwidth of interaction with digital information such that, just like you see here, you can interact … Well, not here, with the digital space in the same natural and high bandwidth way that you interact with your physical space. The way you do that is using computer vision and AI to track your fingers in space and bring that input in virtual reality or augmented reality in this case.

Mira Murati: Before that, I was at Tesla for almost three years leading the development and launch of the Model X. That’s enough about me. I’ll touch a bit about on the AI landscape as a whole, just to offer a bit of context on the type of work that we’re doing with our Reinforcement Learning team. Then I’ll talk a bit about the impact of this work, the rate of change in the field as well as the challenges ahead.

Mira Murati: As you know, the future has never been bigger business. Every day we wake up to headlines like this and a lot of stories talking about the ultimate conversions where all the technologists come together to create the ultimate humankind dimension, that of general artificial intelligence. We wonder what this is going to do to our minds and to our societies, our workplaces and healthcare. Even politicians and cultural commentators are aware of what’s happening with AI to some extent, and politicians like this, to the extent that there’s a lot of nations out there that have published their AI strategies.

Mira Murati: There is definitely a lot of hype, but there is also a ton of technological advancement that’s happening. You might be wondering what what’s driving these breakthroughs. Well, so a lot of advancements in RL are driving the field forward and my team is working on some of these challenges through the lens of reinforcement learning.

Mira Murati: Both Brooke and Lilian did a great job going over reinforcement learning so I’m not going to touch too much upon that, but basically, to reiterate, it is you’re basically learning through trial and error. To provide some context for our work, I want us to take a look at …

Mira Murati: Oh, okay. There’s music. I wanted to take a look at this video where first you see this human baby, nine months old, how he is exploring the environment around him. You see this super high degrees of freedom interaction with everything around him. I think this is four hours of play in two minutes. In some of the things that this baby does like handling all these subjects, rolling around all this stuff, this is almost impossible for machines to do as you saw from Lilian’s talk.

Mira Murati: Then … Well, he’s going to keep going, but let’s see. Okay, now that … What I want to show you is … Okay, this is not working, but basically, I wanted you to show you that by contrast, so you have this video game over there where you would see this AI agent that’s basically trying to cross this level and makes the same mistakes over and over again. The moral of the story is that AI agents are very, very limited when they’re exploring their environment. Human babies just nine months old have this amazing ability to explore their environment.

Mira Murati: The question is, why are humans so good at understanding the environment around them? Of course, humans … We have this baby running in the playground. Of course, humans are very good at transferring knowledge from one domain to another, but there is also prior knowledge from evolution and also, from your prior life experiences. For example, if you play a lot of board games and I asked you to play a new one that you have never seen before, you’re probably not going to start learning that new game from scratch. You will apply a lot of the heuristics that you have learned from the previous board game and utilize those to solve this new one.

Mira Murati: It’s precisely this ability to abstract, this conceptual knowledge that’s based on or learned from perceptual details of real life that’s actually a key challenge for our field right now and we refer to this as transfer learning.

Mira Murati: What’s the state of things? There’s been a lot of advancements in machine learning and particularly in reinforcement learning. As you heard from the talks earlier, new datasets drive a lot of the advancements in machine learning. Our Reinforcement Learning team built a suite of games, thousands of games, that in itself you think playing video games is not so useful, but actually, they’re a great test bed because you have a lot of problem-solving and also content that’s already there. It comes for free in a way.

Mira Murati: The challenge that our team has been going after is how can we solve a previously unseen game as fast as a human, or even faster, given prior experiences with similar games. The Gym Retro dataset helps us do that. I was going to say that some of the games look like this but the videos are not quite working. But in a way, the Gym Retro dataset, you can check it out on the OpenAI blog, emphasizes the weaknesses of AI which is that of grasping a new task quickly and the ability to generalize knowledge.

Mira Murati: Why do all these advancements matter and what do the trends look like? It’s now just a bit over 100 years after the birth of the visionary mathematician Alan Turing and we’re still trying to figure out how hard it’s going to be to get to general artificial intelligence. Machines have surpassed us at very specific tasks but the human brain sets a high bar for what’s AI.

Mira Murati: In the 1960s and ’70s, this high bar was a game of chess. Chess was long considered the summit of human intelligence. It was visual, tactical, artistic, intelligence, mathematical, and chess masters could remember every single game that they played, not to mention that of their competitors, and so you can see why chess became such a symbol of mastery or a huge achievement of the human brain. It combined insight and forward planning and calculation, imagination, intuition, and this was until 1996, when the Deep Blue machine, chess machine from IBM was able to beat Garry Kasparov. If you had brought someone from the 1960s to that day, they would be completely astonished that this had happened but in 1996, this did not elicit such a reaction because in a way, Deep Blue had cheated by utilizing the power of hardware of Moore’s law. It leveraged the advancements in hardware to beat Garry Kasparov at chess.

Mira Murati: In a way, this didn’t show so much the advancements in AI, but rather that chess was not the pinnacle of human intelligence. Then the human sights were set on the Chinese game of Go, which is much more complex and just with brute force, you’d be quite far from solving Go, the game of Go with brute force and where we stand with hardware today. Then of course, in 2016, we saw the DeepMind’s AlphaGo beat Lee Sedol in Korea and that was followed by advancements in AlphaGo Zero. OpenAI robotics team of course, used some of the algorithms developed in the RL team to manipulate the cube and then we saw very recently, obviously, the Dota 5v5 beat the world champions.

Mira Murati: There’s been a very strong accelerating trend of advancements pushed by reinforcement learning in general. However, there’s still a long way to go. There are a lot of questions with reinforcement learning and in figuring out where the data is coming from and what actions do you take early on that get you the reward later. Also issues of safety, how do you learn in a safe way and also how do you continue to learn once you’ve gotten really good? Think of self-driving cars, for example. We’d love to get more people thinking about this type of challenges and I hope that some of you will join us in doing so. Thank you.

Amanda Askell speaking

Research Scientist Amanda Askell on the Policy team gives a talk on AI policy at OpenAI Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Amanda Askell: Okay, can everyone hear me? Cool. We’ve had like a lot of talks on some of the technical work that’s been happening at OpenAI. This talk is going to be pretty introductory because I guess I’m talking about what is quite a new field, but as Ashley said at the beginning, it’s one of the areas that OpenAI focuses on. This is a talk on AI policy and I’m a member of the policy team here.

Amanda Askell: I realize now that this picture is slightly unfortunate because I’m going to give you some things that look like they’re being produced by a neural net when in fact this is just an image because I thought it looked nice.

Amanda Askell: The core claims behind why we might want something like AI policy to exist in the world are really simple. Basically, AI has the potential to be beneficial. Hopefully, we can agree with this. We’ve had lots of talks showing how excellent AI can be and things that it can be applied to. AI also has the potential to be harmful so I’ll talk a little bit about this in the next slide but you know we hear a lot of stories about systems that just don’t behave the way that they’re creators intended to when they’re deployed in the world, systems that can be taken over by people who want to use them for malicious purposes. Anything that has this ability to do great things in the world can also be either misused or lead to accidents.

Amanda Askell: We can do things that increase the likelihood that AI will be beneficial so hopefully, that’s also fairly agreed-upon. But also that this includes making sure that the environment the AI is developed in is one that incentivizes responsible development. They’re like nontechnical things that we can do to make sure that AI is beneficial.

Amanda Askell: I think these are all like really simple and this leads to this idea that we should be doing some work in known technical fields just to make sure that AI is developed responsibly and well. Just to like kind of reiterate the claims of the previous slide, the potential benefits of AI are obviously kind of huge and I feel like to this audience I don’t really need to sell them but we can go over them. You know language models provide the ability potentially to assist with writing and other day-to-day tasks.

Amanda Askell: We can see that we can apply them to large complex problems like climate change potentially. This is the kind of like hope for things like a large scale ML. We might be able to enable like innovations In healthcare and education so we might be able to use them for things like diagnosis or finding new treatments for diseases. Finally, they might drive the kind of economic growth that would reduce the need to do work that people don’t find fulfilling. I think this is probably controversial. This is one thing that’s highly debated in AI ethics but I will defend it. I’ve done lots of unfulfilling work in my life and if someone could just pay me to not do that, I would have taken that.

Amanda Askell: Potential harms like language models of the same sort could be used to like misinform people by malicious actors. There are concerns about facial recognition as it improves and privacy. People are concerned about automation and unemployment if it’s not dealt with well. Like does this just lead to massive unfairness and inequity? Then people are also worried about things like decision making and bias. We already see in California that there’s ML systems being used for things like decisions about bail being set but also historically, we’ve used a lot of systems for things like whether someone gets credit. I mean so whether your loan’s approved or not given that there’s probably a huge amount of bias in the data and that we don’t know yet how to completely eliminate that, this could be really bad and it could increase systemic inequity in society, so that’s bad.

Amanda Askell: We’re also worried about like AI weapons and global security. Finally, just like a general misalignment of future AI systems. A lot of these are just like very classic examples of things that people are thinking about now, but this should just … We could expect this to be the sort of problems that we just see on an ongoing basis in the future as systems get more powerful.

Amanda Askell: I don’t think AI is like any different from many other technologies in at least some respects here. How do we avoid building things that are harmful? Doing the same kind of worries just apply to like the aviation industry. Planes can also be taken over by terrorists. Planes can be built badly and lead to accidents. The same is true of like cars or pharmaceuticals or like many other technologies with the potential to do good, it can end up … There can be accidents. It can be harmful.

Amanda Askell: In other industries we invest in safety, we invest in reducing accidents, we invest in security, so that’s like reducing misuse potential, and we also invest in social impact. In case of aviation, we know are concerned about things like the impact that flying might have on the climate. This is like the kind of third sort of thing that people invest in a lot.

Amanda Askell: All of this is very costly so this is just a kind of intro to like one way in which we might face problems here. I’m going to use a baking analogy, mainly because I was trying to think of a different one and I had used this one previously and I just couldn’t think of a better one.

Amanda Askell: The idea is, imagine you’ve got a competition and the nice thing about baking competitions, maybe I just have watched too many of them, is like you care both about the quality of what you’re creating and also about how long it takes to create it. Imagine a baking competition where you can just take as much time as you want and you’re just going to be judged on the results. There’s no race, like you don’t need to hurry, you’re just going to focus purely on the quality of the thing that you’re creating.

Amanda Askell: But then you introduce this terrible thing, which is like a time constraint or even worse, you can imagine you make it a race. Like the first person to develop the bake just gets a bunch of extra points. In that case, you’re going to be like well, I’ll trade off some of the quality just to get this thing done faster. You trade off some quality for increased speed.

Amanda Askell: Basically, we can expect something similar to happen with things like investment in areas like the areas that I talked about in the previous slide, where it’s like it might be that I would want to just like continue investing and making sure that my system is secure essentially like forever. I just never want someone to misuse this system so if I was given like 100 years, I would just keep working on it. But ultimately, I need to produce something. I do need to put something out into the world and the concern that we might have is that competition could drive down the incentive to invest that much in security.

Amanda Askell: This, again, happens across lots of other industries. This is like not isolated to AI and so there’s a question of like, what happens here? How do we ensure that companies invest in things like safety? I’m going to argue that there are four things. Some of the literature might not mention this one but I think it’s really important. The first one is ethics. People and companies are surprisingly against being evil. That’s good, that’s important. I think this gets not talked about enough. Sometimes we talk like the people that companies would just be totally happy turning up at like 9:00 a.m. to build something that would cause a bunch of people harm. I just don’t think that people think like that. People are … I have fundamental faith in humanity. I think we’re all deeply good.

Chloe Lin software engineer OpenAI Girl Geek Dinner

Software Engineer Chloe Lin listens to the OpenAI Girl Geek Dinner speakers answer audience questions.  Photo credit: Erica Kawamoto Hsu / Girl Geek X

Amanda Askell: It’s really great to align your incentives with your ethical beliefs and so regulation is obviously one other component that’s there to do that. We create these regulations and industry norms to basically make sure that if you’re like building planes and you’re competing with your competitor, you still just have to make your planes. You have to establish that they reach some of … Tripped over all of those words.

Amanda Askell: You have to establish that they reach some level of safety and that’s what regulation is there for. There’s also liability law and so companies have to compensate who are harmed by failures. This is another thing that’s driving that incentive to make sure your bake is not going to kill the judges. Well, yeah, everyone will be mad at you and also, you’ll have to pay a huge amount of money.

Amanda Askell: Finally, the market. People just want to buy safe products from companies with good reputations. No one is going to buy your bake if they’re like, “Hang on, I just saw you drop it on the floor before you put it into the oven. I will pay nothing for this.” These are four standard mechanisms that I think are used to like ensure that safety is like pretty high even in the cases of competition between companies in other domains like aviation and pharmaceuticals.

Amanda Askell: Where are we with this on AI? I like to be optimistic about the ethics. I think that coming to a technology company and seeing the kind of tech industry, I’ve actually been surprised by the degree to which people are very ethically engaged. Engineers care about what they’re building. They see that it’s important. They generally want it to be good. This is more like a personal kind of judgment on this where I’m like actually, this is a very ethically engaged industry and that’s really great and I hope that continues and increases.

Amanda Askell: With regulation, currently there are not many industry-specific regulations. I missed an s there but speed and complexity make regulation more difficult. The idea is that regulation is very good when there’s not an information asymmetry between the regulator and the entity being regulated. It works much less well when there is a big information asymmetry there. I think in the case of ML, that does exist. It’s very hard to both keep up with like, I think for regulators keeping up with contemporary ML work is really hard and also, the pace is really fast. This makes it actually quite difficult as an area to build very good regulation in.

Amanda Askell: Liability law is another thing where it’s just like a big question mark because like for ML accidents and misuse, in some cases it’s just unclear what existing law would say. If you build a model and it harms someone because it turns out that there was data in the model that was biased and that results in a loan being denied to someone, who is liable for that harm that is generated? You get easier and harder cases of this, but essentially, a lot of the kind of … I think that contemporary AI actually presents a lot of problems with liability law. It will hopefully get sorted out, but in some cases I just think this is unclear.

Amanda Askell: Finally, like market mechanisms. People just need to know how safe things are for market mechanisms to work well. In the case of like a plane, for example, I don’t know how safe my planes are. I don’t go and look up the specs. I don’t have the engineering background that would let me actually evaluate, say, a new plane for how safe it is. I just have to trust that someone who does know this is evaluating how safe those planes are because there’s this big information gap between me and the engineers. This is also why I think we shouldn’t necessarily expect market mechanisms to do all of the work with AI.

Amanda Askell: This is to lead up to this … to show that there’s a broader problem here and I think it also applies in the case of AI. To bring in a contemporary example, like recently in the news, there’s been concern. Vaping is this kind of like new technology that is currently not under the purview of the FDA or at least generally not heavily regulated. Now there’s concern that it might be causing pretty serious illnesses in people across the US.

Amanda Askell: I think this is a part of a more broad pattern that happens a lot in industries and so I want to call this the reactive route to safety. Basically, a company does the thing, the thing harms people. This is what you don’t want on your company motto. Do the thing. The thing harms people. People stop buying it. People sue for damages. Regulators start to regulate it. This would be really uninspiring as your company motto.

Amanda Askell: This is actually a very common route to making things more safe. You start out and there’s just no one who’s there to make sure that this thing goes well and so it’s just up to people buy it, they’re harmed, they sue, regulators get really interested because suddenly your product’s clearly harming people. Is this a good route for AI? Reasons against hope … I like the laugh because I’m like hopefully, that means people agree like no, this would be terrible. I’m just like well, one reason, just to give like the additional things of like obviously that’s kind of a bad way to do things anyway.

Amanda Askell: AI systems can often be quite broadly deployed almost immediately. It’s not like you just have some small number of people who are consuming your product who could be harmed by it in a way that a small bakery might. Instead, you could have a system where you’re like I’ve built the system for determining whether someone should get a loan. In principle, almost every bank in the US could use that the next day and that’s –The potential for widespread deployment makes it quite different from technologies where you just have a really or like any product where you have just like a small base of people.

Amanda Askell: They have the potential for a really high impact. The loan system that I just talked about could, basically, could in principle really damage the lives of a lot of people. Like apply that to things like bail systems as well, which we’re already seeing and even potentially with things like misinformation systems.

Amanda Askell: Finally, in a lot of cases it’s just difficult to attribute the harms and if you have something that’s spreading a huge amount of misinformation, for example, and you can’t directly attribute it to something that was released, this is concerning because it’s not like this route might work. This route actually requires you to be able to see who caused the harm and whenever that’s not visible, you just don’t expect this to actually lead to good regulation.

Amanda Askell: Finally, I just want to say I think there are alternatives to this reactive break things first approach in AI and this is hopefully where a lot of policy work can be useful.

Amanda Askell: Just to give a brief overview of policy work at OpenAI. I think I’m going to start with the policy team goals just to give you the sense of what we do. We want to increase the ability of society to deal with increasingly advanced AI technology, both through information and also through pointing out mechanisms that can make sure that technology is safe and secure and that it does have a good social impact. We conduct research into long-term issues related to AI and AGI so we’re interested in what happens when these systems become more powerful. Not merely reacting to systems that already exist, but trying to anticipate what might happen in the future and what might happen as systems get more powerful and the kind of policy problems and ethical problems that would come up then.

Amanda Askell: Finally, we just help OpenAI to coordinate with other AI developers, civil society, policymakers, et cetera, around this increasingly advanced technology. In some ways trying to break down these information asymmetries that exist and it can cause all of these problems.

Amanda Askell: Just to give a couple of examples of recent work from the teams to the kind of thing that we do. We released a report recently with others on publication norms and release strategies in ML. Some of you will know about like the GPT-2 language release and the decision to do staged release. We discussed this in the recent report. We also discussed other things like the potential for bias in language models and some of the potential social impacts of large language models going forward.

Amanda Askell: We also wrote this piece on cooperation and responsible AI development. This is related to the things I talked about earlier about the potential for competition to push this bar for safety too low and some of the mechanisms that can be used to help make sure that that bar for safety is raised again.

Amanda Askell: Finally, since this is an introduction to this whole field, which is like new and emerging field, here are examples of questions I think are really interesting and broad but can be broke down to these very specific applicable questions. What does it mean for AI systems to be safe, secure, and beneficial and how can we measure this? This includes a lot of traditional AI ethics work, like my background is in ethics. A lot of these questions about like how you make a system fair and what it means for a system to be fair. I would think of that as falling under the what is it for a system to be socially beneficial, and I think that work is really interesting. I do think that there’s just this broad family of things there are like policy and ethics and governance. I don’t think of these as separate enterprises.

Amanda Askell: Hence, this is an example of why. What are ways that AI systems could be developed that could be particularly beneficial or harmful? Again, trying to anticipate future systems and ways that we might just not expect them to be harmful and they are. I think we see this with the existing technology. Maybe it’s like trying to anticipate the impact that technology will have is really hard but like given the huge impact that technology is now having, I think trying to do some of that research in advance is worthwhile.

Amanda Askell: Finally, what can industry policymakers and individuals do to ensure that AI is developed responsibly? This relates to a lot of the things that I talked about earlier, but yeah, what kind of interventions can we have now? Are there ways that we can inform people that would make this stuff all go well?

Amanda Askell: Okay, last slide except the one with my email on it, which is the actual last slide. How can you help? I think that there’s this interesting, this is just like … I think that this industry is very ethically engaged and in many ways, it can feel like people feel like they need to do the work themselves. I know that a lot of people in this room are probably engineers and researchers. I think the thing I would want to emphasize is, you can be really ethically engaged and that doesn’t mean you need to take this whole burden on yourself.

Amanda Askell: One thing you can also do is advocate for this work to be done, either in your company, or just anywhere where people are like … in your company, in academia or just that your company is informed of this stuff. But in general, helping doesn’t necessarily have to mean taking on this massive burden of learning an entire field yourself. It can just mean advocating for this work being done. At the moment, this is a really small field and I would just love to see more people working in it. I think advocacy is really important but I also think another thing is you can technically inform people who are working on this.

Amanda Askell: We have to work closely with a lot of the teams here and I think that’s really useful and I think that policy and ethics work is doing its best, basically, when it’s really technically informed. If you find yourself working in a position where a lot of the things that you’re doing feel like they are important and would benefit from this sort of work, like helping people who are working on it is a really excellent way of helping. It’s not the only thing that you can do is spend half of your time doing the work that I’m doing and the others on the team are doing. You can also get people like us to do it. We love it.

Amanda Askell: If you’re interested in this, so thank you very much.

Brooke Chan, Amanda Askell, Lilian Weng, Christine Payne, Ashley Pilipiszyn

OpenAI girl geeks: Brooke Chan, Amanda Askell, Lilian Weng, Christine Payne and Ashley Pilipiszyn answer questions at OpenAI Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X 

Audience Member:  I have a question.

Amanda Askell: Yes.

Audience Member: For Amanda.

Amanda Askell: Yes.

Audience Member: Drink your water first. No, I think the ethics stuff is super interesting. I don’t know of a lot of companies that have an ethics department focused on AI, and I guess one thing that I’m curious about is, like you pointed out like your papers but like, and I know you talked about educating and all this other stuff but what are you guys…do? Do you know what I mean? Other than write papers.

Amanda Askell: Yeah.

Ashley Pilipiszyn: Oh, Christine.

Amanda Askell: Which one? Yeah, so I think at the moment there’s like a few kind of rules. I can say what we do but also what I think that people in these roles can do. So in some cases it can be like looking at what you’re building internally. I think we have like the charter and so you want to make sure that everything that you’re doing is in line with the charter. Things like GPT-2 and release decisions, I think of as a kind of like ethical issue or ethical/policy issue where I would like to see the ML community build really good norms there. Even if people don’t agree with what OpenAI try to do with its release decisions, it was coming from a place of trying to build good norms and so you can end up thinking about decisions like that.

Amanda Askell: That’s more of an example of something where you’re like it’s not writing a paper, it’s just like thinking through all of the consequences of different publication norms and what might work and what might not. That’s like one aspect, that’s the kind of like internal component. I think of the external component as like, on the one hand it’s just like writing papers so just being like what are the problems here that people could work on and in a lot ways that’s just like outreach, like trying to get people who are interested in working on this to work on it further. For that, there’s a few audiences, so you might be interested in attracting people to the field if you think that there are these like ongoing problems within both companies and maybe with other relevant actors. Like maybe you also want people going into government on this stuff.

Amanda Askell: But also just like the audience can be internal, to make people aware of these issues and they can also be things like policymakers, just inform of the kind of structure of the problem here. I think of it as having this kind of internal plus external component and you can end up dividing your time between the two of them. We spend some time writing these papers and trying to get people interested in these topics and just trying to solve the problems. That’s the nice thing about papers is you can just be like, what’s the problem, I will try and solve it and I’ll put my paper of an archive. Yeah, and so I think there’s both of those.

Amanda Askell: It’s obviously fine for companies to have people doing both, like if you haven’t and I think it’s like great if a company just has a team that’s just designed to look at what they’re doing internally and if anyone has ethical concerns about it, that team can take that on and own it and look at it. I think that’s a really good structure because it means that people don’t feel like … if you’re like just having to raise these concerns and maybe feel kind of isolated, that’d be bad but if you have people that you know are thinking about it, I think that’s a really good thing. Yeah, internal plus external, I can imagine different companies liking different things. I hope that answers the question.

Rose: My question is also for Amanda. So the Google AI Ethics Board was formed and disbanded very quickly kind of famously within like the span of less than a month. How do you kind of think about that like in the context of the work that OpenAI is doing and like how do you think about like what they failed at and like what we can do better?

Amanda Askell: This was a really difficult case so I can give you … I remember personally kind of looking at this and being like I think that one thing that was in it … I don’t know if people know the story about this case but basically, it was that Google formed a board and they were like, “We want this to be intellectually representative,” and it garnered a lot criticism because it had a person who was head of the Heritage Foundation, so a conservative think-tank in the US, as one of its members, and this was controversial.

Amanda Askell: I remember having mixed views on this, Rose. I do think it’s great to … Ultimately, these are systems that are going to affect a huge number of people and that includes a huge number of people who have views on how they should be used and how they should affect them. They’re just very different from me and I want those people to be represented and I want their views on how they do or do not want systems to affect them to be at the table. We talked earlier about the importance of representativeness and I genuinely believe that for people who have vastly different views for myself. If they’re affected by it, ultimately, their voice matters.

Amanda Askell: At the same time, I think I also … there’s a lot of complicating–you’re getting my just deeply mixed emotions here because I was like, there’s a strange sense in which handpicking people to be in the role of a representative of a group where you’re like, I don’t know, we select who the intellectual representatives are also struck me as somewhat odd. It’s a strange kind of … It set off my old political philosophy concerns where I’m like, “Oh, this just doesn’t …” It feels like it’s imitating democracy but isn’t getting there. I had and I was also just like plus the people who come to the table and there are certain norms of respect to lots of groups of people that just have to be upheld if you’re going to have people with different views have an input on a topic.

Amanda Askell: I think some of the criticisms were that people felt those norms had been upheld and this person had been insulting to key groups of people, the trans community and to immigrants. Largely, mixed feelings where I was just like I see this intention and it actually seems to me to be a good one, but I see all of these problems with trying to execute on it.

Amanda Askell: I can’t give an awesome response to this. It’s just like yeah, here it is, I’ve nailed it. It’s just like yeah, these are difficult problems and I think if you came down really strongly on this where it was like this was trivially bad or you were like this was trivially good, it just feels no, they were just like there are ways that I might have done this differently but I see what the goal was and I’m sympathetic to it but I also see what the problems were and I’m sympathetic to those. Yeah, it’s like the worst, the least satisfying answer ever, I guess.

OpenAI Girl Geek Dinner audience women in AI.

OpenAI Girl Geek Dinner audience enjoys candor from women in AI.  Erica Kawamoto Hsu  / Girl Geek X

Audience Member: Hi, I have a question for Brooke. I’m also a fan of Dota and I watched TI for two years. My question is, if your model can already beat the best team in the world, what is your next goal?

Brooke Chan: Currently, we’ve stopped the competitive angle of the Dota project because really what we wanted to achieve was to show that we could get to that level. We could get to superhuman performance on a really complex game. Even at finals, we didn’t necessarily solve the whole game because there were a lot of restrictions, which people brought up. For example, we only used 17 out of the you know 100 and some heroes.

Brooke Chan: From here, we’re just looking to use Dota more as a platform for other things that we want to explore because now we know that it’s something that is trainable and can be reused in other environments, so yeah.

Audience Member: Hi, my question is about what are some of the limitations of training robots in a simulator?

Lilian Weng: Okay, let me repeat. Question is, what’s a limitation of training the robot-controlled models in the simulation? Okay, there are lots of benefits, I would say, because in simulation, you have the ground rules. You know exactly where the fingertips are, you know exactly what’s the joint involved. We can do all kinds of randomization modification of the environment. The main drawback is we’re not sure what’s the difference between our simulated environment and reality. Our eventual goal is to make it work in reality. That’s the biggest problem. That’s also what decides whether our sim2real transfer going to work.

Lilian Weng: I will say one thing that confuse me or puzzles me personally the most is when we are running all kinds of randomizations, I’m not sure whether it’s getting us closer to the reality because we don’t have a good measurement of what the reality looks like. But one thing I didn’t emphasize a lot in the talk is we expect because we design all kinds of environment in the simulation and we asked the policy model to master all of them. There actually emerges some meta learning effect, which we didn’t emphasize but with meta learning, your model can learn how to learn. We expect this meta learning in fact to empower the model to handle something they’d never seen before.

Lilian Weng: That is something we expect with domain randomization that our model can go above what it has seen in the simulation and eventually adapt to the reality. We are working all kinds of technique to make the sim2real thing happen and that’s definitely the most difficult thing for robotics because it’s easy to make things work in simulation. Okay, thanks.

Audience Member: I was just curious as kind of another follow-up question to Brooke’s answer for earlier but for everybody on the panel too. What do you consider to be some of the longer-term visions for some of your work? You did an impressive thing by having Dota beat some real people but where would you like to see that work go or what kinds of problems do you think you could solve with that in the future too, and for some other folks on the panel too?

Brooke Chan: Sure, I would say that pretty honestly when we started the Dota project we didn’t actually know whether or not we would be able to solve it. The theory at the time was that we would need a much more powerful algorithm or a different architecture or something in order to push it kind of all the way. The purpose of the project was really to demonstrate that we could use a relatively straightforward or simple algorithm in order to work on this complex game.

Brooke Chan: I think going out from here, we’re kind of looking into environments in general. We talked about how Dota might be one of our last kind of games because games are still limited. They’re helpful and beneficial in that you can run them in simulation, you can run them faster but we want to kind of also get closer to real-world problems. Dota was one step to getting to real-world problems in the parts that I talked about like the partial information and the large action space and things like that. Going on from there, we want to see what other difficult problems you could also kind of apply this sort of things to. I don’t know if other people …

Christine Payne: Sure. In terms of a music model, I would say kind of two things I find fascinating. One is that I really like the fact that it’s this one transformer architecture which we’re now seeing apply to lots of different domains. The fact that it can both do language and music and it’s really kind of interesting to find these really powerful algorithms that it doesn’t care what it’s learning, it’s just learning. I think that that’s going to be really interesting path going forward.

Christine Payne: Then, also, I think that music is a really interesting test for like we have a lot of sense as humans so we know how we would want the music to go or we know how the music affects us emotionally or there’s all this sort of human interaction that we can explore in the music world. I hear from composers saying well, they want to be able to give the shape of the music or give the sense of it or the emotion of it, and I think there’s a lot of space to explore in terms of it’s the same sort of thing we’ll want to be able to influence the way any program is going to be, how we’ll be interacting with a program in any field but music is a fun area to play with it.

Ashley Pilipiszyn: Actually, as a followup, if you look at all of our panelists and everything everyone presented too, it’s not just human and AI interaction, but human and AI cooperation. Actually, for anyone who followed our Dota finals event as well, not only did we have a huge success but, and for anyone who is a Dota fan in the crowd, I’d be curious if anyone participated in our co-op challenge. Anyone by chance? No, all right. That’s all right.

Ashley Pilipiszyn: But actually, being able to insert yourself as being on a team with OpenAI Five and I think from all of our research here we’re trying to explore the boundaries of, you know what does human AI cooperation look like and I think that’s going to be a really important question going forward so we’re trying to look at that more.

Speaker: And we have time for two more questions.

Audience Member: Thank you. Just right on time. I have a question for you, Christine. I was at a conference earlier this year and I met this person named Ross Goodwin who wrote using a natural language processing model that he trained a screenplay. I think it’s called Sunspring or something like that. It’s a really silly script that doesn’t make any sense but it’s actually pretty fun to watch. But he mentioned that in the media it’s been mostly–the credit was given to an AI wrote this script and his name was actually never mentioned even though he wrote the model, he got the training data. What is your opinion on authorship in these kinds of tools that … also the one you mentioned where you say you’re actually composing? Are you the composer or is the AI the composer? Should it be like a dual authorship?

Christine Payne: That is a great question. It’s a difficult question that I’ve tried to explore a little bit. I’ve actually tried to talk with lawyers about what is copyright going to look like? Who owns pieces like this? Because in addition to who wrote the model and who’s co-composing or co-writing something, there’s also who’s in the dataset. If your model is imitating someone like are they any part of the author in that?

Christine Payne: Yeah, I mean I have my own sort of guesses of where I think it might go but every time … I think I’m a little bit [inaudible 01:37:11] in terms of the more you think about it, the more you’re like this is a hard problem. It’s really, like if you come down hard on one side or the other because clearly, you don’t want to be able to just press go and have the model just generate a ton of pieces and be like I now own all these pieces. You could just own a ridiculous number of pieces, but if you’re the composer who has carefully worked and crafted the model, crafted … you write a little bit of a piece, you write at some of the piece and then the model writes some and you write some. There’s some interaction that way, then sure, that should be your piece. Yeah, I think it’s something that we probably will see in the near future, law trying to struggle with this but it’s an interesting question. Thanks.

Audience Member:  Okay, last question. Oh no.

Ashley Pilipiszyn: We’ll also be around so afterwards you can talk to us.

Audience Member: This is also a followup question and it’s for everyone on the panel. Could you give us some examples of real-life use cases of your research and how that would impact our life?

Ashley Pilipiszyn: An example.

Christine Payne: It’s not an easy one to close on. You want to take it. Go for it.

Lilian Weng: I will say if eventually we can build general purpose robots, just imagine we use the robot to do a lot of dangerous tasks. I mean tasks that might seem danger to humans. That can definitely reduce the risk of human labors or doing repeated work. For example, on assembly line, there are some tasks that involve human hands, but kind of boring. I heard from a friend that there are a lot of churn or there’s a very high churn rate of people who are working on the assembly line, not because it’s low pay or anything, most because it’s very boring and repetitive.

Lilian Weng: It’s not really good for people’s mental health and they have to–like the factory struggle to hire enough people because lots of people will just leave their job after a couple months or half a year. If we can automate all those tasks, we’re definitely going to leave others more interesting and creative position for humans to do and I think that’s going to overall move the productivity of the society. Yeah. That’s still a very far-fetched goal. We’re still working on it.

Amanda Askell: I can also give a faraway thing. I mean I guess my work is,, you know with the direct application, I’m like, “Well, hopefully, ML goes really well.” Ideally, we have a world where all of our institutions are actually both knowledgeable of the work that’s going on in ML and able to react to them really well so a lot of the concerns that people have raised around things like what happens to authorship, what happens to employment, how do you prevent things like the misuse of your model, how can you tell it’s safe? I think if policy work goes really well then ideally, you live in a world where we’ve just made sure that we have all of the kind of right checks in place to make sure that you’re not releasing things that are dangerous or that can be misused or harmful.

Amanda Askell: That just requires a lot of work to ensure that’s the case, both in the ML community, and in law and policy. Ideally, the outcome of great policy work is just all of this goes really smoothly and awesomely and we don’t have any bad things happen. That’s like the really, really modest goal for AI policy work.

Brooke Chan: I had two answers on the short-sighted term, in terms of just AI being applied to video games, AI in video games historically is really awful. It’s either really just bad and scripted and you can beat it easily and you get nothing from it or it’s crazy good because it’s basically cheating at the game and it’s also not really that helpful. Part of what we found out through the Dota project was people actually really did like learning with the AI. When you have an AI that’s at your skill level or slightly above, you have a lot of potential, first of all, to have a really good competitor that you can learn from and work with, but also to be constantly challenged and pushed forward.

Brooke Chan: For a more longer-term perspective, I would leverage off of the robotics work and the stuff that Lilian is doing in terms of the system that we created in order to train our AI is what is more general and can be applied to other sorts of problems. For example, that got utilized a little bit for the robotics project as well and so I feel it’s more open-ended in that sense in terms of the longer-term benefits.

Christine Payne: Okay and I’ll just wrap up saying yeah, I’ve been excited already to see how musicians and composers are using MuseNet. There are a couple examples of performances that have happened now of MuseNet pieces and that’s been really fun to see. The main part that I’m excited about is that I think the model is really good at just coming up with lots and lots of ideas. Even though it’s imitating what the composers might be doing, it opens up possibilities of like, “Oh, I didn’t think that we could actually do this pattern instead.” Moving towards that domain of getting the best of human and the best of models I think is really fun to think about.

Ashley Pilipiszyn: So kind of how I started the event this evening, our three main research areas are really on these capabilities, safety, and policy. You’ve been able to hear that from everyone here. I think the big takeaway and a concrete example I’ll give you is, you think about your own experience going through primary education. You had a teacher and you most likely … you went to science class then you went to math class and then maybe music class and then art class and gym. You had a different teacher and they just assumed, probably for most people, you just assumed you’re all at the same level.

Ashley Pilipiszyn: How I think about it is, we’re working on all these different kind of pieces and components that are able to bring all of these different perspectives together and so a system that you’re able to bring in the math and the music and the gym components of it but also able to understand what level you’re at and personalize that. That’s kind of what I’m really excited about, is this human AI cooperation component and where that’ll take us and help unlock our own capabilities. I think, to quote from Greg Brockman, our CTO, that while all our work is on AI, it’s about the humans. With that, thank you for joining us tonight. We’ll all be around and would love to talk to you more. Thank you.

Speaker: We have a quick update from Christina on our recruiting team.

Ashley Pilipiszyn: Oh, sorry.

Christina Hendrickson: Hey, thanks for coming again tonight. I’m Christina. I work on our recruiting team and just briefly wanted to talk to you about opportunities at OpenAI. If you found the work interesting that you heard about from our amazing speakers tonight and would be interested in exploring the opportunities with us, we are hiring for a number of roles across research, engineering and non-technical positions.

Christina Hendrickson: Quickly going to highlight just a couple of the roles here and then you can check out more on our jobs page. We are hiring a couple roles within software engineering. One of them, or a couple of them are on robotics, so that would be working on the same type of work that Lillian mentioned. We are also hiring on our infrastructure team for software engineers, as well, where you can help us in building some of the world’s largest supercomputing clusters.

Christina Hendrickson: Then the other thing I wanted to highlight is one of our programs. So we are going to have our third class of our scholars program starting in early 2020. We’ll be opening applications for that in a couple weeks so sneak peek on that. What that is, is we’re giving out eight stipends to people who are members of underrepresented groups within engineering so that you can study ML full-time for four months where you’re doing self-study and then you opensource a project.

Christina Hendrickson: Yeah, we’re all super excited to chat with you more. If you’re interested in hearing about that, we have a couple recruiting team members here with us tonight. Can you all stand up, wave? Carson there in the back, Elena here in the front, myself. Carson and I both have iPads if you want to sign up for our mailing list to hear more about opportunities.

Elena Chatziathanasiadou waving

Recruiters Christina Hendrickson and Elena Chatziathanasiadou (waving) make themselves available for conversations after the lightning talks at OpenAI Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Christina Hendrickson: Thank you all again for coming. Thanks to Girl Geek X. We have Gretchen, Eric, and Erica here today. Thank you to our speakers: Brooke, Amanda, Lilian, Christine, Ashley, and thank you to Frances for helping us in organizing and to all of you for attending.

Ashley Pilipiszyn: Thank you, everybody.


Our mission-aligned Girl Geek X partners are hiring!

Girl Geek X Bosch Lightning Talks (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Tara Dowlat, Seow Yuen Yee, Yelena Gorlin, LisaMarion Garcia, Panpan Xu, Shabnam Ghaffarzadegan, Sun-Mi Choi

Bosch girl geeks: Tara Dowlat, Seow Yuen Yee, LisaMarion Garcia, Sun-Mi Choi, Yelena Gorlin, Panpan Xu and Shabnam Ghaffarzadegan at Bosch Girl Geek Dinner in Sunnyvale, California.   Erica Kawamoto Hsu / Girl Geek X

Transcript from Bosch Girl Geek Dinner – Lightning Talks:

Angie Chang: We are really excited coming to Bosch to be listening to so many amazing girl geeks tonight.

Dr. Hauke Schmidt: We are very happy to host the Girl Geek dinner as a celebration of gender diversity, and I’m very proud of the team here who has put all this together.

Dr. Uma Krishnamoorthy: How many of you came here looking for headphones, acoustic systems in our demos? We’re not that company. You may have gone outside and you may have seen our car, autonomous car, so I don’t have to speak to our autonomous driving effort.

Dr. Seow Yuen Yee: Have you ever thought of how does the car know when to deploy these airbags? This is thanks to the airbags control unit in the car. It house a tiny little sensors which we call accelerometers.

Tara Dowlat: Did you guys know that at least every single one of you in this room, in your pockets or in your bags, have at least one sensor from Bosch on you? It’s a fun fact.

Dr. Yelena Gorlin: Each new generation of a battery management system looks to increase the charging speed of our device without having an effect on its lifetime.

LisaMarion Garcia: Each of these individual sectors provide us different opportunities to incorporate AI, either as a feature of a product that we sell or as part of the process of producing that product.

Dr. Shabnam Ghaffarzadegan: So our idea is asking human and machine to work together to empower their both abilities with much more perception and knowledge, and also to make a better machine to help us in our everyday life.

Sun-Mi Choi: So how many of you are using ride hailing apps to get from A to B on a regular basis? Mobility is also getting more user centric. The consumer is more and more changing from owned to shared.

Dr. Uma Krishnamoorthy: Big goals here. 2020, the goal is all of our electronic products will be connected. And in 2025, all our products are going to either possess intelligence or AI will have played a key role in their creation.

Angie Chang: Thanks for coming out tonight. I’m Angie Chang, founder of Girl Geek X. We’ve been hosting Girl Geek dinners up and down from San Francisco to San Jose for the last 11 plus years. We are really excited to be coming to Bosch to be listening to so many amazing girl geeks tonight.

Gretchen DeKnikker: I got my own microphone. You guys have no idea what that means. I’m Gretchen. Thank you. How many of you, it’s your first Girl Geek dinner? Good. Okay. So like she said, we do them every week. We also have a podcast, so pull out your phone now and go to your favorite podcast app and then rate it and then write a review or send us a message and say, “This is how it could be better.” Because we’re only doing it so it’ll be awesome for you guys. Right?

Gretchen DeKnikker: Then we also recently opened a little swag store on Zazzle. So there’s all sorts of cute things. I only have one or two cute things tonight. Cute water bottle.

Gretchen DeKnikker: I know. It’s ridiculous. Oh, I kind of had stuff with … There are more designs than this one. Apparently I only just brought things … But it’s a fanny pack. It’s so cute. Okay. Got it. So I’m going to try something new tonight. Who’s found a job through Girl Geek? No one? Okay, get out. Okay, has anyone got a … Oh, you did.

Audience member: No.

Gretchen DeKnikker: Oh. No, definitely not. That’s awesome. Okay, anyone found a job lead? Oh, okay.

Audience Member: I found candidates through Girl Geek.

Gretchen DeKnikker: You found candidates. Okay. So if you guys want to email us, I have these things and you can’t buy them. You can only get it from me. These adorable socks. So if you want to tell us, we would love to feature your story about finding a girl geek, a job through Girl Geek Dinner or something that you built and we want to have little community features and stuff. If you do it, you get those socks and it’s the only way in the world to get the socks.

Gretchen DeKnikker: Okay, so without further ado, how great is this space? This has been so awesome so far. You guys enjoying it? All right, so without further ado, we are bringing this gentleman right here.

Dr. Hauke Schmidt: Thank you very much, and welcome to Bosch. So my name is Hauke Schmidt. I’m the head of corporate technology research for Bosch here in North America. And I’m also the site leader for the innovation center here in Sunnyvale. A few words about the company for those of you who don’t know Bosch all too well. We have our roots in the automotive business, so we’re actually the largest automotive supplier in the world.

Dr. Hauke Schmidt: And very likely, if you open your car, there are a couple of Bosch components inside. You also might know us from household appliances or power tools. We’re also a leading IoT company, as you saw in the videos, here. And we’re driving product and services innovation in the areas of mobility, industrial, and building technologies.

Dr. Hauke Schmidt: One interesting part about Bosch is the ownership structure. We are privately held. We’re a very large multinational out of Germany and privately held. And mostly to the largest part, owned by the Robert Bosch Foundation. And the Foundation then also takes all of the profits and earnings we create and puts them to use in charitable projects. So this gives us an extra motivation to work hard and provide good results.

Dr. Hauke Schmidt: The site here, we’ve been in Silicon Valley for 20 years now. We have our 20th anniversary this year. We moved into this building one and a half years ago so this is now our new home here with a nice Bosch sign outside as well. We have about 200 scientists, engineers, and experts on site, and these experts cover a broad variety of different functions of the company. We have here everything from corporate research, venture capital technology scouting, prototyping, product development, but we also have product sales and engineering services here on site that we offer into the local industry around us.

Dr. Hauke Schmidt: For us diversity is an important thing. We have associates here from a very broad variety of different ethnic backgrounds, also from experts in a large number of different technology fields. So today we are very happy to host the Girl Geek dinner as a celebration of gender diversity and I’m very proud of a team here who’s put all this together since I’m also the executive champion at the Women at Bosch Group here on site as well.

Dr. Hauke Schmidt: Thank you. So with that ,without further ado, I would like to hand over to Uma who has her own microphone to kick off some of the lightning talks that we’ll listen to right now. Thank you.

Dr. Uma Krishnamoorthy: Can you hear me now?

Audience: Yes.

Uma Krishnamoorthy speaking

Director of Research Dr. Uma Krishnamoorthy gives a warm welcome to the crowd at Bosch Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Dr. Uma Krishnamoorthy: Okay. First welcome from my side. My name is Uma Krishnamoorthy and I am a director here at Bosch RTC. We are part of corporate. We, me and my department, are part of corporate research of the bigger Bosch. My particular groups are focused on microsensor systems technologies and multiphysics modeling and simulation areas of research.

Dr. Uma Krishnamoorthy: So today, this works, my role is very easy. It’s going to be a bit longer than the others but my role is relatively easy. I’m going to be giving your introduction to Bosch from a broader scale than what hopefully Hauke did. Then, of course, I’m going to lead into the Internet of Things and how we play a role in there.

Dr. Uma Krishnamoorthy: Hauke unfortunately told you what we do, so I’m going to ask anyway. How many of you already were aware of what Bosch does and what our products are before you came to the dinner today? Oh, that’s quite a few. Okay. The reason I ask, how many of you came here looking for headphones, acoustic systems in our demos or lens solutions? We’re not that company.

Dr. Uma Krishnamoorthy: Yep, we are Bosch. Who are we? First thing, we’re very diverse and the range of products we cover is very broad. I’m going to try to cover some of it today from the perspective of IoT. I’ll start off with this slide here, market figures. Bosch, exactly as Hauke mentioned, is from– originally started by Robert Bosch in 1886. So we’re over 130 years old.

Dr. Uma Krishnamoorthy: Yeah, we’re pretty old. We started in Germany, but as you can see we’re a global company. We have been in the Americas since 1906, I believe, over 100 years old. Very, very long time, very well established manufacturing company. We’ve made a very huge reputation in creating high quality products.

Dr. Uma Krishnamoorthy: We have 268 manufacturing sites across the world. Of course, we have a lot of representation in Asia-Pacific also. I wanted to draw your attention to that number right in the middle, 409,881 associates. That’s a huge number. Just to give you an idea, you take all of the associates at Alphabet, all of the associates at Apple, combine them, multiply it by approximately two. Okay, you’re all Girl Geek so approximately 1.78. And that will be the number of associates at Bosch. This was of course from 2018, so we are huge.

Dr. Uma Krishnamoorthy: To give you an idea of scale. So what do we do? I’m going to try to answer that question with this slide. You may be aware of our products in the consumer goods business. You may have seen our dishwashers, washing machines, maybe some coffee makers, many household appliances, power tools. Very popular there and a leading supplier. We also work in energy and building technology. What is this?

Dr. Uma Krishnamoorthy: Here’s a leading manufacturer of security communication technology. We actually make energy efficient heating products. This is a bigger business in Germany maybe than here, so we’re very well known for that. Or Europe, not Germany. On top of that, Hauke already mentioned mobility solutions.

Dr. Uma Krishnamoorthy: Sixty percent of our sales come from the mobility solutions business. This includes automotive and also consumer electronics. Essentially things like sensors that go in your cell phone, smartwatches, things of that sort. We’re a leading provider of that too.

Dr. Uma Krishnamoorthy: Surprising to me, I’ve been with Bosch for four years so this was a bit of a surprise, industrial technology. We also make a variety of industrial technologies. What does this mean? If you’ve ever been to the Jelly Belly factory, on the way back from Tahoe, you know, it’s a good stop.

Dr. Uma Krishnamoorthy: So if you stop there and look around, take a tour of the factory floor, you will see Bosch equipment, packaging equipment. I believe they might have been sorting the jellybeans, but I can’t remember exactly. So we are pretty broad and you’ll see us in many places, unexpected places. That’s how broad we are.

Dr. Uma Krishnamoorthy: To give you an idea of our culture, Hauke already mentioned our founder, Robert Bosch. We strongly follow the values of our founder Robert Bosch, which comprises of quality and innovation which is what our products are known for. This may not be as well-known over here in the US, but it’s known in Germany for sure, is the aspect of social commitment.

Dr. Uma Krishnamoorthy: Robert Bosch himself gifted the Robert Bosch Hospital to the City of Stuttgart back in 1936, which stands to this day. A lot of very important medical research is done there, including, I believe … I can’t remember all the details but a variety of really good medical research is done there.

Dr. Uma Krishnamoorthy: As Hauke mentioned, we’re privately held. Ninety percent of our shares are held by this Robert Bosch Foundation and this foundation fundamentally finances work that addresses social challenges. So they focus on areas like healthcare, science, society, education, international relations, all about society and life.

Dr. Uma Krishnamoorthy: They have provided, the number’s right there. 153-ish million euros to project grants that are in these areas. So, they really put their money where their values stand. That’s the message there. As I mentioned, one of the one of the cornerstones of Bosch is our innovation. We’re worldwide but we also have a very strong commitment to innovation. We have a, I don’t have the numbers here, a very large number of associates. Believe it was in 65,000 number range of associates who work in R&D across the company.

Dr. Uma Krishnamoorthy: Some of those actually work under a separate division called corporate research, which we’ve alluded to in the past and what you see in the background here is our campus that was recently built in Germany specifically for corporate research that services all of the Bosch groups,, fundamentally, almost all of them.

Dr. Uma Krishnamoorthy: And, what you really … I would like to highlight this one sentence over here our objective. Our motto is invented for life which is pretty much self-explanatory. So everything we do is about the quality of life, enhancing the quality of life through technology. I would like to say one more thing about this. Recently–I’ll have to … Mind me if I refer to my notes. Only because our CEO recently announced that we Bosch were going to be the first carbon-neutral industrial enterprise from 2020. That is a huge statement, and we’re all committed to delivering on that.

Dr. Uma Krishnamoorthy: What we came for, that was the introduction very briefly. I’ll try to go through this pretty fast. IOT at Bosch. This is going to essentially be kicking off a series of tech talks centered around IoT for Bosch. I’m only going to set it up for them. The real speakers will come after me.

Dr. Uma Krishnamoorthy: So what does IoT mean for Bosch? As many of you know, IoT is about creating better customer experiences through connectivity. And Bosch plays a very big role in it because we make a variety of products and we’re connecting them to make our customers get a better experience out of it, fundamentally. That’s the simplest way you can think about it.

Dr. Uma Krishnamoorthy: In the process, though, what we are noticing is industries are transforming, and we are playing a key role in this transformation at Bosch. So how are we playing in this field? Just giving you a sampling over here. You may have gone outside and you may have seen our car, autonomous car, so I don’t have to speak to our autonomous driving effort, our driver assistance efforts. There’s many of those that are ongoing that are widely shared.

Dr. Uma Krishnamoorthy: But on top of our mobility efforts we also work in the smart city area. We have products in all of these areas so connecting them and providing customer experiences goes beyond mobility into smart city, into buildings, industry, industry 4.0. But one of the key things for us, for our connected Bosch systems across these domains is we are creating intelligent user centric solutions without compromising safety or data security. Those are big messages that we carry and we essentially put into all our products.

Dr. Uma Krishnamoorthy: What is Bosch’s IoT vision? Again a borrowed slide. You will see big goals here. 2020, the goal is all of our electronic products will be connected. We’re going to continue working across a variety of domains and in 2025 all our products are going to either possess intelligence or AI will have played a key role in their creation. So AI is closely tied to our IoT.

Dr. Uma Krishnamoorthy: A few examples, I’ll have to go very quick. She just told me I have five minutes left. Quick examples, home appliances. Series 8 oven. It’s an oven, yes, but it’s also a microwave, it’s also a steamer, and it’s connected. So you can bake a cake–if you have the right app–you can bake a cake in it from your phone, and I’ll leave it there.

Dr. Uma Krishnamoorthy: This app is apparently not available everywhere but it is there, the technology is there. Mobility, you already mentioned that powertrains is one of the big areas we contribute in for the automotive business. Electric powertrains is our big area of work now. One thing I’ll show here is we are taking it beyond just electrification of cars, we’re actually moving into other powertrain systems for other vehicles such as two wheelers and trucks.

Dr. Uma Krishnamoorthy: Another aspect here is beyond just building EV vehicles, we’re also looking at connecting these vehicles. So anybody using an EV vehicle cares about charging them. So we actually have an app. Bosch has an app that’ll let you find up to 20,000 charging stations, which is very convenient, in five countries. I believe that will be increasing as this gets used more.

Dr. Uma Krishnamoorthy: Last but not least, the example automated valet parking. This came out recently. I had a beautiful video on this. It took too long so I’ll just tell you in two sentences. Automated valet parking. It’s like a mini autonomous vehicle that you can use in a parking garage.

Dr. Uma Krishnamoorthy: You bring your car to the garage, you walk out of it, hit the park button on your phone, the car will go park itself. When you are done with your dinner or whatever else, you come back to the garage. Say pick up the car. The car will drive itself to you. You can get in it and go home. That’s the idea and it’s actually real and they already rolled it out. So, that’s an example of some of the innovation we contribute to.

Dr. Uma Krishnamoorthy: Now I’ll be talking to you about some of the elements of IoT, not for very long. We have tech talks following me, they’ll go into all the details. So here, I’m going to talk briefly about transformation from the things to IoT. I’ve already mentioned that we make a lot of things here at Bosch across many domains. But one of the fundamental things we do is in the hardware. Sensors is a big area for Bosch, we are one of the enablers–sensors are the enablers for the Internet of Things and we’re one of the leaders in building micro sensors. Bosch Sensor Tech, in fact, is the part of Bosch that builds them, and you’ll be hearing a lot more about that from Tara right after me.

Dr. Uma Krishnamoorthy: Sensors are the data collectors. They are your direct connection to your products, they collect the states of your products, whatever they are. Then, another aspect of it that is kind of hidden, but is very important as batteries. So we need batteries to charge all of our things and our sensors and our phones, everything else. So that’s another aspect that we will be talking about soon. Yelena will be talking about it, I believe.

Dr. Uma Krishnamoorthy: Bosch has a strong background in the hardware aspect of manufacturing and in sensors products. So we understand that, the cause and effect. That’s our core business. So, what else is there to be done in IoT? It’s all about the connectivity. So once you have the data, you have to connect to it. We have the data collectors.

Dr. Uma Krishnamoorthy: So the next thing you need is to analyze the data and to create some–once you acquire the data you want to provide some, I guess models, right, and some plans on essentially understanding the data and to potentially predict what’s going to happen for whatever system you’re working with. So that’s where our AI comes into play right, and LisaMarion will be talking about that. She’s part of our BCA, Bosch Center for Artificial Intelligence.

Dr. Uma Krishnamoorthy: Then, finally, it all comes down to the user and the user interface. So that portion will be handled. It’s an important portion but that portion will be handled by … Panpan and Shabnam will be talking about that. They’re a part of our human machine interface, we used to call the interactions, human machine interaction group.

Dr. Uma Krishnamoorthy: So fundamentally we are integrating our hardware with AI, our IoT products and our sensors and that’s in a very, very high-level picture of what Bosch does in IoT. I’m going to stop there and hand the microphone on to Tara. So Tara and Seow Yuen Yee will be talking about sensors next and they will introduce the next speakers. So thank you very much.

Seow Yuen Yee and Tara Dowlat

Senior Research Engineer Dr. Seow Yuen Yee and Senior Account Manager Tara Dowlat give a talk on sensors for IoT at Bosch Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Tara Dowlat: Hi everyone, my name is Tara Dowlat and I’m part of Bosch Sensor Tech. I’m part of the team that focuses on consumer electronic sensors and I’m an Account Manager, part of sales team.

Dr. Seow Yuen Yee: Hi everyone, my name is Seow Yuen. If it’s hard to pronounce you can call me SY. I’m the senior research engineer here in the corporate research. I’m part of Uma’s team. What I do is I make sensors and these sensors go to your car, your home and your phone. So I’ll tell you more about it later.

Tara Dowlat: So, did you guys know that at least every single one of you in this room in your pockets or in your bags have one sensor and most like the majority of you guys had least one sensor from Bosch on you? It’s a fun fact. Let me tell you that sensors are all around us. We might notice it, we might not, but these tiny, tiny little devices are actually pretty commonly used.

Tara Dowlat: They’re made out of micro electromechanical systems. They go also known as MEMS. These devices are made out of silicon. Silicon is the same exact material we use for semiconductor chips and they are used for really complex circuits or switches that we use in our industry today.

Tara Dowlat: If you look at the picture to the right side over here, this shows the structure of a MEMS and you can see that within a thickness of a hair line how many tiny little springs we’re able to fit in there. That’s a MEMS structure for you and typically these devices are within millimeter square. So we can see that how detailed and small these structures are and I find it personally very impressive.

Dr. Seow Yuen Yee: How are sensors made? The process starts with the silicon ingot that you can see on the left there and then it is later cut into thin slices that we call the silicon wafers. So this is an example of the silicon wafers. By itself it is not useful until we are able to process on it to make intricate features. We are able to do this thanks to our Bosch colleagues Franz Laermer and Andrea because they invented the deep reactive ion etching in 1996.

Dr. Seow Yuen Yee: It is now known as the Bosch Process because it has the ability to create a high aspect ratio profile in the silicon wafers. How high is a high aspect ratio and how tiny is tiny? Here’s an example that is the width of these trenches as five micron wide and the height–the deep is 50 micron deep. So you can imagine how small all these features are.

Dr. Seow Yuen Yee: Accelerometer, we’ll tell you later about it. It’s an example of a type of sensors that we are able to create using this process and Tara will tell you more about the sensors and other sensors, about accelerometers other sensors.

Tara Dowlat: So just as SY mentioned, we have a family of classical sensors known as motion sensors. We have magnetometers, accelerometers, gyroscopes, the combination of two that would be an IMU or you put all of the three together it’s known as nine degree of freedom or absolute orientation.

Tara Dowlat: But why do we care about these sensors in general? What’s the application or how do they improve our lives? Well the most classical approach was the use of sensors and automobiles. You guys might have heard about ABS, ESP or even tire pressure monitoring system on newer cars. These are sensor applications. Without the sensors on your cars, you guys would not have these safety functionalities.

Tara Dowlat: Let me ask you this. If you had the choice between a sports car, a sedan, or SUV for safety of your family which class of car would you guys probably pick?

Audience: SUV.

Tara Dowlat: Okay. Let me tell you. Twenty years ago that was not the concept. SUVs and safety were not two words used in the same sentence. Actually these cars were known to be rolling over on the road and actually not safe at all. So what changed since then? The use of a gyroscope on the car is enabling them to stay stable on the road and not roll over. That makes them safe.

Tara Dowlat: Within 20 years or so the market and perception has changed so much that all of you guys think SUV is the best choice to go with. That’s the use of sensor. But, also the modern applications. Take autonomous driving, everybody in the news is talking about it. Autonomous driving would have not been possible without sensors or even more commonly used applications like Park Assist when you tell your car please park it for me in this tight spot. That’s using your sensors in the car, or when you’re trying to drive on the road and hopefully you guys are paying attention and it’s not dismissing the traffic or texting but more modern cars have this functionality that it actually tells you please slow down there’s an object in front of you. Don’t switch lane there’s an object next to you. These are the functionalities that modern cars have because of use of sensors in them.

Dr. Seow Yuen Yee: Applications that Tara mentioned there’s one more applications that should be familiar to all of you which is the airbags deployment. From 1987 to 201,8 more than 50,000 lives has been saved by airbags according to the US Transportation–Department of Transportation. Have you ever thought of how does the car know when to deploy this airbags?

Dr. Seow Yuen Yee: This is thanks to the airbags control unit in the car and in this control unit it has a tiny little sensors which we call accelerometers. When there’s movement like this impact in your car during the accident this [inaudible 00:29:32] this sudden impact.

Dr. Seow Yuen Yee: So let me show you the video of how it works. The accelerometer chip here contains of two parts, that’s the circuit chip and the MEMS sensors. In the MEMS sensors you can see the blue part is the movable part and the red part is the stationary part.

Dr. Seow Yuen Yee: When there’s movement in your car the blue part will move relative to the red part and from there it caused the relative capacitance change between these two parts. This capacitance change can then be sent to the airbag unit here which will deploy the airbags. For that it will protect you.

Dr. Seow Yuen Yee: The sensing part itself takes around 15 to 30 milliseconds time to sense it and the airbags will deploy from 60 to 80 milliseconds. So that’s how fast it is that can deploy to protect you.

Tara Dowlat: So, a more modern recent application for sensors are consumer electronics, specifically smartphones or tablets. You guys have might noticed over the past few years that actually the cameras have improved quite a bit in terms of picture quality. I hate to take all the credit for the sensors but they did play a part.

Tara Dowlat: You guys have might noticed that when you’re trying to take a picture you’re trying to zoom in and historically I was one of the people that would move the camera back and forth trying to get the best photo and then making sure that my picture’s not blurry. Well today the cameras do that for you and part of it is because of the image stabilization and the sensors that they use with the cameras. That’s one of the applications that uses a sensor.

Tara Dowlat: But another more commonly used one. When you go from horizontal to vertical on your phone when you’re looking at pictures and videos this is something that probably most of us use every day. That’s a use of a sensor on your phone. Or this one I’m a personal huge fan–navigation.

Tara Dowlat: I’m always lost and somehow people trust me to put me in charge of direction. But the reality of it is with my phone, if there is no magnetometer on it I’m looking at the direction and I don’t know if it says right is it really my right or my left.

Tara Dowlat: But a magnetometer on my phone would be able to tell me where is the true north and at what point do I need to truly turn right or left. That’s a really helpful application for most of us that we probably use and don’t commonly notice that it’s a sensor on there.

Dr. Seow Yuen Yee: One other thing is as you all know that GPS hardly works inside the building. In the case of an emergency, especially in tall buildings, it is very critical for the emergency first responder to know exactly where you are and this includes what floor you are in. The GPS do not give you this kind of information but our Bosch pressure sensor comes to rescue.

Dr. Seow Yuen Yee: Because of the as you increase the elevation, the altitude the air pressure decreases and this tiny change of pressure can be sensed by our Bosch pressure sensors. So let me show you another video of how the pressure sensor works. Again in the package it has two chip where there’s a circuit chip and the MEMS sensors.

Dr. Seow Yuen Yee: This time the MEMS sensors consist of a pressure sensitive membrane and on which there is four resistors which are connected in a wisdom bridge formation. As there’s the pressure change the shape of the membrane changes due to the pressure and the resistance is changed due to the change of the membrane.

Dr. Seow Yuen Yee: This resistance change is measured as water changes which ranged from one to five and this water changed correlates to the pressure and this pressure would tell you which elevation you are in. The information from this will be sent to the first responder and they will come to rescue you.

Tara Dowlat: Just as SY mentioned, pressure sensor belongs to another family of sensors that are getting quite commonly adapted nowadays, they belong to environmental sensors. That includes temperature, humidity, gas, or a combination of all those together as one single sensor.

Tara Dowlat: But how did they become so popular nowadays? Well, we are all health aware nowadays. I think most of you guys might be interested, but by show of hands how many of you guys track how many steps you’ve taken or how many stairs have you climbed today? Majority of you. Well, I guess most of us has invested in either a fitness band or a smartwatch or look at it on our phones.

Tara Dowlat: When you go under health application it tells you how many steps you’ve taken. That’s an accelerometer on your phone or on your device. Or if you’re interested in knowing how many climbs of stairs you’ve climbed today. Well, that’s a pressure sensor for you that gives that app information. But it’s not just about humans.

Tara Dowlat: So, I recently heard about a cool application from one of our potential customers that they are trying to put this step tracking option on their chicken. You would wonder why. But, I guess when you go to these stores you notice that there is like advertisement for eggs that are range free and organic, that extra dollar amounts that they are charging is justified because these chickens are taking more steps.

Tara Dowlat: The more steps they take, the healthier your chicken. But today we’re here for IoT and how does the sensor relate to IoT. How does that impact me as an individual? How does it change the quality of my life? I can take the example of a smart home. This belongs to the IoT category. Without the use of all these sensors, smart homes would not be possible. Let’s focus on my case specifically and I think some of you guys might relate.

Tara Dowlat: I’m here with you in the evening or the afternoon today. I will spend some time to drive home and during this drive I would be probably sitting in traffic, it’s hot and I’m thinking I wish when I get home that my Roomba has cleaned the floor. So IoT would be able to enable that.

Tara Dowlat: I wish that the AC has been running for the past 30 minutes because I’m somewhat environmental friendly but not extremely. I still like a cool room. So I’ll take that and I can make sure that a cup of coffee is waiting for me while I watch my last show before I go to bed. That’s a smart home for you.

Tara Dowlat: For IoT to be enabled we need to make sure that all these sensors are effectively and efficiently communicating. But then it becomes a matter of power consumption. That’s why Yelena would introduce battery management, which is a really important topic here at Bosch for us. Thank you.

Yelena Gorlin speaking

Senior Engineer Dr. Yelena Gorlin gives a talk on enabling IoT for batteries at Bosch Girl Geek Dinner.   Erica Kawamoto Hsu / Girl Geek X

Dr. Yelena Gorlin: Hi, my name is Yelena Gorlin and I work in corporate research. As Tara and Seow Yuen just mentioned, we will now switch topics and I will introduce a research topic that we have here at Bosch. It focuses on batteries and specifically battery management systems.

Dr. Yelena Gorlin: Before going into the details of the topic, I wanted to take a moment and quickly introduce to you my home department in order to give you an idea what type of associates are working on the project and also what is our overarching purpose for the everyday work that we do.

Dr. Yelena Gorlin: My home department at Bosch is called energy technologies and we have three areas of research competency and they include electrochemical, modeling, characterization and controls, automatic computation and additive manufacturing. As you can imagine, the associates involved in these areas come from a diverse research background and we actually have research experience from leading academic institutions, both in the US and Germany.

Dr. Yelena Gorlin: We’re specifically strong in the areas of chemical engineering, system controls, material science, and electrochemistry. What unites us all is our interest to work on future energy technologies with the goal of reducing the global carbon footprint.

Dr. Yelena Gorlin: Recently we came up with a new motto for ourselves and it’s putting low-carbon options on the global energy menu. Our department sees the topic of battery management systems, both as a contributor to de-carbonization of our society and also as an enabler to our connected future. But you’re probably now wondering what exactly is a battery management system and how can it be so important to our future.

Dr. Yelena Gorlin: So as the name already gives it away and as I mentioned in the beginning, battery management systems have to do with batteries. Probably all of us in this room have been in a situation that seemed quite dire simply because our phone or maybe our smartwatch, our computer or our car has run out of its battery.

Dr. Yelena Gorlin: In such a situation, we were probably wishing that we could recharge our battery as quickly as possible to bring the device back to life. Well, it turns out it’s not so difficult to recharge a battery very fast once in its life. But what is difficult is to be able to offer consistent fast charging without introducing any aging effects.

Dr. Yelena Gorlin: As you probably have guessed, one of the important functions of the battery management system is to offer precisely this capability at battery management system or as we call it BMS for short controls the operation of the battery. So how fast it charges and discharges and each new generation of a battery management system looks to increase the charging speed of our device without having effect on its lifetime.

Dr. Yelena Gorlin: You can imagine that advances in this area can reduce our anxiety about how long our devices can last and as a result contribute to electrification of our society both in IoT and mobility sector and contribute to its de-carbonization. Now I hope I was able to convince you that battery management systems are very important and very significant to our future and I wanted to take a step back again and bring you to my department and our approach to this future product.

Dr. Yelena Gorlin: At its core, our approach draws on the expertise available within the department, and we rely on the different areas of background, especially in research. As I mentioned, we have chemical engineers, we have control engineers, we have material scientists and electric chemists and we primarily combine three areas and its electrochemical modeling, experimental characterization, and controls.

Dr. Yelena Gorlin: Our typical project workflow starts with the development of an electrochemical model and involves a variety of equations and parameters. We then design and execute experiments to measure these specific parameters and combine them together with a model to form what is known as parametrized model.

Dr. Yelena Gorlin: This parametrized model serves as the basis for the next generation BMS and is used to generate new control algorithms. These control algorithms are what is going to allow us to charge our devices, so our watches, our phones, our computers, and our cars at faster speeds and therefore increase our confidence in all of these IoT components and contribute to the development of our connected future.

Dr. Yelena Gorlin: Thank you very much for your attention. I will now pass the mic to LisaMarion who will tell you about artificial intelligence.

LisaMarion Garcia speaking

Software Engineer LisaMarion Garcia gives a talk on artificial intelligence at Bosch Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

LisaMarion Garcia: Hi, everyone. My name is LisaMarion I work at the Bosch Center for Artificial Intelligence here in Sunnyvale. So, we have a lot of opportunities for AI at Bosch. As my previous colleagues have mentioned, we cover a wide variety of different sectors from mobility, industrial, building, and consumer goods. Each of these individual sectors provide us different opportunities to incorporate AI, either as a feature of a product that we sell or as part of the process of producing that product.

LisaMarion Garcia: As Uma had mentioned before, that is a major goal for Bosch, to by 2025 have all our products either possess some artificial intelligence as part of their features that we provide to the consumers or as we produce them we are using AI.

LisaMarion Garcia: What we need to introduce AI into our products or our processes is–what gets discussed mostly when people are talking about artificial intelligence tends to be focused on the algorithms more. So that’s basically how you actually train a system to be able to learn by itself, how a car can drive itself, for example.

LisaMarion Garcia: We do work on that in-house as well. The Bosch Center for Artificial Intelligence has a pretty sizable research team that is currently working on state-of-the-art research topics. But additionally to actually get it from an idea, from a theoretical idea, into a product we need both compute resources, which we of course have access to, and most importantly, we need data.

LisaMarion Garcia: So, one of the advantages that being such a large company gives us, especially a company that covers so many different sectors is that we have access to a bunch of different types of data. BCAI overview, I guess. Our general mission is to help reach that goal, obviously, of introducing AI into the different areas.

LisaMarion Garcia: I’ve already covered our research team. We also have an enabling team which are–you can kind of think of them as AI evangelists. They go out to the different business units and kind of teach them about what machine learning is, how it can help in their products, what kind of data they need to be collecting if they want to be able to gain relevant insights from it.

LisaMarion Garcia: Then we have the services team which is where I work. We focus more on applied AI. So what we do is we consult with various business units within Bosch who have use cases or interested in introducing machine learning into their products or processes and we basically help them take that from an idea to a reality.

LisaMarion Garcia: We cover these four different areas. I’m going to briefly describe kind of each one. We have a bunch of different projects ongoing right now. But for an example in the manufacturing domain, something that we do is we work with optical inspection, which is where we put a camera in the production line at Bosch’s many plants and we basically collect images of the parts as they come through and try to perform or try to train a model to do automated part inspection. So basically being able to tell if a part is passing or failing by just looking at an image of it.

LisaMarion Garcia: In the engineering space, we do some work around gaining insights from data that is collected as we develop a new sensor, for example, for a new product or if we are trying to add kind of a smart home type of functionality to an existing appliance that Bosch already makes.

LisaMarion Garcia: For supply chain management and controlling we have a financial forecasting platform that basically looks at all of Bosch’s financial data and can make predictions about future sales. Then intelligence services, which I’m going to go into slightly more detail on since that is more of what I have worked on recently.

LisaMarion Garcia: So AI for mobility is obviously a hot topic. We have two main groups at Bosch that are working on that. We have for my friends that work in the autonomous driving space you may be familiar with the L3 to L5 kind of designations.

LisaMarion Garcia: So we have a driver assistance functions which are going to be your L3 and below. Those are things like automated braking when you detect a hazard on the road or lane keeping. Kind of those functionalities that already exist in your car. We also have autonomous driving group, which is the car outside, which would be the car driving itself.

LisaMarion Garcia: Some collaborations that this group has done with BCAI that I’ve been involved with have been lane keeping. So if you see the top image, we basically take a semantic segmentation map of a scene and basically use that to keep the car on the road. We also do hazard detection.

LisaMarion Garcia: So if you look at these two images in the middle, the one on the left is mostly clear windshield, the one on the right the windshield has been obscured with some droplets of water. A human looking at these two images can clearly tell that they’re the same scene. We basically our brains have a really good way of mentally deleting the information that you don’t need.

LisaMarion Garcia: It’s very difficult for a computer to do the same thing. That’s one of the main challenges when we’re training algorithms to be able to see, for example, for driving a cart. So we’ve done some work around helping either make the model itself more robust to these kinds of disturbances or basically just having some kind of a sense so that the car knows when one or more of the cameras has been had its vision obscured.

LisaMarion Garcia: Then the last topic, which I wanted to cover in slightly more detail, is the data privacy compliance topic. So I’m not sure how many of you are aware of the GDPR regulation. Yes, okay, a lot of nodding. So that’s a really important law that was passed by the EU which basically … The general gist of it is that any company that is collecting personally identifiable information from people without their consent basically needs to delete that data every six months or somehow you scrub the personally identifiable information.

LisaMarion Garcia: For our automotive topics, that mainly covers human faces and license plates. So what we did to help our business units and prevent them from throwing away their data every six months is we developed a tool using deep learning to be able to identify, locate the faces and license plates in the data that was generated by the proprietary Bosch sensors and blur those out of the image.

LisaMarion Garcia: So basically what we are doing is helping them generate training data that they can use long term and also store, which will help them basically consistently validate their work over time. So, yeah, just AI for your AI. That’s kind of the overview of what Bosch is doing in regards to AI. I have kind of mostly talked about how we spread AI internally and now I’m going to bring the user back into the conversation and pass off to my colleagues to talk about human machine collaboration. Thank you.

Shabnam Ghaffarzadegan speaking

Research Scientist Shabnam Ghaffarzadegan gives a talk on human machine collaboration research at Bosch Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Dr. Shabnam Ghaffarzadegan: Hi, my name is Shabnam. I’m a research scientist here at Bosch working in human machine interaction group and I’m very excited to be here with my colleague Panpan Xu who is our group too.

Dr. Panpan Xu: Hello everyone. I’m Panpan, I’m also working on the human machine collaboration topic at Bosch Research. So today Shabnam will first give an introduction of what are the topics we have been working on.

Dr. Shabnam Ghaffarzadegan: The topic we are really excited to work here at Bosch is human machine collaboration. If you think about everyday life there’s so many tasks that human is so good at but machine usually has so much trouble doing them. Also there are so many tasks, let’s say repetitive tasks, that machine might be so good at doing them very accurately but human would be having so much trouble to perform them in a short amount of time.

Dr. Shabnam Ghaffarzadegan: So our idea is asking human and machine to work together to empower their both abilities to make a superhuman with much more perception and knowledge and also to make a better machine to help us in our everyday life. Here at Bosch, we do focus on many core technologies such as robotic manipulation, text mining, audio analytics and visualization. We do apply these technologies to so many different use cases such as IoT industry 4.0, smart home, and smart cars.

Dr. Shabnam Ghaffarzadegan: How we do? So here first I’m going to introduce you how AI can help humans. So our goal is empowering human capabilities. What we do in our group is that we take different modalities that we see in the environment such as visual clues, text and audio and speech that we hear around ourselves and we combine this information with domain knowledge, context knowledge and user knowledge and we translate them to some specific applications such as personal assistants, conversational AI, and augmented reality.

Dr. Shabnam Ghaffarzadegan: As I mentioned, our goal is empowering human with domain specific AI. Here our focus on one of the use cases we work that I focus on personally, which is intelligent audio analytics. If you think of course the speech is one of the main … No, it’s okay. We can continue hearing that. It’s fine.

Dr. Shabnam Ghaffarzadegan: Okay, what I wanted to say was that if you think about speech, of course, it’s one of the main input and the way of communicating with outside world as a human, right, but there are so many other sounds that we can hear in the environment such as the sample of sounds you just heard. Right?

Dr. Shabnam Ghaffarzadegan: By these sounds you can guess kind of what kind of environment you were at. Were you at the beach or where you at a restaurant, right, just by listening to the noise in that environment or you can guess what kind of machine are you operating. Is that machine is working in a right mode or is it broken? Right?

Dr. Shabnam Ghaffarzadegan: So here in our group we focus on signal processing and machine learning techniques to discover three kind of sounds. The first one is environmental sounds. As you heard, is it beach, is it in the office, is it in a restaurant? The second one would be machine sounds. Right?

Dr. Shabnam Ghaffarzadegan: We hear, we listen to the different machines in the environment and we try to recognize if they’re malfunctioning or working in the right state. And finally human sound, but non-speech human sound. Imagine you might be coughing or sneezing and that might be a clue that you might have some health issues and you might want to go to a doctor. Right?

Dr. Shabnam Ghaffarzadegan: So the audio analytics field is kind of newer compared to vision or speech technology that already exists so we have so many challenges at this field and the main one would be lack of data as always existing artificial intelligence and also we need to be really robust toward the other different kind of noise and environments that we are at.

Dr. Shabnam Ghaffarzadegan: Here’s some of the use cases we work on. The first one we can focus on physical security and automation. You think that in most places the physical security systems are based on cameras but there might be so many situations cameras might fail. Let’s say, if it’s dark at night or if it’s foggy so the camera might not see what’s happening in environment. But also there are some events that camera is visual clues are not able to capture them.

Dr. Shabnam Ghaffarzadegan: Let’s say gunshot. Right? With a camera if the gunshot is not in the visual field you can’t basically [inaudible 00:54:23]. So, our idea is including microphone to a camera to understand more information about our environments. In this case, such as gunshot, glass break, and a smoke alarm can be sounds that can alarm our physical security system.

Dr. Shabnam Ghaffarzadegan: The next use case is industry 4.0. As I mentioned, we would like to put microphone in our plants and listen to the machines that working on those plants. For this, this is a very easy step to move toward industry 4.0 since the only thing we need to do is basically we put a MEMS microphone on these devices and just listen to them to see if they are operating correctly or not.

Dr. Shabnam Ghaffarzadegan: The third one would be an automotive sensing and diagnosis. Of course, autonomous cars, they are hot topics these days and they are having so many sensor already on them such as radar, camera. But we believe that autonomous cars needs to have the hearing sense as well. One of the important use case would be for example hearing emergency vehicles if there is siren happening for example police car or ambulance so these autonomous cars needs to understand these sounds and act accordingly.

Dr. Shabnam Ghaffarzadegan: Another use case can be listening to your car parts, for example, your car engine. If you go to repair shop so many of the very experienced repair shops they just listen to your engine and they would guess if you have a problem, so this is our idea to do that automatically.

Dr. Shabnam Ghaffarzadegan: Finally to give you some idea how we perform these acts. So basically we do use microphones to get this raw audio input from the environment. This information, we do some signal processing to enhance this signal to remove some environmental noise that we don’t want them and we do use domain knowledge, meaning that we do look into what kind of environment we are performing.

Dr. Shabnam Ghaffarzadegan: Are we in a factory? Are we in a house? Are we in a car? Based on that we extract some features and finally we do machine learning and AI to detect what kind of audio events was in the environment. Next my colleague, Panpan, she will explain now how human can help AI.

PanPan Xu speaking

Lead Research Scientist PanPan Xu gives a talk on human machine collaboration research at Bosch Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Dr. Panpan Xu: So, here comes the other side of story, how can human help make AI more intelligent and more reasonable to the humans. So, our approach is actually very much human in the loop method for big data analysis which we call visual analytics. Visual analytics is actually a technique which combines technologies from many different fields and one of these field is data mining.

Dr. Panpan Xu: With data mining we basically trying to gain insights from data with automatic algorithms and identify the patterns inside it. The other technique is visualization. Basically, we can draw the chart to show different trends and patterns detected by the data mining algorithms and then show or present to the users.

Dr. Panpan Xu: Most important part is user interaction. Actually, in this user centric approach we want to really take in users’ input or users’ knowledge into the data analysis process so it does not appear as a black box choose users. So, one use case that is very much related to this visual analytics topic is expandable AI.

Dr. Panpan Xu: Basically, in most of the cases we use AI as a black box. Basically the machine learning model takes the input and then produce some output to–For example, in autonomous driving we take the video input from the camera and then the steering wheel will take the corresponding directions or in medical diagnostics solutions the AI usually take an image and then tells the doctor or the patient what kind of disease it is.

Dr. Panpan Xu: But this kind of black box approach is usually not much reliable or people do not really want to use the machine learning model as a black box. So, with visual analytics we can present the explanation to the users actually and then the user can provide feedback to the model and continuously improves model until the model becomes transparent or explainable for the users.

Dr. Panpan Xu: Why this is important as I explained, we have these fairness issues because we want to know AI is making its decisions based on some meaningful features instead of other features like gender which can make this model unfair to certain populations and also we want to make this model robust.

Dr. Panpan Xu: On the other hand. There’s also this GDPR regulation which requires every decision made by AI to be explainable to the humans. So the user have the right to assess explanation to the decision made by an algorithm.

Dr. Panpan Xu: So now let’s go in on our deeper technical dive to look at a recent research paper we have published at ACM [inaudible 01:00:04] this year and which is about interpretable and steerable sequence learning. And that has application in many different AI fields like text mining or medical diagnostic sensor.

Voiceover: Recurrent neural networks have shown impressive performance in modeling sequence data. They have been successfully used in a lot of applications, sentiment analysis, machine translation, speech recognition and so on. However, they are considered as black boxes since it is very difficult to explain their predictions. Without explainability it could cause trust and ethics issues.

Voiceover: How can I trust the predictions coming out of a black box? These problems will limit the applications of these deep learning models in various decision-making scenarios. For example, a data scientist has developed a sequence prediction model to predict the risks of future problems of a car based on its historical faults.

Voiceover: However, the mechanics and repair shops may find it difficult to choose the right maintenance strategy with just prediction results. Sometimes they even suspects that the modeling is wrong. The need for explanation is pervasive in such decision-making processes. The predictive model serves as a smart analysis module rather than an automatic end-to-end solution.

Voiceover: Our idea is to explain the predictions by providing similar examples. Such case based reasoning strategy is commonly used in our daily life. For example, why classify a restaurant review, “Pizza is good but service is extremely slow” as negative? This is because it is similar to two prototypical negative sentences, good food but worse service and service is really slow.

Voiceover: We use sequence encoder R which encodes the input sequence into a fixed length embedding vector H. The model learns K prototype vectors that are most representative in the embedding space. We compute these similarities between H and the prototype vectors. The similarity scores are used as a source for prediction. To ensure that the prototypes are readable, we project the prototype vectors to their closest training samples every few epics.

Voiceover: To further improve interpretability, we’ve simplified the prototype sequences using a beam search based algorithm. To utilize expert knowledge, we design an interaction scheme which allows human users to incorporate their domain knowledge into the model. We build interpretable and steerable sequence models for vehicle fault predictions, sentiment analysis, protein classification, and heartbeat classification.

Voiceover: You can get explanations to the accurate predictions on the fly.

Dr. Panpan Xu: I would like to thank [inaudible 01:03:03] for the very nice voiceover of the video. So, if you have any questions about the paper you can search it online. So there’s the title below at the bottom of this slide. So, now let’s move on to the next topic and see how Bosch is enabling a new area of mobility with our presenter Sun-Mi here.

Sun-Mi Choi speaking

Director of Business Development & Strategy Sun-Mi Choi gives a talk on changing mobility with progressive mobility players at Bosch Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Sun-Mi Choi: Hello. Also from my side I guess I’m the last turn. I hope you guys are still with me. That was a little bit too silent. Are you still with me?

Audience: Yes.

Sun-Mi Choi: Okay, good. Thank you. I know it’s late. My name is Sun-Mi Choi. So please just call me Sunny. I’m Sunny from Sunnyvale so it’s easy to remember. I’m responsible for business development strategy within a newly established group. We are probably the youngest group within Bosch. We are eight months old so we were born beginning of this year and probably also the smallest group and we are called progressive mobility players, short PMP.

Sun-Mi Choi: I will tell a little bit more about it later but basically what we do is focus on new mobility startups because we see the mobility world is changing a lot. A lot of new players are entering the market and we are focused on two players which are new electric vehicle manufacturers and at the same time also on mobility service providers.

Sun-Mi Choi: Today we’ve heard a lot about innovative amazing technologies, learning about sensors, learning about battery management solutions, artificial intelligence, and human machine collaboration. I’ve been with Bosch seven years but I didn’t know that we had so much capability in-house. I just moved here beginning of this year so it’s amazing to see how much capabilities we have.

Sun-Mi Choi: I would like to bring in a little bit of a different perspective. Basically bringing a little bit the market perspective customer needs to explain and verify why these capabilities are so important for Bosch and also for the future of mobility.

Sun-Mi Choi: So, before I start, I would like to give a little bit of a bigger picture of why the mobility is changing and what are the driving forces behind.

Voiceover: Our world is changing and this change is visible across the globe. More than 50% of our population now lives in cities. These cities are growing, as is the share of older people in them, while space to live is becoming ever more precious. More and more goods and people need to be transported, pushing the traffic infrastructure to its limits and increasing pollution and noise levels.

Voiceover: But the world is waking up. Regulations are calling for stricter limits and cleaner solutions. A transformation has started, powered by new technologies and services. In a world where everything is connected, mobility is being re-imagined. Solutions like traffic management combined with cleaner and more efficient power trains and the benefits brought by automated driving will make our cities sustainable and livable.

Voiceover: Bosch is driving this change and shaping the future. The future of mobility.

Sun-Mi Choi: Trends they are not new for you. But it’s still very important to understand the fundamental driving forces behind it because this actually has a really big impact on Bosch. Because as we learned from Uma, the mobility part makes 60% of our revenue and all of these changes make a huge change or an impact also our business model if we want to maintain sustainable for the future.

Sun-Mi Choi: So air pollution, congestion, urbanization, and also what we see a changing consumer behavior, all of these factors are really shaping a new focus for us in the mobility area, which we call electrified, automated, connected, and also shared and personalized, which you probably experience and also live every day.

Sun-Mi Choi: At the same time, mobility is also getting more user centric. The consumer is more and more changing from owned to shared. So how many of you are using ride hailing apps to get from A to B on a regular basis? So I see not everyone, but I see a lot of hands raised. So this has become an integral part of how we move from A to B because it brings convenience, especially in congested cities.

Sun-Mi Choi: Also, consumers become more individual and personalized and more importantly, they always want to stay connected. This all relates to mobility and new players, startups see this change and these trends as basically opportunities to come into the mobility market. Because now new capabilities are required and this disrupts the whole mobility value chain also from our Bosch perspective.

Sun-Mi Choi: So what does it mean for us? We also need to understand what these new players are about to develop, what is their thinking. How do they approach innovation? That’s why as mentioned in the beginning we are focusing on new EV based customers.

Sun-Mi Choi: So probably a lot of you know Tesla in this area. So really young companies who are starting vehicles from scratch or the second customer segment is mobility service based customers. So, all companies who provide mobility as a service, the ride hailing apps, car sharing and so on.

Sun-Mi Choi: What we see is that they have quite of a different DNA, they have different requirements. That means also for Bosch, we need to understand the requirements and adjust also the way how we approach customers. Because these young customers, they act differently, they drive innovation differently than the VW or Mercedes driver that we’ve been dealing with for the past hundred years.

Sun-Mi Choi: So it’s time to change and it has also a big transformational impact on us. So, we see in the shared space, for example, the one customer segment we are focusing on is huge change. If you look at an annual number of ride hailing rides you see a tremendous growth over the past four years. It’s been grown more than 60%.

Sun-Mi Choi: From a user perspective, you also see a good reason why they are switching from ownership to shared. One of the reasons is because 96% of the time your asset stands idle. The car is parked, you’re at work, it stands idle for eight, nine, 10 hours while you sleep also. This this is a waste of assets.

Sun-Mi Choi: So people are looking for alternative modes to move, alternative modes how to utilize their assets in a most, more efficient way. So also this is one indication for why people are moving towards shared. Last but not least, from an investor perspective, if you look at how much investments have flown into this area over the past four years only more than 80 billion US dollar have been invested into the ride hailing market.

Sun-Mi Choi: This is humongous. This is likely to grow further. So, this shared mobility will happen. So how do these new customers take, what are the pain points, what are the requirements? These are just some of the requirements or pain points that we identify when speaking to the customer. So operational costs for these ride hailing companies is a sure thing.

Sun-Mi Choi: How can we become profitable? How can I optimize my operations? Second point is how can I ensure safety and security for their passengers, especially when we go towards robo taxis, it will not have a driver anymore being able to control the ride. So we need technology to basically operate and also ensure the safety even without a driver.

Sun-Mi Choi: Third is there are so many players arising, I need to differentiate. If I want to survive in this market I need to have a good differentiation point. So personalization, how to ensure that your ride is individual and a really great experience is one important differentiator that we have identified.

Sun-Mi Choi: For all these pain points, for all these requirements that we see, it kind of makes sense where you bring now the puzzle pieces together of the capabilities that we’ve seen from sensors which connect the cars, can connect the car and the user and a lot of other use cases that we’ve learned today.

Sun-Mi Choi: Battery management solutions is super important because we see a strong push towards electrification pushed by the government. Also end users are looking for environment friendly solutions. Also a lot of these ride hailing companies tend to establish their own EV fleets.

Sun-Mi Choi: So range anxiety and also improving the battery lifetime what we learned today are super, super crucial for the customers in the market. Autonomous driving was something that was mentioned. So a lot of these companies are also going towards robo taxis. So artificial intelligence is also human machine collaboration to really ensure that there is a safe and also unique experience between the human and the machine will be very relevant.

Sun-Mi Choi: When we look at the customer and the market and the customers, we see that these capabilities will be important for the future to come. So I’m very proud to see that we are working on these very future-oriented topics. This is the way how we would like to tackle the new era of mobility.

Sun-Mi Choi: So basically in summary, with these capabilities enable the vision of our mobility customers not only the new ones, of course, also the existing customer base. Second, we want to innovate and co-create with these customers together. Because even though we have the best technology that might be requirements that we may not have seen so we need the customer input to even more improve the technology and also the use case.

Sun-Mi Choi: Last but not least, important point is really to understand and translate what the customers tells it to us into technology. That’s why it’s a good collaboration to have technology and also sales and the market proximity close to each other so that there is always an inter-linkage and a bridge between technology and also market need.

Sun-Mi Choi: So, we’ve talked a lot about AI, about new customers, about innovation, but I think it’s also important to really close with the core, with the tradition to not forget about the core business and also the roots where this company is found on. So two values from Robert Bosch, the founder, since 1886, have been that he says, “I have always acted according to the principle that I would rather lose money than trust.”

Sun-Mi Choi: So the trust to the customers, to the market, providing safety is one really crucial element. Second point for doing business also with our customers is integrity. Integrity of the promises we make to our customers in regards to quality and also in terms of the promises that we make to them. This to the founder and the values still hold today our prioritizing this versus just having a short-term transitory profit.

Sun-Mi Choi: So I would like to remind us all of us when we speak about future topics to think about the core values as well because these are important. This is how I would like to close the presentation. Thank you very much for the one hour attention. So you have been an amazing crowd.

Sun-Mi Choi: I went a little bit over time, so thanks a lot for your patience. I think we had great presentations here today. I would like to thank all of you on behalf of the whole team for coming to our Sunnyvale site, for showing interest in our portfolio, in our technologies. And we would be happy to see you again, also to mingle and network after and to see if we have some collaboration opportunities.

Sun-Mi Choi: Last but not least, of course, I would like to thank all the staff, the presenters, and all the people who have helped to support making this event happen. It was a lot of work. So let’s have a nice evening and please don’t leave too quickly. Thank you very much.

Uma Krishnamoorthy, Hauke Schmidt

Like a Bosch: Tara Dowlat, Seow Yuen Yee, Yelena Gorlin, Panpan Xu, LisaMarion Garcia, Shabnam Ghaffarzadegan, Sun-Mi Choi, Uma Krishnamoorthy and Hauke Schmidt.  Erica Kawamoto Hsu / Girl Geek X


Our mission-aligned Girl Geek X partners are hiring!

Girl Geek X Clover Lighting Talks & Panel (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Mary Uslander, Ellen Linardi, Rachel Ramsay, Meghana Randad and Bao Chau Nguyen speaking

Clover girl geeks: Mary Uslander, Ellen Linardi, Rachel Ramsay, Meghana Randad and Bao Chau Nguyen speak on a panel at Clover Girl Geek Dinner in Sunnyvale, California.   Erica Kawamoto Hsu / Girl Geek X

Transcript of Clover Girl Geek Dinner – Lightning Talks & Panel:

Gretchen DeKnikker: Hi, I’m Gretchen, I’m with Girl Geek X. Welcome. How many of you guys, this is your first event? Oh wow, that’s so many. We’ve been doing these for about 11 years. We’ve done over 200 of them. We do them almost every week, up and down the peninsula, so hopefully you should be on our … That’s all right, I can definitely talk over that. We do them every week and you should come because you get to see amazing women, you get to meet amazing women, and you get to feel inspired so that you can go back and fight the good fight every single day, right? Yes.

Gretchen DeKnikker: We do a podcast also, if you want to check it out. We take like little clips from these events, and then we chitchat around them. So, there’s like finding a mentor, and what’s the right way to use the word intersectionality, and all sorts of really important life skill things. Definitely find it, rate it, keep it, and tell us if it’s any good, because we’ve never done a podcast before so we’re still figuring it out. Then finally, we just launched a store on Zazzle with all of our cute little Pixie things. You guys haven’t seen a lot of them because they weren’t on the branding for this, but it’s super cute.

Gretchen DeKnikker: Can I borrow you because I love your hair? Can you hold this for a second? I love her. We have this cute fanny packs and a little bag that you could put cosmetics, but you could also put Sharpies or something less female in, and water bottles. All sorts of stuff, and they have our little Pixie characters, they say, “Lift as you climb.” That’s it, we’re good. That’s all the things that are in my bag. You were an awesome assistant, everyone give her a hand.

Gretchen DeKnikker: This space is awesome. I’m so excited for the content because everything that we’ve experienced thus far has been really amazing, right? Yes, you ate, you had your… They’re not quite awake yet, but we’re going to get them there. I am not a good warm up for this, apparently. Without further ado, please welcome Jennifer Oswald from Clover, who’s the head of People Operations.

Jennifer Oswald

Head of People Jennifer Oswald welcomes the sold-out crowd at Clover Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Jennifer Oswald: Hi, everyone, I’m going to try and navigate a lot of different technology while I’m up here. I’m Jen Oswald, and it’s my pleasure to have you all here to kick off our collaboration with Girl Geek X. This is an event on unconscious bias. I’d like to thank you for attending and I can’t wait to hear what takeaways you have from this event. We know that events like these can impact your lives and have a lasting effect on not only your professional life, but also your personal life.

Jennifer Oswald: Our agenda this evening is as follows. First, me, I’m your introduction and welcome. Then we’re going to look at what we do. We’re proud to showcase a bit on what we do here at Clover. You’ll also be meeting our CEO, who will talk you through that. We’ll have lightning talks as well that will show you a little bit more about our product. Next we’ll be featuring our panel discussion on unconscious bias, and then lastly, we want to make sure you still have time to network, and don’t forget your swag.

Jennifer Oswald: Maybe a silly question, but who is confused by me being up here today introducing unconscious bias? You don’t have to raise your hand, you can just think it if you want. Would it surprise you to know that I grew up identifying as two races, Native American and Caucasian? That was before a DNA test. More to come about that later. When biases come to mind, what did you think when you saw my picture before this event? What did you think when I came up here? That is unconscious bias, it’s bias happening in our brains making incredibly quick judgments and assessments of people and situations without us even realizing.

Jennifer Oswald: They can be influenced by our background, our cultural environment and personal experiences, and resolving feelings and attitudes towards others based on race, ethnicity, age, appearance, accent, et cetera. Also termed as implicit social cognition, this includes both favorable and unfavorable responses and assessments activated without an individual’s awareness, or intentional control.

Jennifer Oswald: How did I get here? That’s little baby Jen and that’s my mom. As you can probably see, she was a very, very young mom. She had me at a young age, she worked the night shift and we lived in the projects aka, subsidized housing. That’s a picture of Iowa City, Iowa. We were on food stamps and we struggled to get by. Even at a young age, I knew what it was like to struggle. Then the classic story, mom meets dad, he adopted me at about age six and life was a little more middle class and a little more in the middle of nowhere.

Jennifer Oswald: I grew up in Palmer, Iowa. This is a picture of our downtown. That is the one gas station, right next to it was the grocery store/where everybody went to have coffee in the morning. I was in a town of 256 people, so how diverse do you think that was? Here I am, I’m the only adopted person in the whole town, mixed, left-handed, and female. How many do you think were college grads? I was supposed to get married, raise three to five kids, maybe have a job after I took care of the kids and at the very least, I should be a great cook and make sure that everyone is well fed. So, what do I have? I have a college degree, an almost masters, zero kids except for my fur babies, zero husband, and I just moved from the Silicon Hills, Austin, Texas, to Silicon Valley.

Jennifer Oswald: My unconscious biases tell me that men should have a career, women should stay home and raise a family. Being adopted means you don’t really have a family like others. Men should make the money, women should tend to the family. Once poor, always poor. You should write with your right hand because everyone else does. Men are better at math and science. Yet here we are at a tech company with a panel of amazing females to tell you about their experiences and biases they’ve encountered, and how they proved many of my own unconscious biases wrong.

Jennifer Oswald: We all have unconscious biases. It comes from our culture, it comes from our families, it comes from our family’s families, yet once recognized, we can overcome them. So here I am, a place I shouldn’t even tried to get to, kicking off an event for an amazing company that says FU bias, and we’re working to overcome and support diversity and inclusion. No matter what the package looks like on the outside. We hire the book, not the cover. On that note, I want to introduce the person responsible for creating such a great place for like-minded people to come together. In fact, in 2019 he was nominated for two awards, Best CEO for Women, and Best CEO for Diversity, and we just think he’s the best. I’d like to welcome John Beatty, our Clover CEO.

John Beatty speaking at Clover Girl Geek Dinner

CEO John Beatty talks about the change that needs to happen in the world at Clover Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

John Beatty: Thanks, Jen. Welcome everyone to Girl Geek X. You know, I get the opportunity, of course I have to promote my own company. There could be no better promotion of Clover than what you just saw with Jen. She’s our new Head of People, and I think she’s absolutely amazing. Really excited to grow our people function here, so thank you very much, Jen.

John Beatty: First I’m going to just tell you a little bit about what we do here. You’ve probably encountered a device that looks very much like this. We are all across America, we’re also in a number of other countries. Thank you. We build absolutely beautiful cloud based point of sale hardware and software and systems. I’ll tell you the reason why we did this, this is going back, we started Clover about eight years ago. What we saw was a bunch of really ugly, really insecure, really closed systems and there was … on the counter at all these restaurants and retailers and services companies. We were trying to bring some innovations into that market and just ran into a bunch of brick walls.

John Beatty: We started talking to business owners and we realized they absolutely hate their systems, they keep having data breaches, the systems really don’t help them run or grow their businesses very efficiently. We thought that was a very interesting problem to solve. We love small businesses and recognize that a lot of small business owners are just trying to do what they love and they need technology to support them. We have many, many … We’ve manufactured over 1 million devices. The US is our largest market, so you have almost certainly encountered one of our devices.

John Beatty: On the consumer side, we have a very engaging consumer experience. First, the consumer journey starts off typically signing up for a loyalty program. You’ve probably seen one of these as well, you just type in your phone number and then we extend that consumer journey–if we could could go to the next slide, all all the way to the mobile phone. We have a very highly rated mobile app as well. It starts off with loyalty, but of course we also have Bluetooth beacon enabled payments. You can walk into a store, you don’t even take it out of your pocket. They know you’re there, they know what you like. You don’t even have to pay. You just say, “I’d like to pay with Clover,” and you walk out. It’s a very magical experience.

John Beatty: On the other side of the counter, they have a Clover device. Your profile picture will show up there, a little bit about your history, how often you’ve been there and what you like. We’re really building an absolutely fantastic end-to-end experience both for the merchant and the consumer.

John Beatty: Now, we also have an app marketplace that helps businesses run and grow their businesses. We take a lot of the … We make a lot of the mundane, very simple. We have a number of partners in categories like payroll. If you want to make your life very easy as a business owner and get all the employee information and get it into your payroll system, we make that very seamless. We work with best-of-breed other companies and we partner with many of them here in the market.

John Beatty: That is enough about Clover. I know I get a few minutes here of corporate shilling, so thanks for bearing with me. First, I want to talk a little bit about, what does it take to win one of these awards? Let me just tell you, when I first saw the news that I’d won these awards, I had two thoughts. The first is, “Well, that’s really cool. I’m very proud of that.” Then the second is like, “How did that happen?” To be completely honest. So first, to talk just a little bit about the pride that I felt. These middle meant a lot to me, both personally and professionally.

John Beatty: Personally, I have a–I have a wife. My wife is right here in the front row. She’s a scientist who’s now in business development. Very accomplished in her field. I also have a six year old daughter, and I also have two boys, four and two. I’m not going to go into any details. Let’s say, my wife has run into some professional situations that are absolutely outrageously unacceptable. I think the world has made a tremendous amount of progress in being more fair and just over the last 50 years, but there’s a lot of work left to do. And with all of my kids, both my girl and my boys, I’m very … When they grow up and they see that I’ve done things like this, I’m very proud that I can say I helped make the world more fair and just. That means a lot to me personally.

John Beatty: I asked the question, what does it take to win one of those awards? Honestly the answer is, not enough. The bar is actually just too low. I will say we try very hard at Clover on diversity and inclusion, but we are a small company. Just a short number of years ago, we were a very small startup just trying to survive. Most of your thoughts on, how do I not die, not, how do I create the world’s best culture?

John Beatty: Now that we’ve grown up a little bit, now we are very focused on building out those programs. We’re out of the almost dying category and into the very successful category. I’m very proud that we’re doing events like this tonight. But, this is very recent for us to actually build these institutions. We have a Women in Tech Group here at Clover, and that’s a very grassroots effort. It’s building and it’s building, and we’re really getting a lot of great programs here.

John Beatty: I could win this award with honestly not doing that much proactively, just avoiding the unforced errors and making sure we squash any bad behavior that we see, it means the bar’s probably too low. That’s the Clover story. If you could just jump of course, I’m going to show one more time. We have recruiters standing by. Alicia, John, they are waving at you right there. They would love to talk to you and of course, Clover.com/careers.

John Beatty: I’m going to introduce Rachel. Rachel, why don’t you come on up? Rachel is on our software engineering team on our Payment Terminal API and she will tell you a little bit more about what she does in a lightning talk.

Rachel Antion speaking

Software Engineer Rachel Antion gives a talk on semi-integrations and how it fits into the business at Clover Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X  

Rachel Antion: Hi, my name is Rachel Antion and I’m a software engineer here at Clover on the semi-integrations team, which is our internal name for the Payment Terminal API so if I use them interchangeably, that’s why. Overall, we make about 2 billion card transactions every year, which amounts to be about $100 billion on over 1 million devices sold in seven countries, and we are approaching 5% of Visa and MasterCard volume worldwide, which I think is pretty impressive considering we’re only in seven countries right now. Of that, 2.5% of those transactions are processed via the Payment Terminal API, which might not sound like a lot until you think that it’s about $2.5 billion, and it’s growing every year.

Rachel Antion: Can you click it? Some of those transactions are coming from integrators that you probably recognize like Amazon, the Las Vegas Convention Center, the stadiums of the Philadelphia Eagles, the Seattle Seahawks, and the New York Mets. All of these integrators created their own solution customized to their individual business needs. Here is a specific example of a solution built with the Payments Terminal API. This is a beautiful point of sale created by Hy-Vee that’s totally customized to their individual business needs. But in order to appreciate just how awesome this is, you might need to know a little bit more about the Payment Terminal API, where it came from, and how it works.

Rachel Antion: People have been taking payments for pretty much as long as people have been around and as we progress, the way that we take payments also has to progress. When credit cards were first introduced, there was not a lot of security, but as the age of the internet progressed, so did the need for that security. Older point of sales basically consisted of some kind of UI attached to a magstripe reader that would send unencrypted data to the point of sale, which might make all of you uncomfortable because it led to things like the data breaches that started in 2010.

Rachel Antion: Clover knew that there had to be a better way to take secured payments without making companies throw away all the hard work they put into developing their point of sale systems. That solution was the Payments Terminal API, which allows you to use a Clover device as an external payment device. Your point of sale gets a Clover payments API, and Clover provides the PCI compliance. Basically, you make the point of sale and Clover takes care of the rest. All the point of sale needs to worry about is creating the order and making sure the right amount gets sent to the Clover device.

Rachel Antion: We have two different flavors, if you will, of the Payments Terminal API. We have Native or takeover that lets you create your own app that runs directly on the Clover device, and we have Remote that lets you run it on pretty much any device. We have SDKs and Android, iOS, Windows, and JavaScript so the possibilities are pretty endless. That beautiful point of sale I showed you earlier is actually an example of a takeover model. You can see it here running on our Clover station.

Rachel Antion: Who exactly is the Payment Terminal API for? Its for someone who has an existing point of sale. Maybe everybody’s already trained, they know how to use it and it works just fine, but they want to use a Clover device to take payments because it’s faster. It’s someone with a specific business case, a hotel, a restaurant, a mom and pop shop. They’re all going to have different payment needs and it makes sense that they might want different apps. It’s for someone who wants more control over the process. It’s possible that you need different payment flows, even within the same business.

Rachel Antion: For example, at salon, how you pay for a service and just a product might be different. You probably don’t need a tip and signature if you’re just buying a bottle of shampoo, but you do when you’re buying your snazzy new haircut. Or, it’s someone who just wants to build their own app. If you think this might be you or you have any other questions, I’d be happy to chat with you after. I’m going to turn this over to Wako who’s going to talk to you about empathy here at Clover.

Wako Takayama speaking

User Research Lead Wako Takayama gives a talk on fostering customer empathy at Clover Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Wako Takayama: Hi everyone, my name is Wako Takayama and I lead the user research group here at Clover. John and Rachel introduced you to our product and the technology, so I am going to focus on the people who use our products and services here. Business owners like Thomas, who runs Poorboy’s Cajun Kitchen, which is just a few miles from here. You may have been there, very good food. And, Olivia from Theory Salon, which is in Woodstock, Georgia.

Wako Takayama: As with a lot of companies, we at Clover, we face the challenge that we build products for people who do jobs that we don’t do. These small business owners like Thomas and Olivia, they have a lot of things on their plate, they’re juggling a lot of things. They make all the decisions about their business, where are they going to open their store? What’s their product? What’s the price they’re going to sell things at? They have to hire, they have to fire.

Wako Takayama: Here we have one of our local businessmen. He needs to set up his own Clover system. He takes orders, he delivers food, he’s checking inventory, and then he has to call the vendor to make sure that he has stuff to sell, so a lot of stuff. This is just what we call front of house. Then there’s back of house. It’s all the office management stuff, lots of stuff that these business owners have to do.

Wako Takayama: For us to do our jobs as designers, engineers, marketers, we really need to know a lot about what these people do. We need to know that because that’s what we base our work on, the building, the designing that we do. The user research team, my colleagues and I, we help by doing formal research studies and, we work on fostering company empathy across the whole company.

Wako Takayama: But first, what is empathy? I’m going to read this to you, the ability to step into the shoes of another person aiming to understand their feelings and perspectives, and to use that understanding to guide our actions. The key here is that empathy allows us to get beyond our biases. One way we’re doing this, I’ll tell you quickly, is that we foster empathy at Clover starting on day one at the company. If you were to join Clover, you’d join the Merchant Empathy Program. This is a way to step into the shoes of a new Clover merchant. During the first week, you would work with your fellow new hires to dream up a business, set up a Clover system. You can see one of our designers really went over the top and he created this beautiful menu, and then take orders and payments.

Wako Takayama: I’m a researcher, so of course I send out surveys after things. I found out that this program has had a really great impact. One engineer said, “There were a couple of issues I worked on as I joined the team and due to my knowledge of the system from the session I was able to figure out a couple of issues easily.” That’s fantastic, right? Another engineer said, “It has helped me feel more connected to the customer and the company, and has helped me feel a little closer to the customer.” That’s really the key. We want to all feel closer to the customer, that we understand them, that we are serving them.

Wako Takayama: Imagine what stepping into the shoes of the user of your product or service could look like. How can you foster empathy for the person who’s using the product that you’re working so hard to build? If you’d like to brainstorm with … If you’d like me to brainstorm with you about some ideas, I’d be happy to do that, just come find me afterward. And, if you haven’t already had a chance to touch and step into the shoes of our Clover merchants, you can do that over there to get your schwag, and also just to play around with our product. Thank you.

Wako Takayama: Now I’d like to introduce Kejun Xu.

Kejun Xu speaking

Product Design Manager Kejun Xu gives a talk on thinking like a designer at Clover Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Kejun Xu: Thank you, Wako. Let me see if I can make this magical work somehow. Let me give it a try. Nope, doesn’t like me. All right, hi, everyone. My name is Kejun Xu. I’m a Product Design Manager here in Clover. I want to talk about how we design at Clover today, and you don’t have to be a designer to think design. You may ask, well … Next please. What is design thinking?

Kejun Xu: Actually, first of all, let me start with some numbers. It’s quite interesting. A few years ago, a team of researchers looked at how design impacted the organizations across S&P 500 companies. What they found was that of the top 20 companies, including Apple and Coca-Cola, who made it to the list, who are considered as design-centric, their stocks performed 211% over S&P 500 Index. This is compelling data.

Kejun Xu: You may ask, well, what is design thinking? Fortunately, we didn’t invent the term. You can search tons of information and technology out there. But basically, it’s a framework to foster innovation and collaboration. It starts from empathizing with your target audiences all the way to testing and evaluation. Wako talked a lot about merchant empathy. A lot of us joined at Clover without any knowledge about restaurant or SMBs, including myself, so we would go out for day trips and we’d go talk to the restaurant owners and managers. We’ll learn about their lives and their challenges. We also would go and shadow them and see how they would ring up an order on the Clover station, or how they would take payments …

Kejun Xu: Oh, it works? Can I have it? I’ll try it. This was a trip that my product manager, my researcher, and I went out and shadowed the merchants and see how they would take payments at the table. Still doesn’t like me. Sometimes when things are disconnected, we’ll go out and talk to them and see how much the pain point was. There are also other insights and data that we just couldn’t get by sitting here at our cubicles or in the office. By looking at this sheet of paper, the restaurant owner would know exactly what’s going on with this restaurant. It’s actually a pizza restaurant out there in Sunnyvale called Tasty Pizza.

Kejun Xu: That owner would know exactly what their customers ordered, where’s the order coming from, is Uber Eats or is it from DoorDash, was it paid or not? With all that forward data … I’m going to just do it myself, we’ll come back to the office and sit down as a team and really scope the problem. I’m really proud to say that every sticky note out there that you see our team put up, it connects to a real world problem. Then we’ll also sit down with the team to sketch the ideas all together. Like I said, you don’t have to be a designer in order to design. One of the sketches that got the most [inaudible] vote on is actually from one of our engineers.

Kejun Xu: This is where the design team will come into play. We would turn the ideas and all the concepts and sketches into clickable prototypes. We would then present the prototypes and we’ll do usability testing around it. Some of the testing that we’ve done are in house. We will invite merchants to our office and give them a tour and in the meantime, help us usability test or prototype. Sometimes we’ll go back to the restaurant, and we’ll go back and talk to them and test the prototype in their natural environment. A lot of times, we also do our usability testing remotely in remote sessions through GoToMeeting or Google Meet because we know that we live in this place called a bubble of Silicon Valley.

Kejun Xu: Well, design apparently doesn’t stop here. We shepherd through the entire development process. What this really enables us is that design get to sit at the forefront of the conversation and everyone get to sit at the forefront of the conversation. It allows product managers, engineers, marketers, researchers, designers, and everyone on the team and cross functionally align our goals, and that’s a recipe for high performing teams. You have to add a very special flavor to how we make design here at Clover, and it’s really that we make this a fun process to work on and if you haven’t noticed, we have an open bar at that corner. What’s more fun than sipping on a glass of Mimosa, then sketching your next product idea? Thank you.

Kejun Xu: Next up, I want to introduce our lovely panel for tonight with a topic of navigating conscious and unconscious bias and I want to introduce our moderator for tonight, our engineering director Bao Chau Nguyen. Welcome.

Bao Chau Nguyen speaking

Director of Engineering Bao Chau Nguyen introduces the panel of Clover leaders at Clover Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Bao Chau Nguyen: Good evening everyone. My name is Bao Chau Nguyen and I lead several engineering teams here at Clover, the Clover mobile apps point of sale and the app market web apps. The topic of conscious and unconscious bias had never been more prevalent than right now. From the current political landscape to the social movements, we are immersed in this topic, sometimes not by choice. We’ve come a long way in identifying biases, but we’re not close to eliminating or overcoming them consistently.

Bao Chau Nguyen: I want to show you a research study that I ran across on this topic. Imagine a fake company having a 1% performance bias towards gender. The impact of this 1%, they’re starting out with 50:50 men-women distribution across all career levels and this company rates women from one to 100, and men from one to 101. Over 20 simulations, the company is now skewed with fewer women at top levels. Now imagine running more simulations, the number is going to be a bigger gap.

Bao Chau Nguyen: We know this is a fake company, but we also know 1% bias is not realistic. Having been a young immigrant to America, I faced many biases over the years in all aspects, from classrooms, to just vacationing outside of California, to workplaces. I wanted to make sure that tonight’s panel will have a heart to heart conversation with you and whether you have experienced a bias or not, you can walk away with more awareness and some learnings on how we can become allies to one another. You want to speak up when you see these microaggressions and stand up for each other, because together we are stronger.

Bao Chau Nguyen: With that, I’d like to introduce our panelists, Mary Uslander, Ellen Linardi, Rachel Ramsay, and Meghana Randad. Let’s start ladies, welcome. Would you talk a little bit about your role here and, what was your initial reaction when you were invited on this talk?

Mary Uslander: Yes. Hi, everyone. My name is Mary Uslander. I’m actually from our New York office and I lead commercialization, client experience and work closely with the Clover team. I’m actually part of Fiserv, the parent company. For me, the topic was really around inclusivity and how you use it to an advantage, to really build diverse teams for success. I’m really excited to talk more about that.

Ellen Linardi: Hi, Ellen Linardi. I head the product team here at Clover. When Bao Chau approached me about being in the panel, it was interesting. I think I’ve always had a very interesting relationship with bias, both having seen a lot of it and we’ll chat more about that a little bit later, but also how it made me feel, then how I reacted to it and how I find what you do with the bias that is ultimately always going to be there leads a lot to the outcome. Hopefully we get to chat a little bit about that and we find it valuable. Excited to be here.

Rachel Ramsay: Hi, my name is Rachel Ramsay. I’m a developer advocate here at Clover. I also work very closely with our data analytics team. When you invited me to be on this panel, I was excited because up until I was 25, I thought I was going to be a sociologist, so I feel that I bring a more structural perspective than a lot of people have.

Meghana Randad: Hi, I’m Meghana Randad and I am a software engineer on the payments team here. When I was first invited to talk about this topic by Bao Chau, I was really excited and very happy because this is one of the topics which is very close to my heart. I have always been an advocate for women against inequality, against bias, and a lot of things we are going to talk here. Just coming from a very different background of being an immigrant and a woman and just an engineer, I face it every day, so thank you for having me. Honored to be here.

Bao Chau Nguyen: Great. Where can I start? This is a question for all of you. Would you share a time or a setting where you experienced a gender or an affiliation bias? How did that make you feel and how did you overcome that? We can start with you.

Meghana Randad: When I was growing up, the part of the world that I grew up in, in India, it was a norm and it was also common that women should get a college degree to find a better husband, not to find a better job, and then run the home. People often ask me, “Why do you want to work so hard? Why do you want to have a career when all you can do is support your husband, be home so he can really focus on his work?” A very fundamental assumption that women cannot, are not really so capable to work outside home and can’t have a career was very upsetting.

Meghana Randad: I had to overcome that many times in my life. To me, the key really is to believe in yourself. Sometimes you have to do what you have to do. If you want to get something, if you have a goal that you need to achieve, you have to be persistent and sometimes it could mean challenging the status quo. I was the first woman engineer in my family, and the first one to travel abroad, come to a new country all alone to pursue my career. It’s very easy when you have a defined path, but it’s really hard when you know where you want to be, but nobody to guide you or mentor you, so really all you can do is to believe in yourself.

Bao Chau Nguyen: I really can relate to that. My parents came here and had to start their career all over. They were teachers and then they came here, they had to go to back to school for a different degree and different occupations, so I applaud you, Meghana. Rachel?

Rachel Ramsay: Yeah. I’m an older millennial. I say that because I feel like a lot of women my age, when we were in middle school and when we were in high school, we were learning HTML, we were learning CSS, we were learning JavaScript because we were making our own websites back in the web 1.0 days, yet of all my friends and I who did that, no one was like, “That’s front end web design. You can make a lot of money doing that.” No one else was like, “There are other programming languages that you might enjoy.”

Rachel Ramsay: Out of my friends, none of us ended up pursuing it in college or as a career. I sort of backed into tech by going to a boot camp. But even once you get your foot in the door, once you’re the diversity in D&I, it can be hard to stay technical. Because people say, “You have such great people skills, maybe you want to go into management,” or “You’re a great communicator, have you thought about technical writing?” So, it can be very hard to say, “My North Star is,” whatever it is for you. I want to be a principal engineer and stay on that, stay in technical working with your manager to say, “I want to get the promotion, what do I have to do? Where are the opportunities?” You really do have to run your own career sometimes.

Ellen Linardi: I think from my perspective, a lot of the stuff that Meghana and Rachel both talked about are certainly true. I grew up in Indonesia, in a town not very different than what Jen showed. We had seven, about 7-Eleven looking thing and if I get in trouble at school, by the time I get home my mom knows about it. I don’t know how, but it’s a very small town. It was similar expectation with Meghana was saying, grow up, get married, make sure the man takes care of you.

Ellen Linardi: While I have a lot of stories I think on on biases that I’ve seen, what I wanted to share was probably an experience I had early in my career when I was in Intuit. I started out as an engineer there and loved coding. I was a keyboard hogger. When someone’s coding or trying to solve a problem too long, I get anxious and it’s like, “Let me try, let me try.” I knew I was very comfortable, I enjoyed that a lot.

Ellen Linardi: The other thing that was quite interesting, and I think this is something a lot of females can identify with, I was a good communicator, I like to organize, I pay a lot of attention on how everybody else feels so I kind of try to make it a team decision, make sure everyone’s included. So, one day one of my colleague came to me and told me, it was like, “You know, you’re an okay developer, but it’s all because you’re a good talker.”

Ellen Linardi: It was meant as a dig and I think the thing that I really wanted to share here is, at that point you have a decision. You could take it as a dig, or you could take it as a compliment. I chose to take it as a compliment at the time and I said, “Thank you very much. It is a skill so if you ever need help, I’ll be happy to help you in that area.” The thing I wanted to share there is that we are all going to run into bias, especially unconscious bias, and it’s called unconscious for that reason.

Ellen Linardi: It is going to be there, and I think we’re going to have a lot of opportunity to decide what you do with it. You either let it drive you and change the decision you have, to the point of focusing on where you want to go. Take it how you want it, and the bias folks have are not always bad. If someone say, “You’re Asian, you must be good in math,” maybe you are, you’re like, “Yes I am, thank you.” I just think that one of the way that I’ve approached some of the biases is not always negative, it’s simply a perception people have had going to that interaction with you and their experience of how they thought you should be.

Bao Chau Nguyen: Did you remember some of the responses after your-

Ellen Linardi: I never heard that line again after and I could tell you, certainly being a good communicator has gotten me to where I am. It hasn’t held me back, so I suggest that if you guys have felt biases or people saying things that you know, you’re female, you must be good in this, just say, “Thank you, that’s awesome. I’m good in that and this.”

Mary Uslander: I wanted to share more … First of all, having conversations like this is critically important and I’m just thrilled that everybody’s here. I think this is a conversation that we have to keep having. From my perspective, what I try to do is constantly make people aware that maybe they’re thinking about things a certain way, because of some unconscious bias. Whether it’s working with my male colleagues if we’re in the middle of merging with a new company, and people are making their decisions or judgments about individuals. It’s always interesting about how they talk about the women versus how they talk about the men.

Mary Uslander: When they’re senior women who are very strong, and very powerful, and very opinionated, and very inquisitive and are asking hard questions, there’s always a different value judgment on that individual versus if John was sitting down and really asked all those hard questions, “Why did you think about it this way? Why are you doing that?” That’s part of what you do. It’s really important to–in a right way, but just say, did you think about … Are you judging this person differently because they are a woman?

Mary Uslander: It’s really being aware of that and personally, I try very hard within my own team and I can see it as well. I have two young analysts, there’s a male and a female and they’re both incredibly smart and very talented. She works her butt off and puts her head down quietly and just gets things done. The young gentleman, he’s great too but he’s constantly putting time on the calendar and just showing me what he’s done. Not in a bad way, but I encourage her to do the same. I think it’s just being aware each other as well, and really trying to keep the conversation going, and how do you use it in a positive way?

Bao Chau Nguyen: Thank you Mary, just hold on to that. I wanted to ask you a follow up question. Having so much experience and leading big teams, in your … What have you noticed in your observations on diversity and how it impacts business outcomes?

Mary Uslander: I would say it’s really important to have different people on your team that do different things, but also come with a different perspective. You want someone like Kejun who’d have a design perspective, somebody who’s going to have a different perspective on, let’s say the merchant or empathy, analytical skills, detail oriented, big picture, creative. But, it’s really the power of that diversity of thought that really helps you get better outcomes.

Mary Uslander: What you also want to have is the commonality of you want people to have similar core values, to be ethical, to be honest, to work hard, to be smart and talented, so you really want to … You want to build your team based on skills and based on talent, but you want that talent to have a very diverse perspective. That really helps you achieve much better goals, because people are challenging you in different ways and arriving in problem solving in unique ways to get a much better result.

Bao Chau Nguyen: Thank you, I love that. Ellen, going back to what you were saying, coming from Indonesia and having that cultural bias of certain things that women have to do, and I know you have two daughters. Are they here?

Ellen Linardi: Wondering around here somewhere.

Bao Chau Nguyen: They’re just being great kids. I wanted to ask you, knowing that cultural bias exists and having daughters, does that impact how you raise them?

Ellen Linardi: I think what actually impact how I raise my kids has a lot to do with how I was raised actually. The interesting thing is while I grew up in a very traditional Asian town, I would say my parents were probably pretty progressive, not very conventional. Partly, my sister and I always … I have one sibling, so we are two sisters as well. My dad never had a son. I think he poured it all into us. He basically told us, “Whatever you want to do, pursue it. If you don’t like something, question it.”

Ellen Linardi: I think it drove my mother crazy somehow because when she told us, “Because I told you so,” we were like, “That’s not a reason.” We were brought up to really question the assumption and I think that was unusual. I think that was unusual in my town, that might be unusual for some of you, but I think questioning the bias and assumption and take it as an opinion at face value, and then deciding for yourself. It’s really a matter of choice. Running a home is not a bad choice.

Ellen Linardi: I think that’s one of the tricky thing, is that a lot of times you could see, your mom’s giving you the value she knew, and she knew how to run a home. That’s the life she could envision for you. To be able to understand the intent behind it and realize the impact that it has but not take it as face value, and be able to insert your own thoughts and your own desire to it, I think that is what I was taught.

Ellen Linardi: For me, I told my parents all the time I grew up to become who I am because of what I think the upbringing that I had and I try to do the same with my kids. I hope to be half as good of a parent as my parents was, but it’s the same thing and I think part of it is that it’s slightly uncomfortable. You tell them to question things, I tell them, “Because mommy told you so,” and before I say it I’m like, “They’re going to tell me it’s not a reason.” But, it’s ensuring that you understand why you’re doing things, and it is for a reason that you accept and you’re aligned with.

Ellen Linardi: It’s not because someone told you, it’s not because you’re scared, it’s not because society expect you to do so, it’s because you want to. I think having that as a compass is what I try to instill in my kids. That’s helped me, hopefully it helps others as well.

Bao Chau Nguyen: Certainly I grew up and my mom expected me to help her in the kitchen, and I always ran off and go do something else. Having two kids, a boy and a girl, I try to be as equal, whether by chores, it’s like, “Both of you clean up your rooms, both of you fold away your own laundry, both of you wash your own dishes.” So, not guiding them towards anything that is specific to their gender that they have to do. Just growing up here and seeing that world, it really helped me raise my kids to.

Ellen Linardi: It was actually what the interesting thing when I first came to the States, and I came after high school, actually. I always thought I was different when I was back home, but my parents kept telling me it was okay to be different. I was also a sick kid, so there was a lot of reason to be different. But when I came here, I realized I was different, but everyone felt a lot more different and being different was okay. I’m like, “That’s awesome, I’m never going back.” Here I am like 20 years later.

Bao Chau Nguyen: Now Rachel, being a lesbian you have twice the potential for bias from gender to sexual orientation. What changes or suggestions would you like to see in an organization to combat these biases?

Rachel Ramsay: Well, it’s easier to be a lesbian in the Bay Area than it was in North Carolina. I do want to call out the ways in which I am privileged, which allowed me to come here. I’m a white woman, I come from an upper middle class background, I’m a cis woman. When I decided like, “The Bay Area is really expensive, I need to get one of those tech jobs,” I was able to say, “I can get a loan to go to the boot camp, but dad, I’m going to be out of work for three months. Can you give me a loan from bank of dad?” Which he did. The question is…

Bao Chau Nguyen: Thank your dad for us.

Rachel Ramsay: Yes, I’ll tell him that. So, how do we create a world where everyone is safe is a really big question, bigger than the question you asked me, so I will limit myself. But, I’m really excited by what Jen is planning, our new head of people ops to include more of a diversity and inclusion training as part of our onboarding, similar to the program that we established for merchant empathy. But it’s not just about new hires, it’s across the company. Every year I get to sit through some trainings that are like, don’t bribe people, don’t sexually harass people.

Rachel Ramsay: I would love to also have a mandatory training like, don’t misgender your colleagues. It’s not just about education, it’s also the policies and the material support that we can provide to our colleagues. Whether that’s little simple steps like normalizing doing your pronouns when you get introduced, whether that’s having a gender neutral bathroom that’s just like a place for non-binary folks. And of course, making trans healthcare accessible. It has to be part of your health coverage and you also have to pair it with a supportive medical leave policy.

Bao Chau Nguyen: Hear that, Jen? She’s working on it. Meghana, you have two little kids. Describe to me balancing work and life, and not having the choices to stay late to work on a project or going out to a team dinner for team bonding. How did that impact you, or how do you feel like it impact you or your career?

Meghana Randad: Most of us feel that 24 hours in the day is not enough. I feel when you have young kids, even 48 hours are not enough. It’s just a lot of physically, emotionally sleepless nights, and being present at work and to be productive at what you do. When my team goes out for happy hours, and happy hours I feel are staying, working late together as a team are ways to bond, are ways to network. Sometimes you talk about things which are not related to work. You talk about your passions, we are in this space together and we are all motivated towards similar goals. You form a sense of community, you feel you belong here.

Meghana Randad: I felt when that happens, the team that I worked in was much more productive. Then being a young mom, being a young mom is incredibly hard. It’s very hard to create that harmonious balance between work and family. I do have to put definitely much more effort for working or even sometimes to just bond with my colleagues. For example, there has been times I had a four year old boy, a five month old baby, I’m on call for production, there’s a fire and I have to deal with it, I have to debug the issue.

Meghana Randad: My sick kid is now refusing to eat, some I’m sitting at the table, trying to get him to eat, a laptop in front of me Slacking and trying to look at all the graphs and debugging our code to figure out what’s wrong, to make sure we don’t fall apart as Clover. At the same time, holding my five month old in another hand and breastfeeding her. She was happy sucking away.

Bao Chau Nguyen: Multitasking to the next level.

Meghana Randad: And all moms have it. It’s not just me. But, I feel very grateful. I have an incredible partner who supports me when you have to stay late at work. For example, today he’s babysitting. I feel equally happy to work for a company, which supports its employees through various life phases. It’s just not flexible hours or maternity perks, it’s more than that. It’s a thinking that’s ingrained in culture here at Clover.

Meghana Randad: In my first week actually, we had happy hour on a Thursday and John Beatty, our CEO, he came up to me and he told me, “Hey, I know you’ve been a new mom and I know how hard it can be because I’m a new parent myself. I understand it’s hard, and I’m here to support you, so let me know if you need anything.” That itself is, that comes back to me every time I feel I’m struggling, and it’s very reassuring to have that support, just not at home, but also at work. I feel happy and cared for.

Bao Chau Nguyen: Wow, that’s a great story. Thank you, John. One last question before we open up to Q&A for everyone. How would you challenge stereotypes, provide some advice to your audience and promote sensitivity and inclusion?

Meghana Randad: As Jen said, we all have unconscious bias. We have amazing unconscious mind, which helps us navigate through a lot of decisions that we make every day. But unfortunately, this unconscious bias that we have against people could lead to make some wrong assumptions about people. Every time I make assumptions about someone, I try to ask myself, why? Why have I made that … Why do I think that way? Do I have enough data to support that? Has that person, does he have skills to do what he needs to do or she needs to do?

Meghana Randad: For me to challenge stereotypes, the keys to keep asking yourself and be really mindful, and be conscious about your biases. Once you’re aware, I think that’s the very first step towards tackling those and to create a very diverse and inclusive environment. It’s very important to have a diverse team, because most people learn from their experiences. To me personally, experiences are most powerful, that’s how I learn.

Meghana Randad: When you create those diverse teams, it can be gender, it can be number of experience, your background, many other things, right? Then people when they interact with each other, their assumptions are challenged a lot of times and they understand perspective of other people. That helps improve the whole culture of inclusion. I feel when you’re creating such diverse teams in workplace, the most important thing is to create a safe place where people can really share their differences and don’t feel that they have to conform to a norm. Really getting that richness in workplace would be the key I guess.

Bao Chau Nguyen: Well said. Rachel?

Rachel Ramsay: I think getting people in the door is not enough, hiring is not enough. You have to be bringing them into an environment that is truly inclusive, truly safe, where they can show up with their whole self and do good work, and come home feeling only the normal amount of exhaustion that you feel. How do you do that? I do think it requires a C suite level buy-in, it requires a buy-in from managers. I’m not a manager, I’m an individual contributor. As an IC, one thing that we can do for each other is we can look out for each other, we can have each other’s backs.

Rachel Ramsay: One time I was in a meeting and whenever I notice like, who gets cut off, who gets assigned the note taking, who gets chosen. You don’t want to white knight for people because it’s their career, but it’s easy to stand up for someone else, probably easier than standing up for yourself. So, there’s always an opportunity to call in a co-worker, to call in a manager.

Ellen Linardi: Let’s see, where do we start here? I think that ultimately, the interesting thing for me, at least from my experience on unconscious bias, is that we all have it. In some ways I say we have unconscious bias to the people that we think have unconscious bias. When certain people approach you in a particular way, you react to them. One of my biggest learning over the years professionally and personally really … I’m a divorced mom as well, so I’ve gone through various life experiences.

Ellen Linardi: Well, in that area is to decouple the impact and intent. The minute you couple the two because of the way someone makes you feel and you start reacting to that personally, emotionally, the conversation really isn’t going to go anywhere. The biggest thing that I really try to do is, I’m like, “Take the impact,” like, “Ouch, that hurt,” and then decouple it and say, “I know you didn’t mean to do that because when you say at the intent it sounds completely bad,” and then even if they mean to so it they’ll be like, “No, no, that was not what I meant to do.”

Ellen Linardi: Everyone take the higher road, but give people a chance to take the higher road. Because, when you tell someone, “I know you’re bad,” they’ll be bad, but when you say, “I know you’re actually good, but what you did was bad,” it gives them a chance to make different choices. I think that’s the first thing, is be aware of how you’re reacting to the unconscious bias. If you react to the unconscious bias by providing your own unconscious bias, it’s like regurgitating the same cycle and it doesn’t really get anywhere.

Ellen Linardi: I think the second thing is when it comes down to bias, the best thing I’ve ever find throughout my career of changing that is by changing the experience that the individual or the people or group in front of you have with whoever you represent. Sometimes I represent an epileptic person, sometimes I represent a divorced mom, sometimes I’m an immigrant, sometimes I’m a female leader, but in whatever context, you have an opportunity to recreate what it meant to interact with who you represent.

Ellen Linardi: When you change that experience, that change perception, that change bias because it is very hard to tell someone, “Change your unconscious bias.” It starts from the experience because that’s where it comes from. I think we all have an opportunity to slowly change that up, both by, I think, providing programs, having structures, and policies and everything that encourages it and making sure people are more aware, but each of us individually also have a chance, I think, on every interaction, to, I think, not continue that bias cycle and try to break it as well.

Bao Chau Nguyen: Yeah, I think we can all be allies. We can always find something that we can ally for each other.

Mary Uslander: A couple of things. One, I try and it’s very hard to do, is listen more. So much with unconscious bias, your brain is going, you’re looking at someone, you’re making a snap judgment. But then if you stop and you actually listen to what they’re saying, it’s overwhelming like, “Oh my God, this person’s amazing and what they’re saying is incredible.” I think for all of us to just stop and really listen, hear, and just try to incorporate that skill into everything you do. That would be one thing I work on every day.

Mary Uslander: I think the other is if you’re either managing people, be aware of always going to the same person. It’s easier said than done because a lot of times you have deadlines, and you need to get things done, and Ellen is the one who can always deliver like that or whomever. But you have to really give other people a chance, and also coach and help them right. Mentoring is another thing we haven’t talked about as much here, but we all know how important mentoring is, and mentoring is everywhere. It’s tonight, right? It’s listening to these amazing women and hearing about John and others, you look around you.

Mary Uslander: Every day, you should look forward and see, what could I take from someone? Whether it’s the person at the front desk or whether it’s the person who’s bringing the coffee, there’s always something to learn. Then if there’s someone who you really admire or respect and you want to spend some time with them, seek them out, ask them if they’d be willing to have a cup of coffee with you. It’s listening, it’s being aware, it’s trying to spread the love around and really help each other out. We as women here have to really continue to help each other and help the men, because sometimes they need a little help and understanding, probably more so than most, but I think it’s our job and responsibility to keep doing and keep advocating.

Bao Chau Nguyen: I know that you are part of many women organizations as well, you’re a big advocate for women. Can you talk a little bit about that?

Mary Uslander: Wnet is another women’s organization. Girl Geek X is amazing, but Wnet is another organization for women in the payment industry. Audrey Blackmon is in the back and she’s one of my fellow board members at Wnet. We really try to do all kinds of advocacy, education, training, webinars. I encourage you to take a look at wnet.org if you’re interested in joining. What we’re going to do is more … We’ll probably do an event here as well, but, any women’s organization or have a lunch and learn in your company. Get people together, have conversations. I think that’s really what we are trying to do here.

Mary Uslander: I just personally want to say about Jen and all of you, thank you. I feel like I’m an honorary Clover member because I’m part of the other side of the company, but I am so honored personally just to be here and to be part of this amazing group. Thank you for having me.

Bao Chau Nguyen: At this time, we’ve wrapped up the panel questionnaire and open up for Q&A.

Natalia: Thank you. I actually thought of not using maybe a microphone because it was so far away. Well, thank you for this. My name is Natalia, and thank you for sharing all the stories and feedback. Unfortunately, unconscious bias is something that affects many people, whoever brings any kind of diversity. I’m really curious about the feedback that you might actually hear from male colleagues, maybe your partners, maybe your husbands, maybe your brothers or fathers. Do they also see that unconscious bias impact them and most importantly, how they deal with it?

Ellen Linardi: I can get that started, I think. I actually am in a lot of rooms where I’m the only female. John knows this and we’ve talked about it. Recently we had a senior leader session with someone of the top product leader in the organization and I walked into a room, I opened the door, I was a little bit late. I opened the door and the room gasped. There was about 50 men in the room, and I was the only female. The guy who set up the meeting looked in the room, he looked at me, we all looked at each other and he’s like … And nobody noticed until I walked in, but–

Mary Uslander: They were all guys.

Ellen Linardi: Yeah, but they were all guys. Then he looked at me and he’s like, “That’s not good.” I think sometimes people don’t realize it’s happening, so I think being there representing it is one thing. A lot of situation, those interactions, I think, once it happens, allows you to highlight and have the discussion about how being present and having different personality from various points where I actually can deliver different values. I do think just the general climate and awareness is helping bring those conversation to the surface, so at least on the …

Ellen Linardi: Even if people don’t notice it all the time, the desire and willingness to have more inclusivity, I feel the tide is changing and it’s there. And the ability for us to actually engage in those conversation in an open way, in a non-biased way on our own and say, “I know we didn’t mean it, but this is just how it looks like right now. What do we do about it?” I think the ability to be inclusive of the solution and to not pass judgment on how we got to where we are today, I think allows everybody to take the high road and look forward on what it needs to look like in the future.

Ellen Linardi: The biggest suggestion I would say in, how do you engage in a discussion about somebody’s bias is to be very, very kind about what their intent is. Even if you’ve felt it multiple times, even if you’re like, “God, that’s so unfair,” the minute you put them in an area where they don’t have a chance to say, “I didn’t mean to do that,” you get a very different reaction and that’s true, like I said, from a personal basis, whether it’s international with your partners or your friends or different community member, all the way to in a professional environment.

Bao Chau Nguyen: I’d like to add on since you mentioned whether our male partners or husband experience bias as well. I think everyone experience it in some form, like it’s a segment that you belong to, that you’re different. Men experience it with race, as well as if men have kids, there’s unconscious bias with men who have kids versus single men. Everyone, everyone experience it and we need to have that open conversation and be receptive to that, that they do feel it to. Anyone else?

Audience Member: You spoke a little bit about being the only, help me understand your perspective on oftentimes being the only person in the room, in my case, the only person of color, sometimes the youngest person in the room, sometimes the person with the highest EQ in the room.

Bao Chau Nguyen: Good for you.

Audience Member: Help me understand your thoughts on being the only and representing all of those people. You spoke about representing all the different aspects, representing all those people while still trying to be yourself and bring your 100% self in that situation or in that room.

Ellen Linardi: I think two things. I’m going to say the first is, it’s important to know who you are, what you are and what you’re not. The best way you can represent whoever you present, whether that’s color, ethnicity, age, or what, it’s still a version of you. It doesn’t make everybody else who’s Asian or female be like me, but it allows people to understand that no matter your color, your gender or your age, the individuality and the differences and the diversity is where it matters.

Ellen Linardi: Really a lot of the things that we talked about on biases, it’s not about, it can’t be all men, or it can’t be all white or anything, it’s that the lack of diversity impact outcome. I think being able to demonstrate how that diverse opinion and approach can change the outcome is important one. That’s number one.

Ellen Linardi: The second thing I would say is, it does come down to choice. Just because sometimes it worked, doesn’t mean it always works. You’ll find yourself sometimes in an environment where you bring your true self, and they don’t want you. That’s not what they want, and that’s a call to action. If you’re being you, and you’re not acting or behaving because you’re afraid of what people’s expectations are, or perceptions or because someone told you so, and you’re just being truthfully your value, your belief, and your talent and your skill and they’re not interested, I guarantee you someone else is. You’re wasting their time and they’re wasting your time.

Ellen Linardi: I would say if you run into a situation where you’re being your true self and that’s not being valued, there’s a better place for you out there. I’ve made multiple choices, both personally and professionally where I was being myself and that, it wasn’t right. It doesn’t make them bad, but it wasn’t right. I think at that point, you have to make the choice of whether you continue in that environment, which is your choice to stay there.

Ellen Linardi: It’s hard to make that choice and say, “Well, they’re not accepting me.” Well, you know that so what are you going to do about it? I think making the choice when you’ve tried and it’s not working is another important one I would say. When you find yourself being the only one who’s represent in whatever group it is, sometimes it’s welcomed, sometime it’s not.

Mary Uslander: I would just add to that. This is a great conversation to. I also think you just … A lot of it is competence and confidence. I can imagine you in a room with all these men even if they’re all white, but just smart, articulate, talented, and once you start talking, I think instead of looking at your exterior, they’re going to start thinking about what you have to say and say, “Oh my God, that’s really great.” I would encourage all of us, right, to say you have to be confident, you have to know your stuff, you have to be prepared. Sometimes we have to be more prepared than others and so do your homework, but just be yourself and try not to get tripped up about that. Just go in with the objective at hand and be yourself.

Meghana Randad: And as Ellen said earlier, sometimes even if you are all of that, all of your authentic self, you’re still not accepted. There will be times. You have to go back and think, how does it affect you? What is your goal here? Does it affect you so negatively that it’s not taking you to your goal, or is there something that you can overcome this resistant by doing something differently and it still be you?

Meghana Randad: If it’s actually hurting your goal and hurting what you want to do, then I would say definitely, as she said, there is a better place for you. Maybe this is not the right place. You just have to sit back and think, is that right for me and does that align with who I am and where I want to be? You can be at a certain place, there can be various paths, so this might not be it.

Audience Member: Hi, I’m [inaudible]. I’ve been in the tech industry almost 20 years now. Started in engineering, went to business school. After that, worked overseas and back here and I find like and back in ’98 sometimes, that it’s been over 20 years and the progress hasn’t happened personally for me. I look at myself as a fresh engineer arriving here in Silicon Valley. The thing that I have realized, and so it’s a comment and I agree with 100% everything that you guys have said, because it’s not just here in Silicon Valley. I’ve seen it in APAC, Singapore, Malaysia, name it which country, I’ve seen it. There’s multiple layers of biases when you work abroad. Switzerland, yes. I left a business school because I didn’t like how they treated women, and this is Switzerland. Right, so it’s all over.

Audience Member: My thing that I have come to a conclusion and I don’t know, I’m opening it up here, is that fundamentally the way–I’m trying to understand neuroscience also here–if fundamentally we were designed with unconscious bias, that’s not fundamentally going to change because it’s like 1,000 years of how the brain was wired to protect us from … To keep us safe. That’s where fundamentally, some of these reactions are. I think what we as women need to learn and some of it, I think Ellen beautifully put it there is, how do we communicate much more effectively as individuals?

Audience Member: Understanding that as the other person has bias, we carry our own biases as well on how we perceive and judge other people and it comes from that fundamental sense of safety and security. That’s my add on I wanted to contribute, is to fundamentally learn ourselves and also most importantly, teach our kids. I have a five year old girl and I want at least in the next 20 years, things to be different for her, what I didn’t have. I want to make sure that we also talk about how we raise the next generation on effective communications because the bias is not going to disappear.

Bao Chau Nguyen: Right, and I think when you catch yourself doing that bias, you can always correct and apologize. That’s the best way, “I didn’t mean that,” or, “I phrased it wrong, let me rephrase that.”

Mary Uslander: And to that point. I do think though, part of what the action has to be is there needs to be more women at the top of the house because if you have more executive and C suite women, they’re going to be more inclined to have less of those unconscious biases and have more women like themselves be part of it. We saw the stats of the 1%, but if you look at the Fortune 500 companies, maybe there’s one or two women CEO. The unconsciousness is, I’m just going to go, we’re going to go to play golf or, I’m going to go down to so and so’s office.

Mary Uslander: It’s just, people are more comfortable with people like themselves, and therefore have the tendency to then promote people like themselves. What we have to do is start changing that, and it’s up to us in our companies to really push leadership to have the training, people like Jen, to make sure our CEOs are aware of this phenomena. We have to start getting more women in leadership positions, we have to get them more on boards. I mean, there’s a whole ‘nother conversation we can have and should have.

Ellen Linardi: I was going to say the other thing that I feel like if you guys are, whether you’re manager or in leadership, is model behavior. Those of my colleague at Clover and Fiserv [inaudible] would know, I’m like unbashfully mommy. I think a lot of times to the point of being the only person in the room, you try to look like everybody else. Whether it’s if everyone go drinking, you go drinking or everyone go golfing, you go golfing or if everyone shows up at seven, you shows at seven, that actually doesn’t help the diversity. Because what it does, it creates a perception that in order to be there, you wake up at seven, you leave at six.

Ellen Linardi: I made a rule that between seven and eight, my kids at home and like I said, I’m co-parenting, there’s time where … I don’t have my parents here. They’re in Indonesia, so I’m on my own. I got to drop off, I get them ready to go to school and if we have a Thursday night and it’s my turn with the kids, they’re right there. So, I think the … Be authentically you, because then you can actually represent the diversity. It’s a little bit unsettling and people will look at you funny, but someone looking at you funny doesn’t actually hurt you.

Ellen Linardi: I think being able to actually represent the diversity and not try to be in the room and try to look like everybody else, is the responsibility that all of us have here. Because I think historically, everybody says the female get to the leadership level and they try to look like everybody else. That doesn’t help. That’s what I would say, I guess.

Audience Member: Hi, thank you very much for sharing your personal stories. My question is about change management. I was wondering if you could give an example at Clover of things within the system that was broken that you got to fix. So, a system that accidentally had unconscious bias embedded in it and affected people of color, women, other marginalized groups, and you were able to address it, because I believe that it is the system we got to fix and not the women because we’re not broken.

Meghana Randad: It’s not my story, it’s a story of my colleague. Last year when I had my baby, another colleague of mine did too. I was lucky to have a manager who was understanding and could support me to that, but she was not as fortunate, so often, she used to get interrupted during her mommy duty times and she was scared, she did not want to bring it up. She was not a leadership level, she was not a manager, she was an individual contributor at a very early stage in her career.

Meghana Randad: But then, we talked about it often. We talked about it in mother’s room and she gathered the courage. I’m very, very, very proud of her to do that and she brought it up to the management. She brought it up to John, I guess. John took action in one day and it was corrected for her. The leadership which created all that discomfort, did not value her as a mother, as a female, and did not support her was corrected right then. This is a story I know very personally for someone.

Bao Chau Nguyen: That concludes our panel for tonight. We still have plenty of networking and swag left to pick up, so enjoy the rest of your evening. Thank you for coming to Clover.


Our mission-aligned Girl Geek X partners are hiring!

Girl Geek X LiveRamp Lightning Talks (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Akshaya Aradhya, Angie Chang speaking

Angie Chang, founder of Girl Geek X, welcomes sold-out crowd to LiveRamp Girl Geek Dinner in San Francisco, California.  Erica Kawamoto Hsu / Girl Geek X

Transcript of LiveRamp Girl Geek Dinner – Lightning Talks:

Angie Chang: Thank you for coming out to the Girl Geek X Dinner at LiveRamp. My name is Angie Chang. I’m the founder of Girl Geek X. We’ve been hosting dinners like this for 10 years up and down San Francisco, San Jose. And I’m really excited to be here tonight to hear from these amazing women and to meet each other over dinner, drinks, and conversation.

Gretchen DeKnikker: So, we also have a podcast, if you guys want to check it out. Check it out, read it, give us feedback. Let us know, we have mentorship, intersectionality, finding career transitions, all of these things. So, definitely go and check it out. And this is Sukrutha.

Sukrutha Bhadouria: Hi, that was Gretchen. She didn’t introduce herself. Yeah, so we started off with dinners, we talked about podcast, and then we made it happen. In the meantime, we started to do virtual conferences, which we’ve had now one every year in the last two years. And fun fact, we now have what is…a Zazzle store with our amazing branded, cool swag, I don’t fit into the T-shirt that I ordered.

Sukrutha Bhadouria: But you could get tote bags, you could get cell phone covers, so it’s really cute. Or somewhere in the back, maybe, you’ll see what our pixie characters look like that up. But if you go to the invite for tonight, you’ll see these little characters that we have represented and we try to be as inclusive as well possible. So, all of our branding is very inclusive. Please share on social media, everything that you hear tonight from our amazing speakers. Use the hashtag Girl Geek X LiveRamp. And we will follow you and retweet and re share, so thank you so much for coming and thank you to LiveRamp.

Allison Metcalf speaking

GM of TV Allison Metcalf gives a talk on how LiveRamp got into the TV game at LiiveRamp Girl Geek Dinner.   Erica Kawamoto Hsu / Girl Geek X

Allison Metcalfe: Hi guys, I get to go first. So my name is Allison Metcalfe. I am the GM of LiveRamp’s TV business. So just for context, what that means, LiveRamp, a couple years ago, we moved away from functional leadership 100%, where I was actually previously the VP of Customer Success. I’ve been here almost six years. I started customer success, I was patient zero A long time ago, and I will never do that again.

Allison Metcalfe: So a couple years ago–LiveRamp has historically been really, we really focus on the digital ecosystem and the cookie ecosystem. And there’s been a lot of changes in the industry that suddenly made TV a very, very compelling opportunity. And so, we launched a TV business that I run. And so, what I’m going to talk to you about now is kind of why we’re in this business and what the opportunity is and why it’s super cool. It’s really fun to be working in TV right now. And hopefully, we’ll get a couple converters from it.

Allison Metcalfe: So, TV is so crazy. Nothing has changed in the world of television in terms of how it was bought, measured, I need a timer here, sorry, in 70 years. So, literally like the way people measured TV and bought TV and demonstrated the success of TV up until a couple years ago was the same as it was 70 years ago, which is a little bit insane.

Allison Metcalfe: As you probably know, you think about yourselves, you are not watching Seinfeld at seven o’clock on NBC anymore. It’s not appointment viewing anymore, you’re streaming it, you’re watching TV really whenever and wherever you want, every single screen that you have, is a TV today, which is really great for us as consumers. Like TV has become very, very consumer friendly. But it’s caused a lot of problems for the industry.

Allison Metcalfe: So number one, is the way we’re measuring it, ratings is really hard to track now, right. Nielsen is the incumbent measure that would say this is how many people watched Seinfeld last night. They were able to do that because of a pretty archaic panel that they had and pretty archaic methodology. But it was accepted. And it worked for a long time. But now, the network–so it’s like NBC is, they’re putting all their money on This Is Us, right? And Nielsen is saying, “This is how many people watched This Is Us last night.” And NBC doesn’t believe them. Because they’re like, “What about all the people that watched it on video on demand? And what about the people that watched it on Hulu and Roku and all these other places where they could be streaming that versus just on appointment viewing, linear television?”

Allison Metcalfe: So, the audience fragmentation is making the networks feel like they are not getting enough credit for the viewership that they are actually driving that translate to they are losing money. And they don’t like that, right. The device fragmentation is also causing problems for brands, because the brands, all they want to do is reach you, right? If they are trying to reach young parents who are in the market for a minivan, they don’t really care where you are. They just want to make sure they’re reaching you.

Allison Metcalfe: TV used to be the easiest way to get phenomenal reach within one buy, right, because everybody was watching Seinfeld at seven o’clock and we knew who they were. Now, we’re all over the place, this creates a big problem. If you’re a brand. You’re like, “Oh my gosh, how much money do I spend on Hulu versus Roku? How much do I put on linear television? How much do I, what other devices,” there’s so many I can’t even think of them all. So, it’s a really big problem for the industry. But it’s good, right? Because change is good. And again, it’s very consumer friendly.

Allison Metcalfe: So what we call advanced TV, is the process of anytime we are using data and automation to buy and sell TV, which again, really was not done before, that sits under the umbrella of advanced TV. This is a roughly $80 billion industry–that’s the TAM in the United States. Historically, for LiveRamp, we made zero dollars from the television industry up until about two years ago.

Allison Metcalfe: So it was a whole new TAM for us, which is very, very exciting. Of that $80 billion that used to be bought and sold in the traditional way up until advanced TV came, now, we’re seeing projections of $3 billion being spent in addressable, which I will explain, close to 8 billion in OTT which is anytime you are watching television, due to your internet connection. It doesn’t matter if it’s on your phone, or your computer or your Smart TV. But if you’re watching it, because of the Internet, and not because of your set top box, right, that’s OTT.

Allison Metcalfe: And then, we’re also seeing a lot of companies like a really interesting trend is a lot of the direct to consumer. Companies like Stitch Fix or Peloton that are 100% digital companies are starting to spend a lot of dollars on television as more advanced strategies are becoming available to them. The other thing that’s happening here, guys, it’s really, really important. Facebook and Google are coming after TV hard, right. They’re like, “We want to keep growing at the rate we’re growing. But we already have like 80 or 90% of the entire digital ecosystem. So how do we keep growing, we’re going to steal money from TV, that’s what we need to do. And we’re going to do that by saying we have all the eyeballs that TV has anyways.”

Allison Metcalfe: And so, that’s another reason that the industry has to change to combat, Facebook and Google. And I think the demise of television is very overblown, as you can see by these numbers here. So, we power the future of advanced TV, when we talk about advanced TV, we’re talking about all of these things. So, addressable TV is literally the idea that you are getting a different ad, than your neighbor, right, Rachel here is big camper, I am not. You shouldn’t waste your dollars showing me commercials for camping equipment, but you should show it to Rachel. So addressable TV is meaning Rachel’s going to get the camping commercial, I’m not, based on my set top box, we power that.

Allison Metcalfe: Data driven linear TV is the idea of, if you have a target audience of say young families in the market for a minivan, we will match that against a viewership data asset so that the buyer can understand that young families in a market for minivans are over indexing to This Is Us and what’s another TV show? Modern Family, and they’re really not watching The Voice, or whatever it may be. So you’re still buying TV in the traditional way, you’re not targeting a household, you are still buying based on content, but you’re buying that content, because you are much more data informed.

Allison Metcalfe: I talked about OTT, digital video, this is clips, this is, Jimmy Kimmel had a great show last night, and there’s a clip of him and his funny joke and we might want to see, you’re all being forced to watch an ad. Before you can see that clip, as you probably all know. And then, probably the most important exciting thing is measurement. So historically, the way TV has been measured has been brand lift awareness, surveys, and reach.

Allison Metcalfe: Now, given the fact that LiveRamp and we have a couple other companies that can do this, too. We recently made an acquisition of a company called Data Plus Math, we can marry viewership data that’s ad exposure data to outcomes. So now Peloton, for example, can say, “Aha, my investment on This Is Us drove this many people to my website, that was a good investment for me. And I’m going to crank it up on This Is Us,” for example. LiveRamp plays in all of these places, a lot of companies that are getting into the TV game usually are only in one or two of these areas.

Allison Metcalfe: So it’s really exciting. I’m going to wrap it up there because we are a little bit crunched for time. And I’m not going to bore you with this. But I hope that was somewhat valuable and interesting to you. And thanks for coming. Thanks.

Tina Arantes speaking

Product Leader of Global Data Partnerships Tina Arantes gives a talk on finding product/market fit at LiveRamp Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Tina Arantes: Okay, Hey, everybody, my name is Tina Arantes, and I’m on the product team at LiveRamp. Been here about five years, so not as long as Allison, but enough to see us go from like 70 people in a little office in the mission to like, mission on mission to three floors here and like over 800 people. So it’s been a crazy ride and on products, we’ve learned a lot.

Tina Arantes: So I’m here to share with you some of the learnings from my product experience here. And primarily, the learning that listening to your customers is the first step in creating awesome products. So this may sound very obvious, like everyone’s probably like, “Duh, how else would you do it?” But when I’m out there like talking to other product managers through interviews, and other ways, it turns out a lot of people aren’t talking to their customers. And it’s actually super important because especially in the B2B business, like I’m selling into marketers, and I’m not a marketer.

Tina Arantes: So if I don’t know, if I’m not my own customer, the only way to figure out and empathize with them is to actually get out there and listen to them. So, I’m also a big fan of design thinking, right? So the only way you can create a product that your customer is going to want to buy is if you first empathize with them, define the problem you want to tackle, ideate to come up with solutions on how to solve it, and then prototype and test. So, the empathize part is actually like the part I’ll focus on first, which is like, how do you get out there and discover what are the problems your customers are actually facing?

Tina Arantes: So let’s jump right into it. How do you actually listen to your customers? The first step is actually just showing up. It sounds simple, but you’d be surprised how many times like you’ll have someone on Allison’s customer success team reached out and be like, “Hey, can you answer this question for this customer about this thing?” And the first thought most teams have is like, “I could, but how about that person does it because I have other important things to do with my engineers.” But actually, a lot of the times, it’s sometimes useful to take advantage of the opportunity to get out there and just meet the user, and start to establish trust with them. So you can ask them your own questions and get to know them better later on.

Tina Arantes: So step one is like just show up, make time in your calendar to find customers that are representative of your user base, and get to know them. So once you’re there, and you’re in the conversation, you can’t just jump right in with the hard hitting questions, right, you have to establish like base of trust. So warm them up, buy them a cup of coffee, introduce yourself, ask them about them a little bit. The way we do this, actually on a larger scale at LiveRamp is through customer advisory boards, where we actually organize getting some of our best customers together into a room, take them off site, somewhere that they can actually spend a few days with us, give us feedback on the roadmap and tell us about some of the biggest problems they’re facing.

Tina Arantes: And that’s been actually one of the really big sources of customer input and feedback that we’ve gotten. So you can do it on a small scale with a cup of coffee or organize like a whole event to get out there and start talking to your users. Okay, so once you have the customer, you warm them up. Don’t again, just jump in there with what you want to say, start listening to what they have to say, I don’t know how many times I’ve just been blown away by like being like, “Okay, what’s keeping you up at night? Like, what are your biggest goals? What can you not solve? Like, how can, how can we help you?” And they come up with all kinds of ideas I would never think of, sitting at my desk trying to imagine what they might want to do.

Tina Arantes: So be an active listener, listen to what they have to say. And don’t try to lead them to the solution you have in your mind. Because you know, you’re so smart, and you know how to solve their problem. But you also should ask juicy questions as well. So once you’ve given them a chance to talk, then you should have done your research and know who you’re talking to and know what kind of questions you can ask to really get at the heart of what you’re trying to solve.

Tina Arantes: So these could be like discovery questions, asking about what areas of problems they’re having to like, help you come up with solutions later on, that could be products. Or if you’re in a stage where maybe you’ve talked to a lot of customers, and you have an idea of a problem you can solve is like throwing it, putting it in front of them and seeing how they react to it. Do they get excited and be like, “Where do I sign? And can I buy this tomorrow?” Or they’re like, “Okay, that’s interesting, like, not that important to me right now.” So yes, you can ask your questions as well, after you’ve done your share of listening.

Tina Arantes: Okay, and after the interview, or after you talk to your customers, what happens next. Now the hard part happens where you have to map it back to everything you’ve heard from every other customer you’ve ever talked to. So definitely write these things down, keep them somewhere, like, I sometimes find notes from customers from five years ago, and I’m like, “Okay, that problem still exists, maybe we should solve it.” And then you start to look for trends, right? You want to see, is it a problem multiple customers are having, like, can I identify 20 customers that are having the same problem? How urgent is it for them?”

Tina Arantes: So people have all kinds of problems, but is it in the top three? Or is it like number 20? And they’re like, “You can solve it for me, but it’s not really going to matter.” And then the important part, like what are they willing to pay for it? You can ask like, “Hey, I have this next month, would you buy it?” And people will let you know, yes or no, there.

Tina Arantes: But let’s get real too, so earlier, I said like a lot of people don’t actually end up talking to their customers for various reasons. Of course, like time is always an issue as a product manager, because you’re running around crazy with your engineering team, like trying to keep sales happy, lots of internal squeaky wheels to keep from driving you crazy. But like you do need to make time to talk to customers. And even once you have the time, like I know, as a PM, all of these thoughts popped into my head, right? Like, what if they don’t want to talk to me? Who am I to like, go knock on the door of a Fortune 500 company and be like, “Can I have an hour of your time?”

Tina Arantes: But like, it turns out, most of the customers really do love talking to product and love providing their input in hopes that it will impact the roadmap and asking their questions to you as well. You can turn it into like a value exchange, like offer your thoughts on the vision of the product in exchange for their input as well. This one’s one of my favorite, like, what if they say bad things about my product? I know like, you get very attached to your work, right, and you don’t want to show up to a customer and they’re just like, “Yeah, no, I hate it. Your baby is really ugly.” Like, no one wants to hear that. Right? It’s terrible.

Tina Arantes: But it’s better to hear it so that you don’t walk around thinking your product is like, the best thing ever, when really like, there are some things you can improve. So, it will happen, like people will say bad things, you just have to deal with it and take the feedback as a gift. And then this one also comes up. I know a lot of product managers are like, “I don’t really want to get on the call. What if they asked me something, that I don’t know the answer to?” It’s like, that will also happen, like every single call, but it’s okay. You just have to be like, “I will find you the answer to that and pull in someone who does know the answer for the next call.”

Tina Arantes: So there’s a lot of resistance to getting out there and talking to your customers, but you got to do it. So what does it actually, what does success look like when you do this right? And when you don’t do this right? So maybe starting with like when you don’t do this right. Definitely over the past few years, I’ve made tons of mistakes, not vetting things carefully enough with customers. One standout in particular where we had a project and we’re like, “Oh, we’ll just make this product go much faster.” Because we had a few customers who were like, “Yeah, that would be great.” Jeff’s laughing back there, because he’s the engineer who built it.

Tina Arantes: So we built it, we launched it, and then no one wanted to buy it. And we were like, “What?” And it turns out, it was a problem for people, but it wasn’t something they were willing to pay for. So now, we always check like, “Oh, great, is the problem like how much would you pay for it at the end?” And it does work sometimes as well. So like we’re working on another product now that we actually got the idea from talking to our customers, different customer advisory boards, they’re like, “How can you help us share data between two partners? And we’re like, “Well, that’s an interesting idea, maybe we could help you there.”

Tina Arantes: And it’s turning out to be more successful and more people are willing to pay for it. Because of the hard work we put in, checking with a really large client base that this is going to be interesting or an urgent problem to solve and something they’re willing to pay for. So that is why I think listening to your customers, as a product manager is one of the most valuable things you can do. And the first step in creating products like people actually want to buy. So yeah. And we’re also hiring on our product team here. Definitely engineering team here. So if you want to chat later about any of this, I’m happy to talk more.

Eloise Dietz speaking

Software Enginere Eloise Dietz gives a talk on lessons learned from becoming CCPA compliant at LiveRamp Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Eloise Dietz: Hi, everyone. My name is Eloise Dietz, and I’m a software engineer here at LiveRamp. I’ve worked here for about two years. And I’m currently on the data stewardship team. Our team is responsible for ensuring that LiveRamp systems use personal and company data ethically. And right now that means working to make sure our systems are privacy compliant. If your company works in personal data, you’ve probably heard of them, GDPR, CCPA. So I’m going to talk a little bit about what this privacy compliance looks like and why it’s relevant to software engineers.

Eloise Dietz: So first, a little bit of background. LiveRamp takes data privacy very seriously, partly because we think it can be a competitive advantage. We work in data onboarding, which means that we help companies advertise to their users online, which means that they can better personalize their ads online. Studies show that consumers actually really prefer this ad personalization and a more of a customized experience. And it can be a guarantee, or it has a higher likelihood of a higher return on investment. However, there’s also losing, people are losing trust in technology companies. And research shows a majority of people worry about how tech companies are using their personal data.

Eloise Dietz: In fact, one study found that 80% of people will leave a brand if they think that they are using their data without their knowledge. So companies in ad tech, like LiveRamp have to deal with this dichotomy. And they need a way to resolve this problem and gain trust back in their users. And I think that GDPR is a really important step in this direction. So, GDPR is a data privacy law that aims to regulate data in the EU, and it took place on May 25th of this year. So CCPA is kind of the California equivalent to this GDPR. And though it has many differences, it also incorporates a lot of the same ideas. It will take effect January 1st of next year.

Eloise Dietz: So a lot of other states are following California’s example, and also have privacy bills in the process. A lot of other countries are also inspired by GDPR around the world and are going through the process of introducing their own privacy laws. More are expected to follow. So as you can see, GDPR is kind of inspiring an overall shift in regulation of data privacy. And in the US alone, 68% of multinational companies have spent between 1 million and 10 million getting ready for GDPR. As CCPA approaches, only 14% of US companies say they are fully compliant despite its similarities to GDPR. They plan to spend another 100000 to $1 million becoming compliant.

Eloise Dietz: So we can see that these laws are really causing a big shift in how companies think about data. And the reason that is, or we can look into why that is by looking at some of the key GDPR requirements. Obviously, GDPR incorporates a lot more than this, but I thought that these were some of the most relevant to software engineers. So, the first is data minimization. Or the idea that we should only collect the data on users that we need to solve a certain task and then delete that data as soon as the task is accomplished.

Eloise Dietz: The next is that data subjects or individuals have certain rights to interact with their data. So they have the right to access the data or retrieve all the data a company has on them, they have the right to restrict processing of that data or opt out, they have the right to delete that data. And they even have the right to rectify the data if they think it is incorrect. Then finally, users have the right to be notified of data collection and the use, that data is going to serve. And if you got a ton of updated privacy policies this year, it was probably from this part of GDPR.

Eloise Dietz: So you seem kind of like standard practices. But they fundamentally change how a lot of companies think about data, the companies in a data graph mode, they might not even realize what personal data they have on people, nonetheless, what it’s useless for and how to collect it and return it to an individual if they asked for it. So this is what data privacy does not look like and what data privacy actually looks like is constantly asking yourself these questions as you build systems.

Eloise Dietz: So the first step is understanding what personal information that you have, and that your system processes. Or associating with that data, why it was collected, where it was collected, and what use it’s going to serve. Data minimization is probably one of the most relevant to software engineers. It means reviewing your data and deleting it, when it is no longer needed. But this also means not logging, personally identifiable information, it means when you store it, not storing it raw, storing it pseudo anonymized, means restricting access to that data to only those who are required to use it.

Eloise Dietz: And it means not using real data in your dev and staging environments. And finally, also automating user rights for deletion, restriction, processing and access. And so at LiveRamp, as we kind of went through this checklist of how to make our systems privacy compliant, we realized that there are some cases where we even need to go beyond the law, beyond GDPR and CCPA, in order to design for the privacy of the end user, not just designed to make our systems compliant by these privacy laws.

Eloise Dietz: So the first one of those instances was reading a privacy vision to hedge against the many data privacy laws that are expected to come out. So, for example, these laws are going to differ. CCPA and GDPR differ in many ways, and sometimes, even completely contradict each other. One example of when they differ, is this right to opt out. So CCPA says people have a right to opt out of data processing, whereas GDPR says people need to actually give their consent and opt in before data is allowed to be collected.

Eloise Dietz: I think that for users, understanding the way that you can opt out. So many different privacy laws is an undue burden on the users. So, LiveRamp decided to have a global opt out repository, where we, if someone wants to opt out an identifier, say a mobile ID, cookie, or email, we pseudo anonymize that information and store it in a global repository. This means that deployments in the EU as well as nationally in the US can check to ensure that they’re not processing data over any identifier that is in this global repository. So going beyond the laws and having a clear privacy vision that opt outs will apply globally not only made our LiveRamp systems more straightforward, but also ensures that the end user is actually receiving the privacy that they’re expecting.

Eloise Dietz: Second, never let privacy come at the expense of security. So in the effort to make users be able to better understand what data companies have on them, laws like CCPA and GDPR may actually be opening up this data to bad actors and more vulnerabilities. For example, the right to access their own data means that someone could make a fake this request and maybe receive another person’s data. So I think users may not understand that this security is at the risk of privacy. And it’s up to the, this privacy comes with the risk of security and it’s up to companies to make sure that this does not happen.

Eloise Dietz: So finally, embedding privacy into the user experience I think is an important place companies can improve on. So especially the ad tech ecosystem is incredibly complicated. This infographic shows the number of ad tech players has increased significantly over the years. Users shouldn’t have to understand how all 7000 players interact in order to understand their data privacy rights. A survey went out after GDPR that asked users what their biggest complaints were and the study found that most people’s biggest complaint was the long overcomplicated privacy regulations.

Eloise Dietz: And though these may be required, sorry, privacy policies. And then though these policies may be required by law, I think that the system should be designed to incorporate the end users privacy in mind, and make it easier to work with the systems in order to find the best privacy policy. So this doesn’t necessarily mean having a accept all or opt out of all policy that often doesn’t work with like most people’s privacy. And it also doesn’t mean having so many different privacy settings where you really have to understand the privacy law in order to understand what you want. It means designing for the end user and creating a concise, intelligible, transparent and easily accessible way of working with the privacy, working with your own privacy settings for that company.

Eloise Dietz: So my end takeaway is to take GDPR and CCPA as a way to rethink your data usage, but also looking beyond these privacy laws and consider the end user when designing your systems in order to truly protect their data privacy.

LiveRamp Girl Geek Dinner

After bites and drinks, girl geeks enjoyed lightning talks from women in various parts of the org at LiveRamp Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Akshaya Aradhya: Now, that the first half of our session is over, does anybody have any questions for the speakers?

Audience Member: Quick question for you. I actually didn’t realize data minimization [inaudible] example because [inaudible] users [inaudible] out [inaudible] that even an option [inaudible] data minimization?

Eloise Dietz: A user opts out, as in the fact that we’re still maybe storing like a pseudo anonymized identifier?

Audience Member: Mm-hmm (affirmative).

Eloise Dietz: So the idea is that personally identifiable information, I think this is right. The idea is personally identifiable information needs to be minimized. But when you pseudo anonymize an identifier, it no longer counts as personally identifiable. So by storing that anonymized version, it no longer kind of counts as the process, I believe, is for opt outs.

Erin Friesen speaking

Software Engineer Erin Friesen gives a talk on destroying an entire build ecosystem to leading the engineering wide initiative to protect and improve that very same system.  Erica Kawamoto Hsu / Girl Geek X

Erin Friesen: Hello, I’m Erin. I’m a software engineer on the infrastructure Platoon, I’m working [inaudible] DevOps. And I have an obsession with making builds easy. It’s absurd. All the engineers here can say that I’ve authored them with everything. So I’m going to talk about how I got to that point, and a lot of the mistakes I made along the way. So next time, you have to do a migration, you don’t have to do them.

Erin Friesen: First off, I’m going to be talking about Jenkins. Jenkins is my best friend. If you don’t–anyone here know what Jenkins is. Yeah. So Jenkins is basically a tool to get servers to do what you want them to do. If you’re like, “I want to deploy this, send it here. I want you to set a cron job, do this, I want you to build this do this.” That’s what it should be. So we start our journey with a horrible Slack message. I snapshoted the wrong thing. And I don’t have a backup, and we don’t have our configurations. We’ve lost our builds.

Erin Friesen: As you can see, Jenkins is on fire there. And our last backup had been 10 months previously, record everything on the master server. And we had just demolished that. So we panicked, we figured it out, we got our builds back, but realizing that we are storing our configurations, the core thing that we need to do to deploy on the thing that if it goes down, it breaks it, not the best situation. So, we came up with a solution, Jenkins files. So basically, it’s codified builds, you put a Jenkins file into your git repository, it lives there, you can take Jenkins down in a heartbeat. I almost did that as a demo. But I didn’t want all those users to panic.

Erin Friesen: And instead of storing your configs in a UI like this, you get seven to eight lines of code. And that’s your entire build configuration, which is pretty awesome. And it’s very replicable. You can version your code, you can pick a library, it’s so much more control over your environment. So previously, these are my steps to get there. Let me say this was one of my first larger, like known visible projects that I’ve ever lead. Here are my steps. I create a product, I just have the teams do it themselves. And then I’m done. Easy, right? Not quite.

Erin Friesen: So first off, I skipped over scoping out the size of the migration. I didn’t realize how large the project was and how different it was. I’ll give you a scope. We have over 250 Java repositories, you have over 150 Ruby on Rails builds. All of these builds have PRs and master builds. So if you do the math, that roughly puts the 700 things that you have to migrate, that you can’t break because if production breaks, you can’t deploy a fix, you’re in trouble. So I didn’t scope out the size of the project. It led to some very troubling times.

Erin Friesen: And the second was, I did not ask for input from engineering team until I was well into development, a lot of about listening to your stakeholders. I didn’t know what they needed, or what they actually wanted from their builds. But I was like, I know better. I’ve seen a Java build. You’ve seen one Java build, you seen them all, right? No, that’s definitely not the case. And lastly, I didn’t ask anyone for help about their experiences with it, what they’d done to actually build it, other people had experienced Jenkins, but I sort of ventured on my own thinking I could plow my own path.

Erin Friesen: That didn’t work out too well, either. And so, a lot of this boils down to I didn’t communicate with people. I didn’t ask them, and I broke a lot of things. And I’m still very sorry, you guys are watching this later. And I think lastly, I assumed that the teams would do the work. Like, I assumed that if I presented the seven lines that I needed to do, everyone would adopt it, everything would work, and everyone would go in the same direction at the same time, and it would be fine. That’s not it. Because guess what, everyone’s builds are different. They’re unique. And they’re just different and unique.

Erin Friesen: And I assumed they would do that. I also didn’t assume that they didn’t want what they had, they wanted something better. Like, you want to build your own solution. And you want to have power over how you deploy and where you deploy. And I didn’t listen to any of that. I mean, I didn’t listen. I also pushed changes without telling people because I didn’t version at first, it was, I didn’t listen, and I didn’t communicate with the team. So that was like the biggest thing if you to take away anything from migration over communicate and like, talk to everyone, and I mean everyone.

Erin Friesen: So these are my steps to a new successful migration. Do your research. I didn’t. So, I didn’t break down my problem. I didn’t even figure out where my share was like, what? Where should I be living? Like, what needs to get done, and what’s broken? What can stay broken? And talking to everyone, I just didn’t think about it. Didn’t break down the problem into injectable sizes. And I couldn’t get the iterative feedback because I didn’t check. I was like, “I’m going to roll into this. And it’ll work.” Which leads into break up the project into bite size. Because if you know what you’re getting into, believe it or not, you can break it up into smaller parts.

Erin Friesen: I’m a rock climber. And so, whenever I go outdoors, I go, and I look at the mountain. I’m like, “Cool, what do I need? I need to be able to solve this section of the climb and the section of the climb.” And this is how I get to every single portion. And I always break it down into bite sized steps because you’re like, “Oh, it’s only one reach, or two reaches or I don’t know, a high knee, like pick a move.” And it works a lot better to get to the top.

Erin Friesen: And if I haven’t said it enough, communicate, just communicate with everyone. I didn’t get feedback early enough. I didn’t iterate on feedback. And I created a doc, a roadmap for it. When I’d already been working on the project for four months, like that wasn’t the efficient way to do it. I got excellent feedback from stakeholders. But it took me too long to get to that point of starting a feedback cycle.

Erin Friesen: The next two come hand in hand. Rollout gradually. And at one point in time, I had 355 PRs open, various repositories, so I created a script to create a PR to inject my one size fits all Jenkins file. And there was no back out, like it’s hard to rewrite those. And it was broken, it was hard because I didn’t version it, I didn’t have an interface. And so, if I had to make a change to a function, I had to make 355 individual commits to everything, they’re starting to get customized. So I didn’t have a rollout plan, which means I also didn’t have a backup plan. If I needed to roll back what I was doing.

Erin Friesen: So, successfully, you need to have backup, you need to be able to bail if a rollout goes bad. And finally, you just iterate and repeat over and over and over and over again. And if you keep these steps in mind, the best thing is, everyone wins. Everyone gets the product they want. You don’t waste cycles on trying to build something that they don’t want. And you actually get help along the way and it speeds it up. So that was me about how to migrate way better than me.

Akshaya Aradhya: Questions for Erin?

Erin Friesen: Part of it, the story, oh, it didn’t have the date on it. It was 2018. November, 2000–no, November, 2017, it was right at the end.

Akshaya Aradhya: Before Thanksgiving, okay. Any other questions? All right.

Rachel Wolan speaking

VP of Product Rachel Wolan gives a talk on the evolution of privacy, discuss what it means to build products intended to protect consumer privacy globally, and the design decisions we make along the way.   Erica Kawamoto Hsu / Girl Geek X 

Rachel Wolan: Hey, everyone, my name is Rachel Wolan. And I’m the VP of Applications for product. And I’ll echo what Tina says, we’re hiring. I’ve been here about five months. And I think Eloise did a great job of kind of helping everyone understand a lot about the regulations of privacy. Today, I’m going to talk a little bit about, like the history of privacy. So I will kick this off by telling you a very private story.

Rachel Wolan: So maybe over Christmas, I got engaged. And before I asked my partner to marry me, said yes, I had to get through her parents. And I was way, way more nervous about this stuff than talking to her. I’ll tell you a little bit about her parents. They’re from Singapore, they’re native Chinese. And I’d met them twice. I had a lot of things going for me. So, I sit down with her parents. And I’ve managed to, it’s Christmas. And I got all the kids out of the house, like they went to the bathroom, is great. I had like 15 minute window.

Rachel Wolan: And I was really looking for, not permission, but their blessing. So I sit down with them. And I say, “Hey, I’d really like to ask your daughter to marry me.” And mom’s like, “Hey, I’m going to sharpen my pencil.” She like, basically pulls out a list of like, 20 questions that she wants to ask me. Just asking me what were your past relationships like, what, like, do you have kids? I’m like, “No, no kids,” “Do you want kids? When are you going to have kids?” Like, all these questions.

Rachel Wolan: And like I think I’m doing a really good job. And this whole time, she’s actually translating in Cantonese to Mr. Chia. And I think, okay, I’m like, her mom’s like holding my hand, things are going really well. And I’m like, “Okay, this is over. She’s about to give me a blessing.” And then all of a sudden, Mr. Chia’s English gets really good. He looks at me, and he says, “What do you do for a living? How much money do you make?” And this is not something that like even I talk to my parents about. And it kind of struck me that privacy is really contextual.

Rachel Wolan: And I tell this story because privacy isn’t like one thing. It’s not something that is just regulated by one country or a group of countries, it’s something that is very meaningful to each individual. It’s different based on your race, your age, your gender, your socioeconomic status, your sexual orientation, where you live, where you’re from, like what religion you grew up in, really everything. And privacy is, each person’s privacy might even change over time.

Rachel Wolan: And, what I think is also, like, an important context about privacy is it’s a relatively new concept. So I’m going to show you guys some really cool technology that has helped evolve privacy. So the first is the printing press. The silent reading was really, one of the first forms of privacy, where people kind of had like, internal thoughts that they weren’t there, maybe they were writing them down, maybe they weren’t writing them down. And that really took like, 500 years to evolve.

Rachel Wolan: Internal walls were huge for privacy. Previously, it had been like, kind of that one room house where people lived, and they kind of all slept in the same bed for a long time in the entire house, and, like, fast forward to the 1900s. And the camera came around. And the concept of the right to privacy actually came to being. And what I think is interesting about this is that we didn’t really even put laws into place around privacy until post Watergate, right, like 1974.

Rachel Wolan: And then fast forward to today, AT&T, is, like, you can pay AT&T 30 bucks to opt out of ad tracking, but most people don’t do that. It’s really, the concept of privacy has evolved. And, I think, really, you have to think about privacy from like the standpoint that there’s a value associated with privacy and people are willing to trade privacy, there is a currency. And how many Millennials are in the room. If I offered you a pizza for three of your friends’ email addresses, would you… That’s what I thought.

Rachel Wolan: And so, I just spent a couple of weeks in China. And if you go to almost any street corner in China, you will see these cameras. And what they’re basically doing is tracking, what do citizens do? Did they walk across the street, did they jaywalk? I jaywalked, like this morning. So my social score will go down. Did they go through a red light, and all of these characteristics are being collected as part of a social privacy score, right, a social credit score. And so, really, in this case, one of the reasons why China introduced a social credit score is because in 2011, I think I saw some stat, two out of three people were unbanked in China, they really wanted to accelerate, people getting credit and being able to buy houses.

Rachel Wolan: And so in 2015, they actually made their data, their privacy data available to eight companies, including like Ant Financial, which is owned by Alibaba. And so today, I was talking to one of my co workers about his social credit score, and he was saying, “Well, I definitely don’t yell at my neighbors, I don’t park in a parking spot that’s not mine. Because that’s going to ding me and I want to, use the whatever the version of TSA Pre check is, right, if you have a high social credit score, you get a better line at the airport, there’s a different car on the train, there’s even a different–you can like skip the line at the hospital.” So there’s a lot of benefits. And, really like privacy can be traded for societal value.

Rachel Wolan: So, then the question is, I did a lot about design in our product org. How many people here have designed apps for Android or products for Android? So you know it’s really freaking hard. And I would say designing privacy is a 10X problem of them. And so, this is actually was a pizza study, where people were, there are 3000 people that were asked to trade their friends’ email addresses for pizza. Like 95% of them did. And that’s kind of like what I think is interesting here, because Tina aptly said, like, ask your customers what they want.

Rachel Wolan: But the most interesting thing about the study is customers actually said, “Oh, no, I would never do that.” Like the people in the study said, “I would never get my private information.” And then they target those same people. And they all did. So, this is one of those situations where you really have to actually think–was anybody in here familiar with privacy by design? Cool. So privacy by design is, it is a framework that you can use in order to start thinking about, does my product really protect the privacy of… So you can think about it at the very beginning and discovery and start asking questions, to try to understand the needs of your users. And look at it as kind of like a review process. We have a data ethics team at LiveRamp. We have what’s called a cake process where you can actually start to think about like, a probe through right before you even start building. Does this match our privacy standards?

Rachel Wolan: And then, I think a lot of the government laws that have been put into place, right, from the perspective that it raised our awareness of–around privacy, but it’s really our responsibility. And so, I’ll leave you with one final thought. So, this is actually privacy. Our phones are just like spraying our private information at all times. And so, like, try this, like brief experiment, turn off location services on Google. Does it still work? So I did this for like two weeks, and it kind of drove me crazy. And what’s interesting about this is, I actually had to go into a separate set of settings to completely turn off location services.

Rachel Wolan: And the cynics may say, “Oh, it’s because Google wants to track you. They want like all your data so they can sell your data, blah, blah.” And I actually think that this was really a design decision. Because they knew that you actually want that blue dot. And you want that blue dot, because you get value from it. You’re willing to trade your value, and maybe even go and kind of look and see. Like maybe you don’t want to trade all of your location data, but maybe some of it, for that value exchange. So, in conclusion, treat data like it’s your own, and make privacy happen by design. Thank you.

Akshaya Aradhya speaking

Senior Engineering Manager Akshaya Aradhya gives a talk on managing a geographically distributed engineering team at LiveRamp Girl Geek Dinner.   Erica Kawamoto Hsu / Girl Geek X

Akshaya Aradhya: Hello, everyone. My name is Akshaya. I’m the IT manager for the integrations group. And I work with people like Jeff, Sean or head of engineering, Andrew, who’s our biggest women ally, here. He has three daughters. And when I told him we are hosting a Girl Geek X event, he’s like, “Woo-hoo.” So, that’s Andrew right there. And Jacob, who’s in my team, he’s awesome. And he’s supporting all of us. And I work with all these people every day. And I want to talk about how I manage distributed teams. And my of champagne.

Akshaya Aradhya: That I want to give a glimpse of how many offices we have globally. So these are camping experience. We have social, there’s a doctor in the office. We have a lot of fun [inaudible]. Our New York office, we’re on Fifth Avenue where all the shopping malls are. Philadelphia. Seattle. Burlington. Arkansas. Erin Bodkins was supposed to be here. But she had another commitment. Paris. There is a lot of French people in my team. London. Asia, Pacific, China [inaudible].

Akshaya Aradhya: Because I knew how loud they were. So, let’s talk about all these teams that you just saw, right? So I manage two teams, I’ll soon be managing four teams. And most of the, like both the teams that I manage are currently in within United States right now, but may spread out to China. So this is the headquarters where most of my team sits, but not all of them. There are some people out there in the New York office. And there’s one in Philadelphia, and, I also talk to the people in Arkansas, because I like them, you saw how fun they were.

Akshaya Aradhya: Some of my team members, like I said, are French and they like going back to France to meet their family and sometimes work out of their homes. And is that normal for LiveRamp? Yes. But you don’t necessarily need to be French to work out of your home. So what do I do first thing as a manager, whenever I, start managing any team, I do it inside, listen first, so I kind of ask them, what are their preferences? Do they have any time commitments? Some people have kids, they need to leave at certain times, some people have soccer practice, some people need to work out for health reasons or for any other reasons.

Akshaya Aradhya: And some people, like not having meetings at a certain time, and we chat a lot during our one on ones. Jacob is nodding his head. He knows why. And so, we have all these preferences. And East Coast people have their preferences. So, how do I manage the priorities? Like how do we all deliver against this shared vision? So, I can go back and make notes. And I’m like, so if we have dedicated set of meetings for the team to talk to each other, that’s number one. You’re all one team. You all need to get along, whether you like it or not. And you need to talk. And how do you establish that, right?

Akshaya Aradhya: Before I started working for LiveRamp, I was working for a company called McKinsey right across the street. And before that, Intuit, and it’s like, each company has its own culture. 

Akshaya Aradhya: At that time, I was married, but I didn’t have kids. So just a piece of cake, right. And then I got pregnant, and then they flew me to Canada, ask me that went. My feet swelled so badly, I couldn’t fit in my shoe. And not that… And I sent a picture to my husband, once I, or two different shoes. And I couldn’t even see it. You know? And I was like, “Yeah, yeah, sure, right. The time difference, just wake up when you’re pregnant, you love waking up when you’re, like then and you like everyone you meet when you wake up. Right?”

Akshaya Aradhya: So that’s how that went. 

Akshaya Aradhya: The culture doesn’t mandate you to go and sit with someone to be productive. You could as well be on blue jeans. You can, like I made my son’s appointment after joining LiveRamp. And then I could come back can take meetings, take knowledge transfers, talk to people, be productive.

Akshaya Aradhya: You’re not judged based on where you work from. Okay, that’s number one. Second thing, as a woman who went through all of this, I kind of make sure that I don’t step on other people’s toes or schedule meetings when somebody has an important thing, okay. And if you’re working with East Coast people, I tell all my teams, you better have those meetings, before 2:00 p.m., Pacific, otherwise don’t have shared meetings. And if you do want to have shared meetings, ask that person, if it’s okay, get the Slack message saying yes, and then you’re going to have that meeting. And, make sure that you don’t keep it as a recurring one. So that’s one thing, coordination.

Akshaya Aradhya: And following the right tools, I mean, you need to, whether you follow Agile or [inaudible], whatever it is, or whatever form of Agile your company follows. I know, Agile means different things for different people. But you need to get your message across to the team, everybody needs to talk, at least for like 10 minutes a day, and share what they’re doing. And, like, after sharing work related things, you want to share anything personal, or any, anything that you want our team to know, like you are engaged or you have a baby or whatever it is right, you can now share it.

Akshaya Aradhya: And, in one of my teams, I tell people, right, just because you’re working out of San Francisco doesn’t mean that you need to sit here till I leave, or sit here till 6:00 to make a point. You’re going to work on flexible time. And I need to see what progress you made. And you’re not blocking anyone and you’re out, right. It’s value to your personal space and time while being productive and accountable. That’s what you need.

Akshaya Aradhya: Again, I’m going to share my version of what works and what doesn’t. So you can as will be micromanaging, go to each person’s desk. Or like you could start off by not asking questions, or over communicating, assuming things and get the wrong thing. And then pass it on to your team, you lose that trust, you lose that trust with, it’s so easy to lose trust when you’re managing distributed teams, then micromanaging. Who loves these people in this room? That’s what I thought. And then people start leaving, and you wonder why and the cycle repeats, if you’re not listening, if you’re not watching your team, the cycle repeats. What works?

Akshaya Aradhya: Get the wrong thing. But you learn and adapt. People make mistakes. It’s okay, as long as you’re not consistently making them, you’re okay, you’re going to learn. And you’re going to share what you learn. Sharing is not on the screen because I run out of space, but you got to share what you learn with your teams, and communicate closer. Talk to them drop. Messages on Slack or whatever messaging service you use, add any relevant process. Relevant process, not process for the sake of process. And relevant process that works for you and whoever you’re working with. Are you peer programming? Are you a software engineer? Does this process work for you? Fine. If you’re in product, maybe you’re talking to customers, there’s a different process that Tina or Rachel may use, I don’t know.

Akshaya Aradhya: But as engineers, especially here in the valley, or New York or all the places that you work, whatever works for you is the best process. That’s what I tell teams and effective collaboration, effective collaboration. Destructive feedback is not effective collaboration. Rambling is not effective collaboration. Putting down others, sarcasm, you’re maybe the best, most intelligent person. But if you’re not nice, you’re out, that’s good as that. So play nice. And teamwork. Teamwork is success according to me. If you don’t work as a team, you work in silo, you may be the best person in the world. But if your team doesn’t see what you do, or if your team doesn’t find value in what you do, you don’t have any business value with the work you’re doing or you don’t grow, you don’t let others grow, you don’t help anybody or mentor people. That’s all contributing to bad culture.

Akshaya Aradhya: One of the things that I really like at LiveRamp when somebody spoke, during my onboarding, was that if somebody sends you an email, you respond quite quickly. It’s–in other companies that I worked at, response right away meant that you’re supposed to work or respond back at some time, right? So now studying at Wharton, Sean, our head of engineering. At his level, or Andrew or even Jacob or who, or Jeff, if you send a message to them, and I work from 1:00 a.m. to 4:00 a.m. because I need to study when my son is sleeping. Some of you may resonate with that. So if you don’t, you can judge and I’m crazy, partly.

Akshaya Aradhya: But that’s my time when both my dogs are asleep, and my son is asleep. That’s my time. Okay, so what do I do? I catch up on all the emails and I told my team, “If I send you a message on Slack, or an email, do not respond to me outside office hours, unless it’s really urgent.” There have been nothing really urgent that needs a response. And I was surprised when I sent a message to Sean one day, and he just responded at 2:00 a.m., I’m like, “What did I see? Did I a response?” And I’m like, “Thank you for messaging.”

Akshaya Aradhya: And it’s like, you may choose to do that. But it’s such your own volition, you’re not forced. And I think I tell all my teams that, “If you see it, ignore it. If you don’t want to, like if you’re sleeping do not wake up, because of me. Snooze your notifications.” Yeah. And basically, there’s a saying, right, you don’t go to work when, something you really like, then you enjoy what you’re doing. It’s not really work or something like that.

Akshaya Aradhya: And I think when you join a company that values your personal space, your ambitions and offers you opportunity to grow. And you love what you’re doing. There was recently a job satisfaction survey at Wharton, where I’m studying, part-time. It’s like, in my group, and when I say group, it’s about seventy people in one section. People did a job satisfaction survey based on so many different metrics. And they were talking about organizational stuff, and how do you grow your teams? What is effective, what’s not, somewhere on this, but in a more lectury fashion.

Akshaya Aradhya: And I took a survey of my past job and this job. And it was one among the top five. And I’m thinking, “Huh, I did that, I think, right?” When you love what you do, your stress goes down, you’re happier, your kid kind of sees you really happy, right? You don’t go crazy. And you can actually do what you want to do, study, pick up a hobby, rock climbing, or do a side project on Android, I don’t know, on whatever you want to do. Don’t do that. So yeah, it’s like, the last thing I want to leave this room with, is like this.

Akshaya Aradhya: Professionally, you set an example for your team. You don’t need to be a manager, each person can be an individual. You set an example for your team. And if you overburden yourself or you don’t enjoy what you’re doing, your team can see it and your productivity goes down. So make sure wherever you choose to work or whoever you choose to work with. Hopefully at LiveRamp, because we have opening, you should choose something that will allow you to grow and be happy at the same time. And that’s what the whole talk was about and what all the speakers and organizers want. And hopefully, after this presentation, you come by and say hi to all of us and hang out with us, ask us questions, learn about us and connect with us. We would love to keep in touch, any case. Thank you.


Our mission-aligned Girl Geek X partners are hiring!

Girl Geek X Aurora Lightning Talks & Panel (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Aurora garage girl geeks

A self-driving car remains in the garage as the Aurora Girl Geek Dinner kicks off with drinks and networking after hours in San Francisco, California.  Erica Kawamoto Hsu / Girl Geek X

Speakers:
Jessica Smith / Software Engineer / Aurora
Haley Sherwood-Coombs / Technical Operations Specialist / Aurora
Elizabeth Dreimiller / Mapping Operations Lead / Aurora
Khobi Brooklyn / VP of Communications / Aurora
Chethana Bhasham / Technical Program Manager / Aurora
Lia Theodosiou-Pisanelli / Head of Partnerships Products and Programs / Aurora
Catherine Tornabene / Head of Intellectual Property / Aurora
Angie Chang / CEO & Founder / Girl Geek X
Gretchen DeKnikker / COO / Girl Geek X

Transcript of Aurora Girl Geek Dinner – Lightning Talks & Panel:

Angie Chang: Okay. Thank you all for coming out tonight to Aurora. My name is Angie Chang, I’m the founder of Girl Geek X. We’ve been hosting these events in the San Francisco Bay area from San Francisco to San Jose for the last 11-plus years, and every week we really love coming out and meeting other girl geeks at different tech companies and hearing them give tech talks that we’re going to be hearing tonight, as well as hearing from them on how they’ve accelerated their careers.

Gretchen DeKnikker: Hey, I’m Gretchen, also with Girl Geek. So, whose first time at a Girl Geek Dinner? Oh. A lot. Cool. Well you should keep coming because they’re awesome. Like Angie said, we do them every week. We also have a podcast that we’d love your feedback on, and we’d love for you to rate it and all sorts of things. We cover mentorship, career transitions, imposter syndrome, getting the definition of intersectionality right, a whole bunch of stuff. So check it out and let us know.

Gretchen DeKnikker: Okay, and then we also just opened a swag store, and it’s a bittersweet story. So we have some really, really cute awesome stuff, and then we have this stuff, which is kind of cute, but poorly printed, so we’re going to find a different place. But in the interim, you can check out these really cool things. Okay, Angie, hold them up. Man, one-armed.

Gretchen DeKnikker: Okay. Water bottle. Cute, right? The little pixie girls? Okay. Notebook. That’s me on the notebook, by the way. That’s my pixie, so if you want to put me in your pocket, that’s the way you take me with you everywhere. And then the fanny pack, which I’m way too old for, but it is so cute. Everybody needs this fanny pack. Oh, and then there’s a little zipper bag. That’s my favorite thing, that’s why we have to show it to them. Look at the little zipper pouch for your pencils and you Sharpies and your Post-Its. Oh, we have Post-Its.

Gretchen DeKnikker: Okay, and iPhone cases. All this crap. Anyway, check it out because we put a whole bunch of work into it and we would love for people to have the stuff that they said they wanted. Okay. So without further ado, so we have got the CEO, his name is Chris Urmson, you can also call him Dr. Chris or Mr. Woke AF, so please join me in welcoming him.

Angie Chang: Oh, and really quickly, this is … okay, really quickly, this is a sold out event, so if you are liking this event, please help us tweet. The hashtag is Girl Geek X Aurora. If there’s something great that he says or any of the girl geek speakers to follow, please help us tweet and share the word that this amazing company is doing really interesting things. Okay do that thing again.

Chris Urmson: Thank you. After that introduction, I feel like I can only fall on my face. So first, thank you for Girl Geek partnering with us to pull this off tonight. Thank you all for coming tonight. This is my first Girl Geek event, and we’re just thrilled to have you here. We’re building something exciting in Aurora, we have this mission of delivering the benefits of self-driving technology safely, quickly, and broadly. We’d love to share that with you.

Chris Urmson: What I’m really excited about is, a lot of time in the press, what you hear about around our company is our founders and about the technology, and I’m proud as hell that we get to show off some of our awesome people today. And I was told I’m allowed to be just blunt about this, we are hiring like crazy, and we are looking for awesome people. So if you enjoy talking to these people and hearing from them, and seeing the work that they’re doing, please come join us. I think you’d love it here, and we would love to have you.

Chris Urmson: So without further ado, I’m going to invite Jessie to come talk about cool stuff.

Jessica Smith speaking simulation

Software Engineer Jessica Smith gives a talk on what her simulation team is working on at Aurora Girl Geek Dinner. Erica Kawamoto Hsu / Girl Geek X

Jessica Smith: I have a mic. So I don’t think I need that mic. Is my other mic on? All right. Sorry. Hi, I am Jessie Smith. I am on the simulation team at Aurora. And we’re going to find out if my clicker works.

Jessica Smith: So a little bit about me is my background is, I’m from Nevada, I’m from Reno, Nevada. I got a master’s degree from UNR in high-performance computing, that weird animation thing is a forest fire simulation, which is what I did my thesis in. I have some other experience in autonomous systems, mainly autonomous drones in grad school, and then on to Uber’s advanced technology group working on simulation, and now at Aurora working on simulation.

Jessica Smith: So I’m going to talk a little bit about what is simulation, and we have three main things that we do on the sim team. We are a developer tool, we do regression testing, and we do problem space exploration. So for developer tool, we build custom tests for developers to help enhance what they do on a day-to-day basis and make them faster at developing the self-driving car software.

Jessica Smith: And then as soon as they land these new features, we go out to make sure, just like every other regression test, that when you land a new one, you don’t break all the old ones. So we also do regression testing. And what I’ll talk about today is problem space exploration, which I think is one of the most interesting things that we get to do at Aurora on the sim team.

Jessica Smith: So, this video here is going to be an example of a log video, and you can see this pedestrian kind of walks into a car, opens the car door, and disappears inside of the car. And so what we’ve done in simulation is extracted the information about the spirit of the scene, and what we can do in sim, which is really, really powerful, is take this interesting encounter, where a man walked in front of the car, and instead say, “What if it’s a mother and a stroller?” And, “What if it’s a person with a bicycle?” And you can actually explore the problem space and make sure that the self-driving car does the right thing, given the insane variation of the inputs to the system.

Jessica Smith: So another example is, we can vary the behavior of the other actors in the scene just based on things like velocity or position, and so you can make sure that the car is capable of making a lane change, when it should lane change in front of another car, between two cars, or behind them, given the state of the other vehicles and what is the safest thing to do.

Jessica Smith: We can also do some sensor simulation, which helps us determine what are the capabilities that our sensors need to have, and what is the fidelity that we need to have of those sensors? Like, do we need to be able to detect … you can’t really see it in this picture because it’s tiny, but you can detect the tiny individual bike spokes on this bicyclist in this sensor simulation. So what we get to build moving forward, and what my team is hiring for, is scaling out simulation. We need thousands and thousands of these tests, and we want to build realistic world modeling, and that’s better act of behaviors in the scene, but also better 3D representation of the world.

Jessica Smith: And then we want to crank the fidelity way up and do really interesting high-fidelity camera simulation. And this image on the far left here is purely synthetic, but I certainly can’t tell the difference.

Jessica Smith: So now I’m going to hand it over to Haley to learn a little bit more.

Haley Sherwood-Coombs speaking

Technical Operations Specialist Haley Sherwood-Coombs talks about machine learning datasets and the perception platform at Aurora Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Haley Sherwood-Coombs: Hey there, I’m Haley, I’m going to talk about machine learning datasets, and our tricks here at Aurora. A bit about me, so I’m in technical operations here and I work under perception platform. I have a background in operations management and information systems from Santa Clara University, and I’ve been here at Aurora since April of 2018.

Haley Sherwood-Coombs: So our team mission is to provide abundant, high quality machine learning datasets to fuel machine learning. And I want to pause on the word fuel. At Aurora, we talk a lot about fueling rockets, which [inaudible] off the saying, “Don’t try to build a ladder to the moon.” What this is getting at is that building a ladder makes very small progress. Small progress which is gratifying to see, but will never practically reach the goal.

Haley Sherwood-Coombs: At Aurora, we believe the way to actually get there is to build a rocket. It will initially appear to make little visible progress, but once carefully built and tested, it will cross the quarter million miles in a matter of days.

Haley Sherwood-Coombs: So how does this fit into the scheme of perception platform, and where I do most of my work in machine learning datasets? So the machine learning datasets are the rocket fuel for our rocket. The metrics are the launch pad, and the models are the engine. So in the machine learning datasets, it’s the creation of meaningful data. So what can we do to input the best data into our models? Metrics is the offline assessment of perception, so making sure and double-checking that the machine learning datasets are going to be great for our models, and accurately assessing these models and having value identification on these.

Haley Sherwood-Coombs: And the models is real time. It’s our Aurora driver. It’s real time action machine learning. So jumping into machine learning datasets. In order to get this data, we have to look at cameras, radar, and LIDAR, and this is where we get the returns for these labels. Our sensors are strategically placed all around our cars to eliminate blind spots and optimize our field of view. Most of the times, we put these so that we never have any blind spots.

Haley Sherwood-Coombs: So looking into data curation a bit more. Our tools allow us to collect high quality annotations, and we care more about high quality and fewer, within a larger amount of lower quality annotations. To curate the best data, we align across our organization. We look across teams, and also organization-wide to see what is feasible, and what will provide the most impact.

Haley Sherwood-Coombs: Diving into a bit of the models here. So here are two examples of our Aurora perception system. Right here on the left, you can see our car. Well when it rolls again, it will then yield to a pedestrian right here. It’s able to track it, stop, and yield, and wait until it passes, and safely drive again. You can also see that it then starts picking up all these other cars that a normal human driver wouldn’t be able to see until it was like mid-way.

Haley Sherwood-Coombs: On the right here, our perception system is tracking cyclists 360 degrees around the car. Normally if you were driving, you would have blind spots and wouldn’t be able to see your cyclist here or here, but having an autonomous system, it’s able to do that.

Haley Sherwood-Coombs: Metrics. This is the quantitative language that binds everything together. So we have our models, we have our data, now we need to make sure that these are doing the best they can. So we look at the impact that every single piece of data has on these models in the machine learning, and identify confusion and what changes need to be made. If something’s right, if something’s wrong, we go back and run another model on it.

Haley Sherwood-Coombs: So finally, where we culminate is the Aurora Driver. As you guys know, it’s our goal to put self-driving cars on the road safely, quickly. Here we go. Thank you. Next up is Elizabeth.

Elizabeth Dreimiller speaking

Mapping Operations Lead Elizabeth Dreimiller talks about the work of the mapping teams at Aurora Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Elizabeth Dreimiller: Hey everyone. So I’m going to be talking a little … pretty briefly about Aurora’s work with high-definition mapping. So a little bit of background around me. I grew up in Ohio, and as a kid, I absolutely loved maps. So whenever I got the opportunity to go to a park or go to a different state, I would just grab a paper map and literally would go home and put it on my wall. And the funny thing about this is, I actually never had a map of Ohio, because it’s so flat and boring, there’s no reason to.

Elizabeth Dreimiller: So that kind of led me on my career trajectory today. I went to school for GIS, geographic information systems in Pennsylvania. And then after school, I went and worked with the mapping team over at Uber before moving on to Aurora.

Elizabeth Dreimiller: So here, you can actually see our mapping software in work. You can see the operator is placing down points, and they’re going to be drawing lines that show the curve placement, where are the paint lines that we need to be paying attention to? So that’s the yellow center divider down the middle. And you’ll see as this image goes on, they’ll be placing lanes that our car pays attention to.

Elizabeth Dreimiller: And a lot of people, when they think about maps, they simply think of how to get from point A to point B. Our maps are that, but also a lot more. Our Aurora Driver needs our maps to understand how it works, or how it relates to the world around it, what it needs to pay attention to. So we’re placing traffic lights and a ton of rich information.

Elizabeth Dreimiller: So a little bit of breakdown about our team. Our mapping team is broken down into two different core teams. We have our engineering team, and they kind of work on making sure the logic is in place, that the Aurora Driver can understand and actually create the tooling that we use. So in the image to the right, you can see an operator moving a lane around to make sure that the trajectory of the lane is appropriate for the vehicle.

Elizabeth Dreimiller: On the other side is our operations team. And operations team is pretty neat. A lot of people think that it’s just creating the map content you see. And you can see all the different rich layers that we have. So we have the ground data, that’s actually LIDAR-processed data. And then we go into traffic lights and all the different lanes and paths. And then finishing off with remissions logic. A lot of rich information.

Elizabeth Dreimiller: But not only are we producing that, but we’re also coordinating all of the collection of this data. We’re making sure we’re running through quality assurance as well as maintaining hundreds of miles of map, and making sure they never go stale.

Elizabeth Dreimiller: So a brief overview of the challenges we face. I’m not going to over all of these, there’s a lot. I’m going to focus on three. So the first one is safety. So we’re producing all of these miles, how do we know that what we’re producing is of quality? And that’s when automatic validation comes into play. So our engineering team and our operations team is working on making sure we have a very good set of validations in place, both automatic and human in the loop, to make sure we’re catching everything.

Elizabeth Dreimiller: So second is quality, and with that comes speed. We want to make sure these hundreds of miles, obviously, are the highest quality, if possible. But also with that, we want to make sure we’re not sacrificing speed. So we want to make sure we’re creating tools and processes that allow us to speed up while maintaining that bar of quality.

Elizabeth Dreimiller: And lastly, policy. As you know if you’ve driven outside the state of California, every state kind of requires a little bit different interaction from their drivers. There’s laws. So we focus on trying to understand how we can create a broad policy on a highway map to fit a large geographic region. And at the essence of it, safely, safely, quickly, and broadly, is all about Aurora. We work on [inaudible] maps.

Khobi Brooklyn: How about now? Oh great. I’m Khobi Brooklyn, I’m on the communications team here at Aurora, so now in the technical part of the business, but in the part of the business that does a lot of work to reach out to folks like you and make sure that you know all the good work we’re doing here at Aurora.

Khobi Brooklyn: So I’m going to bring up a panel of Aurora women who come from all parts of the business, and we’re going to talk a little bit about brand, which is something I know a lot about. That’s what I think a lot about. But the reality is, every single one of us has a brand, and it has a huge impact on our career and how we show up at work.

Khobi Brooklyn: So I’d like to bring on some Aurora folks. We’re getting mic’d up, so it might take just a minute.

Khobi Brooklyn: Okay. All right.

Chethana Bhasha: I can get you … oh, yeah. I’m on.

Khobi Brooklyn: Is that pretty good?

Chethana Bhasha: I think so.

Khobi Brooklyn speaking

VP of Communications Khobi Brooklyn talks about personal brands, citing examples like Beyonce, Alexandria Cortez-Ocasio, and Nancy Pelosi, at Aurora Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Khobi Brooklyn: Okay, cool. So we’re going to talk a little bit about brand and building a personal brand, and what that means, and how that can have an impact on your career. And I think what’s interesting is, a lot of us have a brand, but maybe we don’t think about it because what is a brand? Right? We often think about companies and what a brand is at a company, but the reality is is that we all show up in some way, and so really, when it comes down to it, it’s how you show up.

Khobi Brooklyn: So here are three women that have incredibly strong brands, right? Beyonce is perfection, many would say. Alexandria Cortez-Ocasio, I think, is really real, right? She tells us all that she makes mistakes, but she also is unapologetic. And Nancy Pelosi is a great example of, I’d say, in the last year, she’s done a lot of work to reshape her brand. To be a boss, I would say.

Khobi Brooklyn: But we’re not here to talk about them, we’re here to talk about them. So we’re going to start with … well, and then a woman is really anything she wants to be. So at the end of the day, your brand is whatever you want it to be, and I thought that we could start by talking to these four women, and hear about who they are, and how they think about their brand. And ultimately how, as they’ve shaped their brand through their career, it’s helped them end up at Aurora, and helped them end up in the careers that they’ve had. All of them have really interesting work experience, and have taken very different paths to get to Aurora. So Chethana, we’ll start with you.

Chethana Bhasha: Sounds good. Thanks Khobi. Hello everyone, and welcome to our Aurora space, and then into this space where exciting things happen, as you can see one of the products right there.

Chethana Bhasha: Me, my brand, I should say, if you see me, I’m walking around the whole office talking with cross-functional people, interacting and then building things. I’ve been always curious, I wanted to know where, when I’m building some items, where it ends. So I want to see the end product. So that said, being a controls background engineer, I have worked on many products. And building those products, so I’ve been in the auto industry for the last … or a decade, I should say. And I’ve seen different transformations in the technology, and it’s still transforming, and this is right here. Like me here at Aurora, because we are building the self-driving technology, the Aurora Driver.

Chethana Bhasha: So here, the company, the best part is it’s sort of like an institution, as I’m passionate about learning more and more new things, exploring new spaces, and then be part of the technology, that is what Aurora has provided me. And I’m so excited to be here because, as I said, you can see me everywhere. I’m in packing, and then I have got so many opportunities in my role as a TPM or assistant engineer, or call me anything, I wear different hats every day, every hour, and it’s pretty good to learn things, be challenged, and then make it happen safely, quickly, and broadly. So thanks for that.

Khobi Brooklyn: Cool. Jessie, what’s your brand?

Jessica Smith: So you all heard a little bit about my background. I love simulation. I was kind of bitten by the bug, if you will, in grad school, and I work a lot in a semi-social role at Aurora and in my professional life. But when I go home at night, I usually have to decompress and not talk to another human being, because I’m pretty introverted in general. And so I wear a much more social hat at work, and I do a lot of work in trying to make sure that my team is communicating effectively with our customers who are the motion planning or the perception team. And that isn’t necessarily something that comes incredibly naturally to me, but it’s a role that I fill really well at work.

Jessica Smith: And then I do have to go home and only talk to my dog for a couple hours. So I think that what drew me to Aurora was that we have a lot of opportunity for people to really be themselves and to thrive in whatever environment that they thrive in. And you can find a niche here no matter what your personal brand is or your strengths are.

Khobi Brooklyn: Thank you. And Lia, you had an interesting career. Maybe we could even say you’ve reinvented your brand throughout your career? It’s a leading question.

Lia Theodosiou-Pisanelli: Sure. Oh boy. I don’t know if I’m prepared for that one. Yeah. So I … let’s see, what is my brand? I think one thing that I’ve always been really fixated on is making sure that I am authentic, and true to who I am. And in some cases, that can be a bit serious in the workplace, and I hold myself and everybody to a pretty high standard. But I also make sure that we don’t take ourselves too seriously.

Lia Theodosiou-Pisanelli: And another piece of that is also, I think that it’s really important throughout your career to focus on getting to know people as people. And a big way of doing that is … or, a big benefit of being able to do that is ending, is being in multiple roles where you kind of straddle a line between very different organizations, between very different sort of jurisdictions in some cases in my career between very different countries or political parties. And it’s really kind of evolved over time from when I was in government to when I’ve been doing product and a variety of different companies and scenarios. But the thing that’s tied it together is really being able to connect with people and translating between different worlds. And so that’s what led me here. I had an incredible opportunity to sit at the nexus between, between business and product and technology and to be able to build out a team and a function to really kind of bring all of those pieces together. And so even though I’ve had a lot of different pieces of my career and experiences, all of that has kind of come together to be able to really deliver, I think, something pretty effective here at work.

Khobi Brooklyn: Thank you. And Catherine, you have a very interesting career in spending some time on the engineering side and now on the legal side. And how have you thought about your brand as you’ve changed and evolved?

Khobi Brooklyn, Chethana Bhasha, Jessica Smith, Lia Theodosiou-Pisanelli, Catherine Tornabene speaking

Aurora girl geeks: Khobi Brooklyn, Chethana Bhasha, Jessica Smith, Lia Theodosiou-Pisanelli and Catherine Tornabene speaking on “How to Accelerate Your Career and Increase Your Impact” at Aurora Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X

Catherine Tornabene: So, hi, I’m Catherine, my role here is head of intellectual property and the legal team, but it should be mentioned the, I started my career in engineering. In fact, I was a software engineer back at Netscape back in the day and then went to law school and also obviously worked as a lawyer. And you know, when Khobi asked me this question, my first thought was, well I don’t even remember my Twitter handle. Like I don’t have a brand. And, but you know, thanks to talking with Khobi and her team, I realized, well actually I do. And that there’s really not a lot of people who have, it’s out of a niche expertise. There’s not a lot of people who have the background I do. And so my brand really is that I have a background in engineering and in law and I use both of them really every day in my job. And so it was very interesting. I appreciate Khobi even bringing the question forward cause I think it’s a very interesting question to think about. You know, I encourage you all to think about it. I thought it was a good thought exercise.

Khobi Brooklyn: Well I think building on that often, you know, part of what a brand is, is an emotional connection, right? So it’s how you’re perceived. It’s how we’re perceived in the workplace. And I would say as a woman in business and as a woman and often at tech companies, a lot of, we get conventional methods, right? We get [inaudible] whoa, sorry about that. You know, you’re either too nice or you’re too aggressive or you’re too mean or you’re too sloppy or you’re too proper or whatever, right? The list can go on and on. And I think for me at least, and I think for a lot of us up here throughout our career, we’ve found a way to find that balance of how can we show up at work in a way to to be super effective and so that people listen and we can do really good work. And how do we stay true to who we are? Right. I think, I’ll give you one personal example. I spent the first part of my life being an athlete and every coach I ever had said, you need to be really serious. You’re here to win, put your head down and win. And I literally was told not to smile because it would waste too much energy and I needed to be putting that energy into winning the race.

Khobi Brooklyn: And so that’s how I shaped my brand in the beginning. You know, I was very serious. I never smiled. I was heads down. I was there to win. And then I got into communications and I ended up in meetings with other people and I got feedback that I was way too serious and then I needed to smile. In fact, I was literally told I needed to be a ray of sunshine in every meeting. And I thought to myself like, I’m not a ray of sunshine, that’s not who I am. Like of course I don’t want to be bitchy, but I’m also like, I’m not the sunshine at the table. And it was conflicting. Right? It was super challenging for me to find out how can I be true to who I am, but clearly I need to smile more if I’m going to be effective in the workplace.

Khobi Brooklyn: And I think that’s just one example. I’m sure everybody in this room has some anecdote of a time where they felt they got conflicting messages or they weren’t quite sure like how do I show up in this meeting? Everybody else in this meeting is in sweatshirts, but I love to wear floral prints or you know, seriously or you know, everybody else in this meeting is, is super serious and I like to crack a joke every so often. Is that okay? And so I think that’s something that we all think about. Have any of you ever had conflicting messages and how you work through that?

Chethana Bhasha: I think I can just speak as Khobi just said, I mean she, I’m too serious. Like, and for me like it’s quite opposite. It always worked. I mean keep laughing maybe and get things done. That’s my mantra. But if it needs to be done, I mean it needs to be done. And it’s sometimes like I’m in in the workplace, being like a person. I mean I feel like I need to be straightforward and open, communicate, but the opposite person might not perceive it in a good way probably. But so I have been given an advice from my superiors at my previous company that “Hey, you’re doing a very good job, you get things done but make sure you are a little bit peaceful when talking with people.” And get, okay. So I’ve tried to balance that and then try to balance those emotions and then tried to read and then get at the end, make everyone happy and then work at that same place where you see each other, talk to each other. And that’s that. That has been working so far.

Lia Theodosiou-Pisanelli: Yeah, I’ve definitely gotten that conflicting advice as well. It’s interesting. So I started out my career as a negotiator for the government and I made the mistake of sending an email to a foreign negotiating counterpart that had an exclamation mark in it. And immediately my boss came into my office and said, never put an exclamation mark in an email, you will not be taken seriously. Do not show emotion. You should never have emotion on your face unless it is intentional for the objective you’re trying to get across. Right. And so that was very different from then coming out here to tech. And it’s funny. So I was kind of chiseled into this very aggressive and intense negotiator, which I’m sure none of you can imagine given how effervescent I am right now. But all of the people who work with me, you probably definitely know that I have that in me. But it’s so funny because then I started in tech and one of my first bosses in tech, maybe a month in, sat me down and said, hey, you should really think about like smiley faces, exclamation points, just to soften your tone a little bit because it kind of overwhelms people.

Lia Theodosiou-Pisanelli: And so it’s this funny like, Oh, okay, that is what success is here. And so I think what I keep kind of going back to is what is true to myself as, yeah, I’ll say different days, there’s, there’s a lot of balance that we all have to strike. But I just try to keep coming back to being authentic and being okay with the fact that that version of myself might not be what people expect of me and definitely might not be what people expect of a woman. And so it’s really important to just be OK with the fact that you’re different and not necessarily try to blend in. And so that’s what I’ve tried to hold, hold true to.

Khobi Brooklyn: And speaking of attributes, all of you are building teams and so as you build a team and you meet new people and new candidates, what do you look for? Like what kind of brand are you looking for? Catherine?

Catherine Tornabene: You know, I think, Oh, a lot of my personal career has been driven by that. That sounds really cool. And I look for that. I think intellectual curiosity is wonderful. I love when I get people who are really interested in the world around them and who are interested in how they can have an impact on the world. You know, one of the things I love about Aurora is that we are very mission driven here and that’s something that I look for and that’s some, a lot of people who care about the world around them, it is part of a personal brand and that is something I personally look for and that I enjoy very much in my teammates. We’re lucky to have that here.

Lia Theodosiou-Pisanelli: Yeah. Similar on a similar note, I would say really people who have a growth mindset, you’re not always going to find somebody who has the exact experience and fit for the tasks that you plan to have. But really having somebody who wants to grow and to learn and is willing to challenge themselves, not just in work but who also kind of shows that they want to be better. And it’s okay that maybe some things haven’t done, they haven’t done well in the past. And it’s not that they haven’t done them well, it’s just that things didn’t work out. But they learned from that. I think that’s a really important trait in somebody on the team.

Khobi Brooklyn: Jessie?

Jessica Smith: Yeah, I think, I mean for software we focus a lot on can you program, can you program, can you program? But I also really appreciate it when I ask a candidate something and they don’t know the answer if they’re just honest about like, I don’t know what that is. And then I think it provides an interesting opportunity in an interview to work through a problem together and you get to see a little bit more about is this person teachable and can we actually have a good back and forth? And if I give you a, like a hint or put you on the right path, can you actually go and ask for enough guidance to get to the right answer? So I think I really appreciate honesty in, in the interview environment.

Khobi Brooklyn: And maybe just generally.

Jessica Smith: And generally, yeah.

Khobi Brooklyn: Cool. Chethana?

Chethana Bhasha: So on the same lines, it’s person’s willingness to learn and then also at the same time contribute because it’s on both sides, right? Like you bring your own expertise. Yes you are not expert in all but you are trying to learn more but at the same time you are trying to contribute. So that that’s what most of the time we as a team look forward for like, hey the candidate is willing to learn, have the confidence, but at the same time I mean can contribute what they have learned in their past. Bring those lessons learned. So that’s what we are looking for more to build this awesome product. Yeah.

Khobi Brooklyn: Great. I’ll just add in one for myself. You know, not working in the kind of tech space. Sometimes it’s a little different what we look for, but I would say presence is really important. It’s something that I definitely try to pick up when I meet somebody new. Presence and self awareness. And I think in the tech industry broadly, we’re all doing something new, right? We don’t know the answers to everything. And so there’s a lot of mistakes. So there’s a lot of like, Ooh, we need to rethink that. And I think that takes incredible presence to have the confidence to say “I didn’t do that quite right and I need to do it better.” Or “I think I can do it differently.” And, and I think that that can be a hard skill to build because, it’s intimidating, right? It’s, it sucks to be wrong, but the more that you can get comfortable with it and use it in a positive way, I think makes us even more valuable. We’re gonna open this up to you all, but one more thing before we do is I wanted to ask each of you to share a piece of advice, either a great piece of advice that you’ve received in your career that’s really helped you along the way, or a piece of advice that you’d love to share with this group. Who wants to start? Chethana, go ahead.

Chethana Bhasha: I think what I’ve learned from like in the past was like the, or the mantra. What I usually follow is do the things, do the things in the right way, do it takes time or you face some failures, but at the end you know every single detail of it because if so if you are building a new product then you know, oh it’s the similar lines what I did in the past, this could come up and then there is mistake but that’s fine. I can do it. So that’s one thing which I would like to just as a my, my piece of advice is whatever you are doing, be confident and do it in the right way. Do I take some amount of time and failures.

Khobi Brooklyn: Yeah. Jessie.

Jessica Smith: I think the best piece of advice that someone gave me when I was thinking about a career transition was I was trying to decide should I, what should my next thing be? And it’s really hard to look at where you should go next. And a product manager that I worked with told me you shouldn’t think about your next job. You should think about your next, next job and what jobs do you need to get your next, next job. So you look a little bit further ahead and it’s actually easier to build a roadmap to where you want to be. You know, when your next next job. And so that’s really helped me build out a much more clear picture of where I want to go.

Khobi Brooklyn: Lia.

Lia Theodosiou-Pisanelli: Now I’m going to change [inaudible] ripping it. In terms of kind of looking for next jobs actually and this was, good advice for me as I was thinking about coming here. You know, you can think about is the work interesting and can I make an impact and what will this look like on my resume and all of these things. And all of those are important. But one big thing that is really important is thinking about who are you spending the majority of your waking hours with, right? We’re spending a lot of time together and so think about the people and the culture and the environment and are you going to learn from these people? Are these people going to let you be that authentic self? Are you going to be better? And when things don’t go well, do you feel like these people are going to support you and find the right solution? And so I hadn’t always focused on that. It was important, but I was always kind of blinded by the what is the most interesting, best stuff. Good news is Aurora has all of those things. So it just so happens that the people piece was like the cherry on top. But, no, really, I think, I think the people pieces is really, is really important and that, that was good advice that I received before coming here.

Khobi Brooklyn: Catherine.

Catherine Tornabene: So I think that, I think in this one, one of the most important pieces of career advice I received was once you start down a path, that doesn’t mean you’re fixed on it forever. And sometimes those meanderings that you take along the way actually turned out to be very valuable. So if you want to, if you’re debating a choice in your career or your job, you can always give yourself the choice of saying, you know, I’ll try this and if it doesn’t work out, I’ll try something else. Because I think a lot of the times we feel often like, oh my gosh, if I do this I am down this path and I am never stopping and I’m never off that route. But that’s actually not really how things generally work out. There are very few career paths that are absolutely fixed and you can generally take another route and sometimes you might find that the meandering part is the best fit.

Khobi Brooklyn: Thank you. So we’d love to hear from you all. So if anybody has any questions, please raise your hand. We’ve got mikes I believe around, so maybe you could stand up and just introduce yourself. You want to? Hi.

Aurora Girl Geek Dinner in Aurora garage

Claudia in the Aurora Girl Geek Dinner audience asks for book recommendations for women looking to accelerate their careers.  Erica Kawamoto Hsu / Girl Geek X

Claudia: Hi, I’m Claudia. Claudia [inaudible] and I have a question related to books that you guys have have read in the past that are really impactful. I’m a sucker to to learn more about what you guys have in mind around books that will help career growth.

Khobi Brooklyn: Anybody? Top of your head?

Lia Theodosiou-Pisanelli: I just read a book called The Growth Mindset, which might influence the fact that I look for people with a growth mindset. I found it to be really interesting. Actually. I’m going to be honest, I didn’t read it. I listened to it at a very fast rate. But I found that to be really interesting because it was kind of a way of describing different frames of mind of different people, which helped me to think about how I interact with others. What is my way of approaching things and being open to the fact that I can change that, so that’s a good one.

Khobi Brooklyn: Cool. Anybody else?

Catherine Tornabene: I read a ton, but very few career books.

Chethana Bhasha: That’s what I was going to say too.

Catherine Tornabene: I actually, in a sense. My answer, quite frankly, my answer is that the books that I often finance inspiration from are stories of fiction or I actually pretty much read everything, except I really don’t like brutal murder mystery. But beyond that, and so stories that I’ve read recently have been like for instance stories about, I’ve read a series of stories about Vietnamese immigrants who come to the United States or I actually read recently a story about you know, a mom who gave her child up for adoption. I like just getting in someone else’s mind for a while, I think actually is very good for teaching you mental flexibility in general. So my general advice is not actually a specific book but that the exercise of reading something that describes and gets you into someone else’s life experience is very good.

Khobi Brooklyn: Great.

Shavani: Hi, my name’s Shavani. I just had a quick question about, we talked a bit about all of your brands and what your brands are today, but you know, as you guys mentioned, you come from various backgrounds. How do you guys continue to build your brand? ‘Cause as we all know, it keeps changing every day. So if you guys like, you know, networking or any tips or bits of advice for that?

Chethana Bhasha: Yep. Yep. So, good question. So it’s again as we said, right? Like it’s you who you are. Like I’m in the more, you know, over the years that’s how you know, you get to know yourself like, Hey, who am I or what, what finds yourself like I mean, have your happy. So that kind of, I mean it’s sort of exploration and at one stage you find that Hey, this is me and this is where I have to do. Like for my example, like I started my career, as I graduated from a controls background, I started in the auto industry working on the diesel engines on a small center. But now I’m building the whole vehicle by itself. So because that tended like, I mean, Hey, who am I? Because I’m curious. I want to learn more and then I want to pick, put things together. I want to know where the end product is. So I got to know who I am. So say I’m a system architect or an engineer, now I know like that’s my basis. So that’s what I do, I interact with and collaborate with different stakeholders too because I like it. And then I want to build a product so now I know who I am and what is my passionate. So over time that gets you right there on your path like you know you’ll be happy in what you would be doing.

Khobi Brooklyn: Yep. I think go ahead.

Jessica Smith: That also helps. One of the things that I always find is that if I’m, if I’m too comfortable, I don’t really, I stagnate a little bit and I get, not bored, but I get too used to everything and I have to find something that pushes me out of my comfort zone. And so I will usually target something that I am kind of interested in but like really scares the crap out of me. And then I will go for it and add something to my plate that is completely outside of my comfort zone. And that really has forced me into a lot of situations I never thought I would be in. And it’s made me find out things about myself in terms of what do I want from my career. And the answer has surprised me quite a few times.

Khobi Brooklyn: Yep, absolutely.

Xantha Bruso speaking

Xantha Bruso asks the Aurora Girl Geek Dinner panel a forward-thinking question about the future of jobs.  Erica Kawamoto Hsu / Girl Geek X

Xantha Bruso: Hi, my name is Xantha Bruso. The autonomous vehicle industry didn’t exist that much longer before. And some of you have experience in other autonomous vehicle companies, but some of you didn’t. So how did you leverage the experience you had to enter this industry when in the future? You also know that the jobs in the future that you may have may also not exist currently. And how can you also stay relevant with what you’re doing now for those future jobs?

Catherine Tornabene: Well, I think that, I think that at the end of the day, being able to be comfortable learning things that are outside your comfort zone is really important. And when I look at my career spanned a lot of, I was at Google, I was at Netscape, there’s a lot of, I was often in situations where I didn’t actually know the, I didn’t have an expertise necessarily. And so I think that my general answer to that is that you just have to be comfortable with learning and being comfortable with saying like, you know, I don’t know the answer here, but I can figure it out. And that, you know.

Catherine Tornabene: I think the thing is in the AV space is there’s a great opportunity to learn and it’s developing very quickly. So I think that my answer to that is that I think taking a step back and looking less at the oh, the specific thing is not something you know. And more at, well, you know what? This is a thing I think I can learn. Is how I would approach it at least.

Khobi Brooklyn: Yeah. I think to build on that, I think part of what’s exciting about being in an industry that’s just shaping up and being at a company that is young and growing and shaping is that it’s less about saying, I know exactly how to do this one thing and I do it this way and I’m on this line doing this one thing. But these are my strengths. Here’s what I’m really good at. Here’s the value I can bring and different perspectives that I can bring. And together all of these different experiences and perspectives are shaping a company and helping to shape an industry. And I think that it will continue to evolve. Which one, keeps it super interesting for all of us or anybody in the industry. But also you find new ways to apply your strengths, right? And I think that that’s what’s super exciting about this industry is that you get to think differently all the time.

Chethana Bhasha: Yeah. And just to add, I think I can give my example clearly because I’m coming from a conventional automotive industry. I’ve worked on trucks and on highway and off highway which is completely a conventional [inaudible] was part of it now interspace where we are building the technology to do integrate in those platforms. So I get to see both the sides because I know how it works in the [inaudible] space and which is the technology we are building and how we integrate. So I get my own strength from the industry. At the same time I’m learning like what this technology does and how can we integrate together to have a great product. Yeah.

Audience Member: Oh, social media. Do you do that? What do you do? How cognizant of it are you? What’s your kind of strategy on developing your brand on social media? Thank you.

Khobi Brooklyn: This may sound weird coming from the comms person, but I don’t think you need social media to build a brand. I mean, I think if you want to build a big public presence brand, yeah, you should have a voice and you should find some channels to get your voice out there. But I think you can do a lot of really important work around building your reputation and being known in lots of different ways. I think it’s everything from how you show up to a meeting to what’s your tone over email to going to networking events and meeting people and sharing your thoughts and hearing new people’s thoughts.

Khobi Brooklyn: I think social media is really cool and a whole other conversation, but I think when we think about building our brand, that’s one way to share your brand, but it’s not necessarily fundamental to having a strong brand is my perspective. I don’t know if any of you have big social media presences.

Lia Theodosiou-Pisanelli: I think I tweeted this.

Audience Member: Thank you.

Khobi Brooklyn: And there we have… yeah.

Audience Member: I guess I have more of a practical question. How do you get feedback on if you’re presenting the right brand? Because I found out I’m like a very nice person, but I’m introverted so when people meet me they’re like, she doesn’t like me.

Khobi Brooklyn: I think that’s a great question. I would love to hear how any of you have received feedback. I think, yeah. Let’s hear from you guys first.

Lia Theodosiou-Pisanelli: Trial and error. This is really where it is. It’s like something’s not going well here and I think just really trying very hard to put yourself in somebody else’s shoes and to be aware of how different people are reacting to you. Right? And trying to kind of read the room or read the reaction and realize that, okay, that didn’t feel like it went well. Either I can ask why it didn’t go well or I can just try it a little bit differently this time. Right? So it depends on kind of what your comfort level is. But I think there is no silver bullet here. We all just learn as we go and, you know, that’s my take.

Khobi Brooklyn: Yeah, just to build on that. Kind of paraphrasing what you said, but a lot of it is self awareness, right? And being intentional, right? If you’re like, I’m going to think about how I show up, this matters to me. You start to realize that and pay attention. The way I acted with this person, is it resonating? Am I bringing them along in the way I wanted them to or what have you? I think is a really important thing to pay attention to. I’m sure we’ve all received advice. I know I’ve received tons of feedback on my brand and some of it has been great and some of it I’ve completely disagreed with, right? And so I’ve always had to come back with like, well, what’s true to who I am? What feels right?

Catherine Tornabene: I think the build on that, the piece of that I think is listen to people’s feedback but also have the confidence to say like, no that’s not for me. Because there’s a lot of people who will give feedback that you know, not right for you. I have an example. I remember being told, this is years ago, well you should never as a woman have a picture of your kids on your desk. I remember I took that and I listened… I had a picture of my kids on my desk. I took that and then later I was like, you know, no. That doesn’t work for me. That’s not who I am. I’m not going to do that. So I think be open to it, hear it, but also be true to yourself and say like, no, that’s not who I am. And I’m not going to listen to that.

Lia Theodosiou-Pisanelli: And don’t apologize for who you are.

Khobi Brooklyn: I think we had a question right over here…

Chico: Oh, hi. I’m Chico. And I think my question’s more about like have you ever had imposter syndrome or things like when you get disillusioned with your job because there’s some stressful scenario going on, something like that. So how do you deal with those scenarios and just get over that realize like, okay no I’m actually good at this thing and I can do the thing. So just trying to get over that big hump.

Lia Theodosiou-Pisanelli: What’s imposter syndrome? I’ve never heard that. I don’t think any of us have had that.

Lia Theodosiou-Pisanelli: I think like best advice there for me is assume everybody around you is holding kittens. No, I’m just kidding. Actually somebody did give me that advice and it was great. So I imagine that of you guys sometimes. What I would say is nobody knows everything and you know who you are and you know your experience and what you’ve learned throughout your life better than anybody else and that has made you into who you are. Right? So if I think of everything that’s happened in any of our lives, good or bad, failures, like sometimes we just do things really wrong, right? But that chisels you into who you are and you’re better for it. Right? So think of yourself as like this combination of all the experiences that you’ve had that only you know what those are, right? So nobody gets to say what you’re good at and what you’re not good at and just go for it.

Jessica Smith: I think it also takes a single catastrophic breaking of everything to realize that like, Oh, they didn’t fire me, it’s okay. I’m still breathing, the world still turns. I ruined everything for everybody for a little while, but it’s still all right. And it’s like a learning experience and… not that that ever happened to me in my early, early career, but it made me realize that like it’s going to be okay. Like even if something terrible happens and if you mess up and fall on your face, it’s really going to be okay and it’s okay to make mistakes because everyone does.

Chethana Bhasha: And as Catherine and Jessica and Lia mentioned it’s getting out of your comfort zone, right? Like if you don’t know yourself, like what you are good at or what you can do more. You have to do that. Like, I mean like as, yeah, sure, you didn’t get fired, but like you had to be like present a report in front of the upper management. Own it and then fix it so that builds your confidence.

Khobi Brooklyn: I think also somebody once told me, if you’re in the room, you belong in the room, you know? And I think it’s important to remember. If you’re sitting at the table, if you’re part of that project, you’re there for a reason. So own it and you belong there and somebody else thought you belonged there too. And so it’s just about kind of having that confidence again and just saying like, yeah, I’m here and I belong here. And being there.

Audience Member: I have a question. I guess sort of referring back to the question before this one, which is parsing through feedback, right? You get all sorts of feedback. Someone told Lia to put smiley faces in her email, things like that.

Audience Member: And this is kind of, I guess a tough subject because I think about this a lot. But as a woman, right? We’ve all heard that women get the whole, you’re aggressive feedback or you’re this way. You need to smile more. That type of feedback way, way more like the statistics show that that’s what happens. But sometimes there may be some validity to it. Right? It’s possible. And I think in my head I have that question a lot. If I’m getting the feedback that I’m too aggressive, is that real? Do I actually need to change my behavior? How do I think about this? How do I actually take that advice because it’s showed up in my performance review, so clearly I got to do something there, right? What do I do? And if I suspect that maybe it’s gendered, what do I do about that? Like how do I navigate that? That’s something that I would love to hear how you guys handle.

Jessica Smith: I have also received, “You’re really mean in code reviews.”

Chethana Bhasha: Yeah I think all of us. Yeah, yeah.

Jessica Smith: So I think my strategy for dealing with it is look at the people that I really respect in the company and who I would like to emulate and how do they give feedback and how do I maybe model my feedback on what they do in code review or in any of the communication that you’ve received feedback on and try and find ways to understand that your impact on other people might not be perceived in the way you expect it to be. And whether that’s from you know, a gender reason or you know, an experience level reason. I think that I’ve found success in changing the way that I speak to people by modeling it off of really successful communicators elsewhere in the company and it’s definitely helped me with this exact same problem. And you know, maybe giving like a little bit of positive feedback where you see… if you’re only ever writing like this is broken, this is broken, fix this thing. But you’re never saying like, wow, that was a really clever bit of code. If you have those thoughts, you can also share those thoughts and share the positivity, which helps make it so that you’re not being aggressive all the time.

Khobi Brooklyn: And I would say adding onto that is digging in a little bit. You know, like if you get feedback that you’re too aggressive, then ask why. Like, why? What’s happening or what’s not happening because of that? I think because at the end of the day, to be a good team player, to be a good part of your company and your team is to be effective. And if you’re doing something that’s not effective and maybe people like to call it being too aggressive, there is still something to fix, right? So maybe it’s the wrong label, maybe it’s sort of an offensive label because we women who sort of hear it all the time and it gets annoying. But at the end of the day, if there’s something that’s not working with the people you’re working with, then that’s fair. And that’s probably something to work on, you know? And so I think it’s a little bit of self awareness and ego and being like, okay, something’s not working I need to improve. But maybe pushing whoever you’re getting that feedback from on, well let’s talk more about that. Like let’s talk more about what it is that you’re really saying. I don’t know. That’s something that I have done.

Catherine Tornabene: I think that the other thing I would say is that I think it never hurts to assume positive intent when people are giving you feedback and assume that they actually really are trying to help you and maybe the words aren’t coming out right and maybe someone’s not really skilled at saying it or writing it or whatever. You know, nobody’s a perfect communicator and nobody can always say the right thing at the right time all the time. So sometimes, and of course there’s more career, you do wonder occasionally, you wonder, do you get feedback as a gender? But I think taking a step past and saying like, okay, well what’s the intent here? I’m assuming it’s positive and maybe there’s something here I can grow from and maybe it’s not the thing that was said to me. I mean it’s entirely possible that I’ll go in an entirely different direction.

Catherine Tornabene: But there is something there. And I mean, I don’t know, maybe I think I’m an optimist at heart, but mostly I think people want to help and they mean well and I think thinking in those terms can help you identify the thing that perhaps you want to take from it.

Lia Theodosiou-Pisanelli: One other thing I’d add is collect data, right? So similarly it’s like understand more where that person’s coming from, but then think, okay, if this is in my performance review, then maybe this came from multiple people. Maybe I should talk to a few people and not say, “Hey somebody wrote I was aggressive. Can you tell me if you agree or disagree?” But more along the lines of, “Hey, how do you feel like our dynamic is and are there ways that we could interact better?” Or things like that. And I think by having that with a few people and particularly people who you respect a lot, that will give you more context on something that’s more actionable than just kind of reading into what does this one sentence mean for me? Right?

Khobi Brooklyn: Yeah. Thank you. I think we have time for one more question, but then we have time for lots of questions just over drinks. So I think yes, you, go ahead.

Audience Member: First of all, thank you very much for all of your sharing, your experience and your perspective. It was really great to hear. Several of you here came from really different backgrounds and then transitioned into a new role and you talked a little bit about making those transitions and how your skills carried over and how you brought your backgrounds to your new roles. And I think it’s really great that Aurora is a company that values that and that sees that.

Audience Member: But I was wondering kind of from a branding perspective, if you guys could talk a little bit more about how you repositioned yourself when you made that transition. Because, as you said, you know your skills and your experience, but how do you reposition yourself to reframe that in a way, with your new role.

Khobi Brooklyn: I feel like you two should start.

Lia Theodosiou-Pisanelli: What? I think one way to go about it is to try to understand, okay, where do you want to go and what are the things that you want to do? Right? And then from there it’s trying to understand, okay, well what types of roles are interesting to you in that world.

Lia Theodosiou-Pisanelli: And then the next step, this is my thought process… And then the next step is, okay, well what makes somebody really successful in that role? And that’s usually how I start a lot of conversations because that way you can understand, okay, what are the attributes of a person? What are the things that they can do that mean success for either somebody who’s hiring or even just somebody generally who works at a company that’s interesting or in an industry that’s interesting. And then I think, okay, do I do things like that or do I have experience that can contribute to that? And how can I provide examples of things I’ve done in my past that translate into that. Right?

Lia Theodosiou-Pisanelli: And so I think one of the things about being in the self-driving space, is it hasn’t existed for that long. Right? And there is a finite number of people who have done this before. We have a lot of them here. But what I will say is there really is that openness to finding others because you… But finding people who have experiences that will help us to think about it in a different way. So that’s something Chris focuses on a lot is, how do we have a diversity of viewpoints? And so if you can think about, okay, yes, my perspective is different, but it adds value to whatever problem they’re trying to solve. Think about kind of explaining it in that way. That’s how I’ve thought about it.

Catherine Tornabene: You know, I think in some ways I would pivot it. And I think that the skills, obviously as I switched from engineering into law they’re sort of a different practical skillset.

Catherine Tornabene: But a lot of who I am is still the same. I mean, as a lawyer I’m not really all that different than as I was as a software engineer. And I think that rather than sort of focus on the external concept of necessarily rebranding, I think that I would view all of your collective experiences as you grow as part of your brand. And it’s just additive and it just adds onto your experience and who you are.

Catherine Tornabene: But who you are is you know, core to you and it kind of in a sense like which job you have. It’s just one facet of that. So I think that for me, I can’t say I thought all that much about necessarily repackaging myself as a lawyer. I just actually thought it was kind of interesting, which is how I ended up in law school. Then I thought that like this particular law job was kind of interesting. But in the end, like it’s always been like, oh, this is pretty interesting, but I’m still really the same person. And I think that the idea of brand is quite core to identity and who you are and your job is a big part of that, but there’s a lot more to you. So focusing necessarily, focusing on that will tell your story, I think.

Khobi Brooklyn: Well, thank you so much for coming. It’s been really great to have you and we would love to talk to you more. So stick around for another drink and maybe there’s even some desserts. I’m not sure. But thanks again for coming. We loved having you. And we will talk to you soon.

Khobi Brooklyn at Aurora Girl Geek Dinner

VP of Communications Khobi Brooklyn stays to mingle after the panel discussion at Aurora Girl Geek Dinner.  Erica Kawamoto Hsu / Girl Geek X


Our mission-aligned Girl Geek X partners are hiring!

“Enterprise to Computer (a Star Trek Chatbot)”: Grishma Jena with IBM (Video + Transcript)

Speakers:
Grishma Jena / Cognitive Software Engineer / IBM
Sukrutha Bhadouria / CTO & Co-Founder / Girl Geek X

Transcript:

Sukrutha Bhadouria: Hi everyone, I hope you’ve been having a great day so far. Hi, Grishma. Hi, so yes, we are ready for our next talk. I’m Sukrutha and Grishma is here to give the next talk. Just before we get started, the same set of housekeeping rules. First is, we’re recording. We’re gonna share in a week. Please post your questions, not in chat, but in the Q and A. So you see the Q and A button at the bottom? Click on that and post there. If for some reason we run out of time, and we can’t get to your questions, we’ll have a record of it and it’s easy for us to find later and get you your answers later.

Sukrutha Bhadouria: So please share on social media #GGXelevate and look for job postings on our website at girlgeek.io/opportunities. We’ve also been having, throughout the day, viewing parties at various companies. So shout-out to Zendesk, Strava, Guidewire, Climate, Grand Rounds, Netflix, Change.org, Blue Shield, Grio, and Salesforce Portland office.

Sukrutha Bhadouria: So now, on to Grishma. Grishma is a cognitive software engineer at IBM. She works on the data science for marketing team at IBM Watson. So today her talk is about Enterprise to Computer: a Star Trek chatbot. I’m sure there’s a lot of Star Trek fans out there because I know I am one, and I can’t wait to hear about your talk, Grishma.

Grishma Jena: Thank you, Sukrutha.

Sukrutha Bhadouria: Go ahead and get started. You can share your slides.

Grishma Jena: Okay, I’m gonna minimize this. Alright, can you see my slides? Okay. Hi, everyone, I’m Grishma. As Sukrutha mentioned I work as a cognitive software engineer with IBM in San Francisco. So, a lot of my job duties involve dealing with a lot of data, trying to come up with proprietary data science or AI solutions for our Enterprise customers. My background is in machine learning and natural language processing which is why I’m talking on a chatbot today.

Grishma Jena: I’ve also recently joined this non-profit called For Her, where we’re trying deal with creating a chatbot that could act as a health center, as a resource center for people who are going through things like domestic abuse or sexual violence so I’m very interested to see you know, a totally different social application of chatbot. But for today we’ll focus on something fun. And before I begin, a very happy Women’s Day to all of you out there. So, yeah.

Grishma Jena: When was the last time you interacted with a chatbot? It could have been a few minutes before, when, you know, Akilah was talking and your Alexa probably got activated by mistake and you had to be like, “Alexa, stop.” It could be with Siri. We interact with Siri every day. It could be on a customer service chat or it could be on a customer service call.

Grishma Jena: Basically, there are so many different avenues and applications of chatbots today that sometimes it’s even hard to distinguish if are we talking to a human. Is it a chatbot in disguise of a human? And it’s quite interesting to see where chatbots have come in the past few years.

Grishma Jena: So, this was a grad school project that we did. Our idea was, okay, chatbots are amazing. We really like that they help take some of the workload off humans, but how can we make them seem a little more human, a little less mechanical? Could we give them some sort of a fun personality?

Grishma Jena: And we brainstormed for a bit and we finally came up with the idea, hey, why don’t we, I mean … Well, to be honest we weren’t that big fans of Star Trek, but we did become one during the course of this project and we were like, “Okay, let’s think of Star Trek”. It has a wide fan base and let’s try to not pick one single character from Star Trek but let’s take all of the characters and make this huge mix of references and trademark dialogues and see what kind of personality the chatbot would have.

Grishma Jena: So, like I mentioned, the motivation was to make a chatbot a little more human-like. And we wanted to have a more engaging user experience. So the application of this could be, it doesn’t have to be something related to, you know, like an entertainment industry. It could be also something like a sports lover bot so that would be very chatty and extroverted and it would support your favorite sports team. Or it could be something a little more sober like a counselor bot who is very understanding and supportive and listens to you venting out or asks you about how your day was. So yeah, we chose Star Trek infused personality.

Grishma Jena: So our objective with Star Trek was wanted it to incorporate references from the show. [inaudible 00:05:17] wanted to [inaudible 00:05:20] Spock and live long and prosper. We wanted it to be data driven model, we did not want to feed in dialogues we wanted it to just feed in a corpus and have it generate dialogues on its own. We obviously wanted it to give interesting responses and to have the user engaged because that is one of the things that a chatbot should do, right? So in really simple words, just think of a friend of yours or it could be yourself who is this, you know, absolutely big fan of Star Trek and just transfer that personality to a chatbot.

Grishma Jena: So this is what the schema of our bot look like. We had the user utterance which is basically anything that you say or that you provide as input to the chatbot. And then we had a binary classifier. I’ll delve deeper into why exactly we wanted it, but the main point is that we wanted it to be able to distinguish whether what you’re saying to the chatbot is it something related to Star Trek or is it something a little more general conversation like, “How are you feeling today?” Or “What is the weather like?” And depending on that we had on that we had two different routes which the bot would take to generate a response.

Grishma Jena: So before we begin, we obviously need some sort of data and we decided that we would take all of the data that was available for the different Star Trek movies and the TV series. You’d be surprised at how little data is available, actually. We initially thought of just doing a Spock bot, but Spock himself has very limited dialogues so we just expanded our search to the entire Star Trek universe. And that’s why we took dialogues from movies, TV series. We didn’t want to have any sort of limitations as far as the data was concerned. We ended up with about a little over 100,000 pairs of dialogues.

Grishma Jena: Then we also went and got this database, which is known as the Cornell Movie Database. This database was created by Cornell University, which has a collection of raw movie scripts. It’s just a really good data set to train your bot on, the way how humans interact and what kind of topics they talk about, what are the responses like.

Grishma Jena: And finally, we also had a Twitter data set because we wanted some topics that were related to the ongoing affairs in the world, the current news topics. Because we envisioned that if you had a chatbot then people do like to talk to the chatbot or ask for the chatbot’s opinion on something that’s happening in real time.

Grishma Jena: So the very first component of a chatbot was having a binary classifier. Like I mentioned, we had two different routes for our chatbot. One would be the Star Trek route and the other would be a general conversation route. So we had the binary classifier that would help us distinguish whether whatever the user is uttering or whatever the user is giving as an input is it related to Star Trek or is it general conversation which was getting handled by the Cornell Movie Database. So we used an 80:20, that is the training data set and the testing data set split. And the features that we used were we took the top 10,000 TF-IDF unigrams and bigrams.

Grishma Jena: TF-IDF stands for tone frequency and inwards document frequency. Tone frequency is nothing but how many times a given word occurs in your corpus and inverse document frequency,, it’s kind of a weight that is attached to a word. So think of a textbook or think of a document that you have. Words like prepositions, like the, of, and would occur multiple times. But really words that would be important that would have some sort of conceptual representation, perhaps like the topic of it. Compared to it would be a little rare in occurrence, compared to prepositions, compared to commonly used words, and that’s why they should be given more weightage. So that’s the whole idea behind TF-IDF.

Grishma Jena: Unigrams and bigrams are nothing but you divide the entire document that you have into words. An unigram would be one [bit kilo word inaudible 00:09:17] bigram would be a set of two consecutive words that occur in the document. There’s an example later on in the slide to explain it better. Stop words, when consider stop words are just filler words like I mentioned similar to the prepositions. And we were very happy with the performance of the binary classifier. We were able to get a 95% accuracy on the test set, and we decided that is good enough, let’s move on to the next one.

Grishma Jena: And finally, this is the main core of it, where deep learning comes into play. So with deep learning, we used a model called a Seq2seq which is a particular type of recurrent neural network. So if you can see the image on the right, it is a simplified version of a neural network where you give it an input and it gets an output and that output is also the input for the next cycle, so it’s kind of like a feedback looping mechanism.

Grishma Jena: First, the specific type of neural network that we use, Seq2seq. It was just two recurrent neural networks so just think of a really big component that has two smaller components, which is an encoder and a decoder.

Grishma Jena: So the encoder actually takes in the input from the user and tries to provide some sort of context. What do the words mean? What exactly is the semantics behind the sentence that the user has given? And the decoder generates the output based on the context that it has understood and also based on the previous inputs that were given to it, which is where the feedback mechanism comes into play.

Grishma Jena: So just to go a little deeper into it. This is a representation of what a Seq2seq with encoder and decoder would look like. So the input over here would be, “Are you free tomorrow?” and the encoder takes in that input and tries to understand what exactly is the context or the meaning of this sentence. And finally the decoders understands, okay, this is something someone is asking about either they want to take an appointment or someone’s availability or someone’s schedule. And that’s where the reply is something like, “Yes, I am. What’s up?”

Grishma Jena: So these are some statistics about how exactly we went on training this on AWS. We used a p2.xlarge instance with one Nvidia Accelerator GPU and then we had the Star Trek Seq2seq. So we had one Seq2seq for just Star Trek dialogues and we had another one, the Cornell Seq2seq which is on Cornell data, which is more for just a general conversation purpose.

Grishma Jena: So we went ahead, we generated some sentences, but then we realized that the ones for Star Trek were really good because you’re giving it Star Trek as input so obviously the output is also going to be Star-trekky. But for the general conversation ones, for things like, “What is the weather like?”, “How are you doing today?”, “What is the time?” it was a little difficult for us because obviously the input is not Star Trek related, right? So the output also wouldn’t be Star Trek related, but we wanted this to be a Star Trek chatbot.

Grishma Jena: So we brainstormed a bit and we thought, “Hey, why don’t we try something called a style shifting?” Which is basically like you take a normal sentence, a sentence from the general conversation, and you try to shift it into the Star Trek domain.

Grishma Jena: And the way we did this was, we went through the entire corpus, the data set for Star Trek, and we created a word graph out of it. A word graph would be, just think of it as you pass different sentences in the data set and each of the words would form a node and the edges between them would tell how they occurred in relation to one another. So if they occurred right next to each other or within the same sentence.

Grishma Jena: And along with the words in the node we also had a part of speech tag. So we indicated whether it was an adjective, or a noun, or a pronoun or a conjunction. So let’s say for example our sentence was, “Live long and prosper.” You break it down into four words which are the four different nodes and then we label them with a different part of speech tag and we connected them because they come one after the other in the sentence.

Grishma Jena: So what we did, was after we built out this really huge word graph, we looked it up to insert what could be appropriate words between two given words in the input. So once we had the sentence we would check for every two words in the sentence and see what are the words that we could insert in between to give it more of a Star Trek feel to it to just, you know, shift the domain into Star Trek.

Grishma Jena: We went ahead and we did that and these were the kind of results that we got. “I am sorry” was the input and then the word graph went ahead and inputted “Miranda” at the end. “I will go” and then it inputted “back” at the end of the sentence because “go” and “back” kind of occur very commonly with each other. And similarly for the start of the sentences, it tried to input names like “Uhura” or “Captain”. So one thing we noticed was it really good at inputting names at the start and the end of the sentence and using the character names from the show did end up giving it a slightly more Star Trek feel than before.

Grishma Jena: So we went ahead and we just randomly tried to insert words that occurred more frequently between two words but then we realized that some of the sentences were ungrammatical. So what do we do? We came up with this idea of let us use the word graph as it is and then let’s take some sort of a filter to our responses. So, like I said, we realized that the word graph was giving a few incoherent and incorrect responses. What we did was we went ahead and constructed an n-gram model.

Grishma Jena: So n over here would be unigram, bigram, trigram. You can see the example over here if n is equal to one, which is an unigram, you break down the sentence into just different words so “this” would be one unigram “is” would be another unigram. If n is two, which a bigram, you would take two words that co-occur together. So in this case the first bigram would be “This is,” second one would be “is a” and then similar for trigram it would be “This is a” and then “is a sentence”.

Grishma Jena: So we created an n-gram model which was just to understand what exactly is the kind of dataset that Star Trek has. And then finally we wanted to get a probability distribution over the sequence of words that we have had.

Grishma Jena: So once we get this, we start to filter the responses and we ran the sentences using the bigram models that we trained on the Star Trek data set. Because of this we kind of got a reference type for seeing that what structures are grammatically correct. We went ahead and we get them and the ones that were a little odd sounding or that didn’t really occur anywhere in the data set we went ahead and removed them.

Grishma Jena: Another metric that we used for this was perplexity. So just think of perplexity as some sort of an explainability metric. We went ahead and used that which would help us tell how well a probability distribution was able to predict it.

Grishma Jena: Finally, we have all of the things in place and we have to evaluate the performance of the chatbot. So we came up with two categories of evaluation metrics. The first one was quantitative metrics where we used perplexity, which was mentioned on the first slide. And the second one was we wanted to see often was it using words that were very particular to Star Trek that you don’t really use in normal day life, you know, like maybe spaceship or engage.

Grishma Jena: And the second category was human evaluations where we got a bunch of, user group and we asked them to just read the input and the output and see how good it was in terms of grammar. If the response actually made sense, if it was appropriate. And finally, on the Star Trek style. Just how Star-trekky did it sound?

Grishma Jena: And, we also came across another bot online which is called as a Fake Spock Pandora Bot which was contrary to the way we had. Our bot was data driven this was rule based so it was actually given an input of human generated responses.

Grishma Jena: We wanted to see how good would a data driven model perform as compared to a human generated one. So this is just what the Fake Spock Pandora Bot looked like. And these were the kind of responses that the Pandora Bot gave. If you said, “I’m hungry, Captain” it said, “What will you be eating?” So it’s giving really good appropriate responses because humans were the back end for this.

Grishma Jena: And then, what we did was we went ahead and evaluated the results. And we saw that our bot was performing better for Star Trek style and it also was a little more coherent. For grammar, Pandora Bot was much better and that’s not surprising because humans were the ones who actually wrote it out. For perplexity, the Star Trek perplexity started dialogues were 65, so that was our baseline number and we figured out that the kind of responses our bot was generating that are 60, 60.9 was a little closer compared to Pandora was like, way far off at 45.

Grishma Jena: So we were pretty happy with our performance. I’m just gonna give you a few examples of what the different bots generated. So the yellow ones are the Pandora Bot and the blue ones are the E2Cbot. So let’s see, if the user says, “Beam me up, Scotty” the yellow one, that is the Fake Pandora Bot, gives, “I don’t have a teleportation device” which is a good answer. And the blue one is, “Aye, Sir” which is also a good answer. A little curt, but nothing wrong with it.

Grishma Jena: In the second example if you see our bot answered, “Bones, I like you.” So the “Bones” part is actually come from the word graph which gives it a little more of a Star Trek feel. And the last one over here is the Fake Bot, the human generated one, just says, “I am just an AI chatting on the internet” which is kind of not the response that you are looking for.

Grishma Jena: A few more examples over here. The user says, “My name is Alex” and then the Fake Spock Bot says, “Yes, I know Christine.” I just told you my name was Alex, why would you call me Christine? But our bot says, “What do you want me to do, Doctor?”, which is a better response. And, yeah, these are the kind of responses.

Grishma Jena: I think some of our human focus group people said that they might be correct, appropriate responses, but they might not be factually correct, which was a challenge for us, as well as for the Fake Spock Bot. We didn’t really delve deeper into it because that would kind of dive more into having a question answering system and trying to check if it’s factually correct or not but we tried to make our focus group users understand that it’s just a bot at the end of the day.

Grishma Jena: So finally, we were able to generate Star Trek style text. We were very happy with that, we were able to use the data driven approach which meant we could automate it. And we did figure that it performed better than the human generated responses that Pandora Bot would give, at least on style and at least on the appropriateness. It still needs a little bit of improvement in grammar but we were pretty happy with it.

Grishma Jena: So that’s me. Live long and prosper. And feel free to reach out to me on Linkedin or on Twitter if you have any questions about this. Thank you.

Sukrutha Bhadouria: Thank you, Grishma. This was great. So just to close I just wanted to mention to everybody that you actually sent your speaker submission to us and that’s how we got connected. So thank you for doing that. We got a lot of comments from people who are Star Trek fans, but yeah, what inspired you to build this project?

Grishma Jena: Yes, so this was actually a grad school project. We were taking a deep learning course so all of us had to build a chatbot as an Alexa skill. We brainstormed a lot, and we actually thought that Spock because Star Trek has a really huge fan base so Spock would be a good idea to do. But Spock had very little dialogue in all of the movies and the television series and then we were like, “You know what, let’s not stick to just one character, let’s have the entire Star Trek universe.” And, the bonus was that during my semester, I could continuously binge watch Star Trek and say that, “Yeah, I’m doing research because I want to see how well my chatbot works,” but I was just binge watching to be honest.

Sukrutha Bhadouria: Nice. That’s awesome. Well, thank you so much, Grishma, for your time. We really appreciate it and for your enthusiasm in signing up through our speaker submissions.

Grishma Jena: Thank you so much, Sukrutha.