Girl Geek X MosaicML Lightning Talks (Video + Transcript)

mosaicml girl geek dinner outdoor patio networking dinner palo alto playground global

Over 120 girl geeks joined networking and talks at the sold-out MosaicML Girl Geek Dinner from women working in machine learning at MosaicML, Meta AI, Atomwise, Salesforce Research, OpenAI, Amazon, and Hala Systems.

Speakers discuss efficient machine learning training with MosaicML, reinforcement learning, ML-based drug discovery with AtomNet, evaluating recommendation robustness with RGRecSys, turning generative models into products at OpenAI, seeking the bigger picture at AWS, and more.

Table of Contents

  1. Welcome – Julie Choi, VP and Chief of Growth at MosaicMLwatch her talk or read her words

  2. Making ML Training Faster, Algorithmically – Laura Florescu, AI Researcher at MosaicMLwatch her talk or read her words

  3. Reinforcement Learning: A Career Journey – Amy Zhang, Research Scientist at Meta AIwatch her talk or read her words

  4. Addressing Challenges in Drug Discovery – Tiffany Williams, Staff Software Engineer at Atomwisewatch her talk or read her words

  5. Evaluating Recommendation System Robustness – Shelby Heinecke, Senior Research Scientist at Salesforce Researchwatch her talk or read her words

  6. Turning Generative Models From Research Into Products – Angela Jiang, Product Manager at OpenAIwatch her talk or read her words

  7. Seeking the Bigger Picture – Banu Nagasundaram, Machine Learning Product Leader at Amazon Web Serviceswatch her talk or read her words

  8. 10 Lessons Learned from Building High Performance Diverse Teams – Lamya Alaoui, Director of People Ops at Hala Systemswatch her talk or read her words

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Transcript of MosaicML Girl Geek Dinner – Lightning Talks:

Angie Chang: Thank you so much for coming out. I’m so glad you’re here. My name is Angie Chang and I’m the founder of Girl Geek X. We’ve been doing Girl Geek Dinners in the San Francisco Bay Area for, if you can believe it, almost 15 years now. It’s the first Girl Geek Dinner in over two years, because Julie is a rock star and wanted to do a Girl Geek Dinner in person in the pandemic and we’re like, “Yes!” It was postponed and now in May, we’re finally doing this event. I’m so glad that we have a sold out event of amazing women in machine learning that we’re going to be hearing from tonight!

mosaicml girl geek dinner sold out angie chang speaking girl geek x

Girl Geek X founder Angie Chang welcomes the sold-out crowd to our first IRL Girl Geek Dinner in over two years during the pandemic! (Watch on YouTube)

Angie Chang: I don’t want to steal too much of the time, but I wanted to do a quick raffle of a tote bag that I have. I’m going to ask, who has been to the first Girl Geek Dinner and can name a speaker from that event? Sometimes I meet people who have. Are we making this really hard? Okay, the first Girl Geek Dinner was at Google. We had over 400 women show up for a panel of inspiring women. I just wanted to see because that’s a lot of people. Who thinks they’ve been to the most Girl Geek Dinners in the room? Okay.

Audience Member: It’s actually not me, but like, this clutch was designed by somebody who hates women because its super heavy – and I see that that [Girl Geek X tote] has handles.

Angie Chang: Okay, so we have a full agenda of machine learning lightning talks and I’m going to introduce you to our host for tonight. Julie Choi is the Chief Growth Officer of MosaicML and she is an amazing supporter of women and I would like to invite her up.

Julie Choi: Oh, thank you. Hi everyone. Thank you so much for coming out to the MosaicML Girl Geek Dinner. I am Julie Choi and I actually did go to the first Girl Geek Dinner. I want to thank Angie Chang and the Girl Geek Dinner organization. Angie has just been a pioneer and truly, just a very special person, bringing us together ever since, was that 2010 or something, I don’t know. When was that?

mosaicml girl geek dinner julie choi

MosaicML VP and Chief Growth Officer Julie Choi welcomes the audience. She emcees the evening at MosaicML Girl Geek Dinner. (Watch on YouTube)

Angie Chang: The first event was 2008.

Julie Choi: 2008. Yes. And tonight we have amazing speakers to share with us about machine learning, about engineering, about diversity, and how that can really supercharge productivity in high growth organizations and machine learning research just from some of the world’s best AI companies and organizations.

Julie Choi: Our first speaker of the evening is my dear colleague at MosaicML, Laura Florescu. Laura and I met a little over a year ago. You greeted me at the front door. You were the first face I saw at MosaicML. She has just been an inspiration to me as an ML researcher at our company. Prior to joining MosaicML, Laura actually worked at several unicorn AI hardware startups. Then prior to that, she got her PhD from NYU in mathematics and is just a brilliant lady. Laura, can you come up and tell us about this amazing topic?

Laura Florescu: Thank you.

Julie Choi: Let’s give her a hand.

Laura Florescu: Can you hear me? Hi, thank you so much everybody for being here. Thanks to Julie, Sarah, Angie, Playground, for having this event. Very honored to be here.

mosaicml girl geek dinner laura florescu speaking training nlp

MosaicML AI Researcher Laura Florescu talks about making ML training faster, algorithmically with Composer and Compute, MosaicML’s latest offerings for efficient ML. (Watch on YouTube)

Laura Florescu: As Julie said, I’m a researcher at MosaicML. A little bit about myself. I’m originally from Romania. I came to the states, did my undergrad in math, did a PhD at NYU, and then I got the kind of Silicon Valley bug. And now I’m at MosaicML. And so what we do is develop algorithms and infrastructure to train neural networks efficiently.

Laura Florescu: Basically for the people in the audience who are not into ML, training neural networks is at the core of artificial intelligence. It uses a lot of data. It’s applied to many different fields with image, language, speech, and kind of like a takeaway from this is that, for large powerful models, the training costs for one single run can get in the millions of dollars, to train one such model. And in order to get to a really good model, you need to do several such runs. So the cost can get extremely, extremely expensive.

Laura Florescu: Our belief at MosaicML is that state of the art, large, powerful models should not be limited to just the top companies. As we have seen over the last few years, the costs are getting larger and larger due to both the size of the models, also the data that the models ingest is just exploding.

Laura Florescu: A couple of years ago, a state-of-the-art model, Megatron, actually cost $15 million to train. As you can imagine, startups probably cannot really train models like that. And at MosaicML, this is our belief, that this kind of training should be accessible to other partners as well.

Laura Florescu: That’s where we come in. We want the state-of-the-art efficient ML training. Our co-founders are Naveen and Hanlin from Nirvana and Intel AI. Also, founding advisors from MIT and also founding engineers from leading AI companies. All of us have the same kind of goal to train machine learning basically faster and better and cheaper.

Laura Florescu: Our thesis is, the core of Mosaic is that algorithmic and system optimizations are the key to ML efficiency, right? And then the proof to that so far, is that we are working with enterprises to train ML models efficiently. And we want to enable our organizations to train the best ML models, the cheapest and the most efficient.

Laura Florescu: Some results that we already have. As I said, we want to be agnostic about the kind of models, data that we ingest. For some image classification tasks, we have shown 6x speed ups, like 6x cheaper, faster than like regular training. About 3x faster for image segmentation, about 2x for language models, language generation, and 1.5x for language understanding. We have been around as Julie said for about a year, a year and a few months, and these are already some of the results that we have achieved.

Laura Florescu: A use case is to train NLP models, such as BERT, for those of you who know that, and on our specific platform and without algorithms. Use case is for example, to increase sales productivity. If you see there, on our MosaicML 4-node, with our speed ups, which I’m going to discuss in a little bit, we can see up to 2x speed ups of training such models. And also about 60% training costs reduced by training with both our algorithms and on our platform, on our cloud.

Laura Florescu: The MosaicML cloud, we want it to be the first AI optimized cloud designed specifically for AI and directly to reduce training cost at any layer of the stack. For example, in the training flow, we want to reuse data from past runs. In the models that we’re using, so that’s where the kind of optimized model definitions come in, Composer, which is our open source library. We are doing the algorithmic speed ups, the training. And kind of like at the lower level, we want to also be able to choose the best hardware in order to get to the lowest training cost. At each layer, we are optimizing all of these system optimizations and composing all of those basically leads to the best training runs.

Laura Florescu: As I mentioned, Composer is our open source library. We have a QR code there if you want to check it out. We have about 25+ algorithms that we have worked on and given the name, they compose together, and that’s how we achieve basically the best, 2x to 5x speed ups. And as an example, for a BERT model, we have seen 2.5x speed up for pre-training, which as you can see, goes from nine days to about three days. That’s a huge win. Check it out if you would like. As I said, open source, so we’re always looking for feedback and contributions.

Laura Florescu: We’re open to partnering with any kind of corporate users, for anybody who has vision or language tasks, and then also industry, we want to be industry agnostic and global. Again, we want to optimize basically any kind of models. And as I said, the open source Composer speed up, we’re open to feedback and partnerships for that as well. And of course we are hiring. Yeah, it’s a really great team, really fun, really ambitious. And I’m so honored to work there and we’re looking for all kind of builders, engineers, researchers, products. And thank you very much.

Julie Choi: Thank you so much, Laura. Okay, that was wonderful. Let me just go to the next talk. Our next speaker is Amy Zhang, and Amy comes to us from, she’s currently a research scientist at Meta AI research, and a postdoc I think, you’re not a postdoc anymore, are you?

Amy Zhang: It’s kind of like a part-time thing.

Julie Choi: It’s like, never ending, huh? But you’re on your way to your assistant professorship at UT Austin, amazing, in Spring 2023. And Amy’s work focuses on improving generalization and reinforcement learning through bridging theory and practice. And her work, she was on the board most recently of women in machine learning for the past two years. And she got her PhD at McGill University and the Mila Institute and prior to that obtained her M.Eng. in EECS and dual Bachelors of Science degrees in math and EECS at MIT. Let’s welcome Amy Zhang to the stage. Thank you.

Amy Zhang: Thank you Julie for the really kind introduction and for planning this amazing event. It’s so nice to see people in real life. This is my first large in-person talk in over two years so please bear with me.

mosaicml girl geek dinner amy zhang speaking

Meta AI Research Scientist Amy Zhang speaks about her career journey in reinforcement learning, from academia to industry, at MosaicML Girl Geek Dinner. (Watch on YouTube)

Amy Zhang: Today I’ll be talking about my research which is in reinforcement learning, but I first wanted to just give a little bit of introduction of myself. To me, I feel like I’ve had a fairly meandering journey through academia and industry and research in general. I wanted to give a little bit of introduction of what I’ve done so that if any of you feel like you’re going through something similar, please reach out and I’m happy to chat and give more details. Like Julie said, I did my undergrad at MIT, a year after I finished my masters, I started a PhD at UCSD. It did not go well, through no one’s fault, really. I just felt really isolated and so after about a year, I quit my PhD, meandered through a couple of startups, and then eventually found my way to Facebook, which is now Meta.

Amy Zhang: At Facebook, I initially started on the core data science team. I was a data scientist, but I was working on computer vision and deep learning. This was 2015, still fairly early days in terms of deep learning and everyone is really excited about the gains that it had shown for computer vision at the time. I was working on population density, so we were taking satellite images and doing building detection to find houses, to figure out mostly in third world countries, what the population density was so that we could provide internet to people and figure out what was the best way to provide that internet.

Amy Zhang: After about a year of that, I ended up joining the Facebook AI research team FAIR, as a research engineer. And after about a year of that, I happened to meet the person who became my PhD advisor, Joelle Pineau, who is now director of FAIR, and I got to do my PhD with her at McGill University while still staying in FAIR. Fast forward a few more years and I defended my PhD remotely in the middle of the pandemic last year and am now back in the Bay Area as a research scientist at FAIR and like I said, was part-time postdoc-ing at UC Berkeley.

Amy Zhang: After spending all this time in industry, and having a really great time in industry, and getting all of these amazing opportunities to do my PhD while in industry, with all of the nice resources that that provides, I decided to go back into academia. And, last year I was on the faculty market, on the academic faculty market, and I accepted a position at UT Austin. I will be starting there next year as an assistant professor.

Amy Zhang: With that, I’m just going to jump straight into my research. And this is going to be, again like I said, a very whirlwind, high level overview. I’m passionate about reinforcement learning. I love the idea of agents that can interact with the world, with us, and it can grow and learn from that interaction.

Amy Zhang: Okay, thinking a little bit about what reinforcement learning (RL) can do. I’m personally really excited about the idea of applying reinforcement learning to solve real world problems. To me, this is personalized household robots, having a robot that will do your dishes, clean the house, make your bed, learned autonomous driving so you can just drive in a car without having to actually drive and pay attention to the road, and personalize healthcare, so having like a robo-doctor who knows everything about you and can personalize healthcare and give recommendations for you specifically.

Amy Zhang: Unfortunately we are not there yet, as I think maybe all of us can tell. Deep reinforcement learning has had a lot of successes in the last two years. Maybe some of these things are familiar to you. AlphaGo, where we have an RL agent that was able to beat the best human experts. OpenAI with playing video games, which I’m not very familiar with, but like Dota and StarCraft. These are the things that have been hitting the news in terms of what Deep RL is capable of. But there are still a lot of disappointing failures and I think none of these videos are going to show, but imagine this robot trying to kick this ball and then just falling flat on its face. That’s what that video is. And in the other little cheetah looking thing is supposed to be like tripping and falling. Anyway, this is where we are currently with RL.

Amy Zhang: Why do we still see this discrepancy? How are we getting these amazing gains but still seeing these failures? And what we’re really seeing is that Deep RL works really well in these single task settings, in simulation, when you have access to tons and tons of training data, but it works less well in visually complex and natural settings. Basically we’re not seeing the same type of generalization performance that we’ve been getting out of deep learning in computer vision and natural language processing. My research agenda is mainly about how can we achieve RL in the real world? How can we solve these problems? And, to me, it seems that abstraction is one key to generalization. And I use this type of idea to develop algorithms that have theory-backed guarantees.

Amy Zhang: I’m going to not really talk a little bit about this math, but I’m particularly interested in being able to train reinforcement learning agents that can solve tasks from pixels. Imagine that you have this household robot or this autonomous driving car that is receiving information about the world through a camera, through RGB input, and that’s a big part of how we also perceive the world. There are things in this image that are relevant for the autonomous driver here and there are things that are not. And we want to figure out how can we determine from just a reward signal, what things are relevant versus irrelevant.

Amy Zhang: I’m just going to, as part of this project, we developed this representation learning method using this idea by simulation, and showed that in this type of simulation driving task, which is done in this platform called Carla, so we have just this figure eight simulation environment where this car is just driving along this highway, and there’s lots of other cars in the road and basically, it’s designed to try and drive as quickly as possible. Using the break as little as possible and maximizing throttle while also not hitting anything. And we find that our method which can basically ignore these kind of irrelevant details and figure out what things are irrelevant does much better compared to a lot of existing methods.

Amy Zhang: One really cool thing is that when we look at the representation that we actually learn here, and we look at what kind of observations get mapped to be close together in this representation space, so what information is actually getting captured by this representation, we see that… this is the agent’s point of view. We see that, in these three examples, you’re always on this straight road where you have an obstacle on the right hand side. It doesn’t matter what the obstacle is, but the representation just captures that something is there, which means that you can’t turn right. This is just kind of an example of what our algorithm can do. And unfortunately I’m going to skip over this, because these were just some videos showing what our agent can do.

Amy Zhang: I just wanted to end on talking about some open problems that I’m particularly excited about. I’m particularly excited about compositionality. How can we solve combinatorially difficult problems. And these correspond to a lot of real world tasks that we should care about. When we think about really simple versions of problems like this, you can have a block stacking task. You can have any number of blocks or any combination of blocks and so you can always have new environments that you give to your agent to try and solve.

Amy Zhang: Similarly, moving more towards actual real world problems that we care about. Again, going back to the dishwasher example or an agent that is trying to move boxes around in a warehouse. These are all settings where the exact environment, the exact state that the agent has to deal with, is constantly different. The objects that you want to place in your dishwasher on a day to day basis is always going to look different. How can we get agents that can actually generalize to all of these new states?

Amy Zhang: I think one really exciting direction to go when trying to solve this type of problem is to think about factorization. How can we break down a problem into smaller, easier building blocks? So if we understand how one block, the dynamics of one block moves, so creating a stack of two blocks, right? Babies play around with this sort of thing and then as they get older, they automatically can extrapolate to building like gigantic towers and castles. So how can we take that idea and give it to reinforcement learning? So this is something that I’m particularly excited by.

Amy Zhang: I just wanted to end talking about my sort of wider research agenda. Now that I’m starting as faculty, I have to start recruiting a group. If any of you are interested in doing a PhD at UT Austin or know anybody who is, please send them my way. But when I think about what my research group does, I’m particularly interested in these three applications of reinforcement learning.

Amy Zhang: The first is in robotics, trying to solve manipulation tasks. Going back to that block stacking example, trying to solve locomotion and navigation tasks. How can we build an autonomous driving system purely from first principles, like end to end machine learning? Reinforcement learning has to be a part of that.

Amy Zhang: Natural language processing, so using RL for text generation, being able to extract knowledge from text, when you build an interactive agent, how do you give it information about the world? We learn from textbooks, we don’t want an agent just deployed in the real world with no basic information. Healthcare, how do we build RL agents that can help out with diagnosis and treatment and tackle a lot of the problems that we have there? That’s basically it, very grandiose. I probably won’t make much progress on a lot of these fronts, but this is the dream. And thanks for listening.

Julie Choi: Thank you so much. I think this is very grandiose and amazing that you’re working on it. Amazing. Thank you, Amy, okay.

Julie Choi: We’re going to just clean some of this up. Okay. And we’re going to open up our next talk, which I’m extremely excited about. Let me introduce our next speaker. Our next speaker journeyed probably the furthest to join us tonight… all the way from the east coast, North Carolina, to be here today to deliver this talk. I want to thank you. Thank you.

Julie Choi: Tiffany Williams is a Staff Software Engineer at Atomwise working on AI-powered drug discovery. Prior to Atomwise, Tiffany was a Staff Software Engineer at Project Ronin where she was developing cancer intelligence software. Tiffany earned her PhD in cancer biology from Stanford University and her Bachelor’s in biology from the University of Maryland. Let’s give a warm welcome to Tiffany Williams.

Tiffany Williams: All right. Hello, everyone. I’ll admit, I have some notes here so I can stay on track, but I’m really glad to be here this evening. I was searching through my inbox and realized that I attended my first Girl Geek Dinner in April of 2015. I was fresh out of grad school and a coding boot camp. Eager to form connections, acquired some cool swag, and to be honest, eat some free food and have some drinks.

mosaicml girl geek dinner tiffany williams speaking

Atomwise Staff Software Engineer Tiffany Williams discusses the drug discovery process with AtomNet at MosaicML Girl Geek Dinner. (Watch on YouTube)

Tiffany Williams: I went to grad school to study cancer biology. My research was from the perspective of a molecular biologist exploring the role of a protein as a target for therapy, and skin cancer. And my interests have generally been at that sweet spot, leveraging data, and technology to improve human health. Coming from a biology research background, in my current role as a software engineer at Atomwise, I’m able to look further down that drug discovery process. I’m happy to share with you all today, what that looks like.

Tiffany Williams: Now, I’ll start by giving a very brief primer on the current state of the drug discovery process, as well as a very brief primer on biochemistry. From there, I’ll talk about what we’re doing at Atomwise to make a significant impact in human health, and some interesting challenges ahead of us. I do want to give a disclaimer that I am merely scratching the surface of what could be discussed in drug design, drug development, and even applying ML on top of that. But what I hope you all take away from my time with you all this evening is just general excitement about the possibility of improving human health.

Tiffany Williams: Beyond that, I hope you feel empowered to do even more digging into the drug discovery process, and maybe you’ll feel empowered to even… to find opportunities to make an impact in that space. For me personally, I’m coming from the east coast, but I have a personal attachment to improving the drug discovery process. I’m on the east coast serving as a caregiver now – my mom has endometrial cancer and it’s at a point where there’s only one treatment option. If any of you have ever been in, unfortunately, in a predicament like that, it sucks.

Tiffany Williams: There’s a lot of data out there, technology’s improving, and it would be great if we can leverage that to improve health, right? This diagram depicts the drug discovery process from the basic research to identify a potential drug target all the way to FDA approval. On the right, I’ve noted the average number of years of the different stages in the drug discovery process. In the middle, what it shows are like basically in these different stages, there are certain types of experiments that are done that basically kind of knock out the potential candidates in that step.

Tiffany Williams: Initially you might have like a candidate pool of over like 10,000 drugs, but in an early stage, which is known as like kid identification, computer simulation is used to predict potential drugs ability to bind to a target of interest. These subsequent steps will test for other characteristics like potential toxicity, or efficacy in cell cultures, and animal models. Eventually, hopefully, we end up with a few candidates that reach the human clinical trial phase to verify safety, and any other side effects along with a few other things.

Tiffany Williams: What I hope that you take away from this slide really is that the current basic research to FDA approval drug process takes a really long time. It takes on average, it’s estimated to take about 15 years, and it’s also really expensive for each drug that goes to market, it’s estimated that 2.6 billion is spent.It’s the case that not all of these trials have a happy ending. For every drug that makes it to the market, millions may have been screened, and discarded. We have to improve this process, right? But in order to appreciate how this process can be improved, let me first give that very, very, very brief biochem primer. And I’ll focus specifically on protein interactions.

Tiffany Williams: Up to this point, and then even later in this talk, I’m going to be using some words interchangeably, and I just want to make sure I bring some clarity to what I’m actually talking about. When I say like ligand, or ligand, I’m specifically referring to any molecule that combined to a receiving protein, and that receiving protein is also known as a receptor. And in this presentation, when I use the word or phrase drug candidate, I’m actually referring to that ligand and the receptor would be that the drug target.

Tiffany Williams: One model for how proteins interact is this lock and key model. And the gist of this is that the ligand and the receptor have these somewhat complimentary sites. And the ligand combined to this complimentary site or active site, and basically alter the shape, and or activity of the receptor. And one more thing is that this binding is also referred to as docking. If we know our bodies are made up of proteins, and they have a diversity of functions within the body, but in a disease state, a normal process can become dysregulated. This image on the far right, is taken from a classic cancer biology paper called The Hallmarks of Cancer. The premise of this is that there about 10 biological capabilities that cells take on as they like morph into this more cancerous state. And I’m not going to go over all of these, but I want to highlight that two of these biological capabilities would be like cell growth and like cell motility.

Tiffany Williams: Those are normal functions within the body. But I guess within a diseased state you’ll have overgrowth, or you may have cells that are primarily concentrated in one area, develop the capability to invade into other tissues or metastasize. I say all this to say that like in small molecule drug discovery, what we want to do is actually figure out what sort of structure is needed for a ligand to bind to this problematic protein or proteins, and counteract that like not so great behavior. If this is where Atomwise comes in, right? Or it does, if you don’t know. We want to know how can we efficiently explore this space of all potential chemical compounds to better identify small molecule drugs faster.

Tiffany Williams: Atomwise actually developed AtomNet the first deep convolution neural network for structure based drug design, so that we can actually make better predictions for potential drug candidates earlier in the drug discovery process and faster. I’m highlighting this paper, again, just trying to give high level overviews. Feel free to check this paper out, but what I want you to take away from this is that the AtomNet technology is currently being used in real drug research, in cancer, neurological diseases, various spaces, on the right is, it’s actually a GIF, but since it’s a PDF, it’s not showing up as a GIF, but basically this GIF would show the AtomNet model. It would simulate the AtomNet model, predicting candidate treatments for Ebola. And this prediction that AtomNet made is actually has led to candidate molecules that are now being studied in animal models.

Tiffany Williams: One, despite everything I’ve said up till now, you actually don’t need a background in wet lab research, or chemistry to appreciate what’s happening here. The power of convolute or the power of convolution neural networks, or CNN, is that it allows us to take these complex concepts as a combination of smaller bits of information. And I think if you’re familiar with CNN, or even if you don’t like one area that is really popular, has been computer vision.

Tiffany Williams: I’ll briefly go over like an example of image recognition, and then kind of like try to tie it into how AtomNet works. An image is essentially represented as a 2D grid with three channels, you have red, green, and blue. And this network learns images of objects or faces, for example, by first learning a set of basic features in an image like edges. Then, from there, by combining those edges, the model can then learn to identify different parts of that object. In the case of a face that might be ears, eyes, nose, et cetera. With AtomNet that it’s working in a similar fashion, that receptor ligand pair is represented in a 3D grid, and the channels are essentially the elements that make up protein like carbon oxygen, nitrogen, et cetera. In the case of AtomNet, that the learning of edges is actually the learning of the types of, or predicting the types of bonds between those elements.

Tiffany Williams: Then from there, the compliment to the ear eye detection would be actually identifying more complex molecular bonds. You could say, essentially, that this network is like learning organic chemistry 101. This is powerful because we can then train these models to make predictions about different aspects of the drug discovery process. Like what ligands, or what type of structures are most likely to bind to a certain target? At what strength? What are the additional modifications that we can make to a potential drug candidate to strengthen that bond? Beyond that, it’s not enough just for the ligand to bind to the receptor, it also has to be like biologically relevant in that, let’s say, if we’re looking for something that is treating some neurological disease. We need that ligand to be small enough to cross the blood brain barrier. Or we may need to take into account toxicity or other effects that may happen in the body. Metabolism. We don’t want that small molecule to become quickly metabolized in the body before it has the opportunity to have the intended effect.

Tiffany Williams: These are exciting times, and I’m really, really passionate about the work that we’re doing at Atomwise and any work that is being done at the intersection of health and technology. I wanted to briefly go over some of the projects that my team is currently working on. I work on the drug discovery team at Atomwise, it’s within the engineering team. I think we’re working on some pretty interesting issues. One of my, my teammates Shinji, he recently has been working on bringing best engineering practices, and improving performance of some of our ML tools. Adrian, my teammate, is working on optimizing algorithms to explore a three trillion chemical space. He is also been working on, or has been able to create simulating mocular… mocular, I’m combining words… molecular docking on GPUs.

Tiffany Williams: Then another thing that we’re working on that I actually have more of a hand in, is building a research platform to better enable drug discovery. Oh, I didn’t mention the third person Shabbir, who’s our manager, and he has his hands in a bit of everything. What I hope you take away from this is that I think we’re at an exciting time in today to like really leverage data, and technology to make a major impact in human health. I think there are a lot of challenges, interesting challenges in drug discovery. I hope that you may have been convinced that you actually don’t need a background in chemistry to contribute. There are a lot of transferable skills. If you just know software engineering, or you know ML, or if you’re in product or marketing, there is a place for you in this space.

Tiffany Williams: Finally, really important takeaway is that we are hiring. My team alone, please, if you have any front end experience, if you have backend experience, or if you have a background in computational chemistry, those are some of the positions that are open right now on my team, but then outside of my team, we also are hiring. Definitely check out our careers page. If you have any more questions or are interested in chatting, feel free to reach out to me. I have my LinkedIn handle as well as my Twitter handle, posted here. Then finally, I think I mentioned that I had some references to share. Again, I’m only scratching the surface. There’s so much information out there. I wanted to highlight two Medium articles that were written by former Atom, which is what we call people that work at Atomwise, machine learning for drug discovery, in a nutshell, I highly recommend starting there if you want to do a deeper dive. That’s it.

Julie Choi: Thank you so much, Tiffany. Yes. I think drug discovery is an incredible application domain for deep learning. Really appreciate your talk. Okay, let me introduce our next speaker. Our next speaker is Shelby Heinecke. Shelby is a senior research scientist. Again, I did not touch. Okay.

Julie Choi: Shelby is a Senior Research Scientist at Salesforce Research, developing new AI methods for research and products. Her work spans from theory-driven, robust ML algorithms to practical methods, and toolkits for addressing robustness in applied NLP and recommendation systems. She has a PhD in Mathematics from the University of Illinois at Chicago, and a Master’s as well in Math from Northwestern and her bachelor’s is from MIT. Let’s welcome Shelby.

Shelby Heinecke: Thanks so much. Awesome. I have to give a thank you to Julie for including me in this event, inviting me. This is my first Girl Geek, and not my last Girl Geek event. I’m super excited to be here. Yeah, let me get started.

mosaicml girl geek dinner shelby heinecke speaking recommendation systems salesforce research

Salesforce Research Senior Research Scientist Shelby Heinecke speaks about how to evaluate recommendation system robustness with RGRecSys at MosaicML Girl Geek Dinner. (Watch on YouTube)

Shelby Heinecke: Today I’m going to be talking about evaluating recommendation system robustness, but first I feel kind of like the new kid on the block. Let me just give a little bit of background about myself. I moved here to the Bay Area about a year and a half ago in the middle of the pandemic. Super excited to be here in person to meet people.

Shelby Heinecke: Currently I’m a Senior Research Scientist at Salesforce Research. As Julie mentioned, I work on both research and product. It’s pretty awesome to develop prototypes, and see them in production. Before I was here at Salesforce in the Bay Area, I was doing my PhD in math in Chicago. There, I focused on creating new ML algorithms that were robust. I worked on problems in the space of network resilience. Before that I got my master’s in math, kind of focusing on pure math at Northwestern. Originally I hail from MIT Math, focusing on pure math there. That’s my background. Today’s talk recommendation systems and robustness.

Shelby Heinecke: Let’s get started. A crash course and recommendation systems. So, what is a recommendation system? Well, it consists of models that learn to recommend items based on user interaction histories, user attributes, and or item attributes. Let me give you an example, say we want to build a recommender system that recommends movies to users. Well, what kind of data can we use? We’re going to use the users, previous movies. They viewed we’re going to use the ratings that they rated those movies.

Shelby Heinecke: We’re probably going to take a look at the user’s age. We’re going to look at the user’s location. We’re also going to take into account item attributes. The movie attributes, like the movie title, the movie genre, we’re going to take all that. That’s all of our data, and we’re going to train models. The models can build the recommendation system. As you can imagine, recommendation systems influence our daily lives. We’re all exposed to recommendation systems every day. Just think about purchases. Think about movies.

Shelby Heinecke: Think about songs you’re recommended. Think about the ads that you see every day, people, news, information. We are at the mercy of these recommendation systems and a lot of our decisions are highly influenced by what the recommendation systems decide to show us. Let’s think about the models that we see in recommendation systems. Models can range from simple heuristic approaches, like a rules based approach or co-sign similarity approach to complex deep learning approaches, think neural collaborative filtering, or even now we’re seeing transformer based approaches coming to light.

Shelby Heinecke: With the vast range of models available and how greatly they impact our daily lives, understanding the vulnerabilities of these models of each of these types of models is super critical. Let me get started about recommendation model robustness. As we all know, machine learning models are trained on data, and ultimately deployed to production. And in that process, there are some hidden sources of vulnerabilities that I want to bring to light.

Shelby Heinecke: One of the big issues is that training data may not reflect the real world data. In many cases, we’re training on data that’s been highly curated. That’s been cleaned up. And yeah, and so because of that, when we train a model on that very clean, highly curated data, it’s not going to necessarily perform well when it’s exposed to the messy data of the real world. Real world data has noise. Real world data is just can be unpredictable. As a result, sometimes we train model, we train a recommendation model, but we see poor performance and production. That’s definitely one type of vulnerability we need to watch out for.

Audience Member: Woo! So true.

Shelby Heinecke: What? Okay. Another type of vulnerability. I love the enthusiasm, okay? Another type of vulnerability. As you can imagine, recommendation systems are closely tied to revenue for a lot of different parties for companies, for sellers. There’s an issue that participants may intentionally manipulate the model. Think about creating fake accounts, trying to do things, to get your item higher on the list for customers, things like that. That is a reality and that’s something we have to take into account.

Shelby Heinecke: The last thing I want to bring to light is poor performance on subpopulations. This is a well known issue across all of ML, but I just wanted to bring it to light, to recommendation too, that when we train models and we test on the evaluation set, usually the basic evaluation methods think precision recall F1. We’re computing that on the entire test set, so we’re averaging overall users. And in that average, sometimes we’re hiding…

Shelby Heinecke: Sometimes there’s poor performance on subpopulations that are hidden. For example, a subpopulation that you may care about could be new users, or maybe a users of a certain gender. That’s something that we just need to keep an eye on. We don’t want poor performance on key subpopulations I’ve told you about all these different types of vulnerability.

Shelby Heinecke: What is a robust model? Well, a robust model you can think of it intuitively as a model that will retain great performance in light of all of these potential perturbations, or in light of all of these scenarios. The question is how can we assess the robustness of models? That is where one of our contributions comes in.

Shelby Heinecke: I want to introduce one of our open source repos called RGRecSys, and it stands for robustness gym for recommendation systems. Our library kind of automates stress testing for recommendation models. By stress testing, I mean, you can pick a model, you can pick a data set, and you can stress test it in the sense that you can manipulate the data set.

Shelby Heinecke: You can add in attacks, you can add in noise and so on. I’ll actually go into more detail about that. And you can see in a really simple way how your model, the robustness of your model. As I mentioned, RGRecSys, is a software, is a software toolkit. It’s on GitHub. I’ll talk about that in a second and it’s going to help you assess the robustness of your models.

Shelby Heinecke: One thing that we contribute is that RGRecSys provides a holistic evaluation of recommendation models across five dimensions of robustness. When I say I told you about various types of vulnerabilities, there’s various types of robustness. Our library helps you to quickly and easily test all these different types of robustness for your model. Using our API, you just simply select a model for testing, and then you specify the robustness test that you’re interested in trying along with the robustness test parameters. What I’ll do is I’ll go over the types of tests that are in the library.

Shelby Heinecke: Let’s talk about the different tests that we have in our library. First is around subpopulations and this kind of goes back to what I mentioned in the previous slides. This will allow you the test. What is the model performance on subgroup A versus subgroup B? And this is going to be very useful because, as I mentioned, most of the time, we’re just computing precision recall these usual metrics on the test set, but this gives you an easy way to test performance on specific subgroups. This could be useful, for example, if you want to test you want to test performance on gender A versus gender B, or new users versus old versus like users that have been in the system for longer time. Just some examples there. That is one type of test you can run in our library. The second type of test is around sparsity. If you think about recommendation systems and you think about the items available like a movie recommender, or a purchasing recommender there’s millions of items, and each user really only interacts with a handful of those items.

Shelby Heinecke: Each user is only clicking, only purchasing, only viewing a handful. This is a source of data sparsity. Data sparsity is a huge problem in recommendation systems. It will be good to test the degree to which your model is sensitive to sparse data. That’s one thing you can test with our library. The third test is around transformations. There are a lot of ways that data can be perturbed when you’re training a recommendation system model. For example, maybe you’re gathering data about your users and maybe that data is erroneous in some ways. And because of that, you might want to test if a recommendation model will be robust, if user features, for example, are perturbed. The fourth test that you can test is around attacks. As I mentioned, there’s a lot of reasons why people would try to attack a recommendation system it’s tied to revenue ultimately.

Shelby Heinecke: What you can do with our library is implement some attacks and test how your model performance would change under those manipulations. And finally distribution shift. What we see is that the training data that you train on is often different from the data that you’ll see in production. It’s super important to be able to know, get a sense of how was my model going to perform if it’s exposed to data from a slightly different distribution? You can go ahead and test that with our library., I definitely encourage you to check out our library on GitHub, and feel free to check out the paper for way more details about the capabilities. And with that, thank you so much for listening. It was great to share it. Feel free to reach out.

Julie Choi: Thank you, Shelby. That was great. It’s very important to be robust when we’re doing model deployment. Okay. Angela Jiang, thank you so much for joining us tonight.

Julie Choi: Angela is currently on the product team at OpenAI. Previously, she was a product manager at Determined AI building, deep learning training, software and hardware, deep learning… I think Determined AI was recently acquired by HPE. And Angela graduated with a PhD in machine learning systems from the CS department at Carnegie Mellon University. And we actually have built some of our own speed up methods on your research. It’s an honor to have you here today.

Angela Jiang: Thank you so much, Julie, for inviting me, for the introduction, as well as Sarah, and Angie for organizing the event and bringing us all together too. Yeah. Like Julie mentioned, I’m Angela. I work on the product team at OpenAI.

mosaicml girl geek dinner angela jiang openai

OpenAI Product Manager Angela Jiang speaks about turning generative models from research into products at MosaicML Girl Geek Dinner. (Watch on YouTube)

Angela Jiang: I work on our Applied team where we really focus on essentially bringing all of the awesome research coming out of the org and turning them into products that are hopefully useful, safe, and easy to use.

Angela Jiang: Most of my day, I’m thinking about how to turn generative models from research into products. I thought I’d make the talk about that as well. This might as well be a list of things that keep me up at night, think about it that way. But in particular, what I really wanted to highlight is just some of the observations I’ve had about things that make these generative models unique and tend to have large implications on how we actually deploy them into the market.

Angela Jiang: To start, I want to share a little bit about what OpenAI’s products are. OpenAI does a lot of AI research. In particular, we do a lot of generative models. Over the last two years, we’ve really started to work to bring those generative models into real products. Three examples here are GPT-3, that does text generation; Codex, that does code generation; and most recently, DALL-E, which does image generation. These were all made by DALL-E.

Angela Jiang: To get a little bit more concrete, what these products are is that we expose these models like GPT-3 and Codex via our APIs. As a user, you can go to the OpenAI website, sign up, and then hit these endpoints and start using these models. And here’s an example of how you might use the GPT-3 model. Here we have an example of your input in the gray box. It might say something like, “Convert my shorthand into a firsthand account of the meeting and have some meeting notes.” And you submit this to the model, and then the model will do its very best to return you an output that completes this text. And in this case, the response is essentially a summarization of your notes.

Angela Jiang: Codex works very similarly, except it’s also specialized for code. And in this case, our input is not only text, but it’s also some code, some JavaScript code, and some hints that we want to transform it into Python. And the output is some Python code that does that. DALL-E is very, very new. It’s been out for around a month, so it’s not available in the API or anything yet. But it has a very similar interaction mode where you have, again, an input, which is a description of the image that you want. And then as the output, it provides some images that relate to that description, which in this case is a cute tropical fish.

Angela Jiang: I think GPT-3, Codex, and DALL-E are all really good examples of taking research and seeing that there’s a really big market need for the kinds of capabilities that they expose. Then working as an Applied team to actually transform that research into a productionized product by essentially figuring out where that user value is, designing the product so that we can deliver that user value in a way where users are set up for success, making sure it’s deployed responsibly, et cetera, et cetera.

Angela Jiang: To date, there’s hundreds of applications that have been submitted for production review that is built upon this API. Those are applications like writing assistants like CopyAI if you’ve heard of that, coding assistants like GitHub’s Copilot is built on top of Codex, as well as a lot of other applications like chatbots for video games, question answering bots, etc. This is pretty good validation that these models are not only really exciting research, but are also solving problems in the market.

Angela Jiang: Now, when we think about bringing these to market and turning them into real products, I think there are two interesting properties of these generative models that really change who uses them as well as how we actually deploy them. I’ll talk about how these models are stochastic and how they’re very general. With typical products, you might expect that the results are very predictable and deterministic. For instance, if you’re coding up a payments API, every time you use that payments API, you’d expect it to react in a similar way. This is not how our models work at all.

Angela Jiang: Our models are probabilistic. Every time you submit a prompt, you could get a different response back. If you tweak that prompt ever slightly, then you could get a very different response back. That really changes how you interact with a product and what kind of applications can be built on top of it. Another difference is I think that typical software products that I think about are typically focused on solving a need or a problem very directly, whereas, in these generative models, they’re very, very general. They’re capable of doing a lot of things all simultaneously.

Angela Jiang: For instance, you would think that GPT-3 would be able to summarize legal texts and also simultaneously write poems or maybe even code. I think this is super exciting from a science perspective as we get more and more powerful models, closer and closer to something that’s super general. But from a product perspective, this comes with some challenges because we have to now take one model and one product and have it serve many, many use cases simultaneously.

Angela Jiang: For the rest of this talk, I’ll talk about, again, more detail about how these two properties affect the way that we deploy our products, and I’ll give some examples of each. Okay, starting with stochasticity. Right. Our models, again, are probabilistic, which means that every time you use it, you might get a different result. The result might not be what you want or maybe the result is what you want every third time you try the model.

Angela Jiang: It might be surprising that you can actually build real applications on top of this kind of behavior. What we actually find is that for a lot of tasks, especially very simple tasks, we can have very concrete and reproducible behavior for even tasks where the outputs are imperfect or are only correct some of the time, these results are actually still very useful for many applications in the right context.

Angela Jiang: We find that these models work really well, for instance, in creating productivity tools where there’s a human in the loop. A really good example of this is Codex Copilot. Some of you may have used it before, but this is a screenshot of it in action, where it’s a Visual Studio Code extension and it just gives you auto-complete code suggestions that the user can then tab to accept or reject.

Angela Jiang: This is great for a product like Codex because you have actually an expert there that’s telling you, is this completion good or not? And generating more. What we actually see is that most of the suggestions from Copilot are rejected by the user, but still, developers tell us that this is an integral part of their deployment pipeline or their development pipeline I should say. It does not need to be perfect every time we generate something. Those are cases where you don’t need a perfect result every time. What you’re looking for is just inspiration or getting over writer’s block. In some of the applications built on top of these APIs, you actually don’t have a correct answer to shoot for.

Angela Jiang: Applications that are doing art or creative tasks or entertainment, that’s a case where having variety really is helpful actually. Here I have a DALL-E prompt, which is a bright oil painting of a robot painting a flower. And here are different generations from DALL-E. And in this case, it’s really helpful actually to have probabilistic nature because, for one, I want to have different options. And it’s also really cool to have generations that you’ve never seen before or other people haven’t seen before.

Angela Jiang: I think it’s even counterintuitive that, in this case, we’ve actually learned that these AI systems seem to be doing really, really well and surprising us in creative tasks as opposed to rote tasks, which you might have expected the opposite earlier. Those are some examples where the properties of these generative models affect what applications are built on top, but it also affects how we deploy these models. Going into a little bit more detail, one property is that these products are really, really hard to evaluate.

Angela Jiang: This makes being a Product Manager, among other things, quite challenging because we really need to know how our products perform so we can position them in the correct way to the users, know what to deploy, and tell users how to use them correctly. For example, we have these text models, GPT-3, and they have different capabilities, right? One capability is they might be able to complete text, and another capability is they might be able to edit existing text. There’s some overlap there on which one you should use.

Angela Jiang: It’s really important that we understand exactly how this model performs on different tasks so that we can direct users to use the right tool at the right time. We try really hard to figure out creative ways to evaluate these at scale, given that we often need a human in the loop to be telling us if a generation is good or not.

Angela Jiang: We do things like large-scale application-specific A/B testing. Like we see if we use one model or the other, then which model gets more engagement for this writing assistant.

Angela Jiang: Next, I’ll just give a couple of examples of how generality also comes into the picture here. Like I mentioned, these models are capable of a lot of things simultaneously because the way that they’re trained is that they’re trained for a very long time, for months, on a lot of data. And the data spans the internet, books, code, so many different things.

Angela Jiang: By the time we get this model as the Applied team, we’re really still not sure what kind of capabilities it’s picked up at that time. But like I mentioned, it’s really important that we understand that quickly. This is also really exciting because, at any given time, we can always discover a new capability of the model, and it’s often discovered after training.

Angela Jiang: An example of this is that the original GPT-3 model was built with text for text. And it was really a discovery after the fact that it also happened to be kind of good at code as well. And these signs of life and discovery is what ultimately sparked this idea of a Codex series that specialized in code.

Angela Jiang: As a Product Manager, it’s really critical that we keep on discovering and probing these models to really understand what’s the frontier of what they can do. And again, we are still figuring out what is the best way to go about this. But something counterintuitive that I realized after I started is that traditional user research that we’re used to doesn’t actually work really well for this use case because you might expect that you would go to your users and ask, how is this model? What can you do with it?

Angela Jiang: What we find is that our users, by definition, are working on tasks that the model was already really capable at. Their focus is not on the new capabilities of the model and pushing that frontier oftentimes. What’s worked better for us is working with creatives or domain experts to really hack and probe at the model and see what the limits of it are.

Angela Jiang: That’s the exciting part of generality, that there’s always a new capability around the corner to discover. But there’s also some risk to it because there’s not only all these capabilities in the model, but the model can also generate things that are not useful for you, and it can also generate things that you really don’t want to generate.

Angela Jiang: For instance, GPT-3, when prompted in the right way, can generate things like hate speech, spam, violence, things you don’t want your users to be generating. This is also a big part of our goals as a research and applied organization is to figure out how to deploy these models in a way that doesn’t have toxicity in them.

Angela Jiang: There’s a lot of different approaches for doing this, spanning from policy research to product mitigations to research mitigations, but I’ll highlight two things here. One thing that’s worked really well for us is actually fine-tuning these models after the fact with a human in the loop, telling you exactly what kind of content is good content.

Angela Jiang: What we found by doing this is that we have a much better time at having the model follow the user’s intentions. And we have a much less toxic and more truthful model as a result of it. At this point, every time we deploy a model, we also fine-tune the language model in this way. We also do a lot of research and provide free tooling to help the users understand what the models are generating at scale so they can understand if there’s anything that they need to intervene in.

Angela Jiang: Okay. Finally, even though I think it’s very, very cool that these models can do a lot of different things at once, sometimes it really just can’t serve multiple use cases simultaneously well. Different use cases often will just require, for instance, different ways of completion.A chatbot for children is going to want a very different personality than a chatbot for support.

Angela Jiang: You also just have different product requirements in terms of accuracy or latency or even price. Copilot is a really good example of something that needs an interactive latency so that it can continue to be useful in real time. But contrast that with something like SQL query generation, which doesn’t need a fast latency, but actually just wants the most accurate response that you can give it. What we’ve found is that the combination of two things have allowed us to provide this flexibility to serve the use cases that we need to serve.

Angela Jiang: One is that we don’t just expose the best model that we have or the most accurate model we have I should say, we expose models with different capability and latency trade-offs for the users to choose from. And then we also offer fine-tuning as a first-class product so that you can take a model and then fine-tune it so that it has your personalized tone or your personalized data bank.

Angela Jiang: These are, hopefully, examples that give you a flavor of what it’s like to deploy generative models. This is also just the tip of the iceberg. If any of this stuff is interesting, please feel free to come and chat with me. I should also mention that we are hiring. We are. Thank you so much.

Julie Choi: Thank you so much, Angela. After DALL-E was launched, the productivity went a little down. We were so distracted by the DALL-E. I don’t know whether to thank or curse your team for that.

Julie Choi: Thank you, Banu. It is just a joy to introduce our next speaker. She’s a friend of mine and a former colleague, really an all-star. It’s so wonderful to have you here, Banu. Thank you.

Julie Choi: Banu Nagasundaram is a machine learning product leader at Amazon Web Services where she owns the go-to-market strategy and execution for AWS Panorama, an edge computer vision appliance and service. Prior to AWS, Banu has spent over a decade in technology, building AI and high-performance computing products for data centers and low-power processors for mobile computing. Banu holds a Master’s in EECS from the University of Florida and an MBA from UC Berkeley’s Haas School of Business. Let’s all welcome Banu.

Banu Nagasundaram: Thank you, Julie and team for having me here. I’m super excited to be here. One difference from the other speakers is this is not my area of expertise, the title of the talk. It is something I’m trying to do better at that I wanted to share with you.

mosaicml girl geek dinner banu nagasundaram speaking aws

AWS Product Manager Banu Nagasundaram speaks about seeking the bigger picture as a ML product leader at MosaicML Girl Geek Dinner. (Watch on YouTube)

Banu Nagasundaram: I’m trying to seek the bigger picture at work. I’m a Product Manager and I’m trying to see why companies do what they do and learn more in that process. What I want you to take away from this is how you can also seek the big picture in your roles that you do either as engineering or product leaders.

Banu Nagasundaram: With that, I wanted to share a little bit about the companies that I work with on a daily basis. These are concerns who use machine learning and AI and work with AWS to implement the services in production. This is different from the research that we spoke about.

Banu Nagasundaram: These companies are looking at getting value out of these systems that they put in place, of course, based on the research, but taking into production. What I implore you to think about is put on a hat of a CTO or a CIO in each of these companies and think about how and why you would make the investment decisions in machine learning and AI.

Banu Nagasundaram: For example, I work with healthcare and life sciences team. I learned a lot from the drug discovery talk earlier from Tiffany here, but I do work with healthcare and life sciences team to understand how they can take the vast amounts of health data that they have to translate into patient information that they can use to serve patients better.

Banu Nagasundaram: They use multiple services to personalize, to extract value from the text data, a lot of unstructured data that they have. That’s one category of customers.

Banu Nagasundaram: The second type of customers that I work with include industrial and manufacturing. The key component that they’re trying to improve is productivity and also optimizing their manufacturing throughput.

Banu Nagasundaram: The questions that they ask and they seek to improve include automating visual inspection. How can I improve the product quality across my manufacturing sites? I have thousands of sites in the US. I scale globally. How can I implement this process not only in one site but uniformly across those thousands of sites to achieve something like predictive maintenance on the tools, improve uptime of their equipment, etc?

Banu Nagasundaram: Third set of customers we work with include financial services. They are looking at data to improve or reduce the risk in the decisions that they make. They’re trying to target customer segments better so they can understand underserved populations but lower the risk in making those products and offerings that they want to do.

Banu Nagasundaram: They also look into fraud detection and many applications around financial services. Retail, this is one I work closely with because I work in the computer vision team right now. And retail is trying to use the insights from computer vision products to see how they can reduce stockouts, which is basically when you go to the store, is the product available? Can they sell it to you? How can they manage inventory? Can they keep track of the count or the number of people entering the store?

Banu Nagasundaram: You may have heard about Amazon Go, for example, a store with just a walkout experience. A lot of retail companies are working with us to understand how they can use computer vision to build experiences like that.

Banu Nagasundaram: And it doesn’t stop at a retail store. Think about the operations officer, a centralized person who’s sitting and trying to analyze which region should I invest more on? Which region should I improve security? Should it be in North Carolina? Should it be in California? Or should it be in a whole different country? They’re trying to collect and gather insights across their stores regionally, nationally, internationally to make those decisions.

Banu Nagasundaram: Then there’s media and entertainment too, which we touched upon a little bit around recommendation systems and personalization. Here we work with customers who are looking to improve monetization, who are looking to create differentiation in the marketplace, in the very highly competitive media and content marketplace, through those recommendation systems and personalization that they have.

Banu Nagasundaram: Across all of these customers, the core task as a product manager that I work on is understanding their requirements and then translating it into product features so that they are served better. But what I learned in the process is that it’s so much more in decision-making than just understanding about features or product requirements.

Banu Nagasundaram: It’s about what enables these CTOs to make those decisions is value creation. How can they use all of these AI ML systems to realize value from the systems that they put in place? One simple way to think about what value creation is, is that it’s an aggregation of data, analytics, and IT that brings the machine learning together. But there’s a second part to it, which includes people and processes.

Banu Nagasundaram: What I mean by that is all of this analytics and machine learning and data can help them understand something, but they still have to lean on people to analyze the data that they gather, make improvements in the process in order to recognize and create value. And that’s the workflow for decision-making across all of these companies.

Banu Nagasundaram: We can look at this as two buckets, one in data analytics and IT, and the second one as people and processes. For the computer vision product that I own, I wanted to talk to you about the value chain for the first part, which is the data analytics and IT portion.

Banu Nagasundaram: This might look a complex set of boxes. It is. Once I finish the pitch here, if this excites you, I’m hiring too, growing my team. Hopefully, I do a good job in explaining this value chain.

Banu Nagasundaram: I started my career in the bottom left, by the way, in silicon processor design. Pretty much in the bottom row, trying to understand how silicon design is then aggregated into components and how distributors sell those components to OEMs who are equipment manufacturers, and how from those equipment manufacturers, the equipments reach the consumers, which is through those equipment distributors.

Banu Nagasundaram: My product is currently both an appliance and a service. I do start from the silicon side, working with partners, I can give examples like Nvidia on the silicon side, Lenovo on the OEM side. And then once you have these equipment distributors selling these devices or equipment I should say to infrastructure providers, one of the examples of infrastructure providers is cloud service providers, but it can also be on-prem equipment providers.

Banu Nagasundaram: You then go to the infrastructure providers, which is the second row from the bottom, and these infrastructure providers then build the tools and frameworks either themselves or through the partner ecosystem. Those tools and frameworks essentially put in place to make efficient use of that underlying infrastructure. It’s a motivation for these infrastructure providers to offer the best tool and frameworks so that you can gain that value out of that underlying infrastructure.

Banu Nagasundaram: Then comes the ML services. You have the overall MLOps flow is what would fit in this bucket. Once you have the tools and the frameworks, how can all of these pieces get grouped together to build a robust system that can scale in production? This goes from data annotation, labeling, training to predictions, to model monitoring. “How can you maintain this when it is in production” is a key question for these decision-makers.

Banu Nagasundaram: Then comes the AI services that are built on top of these ML services. AI services can either be services offered by CSPs or it can be services offered by startups or companies who are trying to build the microservices or services on a specific use case. This is where the customer use case comes into play, where you have that specific use case. In case of computer vision, you can think of services like Rekognition or Lookout for Vision. Those are two examples that AWS offers for computer vision services.

Banu Nagasundaram: Then comes the final layer. This is the layer that the customers that I refer to, the CTOs, work closely with. The independent software vendors or ISVs are companies that build these software solutions, aggregating everything that I spoke to you about, but building the software components of it. But the software component by itself is not going to function in a customer’s premise. For the customer to realize value, the solution, the software solution for their use case has to integrate with their existing system. That’s where system integrators come into play.

Banu Nagasundaram: For example, Deloitte. You can think of Deloitte, Accenture, et cetera are system integrators who bring this whole puzzle together for customers, build that reference architecture for that solution. And they’re like, okay, so now we have this architecture in place. Then they bring in the value-added resellers, who are companies like Convergent or Stanley, for example, who take this entire system that’s put in place and roll it into individual sites.

Banu Nagasundaram: This is where it reaches the scale of thousands of sites. Once a solution is put together, proven in a pilot or a production pilot or a proof of concept, when you go to production, across globally, across cities, across countries, the value-added resellers roll the system into place in the customer’s site. But it doesn’t end there. There need to be managed service providers who can offer service contracts to maintain the system in place once it is on a customer’s premise.

Banu Nagasundaram: All of these building blocks in the value chain is what makes ML, machine learning, in production, intangible for a customer to realize value. It is a big journey. And this is the team I’m building who will work with individual partners across this value chain. And if this is something that excites you, we can talk after. In that value chain, we started with value creation. What is it that companies are looking for? We saw the value chain, but the value realization, let’s say the companies went through this process, put this whole system in place.

Banu Nagasundaram: What is it that actually helps them realize that value? When we think about data analytics in IT, many of the ML practitioners that I talk to, talk to me about the output, the visualization, the dashboards, the histograms for decision making, but it doesn’t stop there. That is not sufficient for these companies. There has to be people in those companies who take this output and actually work to achieve an outcome. The outcome is increase in productivity, increasing in throughput. It’s like, if I can know my demand better as a retail store, or if I can forecast the demand better, I can do so much better in my business.

Banu Nagasundaram: That is an outcome that I’m looking for from this data output that I’m getting from the machine learning models. But is that outcome sufficient? No, the outcome is like one place, one time, you are able to visualize that outcome, but you have to scale that outcome globally in order to achieve the impact. The impact that businesses look for is that you have to either increase revenue, reduce costs, reduce their risk, improve their sustainability, or create a competitive advantage. This is what the ML journey in production looks like. While I walked you through incrementally from the beginning on what it would take customers to get there, customers don’t work through individual steps, reach there and see what’s the next step.

Banu Nagasundaram: They actually have to make assumptions along the way and then understand what the impact might be and this is an AWS term that I’m going to throw and work backwards, which is understand what is it that you want and then build out what is it that you can, what you need to do towards achieving that end goal. That’s the big picture that I want to leave you with. In this whole ecosystem of having machine learning in production, can you think big on behalf of the customer, can you seek the big picture that the customer is looking for? If you’re working on a feature, what is the end state of that feature? What is the end state of that business? Who is actually your end customer? Your end customer may not be the team that you’re immediately working with and who is the decision maker for that overall flow?

Banu Nagasundaram: One of the simple frameworks, this might sound silly. It’s super simple, is to ask five whys, which is in a customer discovery or any feature that you’re building. Just ask, why is that outcome important? Why is that output important? How is it going to help? Why is it going to be something that helps the customer? Why is it needed tomorrow? All of these questions is just going to help drive a little bit more clarity into the bigger picture and motivations for the customer on how they make investment decisions and choices in your particular products and features that you’re building. And that’s it. I want to leave you with one fair warning. If you try that five why’s with your partner, that’s on your own.

Julie Choi: All right. Thank you. Banu, that was great. Thank you so much. It is my honor to introduce Lamya Alaoui, a dear friend of mine and I’m so thankful to you, Lamya, for agreeing to give the closing talk of the night. Lamya is currently Chief People Officer at Hala Systems, and she has been committed throughout her career to supporting organizations as they shift behaviors to align their talent strategies with their business objectives. Her corporate background includes over 15 years of experience in talent acquisition and management, where she has had the opportunity to build teams for companies such as Bertelsmann, Orange, Groupon, Google and Microsoft. Her work experiences span North America, Europe, Asia, Middle East, and North Africa. Let us welcome Lamya Alaoui.

mosaicml girl geek dinner lamya alaoui speaking hala systems

Hala Systems Director of People Ops Lamya Alaoui talks about 10 lessons learned from building high performance diverse teams at MosaicML Girl Geek Dinner. (Watch on YouTube)

Lamya Alaoui: Thank you so much, Julie. Hi everyone. Thank you. I don’t see Sarah here, but I want to thank her personally because she has been so patient and Angie as well for inviting me here. How is everyone feeling today? This is the non-tech talk. This is the people talk. A quick background before we get into it. At Hala System we develop early warning systems in war zones. Not the type of things that you broadcast usually when you are in this type of settings, but we’re hiring as well, especially for our AI team. And Julie announced my promotion that no one else knows about, even in the company so, thank you. That stays here, please, in this room, until next all hands, on Wednesday. With that being said, a little bit of background about me. I’m Moroccan.

Lamya Alaoui: I moved to the US about 10 years ago. I will ask a lot of grace because my brain is wired in French. Sometimes there will be French words that will come out from my mouth so please be graceful about it if you can. With that being said, this is one of my favorite things because as a Moroccan who’s half Muslim, half Jewish, went to Catholic school, I thought I got it covered, but I was the lady the first time I landed in Germany because I was not aware that you’re not supposed to kiss people to greet them. But this is one of my favorite pictures to show and talks or even in workshops, because I’m pretty sure it happened to all of us one way or another. We show up and we think that we got it right. It’s what we’re familiar with and actually it’s not what the other person is expected.

Lamya Alaoui: Throughout my career, I have built teams in very, very different countries, different cultures. At Hala we have over 17 nationalities. Altogether we speak 25 languages. It makes it interesting for the meetings, believe me, where you have side conversational Slack in a whole different language, but you still need to deliver officially in English. The translation is super weird sometimes, but those are anecdotes for another time. That leads me to one of the lessons that I learned very early on. It seems obvious that when you’re building teams and you want them to perform, one might assume you want to know your mission and your values, but it doesn’t happen as often as one might think. The mission is basically your GPS. You want people to rally behind a common goal.

Lamya Alaoui: You want them to believe in that and it’s also what’s in it for them when they’re working for a mission. That means that it needs to be aligned with their own values or their belief system. SVlues start very early on. We all have seen or worked for companies where the values are stated. How many of you when you joined the company saw values listed on the website or have been talked to? How many of you were explained to what are the associated behaviors and expectations when it comes to those values? Oh, we have less people. This is one of the first lessons that I always recommend to people to kind of follow. I learned the hard way, by the way, is that know your mission, state your values. Values should come from the leadership of course, but also account for it in the hiring process, which we will get a little bit into it, but be very clear about what are your expectations are when it comes to values.

Lamya Alaoui: If I’m thinking about transparency, for me, it means having information that I need to do my job, but some other people, they think that they need transparency. They need to have access to everything. We all have someone like that in our companies. Don’t we? Setting those clear expectations and associated behaviors are very important. And mission is like literally your North Star. This is why it’s really important. The second one is seek first to understand. This is from Stephen Covey. There is a second part to it which is, then to be understood. When you are building diverse teams, we all come from different backgrounds, we have different understandings, often we speak different languages. We can also come from different cultures, high context or low context and if we don’t have mutual understanding we cannot succeed because then everyone is convinced that they’re right, that their approach is again the one that everyone needs to follow, but that kind of hinders teamwork, which is actually essential to high performing teams.

Lamya Alaoui: Listen. We all think that we listen pretty well, but we actually don’t because as human beings we have been trained, for the last few centuries, to listen to answer. We’re almost never listening for the sake of just listening and there is amazing research about that, that I will be more than glad to share. In working in very diverse teams or diverse teams in general, listen, help, create understanding, respect, which is a very important when it comes to having high performance teams as well. How many of you would say that they listen really, really, really well. Okay. We have four people in the whole room. That gives you something to think about. That’s another, and I’m a good listener, but still a lot of work to do with that.

Lamya Alaoui: Acknowledge that you will face cultural differences. When you join a new team, we all want to get along. Again, its human nature. We want to belong somewhere. We’re super excited. It’s a new job. We just went through the hiring process. We landed the job and then you show up in that meeting or you’re meeting your team and sometimes you see people that you have nothing in common with, and you still try to ignore that, which is again, human nature. Beautiful thing. Just as a manager, as a team member, be prepared that yes, we will have some challenges. We’re not seeing things the same way. One of the best examples that I always give here is like, I’m from what would qualify as a high context culture, which is there is no explicit direct messaging.

Lamya Alaoui: Basically I can be in my office and say, hey, I have a lot of work and I’m hungry. My expectation is that my coworker will just get it and go get me some food. Now, imagine if I’m having someone from a low context culture. In their head, they’re like, I mean, you’re going to go get your food when you’re done. That creates a fracture that it’s not even intentional because there will be resentment from my end. Why didn’t he get it? Having those conversations upfront is quite helpful because then people know again, what to expect. Assess the degree of interactions. When teams are being built, most of the time what happens is no one is thinking about how often the team members will have to work together, at what intensity.

Lamya Alaoui: Before starting to build the teams or adding members to your teams, just make sure that, hey, how often do they have to have, I don’t know, sprints. If it’s once a month, eh, you might want to have people that are pretty much aligned having somehow the same thought process and we’ll get to that in a little bit. If they’re not interacting a lot, you have a lot more room to assemble your team. Always, there are three degrees of interactions. We have low interdependency, which is people are doing their things on their own and sometimes it’s like going in a [inaudible]. Medium, which is a hybrid and then high interdependency, which the output from someone becomes the input for someone else.

Lamya Alaoui: Therefore, things need to be structured in a very certain way in making sure that people get along and understand, again, what’s expected from them. Communicate, communicate, and communicate and when you think you are done, communicate some more. One of the things about communication is that we all think that it happened. Again, the question, oh, disclosure, I ask a lot of questions. I mean, I used to be a recruiter. It comes with the territory basically. How many of you thought in a meeting that you were crystal clear and then two days later you discover that you were not.

Lamya Alaoui: When it comes to communication in teams, again, thank you, Slack, everyone in meetings, or usually where everyone is doing a lot of things at the same time. You’re talking, you think that you got your point across. Always ask a question at the end. What do you think? What are your takeaways? Just to make sure that people are on the same page and teams, high performing one’s, one of the best practices is having action items assigned, even if it’s an informal meeting or if a conversation happened on Slack, on, I don’t know, any other platforms. Even sometimes now, oh, we have also Asana now people add us to Asana and they think that, hey, we’re good.

Lamya Alaoui: No, make sure that the person actually went in, is understanding what you’re waiting for or expecting. Diversity and leadership. Clearly icons still have a long way to go in terms of diversity, but again, high performance teams when it comes to diversity need to see that diversity reflected in the leadership and in the management. Research shows that usually entry level management, there are sometimes good numbers, but as soon as we go to director, VP and above underrepresented groups tend to become less visible or they’re non-existent and C-suites and VP, EVP, SVP type of roles.

Lamya Alaoui: It’s also very important that the leadership reflects the diversity that the company is striving for. Again, an obvious one, hire the right people. This one is very dear to my heart because as I said, I’m a recruiter. Again, we’re only humans. We tend to hire people who are like us, who think like us, who share our values. Anyone wants to venture what happens when you hire people who are like you and your team and you’re building a team of clones, basically from a cognitive perspective. Anyone? Yes.

Shika: You wouldn’t get to learn something different from what you already know.

Lamya Alaoui: Yes. Anyone? Thank you so much. What’s your name?

Shika: Shika.

Lamya Alaoui: Shika. Thank you, Shika for volunteering. When you build a team of clones and this is a tech talk after all, how do you think that you can innovate? How do you think you can perform if everyone is thinking the same way and not challenging the ideas that are being discussed? Chances are very, very, very slim.

Lamya Alaoui: When you’re hiring the right people for your teams you have to look at three things. The first one is the values, because that’s, again, what will be the foundation of the team that you’re building. The second one is what’s missing from your team. An ideal team has five type of people. You have a theorist, you have a strategist, you have someone who is an analyst, you have a manager and you have an implementer. When we hire our teams usually we hire in desperation because you know, you got to deliver on that project.

Lamya Alaoui: No one is thinking about the composition of the team in itself. In tech, my experience showed we have a lot of theorists and strategists and a lot of analysts, sometimes zero implementer and zero managers. Things don’t get done somehow. Always look at what’s missing and finally, the third part is people who are willing to learn, who have a curious mindset and are eager to grow, because this is how actually innovation happens. People who are not afraid basically of questioning the status quo.

Lamya Alaoui: The other lesson is constantly scan for misunderstandings and ways to clarify. Again, here I will go to the low context, high context type of thing so, let me give you an example. I think it might be easier to make my point that way and I will ask a question at the end. In high context cultures or in some cultures when you ask questions, no, is not an acceptable answer. You have to say yes and some others you have to say no three times before saying yes.

Lamya Alaoui: Imagine when you have people from different cultures in the same room and someone is asking a close question that requires a yes or a no. What are the chances, based on what I said, that you will get an accurate answer to your question? 50%, 60%, 20%, a hundred percent. For this always ask open ended questions to give people space to answer without having to break their own boundaries and to make sure that you are getting clarity in the answers. How many of you were in meetings where you felt that you were misunderstood when you were speaking? What did you do? Anyone wants to volunteer again? When you felt that you were misunderstood? Yes.

Angela: Have you summarized in a written notification? I gave action items to the people I wanted clarification from that helped with the misunderstandings.

Lamya Alaoui: Thank you, Angela. One of the things that is really helpful and again, best practice for high performing teams, always summarize and put things in writing and this setting English is the working language. We do have a lot of people, I know in my team, I have a lot of non-native speakers. I’m not a native speaker so to make sure that the message gets across and that it’s clear it’s always in writing, always, which we tend not to do because everyone is working long hours or you have back to back meetings. Then at the end of the day you just want to finally get to do your work and you never summarize in writing what happened in those meetings.

Lamya Alaoui: Finally, the structure. Build a structure that is around diverse teams. When we’re usually building teams we have, we’re looking at what now is referred to as visible diversity, which can be ethnicity, religion, gender and many other things, but we’re never looking at the cognitive one that also has a heavy part, especially in tech. Those are recommendation that I’ve always lived by every time that I join a new company. We do a reintroduction and relaunch of the core values. We redefine them. We create, and this is like a company work, it’s not just someone in their corner doing it, but it’s a collaborative work setting what are the core values, what are the expected behaviors and the ones associated with each one of the values, and then it’s communicated throughout the whole company. When it comes to performance review it needs to be designed with underrepresented communities in mind.

Lamya Alaoui: Performance reviews, if you look at the history of how it was designed, it was designed for a very specific community or population. It doesn’t fit other people. Just leave it at that. Work on creating a performance review management system that is more inclusive. Yeah. Targeted networking. Again, this ties to the diversity and leadership. One of the things, again, hiring in despair. When those roles are open, people tend to hire fast from networks that they know because they don’t have networks that are already established. This is something that is quite helpful. I don’t think there are any people in the HR realm tonight, but all of you can be ambassadors for something like this to be established in your companies. And again, clear expectations around the engagement and the roles. What are the pathways to promotions? Again, if you look at internal promotions or internal applications, woman tend for, this is just an example, women tend not to apply to internal openings when it comes to higher positions in the company.

Lamya Alaoui: Sometimes just because they don’t check all the boxes so the work starts before that, where we actually need to make sure that the job descriptions or the job openings are reflecting and are being thoughtful about underrepresented groups. Finally, this is one of my favorite things, Kaizen. There is always room for improvement. Performance and diversity are a very long journey. Each company is at a different stage or different phase. Just ask questions, be kind because not a lot of people think about it when it comes to those things and be as curious as possible because at the end of the day the only way to know other people is to ask questions. Thank you so much everyone.

Julie Choi: I want to thank all of our speakers tonight and for all of you, for being such an incredible audience. We have people who have been watching these talks outside in the overflow and I’m told that, I think it might be better if we just all go out and just mingle. Get some fresh air after all this time is for us to connect and network. Why don’t we continue the conversation outside? Thank you everybody.

mosaicml girl geek dinner ukranian original borsch tshirt

This site reliability engineer discusses Ukranian borscht or machine learning, or both, at MosaicML Girl Geek Dinner.

mosaicml girl geek dinner speakers tiffany williams banu nagasundaram laura florescu julie choi lamya alaoui shelby heinecke angela jiang angie chang amy zhang

MosaicML Girl Geek Dinner speakers after the event: Tiffany Williams, Banu Nagasundaram, Laura Florescu, Julie Choi, Lamya Alaoui, Shelby Heinecke, Angela Jiang, Angie Chang, and Amy Zhang.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Webflow Girl Geek Dinner – Lightning Talks & Panel Q&A! (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

Transcript of Webflow Girl Geek Dinner – Lightning Talks & Panel Q&A:

Angie Chang: Hi! Welcome to Girl Geek Dinner virtually in a pandemic. This is the Webflow Girl Geek Dinner 2021. My name’s Angie Chang and I’m the founder and CEO of Girl Geek X. Hi, Sukrutha! I want you to introduce yourself.

Sukrutha Bhadouria: Hi! I’m Sukrutha and yes, Angie’s partner in crime whenever I can be a good partner to her. Welcome everyone to the Webflow Girl Geek Dinner!

Angie Chang: I wanted to like really quickly say a bit about what Girl Geek is and kind of go back to the beginning where I started Girl Geek Dinners in like over a decade ago, where I was just really excited to put women on stage at different companies in the San Francisco Bay Area. Now that we are doing this virtually, we can have people join us from around the world. We have people of all genders coming from all cities around California normally. Now we get to have people from all over the world! Thank you so much for coming and if you want to tell us where you’re joining us from in the chat, we would love to see where you’re dialed in from. We not only do Girl Geek Dinners in-person and virtually…

Angie Chang: We also do an annual virtual conference every International Women’s Day – March 8th usually – we are doing our International Women’s Day conference called Elevate. That’s an all day event with lots of speakers and sponsors talking about what they’re working on. It’s like a really all day Girl Geek Dinner, where you learn so much and then you got to meet other women. And you can also… There’s a call for speakers so if you haven’t that email, you should look at the girlgeek.io website and there is a link to apply so you can become a speaker.

Angie Chang: We had at least three speakers, I believe last year, who came in through the submission process. I encourage you to think about what your expertise is and apply to speak. We are reviewing everything in about a month. What else is there? Am I forgetting anything else?

Sukrutha Bhadouria: No, but I did want to say given the Elevate conference is our favorite time of year, we absolutely would love you to apply right away. Don’t overthink it, just do it. There’s a lot of times that we don’t see enough female speakers at tech conferences or conferences in general, just because of unconscious bias that we put on ourselves.

Sukrutha Bhadouria: I feel like sometimes we think it has to be perfect when it doesn’t, it just needs to be authentic. So please go ahead and apply.

Angie Chang: I think it’s about time to bring on our first speaker. Arquay Harris is going to be telling us a bit about Webflow, I’m really excited to hear her speak about the company.

Angie Chang: She is the VP of Engineering at Webflow and previously we knew her in her previous life as a Slack Director of Engineering and you might recognize her we kind of always say, oh, my God, if someone asks us like, “Did someone get hired from a Girl Geek Dinner?” We’re like, “Yes, Arquay. She got hired at the Slack Girl Geek Dinner and has worked there for about four years, I believe. And yeah, I’m just super happy that you’re now at Webflow. You’re going to tell us all about it. I heard so much about no-code and the growing company, so yeah, I’ll pass the baton to you. Welcome, Arquay.

Arquay Harris: Excellent. I hope like I’m not the only person who’s got hired. I think it’s like I’m in good company of all the people. First we’re just going to learn almost everything that you wanted to know about Arquay – consider it your Arquay 101, if you will. But before I get into it…

Arquay Harris: I’ve been going to Girl Geek for a very long time. I’ve considered myself an OG kind of person. I looked at my email and I thought like, what is the earliest Girl Geek that I ever went to? And I found this ticket from 2009 and it was some company called LOLapps. Does anyone even know what LOLapps is? I literally have no idea what that is. I had to go on Wikipedia, I think it’s like a Facebook game or something and I don’t know if it exists but it’s the funniest thing, and I know I went to one before 2009, but this is the oldest one I could find in Gmail and all my Yahoo mail has been deleted. That was pretty funny.

Arquay Harris: I really support the organization. I really love their mission and what they’re trying to do. I really am sincere supporter of Girl Geek. Really quick pronounced R-kway, really sorry to disappoint, it’s nothing exotic like an African princess or anything like that. My parents really just like SEO. I say this a lot and people don’t believe me, but it actually is the truth that’s why I’m named Arquay.

Arquay Harris: I thought a really good introduction would be to talk about my kind of traditional, non-traditional background. Growing up I really loved math and as you can see, I one day dreamed of visiting this island nation of Sohcahtoa, and so I was president of the math honor society, and I really loved math. I went to college to become a math teacher. It’s because I had pretty humble beginnings and I really believe that math is like the kind of great equalizer.

Arquay Harris: You can math and science your way out of poverty so to speak. I had an after-school job though that really introduced me to things like Photoshop and Illustrator. Even though I loved math, I noticed that I was most engaged when I was kind of doing this stuff. I transferred schools and I studied media arts and design, and I got into coding because I didn’t like this process of handing off my designs to someone else. I thought I’m really analytical like I learned to code, I will have this math background, I could learn to code, and so then I started with Flash and then PHP, later and then Python.

Arquay Harris: I later went on to grad school and I did more coding of fine art painting, and so the really interesting thing about me I would say is, even though I’m a developer and I very much consider myself an engineer, I actually have been MFA. I’m a formerly trained designer and it’s really served me well just in my career and in my life to be able to have these informed conversations about topography and color and understand what can be built.

Arquay Harris: I like to tell that story because people often say like, “Oh, I really want to get into coding. I really want to like do this technical thing. I don’t know if I can do it, you can do it. We can all do it. I think there’s no one true direction or path and everyone’s journey is different.

Arquay Harris: This is a good transition into what I do all day. I’ll tell you a little bit about what I do at Webflow. The thing that I really like about Webflow, especially if you hear my story is how Webflow is really invested in kind of democratizing this idea of creating things for the web, visual development platform.

Arquay Harris: Previously there were these gatekeepers, it was like you had to have a CS degree and it was only coders and only people who had like done a certain thing could really create these experiences for the web. I really identify with that mission as well as the fact that I think it really aligned with my design and kind of artsy background. It’s almost a perfect gig for me. I really dig it.

Arquay Harris: Really quickly, I’m sure you’re wondering VP of Engineering. What does this person even do all day? Well, I do this combination of what you might have heard called the Three Ps, which is people, processes, product. Those are the kind of main things that a VP of Engineering does, but process being like how you actually develop software. People is the mentoring piece. Product is the actual strategy of what we’re doing. And then I even have my kind of own framework where I really believe in advocating for the people on my team execution, which is kind of the bread and butter of what engineering managers do.

Arquay Harris: Then these business priorities, because it really matters like you could advocate all day and you could execute all day, but if it doesn’t align with the business priorities, then there’s probably an issue there. So I just wanted to give you a high level, an intro, setting you up for this talk, telling you a little bit about me and my story.

Arquay Harris: I’ll be here later asking questions in the Q&A, if you have any more questions about me or the product or Webflow in general, because we are hiring, we’ll be here talking to you about our open roles and all that stuff. I look forward to talking to all of you and you’re in for a great night. I’m a little bit biased, but really it’s going to be good. So yeah, I’m about to hand it off. Okay. There you go. That’s it. I highly recommend you put puppies in your presentation if you need to take a sip of water. Who’s up next?

Sukrutha Bhadouria: You’re so cool. Oh, my God. Yeah. You’re one of our favorite girl geeks ever, because every time we’re greatly entertained and amazed by what’s going on in your career. Right. So, before my daughter interrupts us Jiaona Zhang is next. Oh, my gosh, she’s an amazing, amazing, cool coster and VP of Product at Webflow, and she’s also an active angel investor or lecturer at Stanford University and of course created Reforge. Oh, my gosh welcome.

Jiaona Zhang: Thank you so much. I’m so excited to be here and that is so adorable hearing your daughter’s voice. All right. I do not have a puppy poster or segue while I drink water and pull up my slides so give me one second.

Sukrutha Bhadouria: All righty.

Jiaona Zhang: So excited to be here. I lead product at Webflow and just a little bit of background on me. I started my career actually in consulting and really wanted to not be advising and actually be on the other side of the table and truly operating. I started my career in product by being a product manager at a mobile gaming company. Definitely not something I thought I would do growing up, but it was a really great way for me to get my hands dirty and learn how to ship things. I spent time at Dropbox, at Airbnb, at WeWork, and then ultimately made it to Webflow.

Jiaona Zhang: I’m so excited similar to Arquay, in terms of being able to work on a mission statement that is really about empowering everyone to build. As someone who didn’t have a technical background, I was an econ major in school being able to create tools so that all of us, no matter what your degree was or what you’ve studied, you can build and you can actually build for the web. That’s just something that I think is so exciting and democratizing. I’m happy to talk about my background a little more later in the Q&A but today I actually wanted to share five lessons in product strategy.

Jiaona Zhang: First of all, product strategy is something that I think a lot of people scratch their heads out they’re like, “What is it exactly? Is it the company strategy? Is there something different like what’s a strategy? How do I know I have good strategy?” What I want to walk through today is what I’ve personally learned over my career in terms of what strategy is, and also how do you really go about bringing that to life and going through some examples there? I don’t believe in progressive disclosure so I’m going to go ahead and share the five lessons at a high level. Then we will go through each one and talk about in more detail

Jiaona Zhang: The five lessons that I’ve learned is first, the most innovative company start with a really bold mission, then this concept of your strategy, and we’ll talk about what? This thing, it really should look like a pyramid from your mission down to your strategy. The next thing I’ve really learned is that it’s really important to articulate real user value before business value. Lesson four, you do not have to do it yourself and then lesson five is you to bring your product tragedy to life. You actually design it into your organization. That’s one of the best ways to execute on it

Jiaona Zhang: Let’s go ahead and get started with each one and talk a little bit more about what each one means. So for the first one, the most innovative companies start with a bold statement. We’re going to do a little bit of interactive at Q&A and I sense that I’m going to ask people to put some stuff in chat. So, first of I’m curious if people know what Tesla’s mission statement is, if you do take a moment and just go ahead and type it into the chat, and we’ll see if anyone does, everyone has it.

Jiaona Zhang: Mark saying, just do it, just do it. It’s whatever you think a lot of people have no idea. Okay. That’s really interesting. Ruling the world. Okay. Boom Boom. Well, I know SpaceX is just to colonized Mars so I’m assuming that Tesla is also very grand. Okay. Caroline has, bingo, essentially to accelerate the world’s transition to sustainable energy. That’s literally exactly what their mission statement is, to accelerate the world’s transition to sustainable energy. And the reason why this is meaningful is because when you have a mission statement that is something like this, it enables you to really innovate towards this fourth star.

Jiaona Zhang: Imagine a world where Tesla’s mission statement was to build the best electric vehicle or to build the best luxury car or to build whatever else, right? Like it would really limit what they do. It would limit the concept of, hey, you know what we actually should do in order to achieve this mission of transition to world sustainable energy, we should have a vehicle, but we also maybe should have solar panels. We should also have charging stations. Like how do we get the world to be using sustainable energy much more? And so when you have something that is much broader than what you’re currently working on a mission that is inspiring and really ambitious, it actually creates that room for innovation and it really allows you to think bigger around how can I achieve that ultimate mission?

Jiaona Zhang: I’m curious if anyone knows what Airbnb’s mission is? How’s the world? Share. It’s really interesting. There’s a very big difference was how’s the world versus share. And I’ll talk a little bit about that later. To make locals share their experience. Okay. Another thing in the vein of sharing. Sharing, okay. Sharing community, something about being at home when you’re not at home. This is actually Airbnb’s mission, which is really to create a world where people can belong anywhere.

Jiaona Zhang: When you are anchored on belonging as your mission as your north star, you’re able to think about all the different ways, all the different aspects of a travel experience that you might want to improve in order to achieve belonging. For example, making that when you feel at home that is a part of belonging somewhere, making sure that you are connecting with locals Tara had the locals and sharing their experience.

Jiaona Zhang: It’s actually part of belonging, making sure you feel it’s part of a community that’s also part of belonging. And so again, when you have a mission statement and that’s where you anchor the company and everything that your company does, you are able to think much broader and open up much more room for, if we were to truly achieve this, what can we do? What are the products we can build? What are the programs we have? And so that actually brings me to Webflow since this is a Webflow and girl geek talk.

Jiaona Zhang: Here at Webflow our mission statement is to empower everyone to create for the web. On top of that, we also really care about making sure that everyone in our company are leading impactful and fulfilling lives while working on this mission statement. The reason why, so I’ll focus more on the first part, which is empowering everyone to create for the web.

Jiaona Zhang: The reason why this is really important and to start here as a part of the product strategy is it is something that we could be working on for the next 100 years. And we will continue to make progress towards this really ambitious mission, getting everyone to be able to engage in the act of creation.

Jiaona Zhang: When we do something like that it also gives us the room to think, okay, to achieve something as ambitious as this, what are the big things we need to do in the short and medium term to ultimately accomplish this journey? When you start with a mission as opposed to we’re going to build X, Y, Z, that’s our product strategy. When we start with a mission, you really get everyone at your company rallied against this is what it means to ultimately long term be successful. These are all the different ways we can innovate towards that, creating much more room for both depth and rep of what you can do as a company.

Jiaona Zhang: All right, the second lesson, it should look like a pyramid. And why say it, I really mean… We just talked about starting with your mission statement that flowing through to your product strategy should look like a pyramid. And so what does that actually look like when you break it down? The first thing is you have your mission statement. So I gave you a few examples. The Tesla example, the Airbnb example, Webflow example. From there, you actually can go and talk about your vision. So if your mission is, what are you ultimately trying to achieve? Your vision is in or we believe that if we do this, that is the best way to help us achieve that mission.

Jiaona Zhang: From there, you actually need to formulate company strategy. And when you have these north stars in place your company strategy will be a lot crisper and focused. And then from there comes your product strategy. And so you can see it’s actually this almost like it’s this nesting doll, this pyramid structure where everything kind of ladders up into your mission statement that we just talked about. So let’s go through a Webflow’s example. So again, this is our mission statement to empower everyone to create for the web. What is our vision? How do we really achieve that over the next five, 10, 20 years? We believe that the best way to achieve it is to build the world’s most powerful no-code development platform.

Jiaona Zhang: Every single word in this sentence actually means something really critical to the way we think about how we approach our vision and also our company strategy. So the first word that I’ll talk about is this concept of power. We believe that in order to empower everyone to create, especially all the different things that you’d want to create, we need to give you power. We need to not just give you a template, but real powerful tools that you can use to customize whatever it is you want to create. We fundamentally believe in no-code, which is instead of asking people to have to learn how to code or all of these different things in order to create, we want to make it much more visual, much more intuitive and give you that abstraction layer.

Jiaona Zhang: And finally, we really believe in order for us to actually achieve our mission, which is to empower everyone to create really anything. We can’t do really do it alone and we have to build this platform. I talk more about that as part of lesson four, but platformization and really creating that platform is a big piece of what we believe we need to do. And as a result, it ladders into our company’s strategy, right? So if this is our vision, how do we then pull that into our company strategy. Okay, what we need to do is ultimately to lean into the power to really, really enable this no-code revolution, and then ultimately create a platform.

Jiaona Zhang: From there, what you actually build, that’s the product strategy and it basically hangs off of the company strategy, which hangs off of the vision, which hangs off of the mission. And so our product strategy, what we actually build what are the features of this no-code platform? What are the ways we can actually bring that power to life? How do we make sure our platform is extensible that comes from our company strategy?

Jiaona Zhang: All right. That was awesome too. It should look like a pyramid and now you kind of have a sense of what is, and we’re going to move on to lesson three, which is, it’s really critical to articulate your user value before your business value. Imagine a world where Tesla’s strategy was, we’re going to build the best electrical vehicle. We’re going to beat Prius, we’re going to beat whatever X vehicle that other brand has. That’s not super inspiring and it also doesn’t ultimately create the best product out there. Imagine if Airbnb was like “We’re going to be booking.com. We’re going to be even better than booking plus Expedia plus Vrbo plus everything combined. That’s also not something that really gets that. This is what our users need, and this is therefore what we have. We can build to fulfill their needs.

Jiaona Zhang: Finally, going back to Webflow, imagine a world where… What we anchored on was we got to beat WordPress, really. We got to beat X thing that’s out there that people are using today. It really limits what you’re able to do, and it limits your innovation because really the most creative, innovative companies are building something that is like leagues beyond what is out there in the market today. Instead here at Webflow, we really think about the user value first. We think about something like everyone who watches Pixar movies, you see this just richness of animation and just what you can create on the screen that was all done via software so that you don’t have to literally create every single person or molecule or snow drop, or Anna’s like dress pattern, right?

Jiaona Zhang: Like you actually have software to scale that. And so that is really anchoring on the user value of how do build something that’s beautiful and delightful for people to watch. Then how do I make it so that every single person who’s, for example, working at Pixar can do it scalably and do it in a way that you can actually create a beautiful movie in X year’s time. And so that’s the same way we think about Webflow which is the value of really getting people to be able to create something that is powerful and ultimately what they’re looking to do. And so the way we think about the user value is we want to give people the building blocks. And here’s an example of some of the building blocks we’ve had in the past. Just illustrated like as an analogy. And you take these building blocks and you actually put them together and create whatever it is that you are looking to do.

Jiaona Zhang: I need something really custom on the UI side and then I need to add our data layer. I need to add some stuff around the CMS. I need to add a storefront. All of these different things I need to add in order to bring my idea to life, we’re giving you the tools to do that. And when you anchor on user value, you can actually see where that can go. You can think of a world where… Today you can build these particular structures, but what if in the future you can actually build a house and a really elaborate house at that.

Jiaona Zhang: If that’s our anchoring around is the user value that we want to generate for our users to be able to go from literally having to code, to being able to put things together and build some of these like really nifty- like planes and trains. Then one day to being able to build everything up to something like this house, this beautiful tailored, polished house. Everything up to something like this house, this beautiful tailored Polish house. If that’s what we believe in, and that’s the user value, then that is really our guiding principle, as opposed to chasing down features with competitors that might ultimately not be the thing we want to benchmark against. It usually isn’t a thing one at benchmark against, because competing against again, for example, WordPress is not going to get us the type of innovation and type of product unlock that we want, as opposed to saying, “Hey, if we want to be able to enable people build something like this, this beautiful house, what would we need to do enable to… in order to enable that?”

Jiaona Zhang: All right, lesson four, which is you don’t have to do it yourself. And I think this is a big lesson for a lot of companies where they have a very ambitious mission and vision, and you look at yourself and you’re like, “Hey, we’re a startup. How are we going to achieve that?” The answer is, you focus on a very critical core and then extending it and allowing your community help you get there. And so we really think about this in terms of an ecosystem.

Jiaona Zhang: What are the native capabilities that are most important to Webflow? And then how do we actually partner, whether it’s with other companies or with a whole community of developers to enable that long tail of use cases. And what’s really interesting, and this actually is almost bringing up lesson three in sharp contrast again, is when you do this, when you really think about the user value first, you then automatically unlock the business value. And so in this case, when you think about the user value of, hey, in order to get that beautiful house to be something that people can build we need to have a partnership with our entire community. That’s user value. You unlock the business value naturally, which is in this case, when you have a community, you really create this wonderful moat against your product, where it’s a lot harder to displace you because so much of your community has these integrations and are deeply embedded in what you provide. And so it actually results in business value naturally when you first focus on user value.

Jiaona Zhang: The last lesson is to design your product strategy into your organization. This is the best way to bring your strategy to life and ensure that you are executing on it. The reason for this is because as you grow as a company, even if you’re beyond just a very small group of people that can just quickly slack each other at all times, it’s really difficult for anyone outside the people working closest to the problem to really understand the best way to solve that problem. So when you design your org in a way that reflects your product strategy, that reflects the type of investments that you want to make in the user value that you want to unlock, you actually empower the people who are working closest to the problem and on the product to make the right decisions for you.

Jiaona Zhang: Here at Webflow, we’ve really mapped our engineering product and design structure to the things that we really believe we need to unlock. For example, you’ll see on here capabilities. How do we build the best first party capabilities for our creators? Then how do we also unlock this ecosystem to extend those use cases? How do we then make sure that anyone working on our community, or sorry, working with our product, they are able to be successful, whether they’re collaborating with each other or they’re going into larger and larger companies that need very… Like much more specific workflows? And then from a growth perspective, end-to-end, segment-by-segment thinking about that life cycle. And then last, but definitely not least, there are the very important foundational investments, infrastructure investments that we need to make to make sure that everything is able to come to life, and that these things are interoperable and connected.

Jiaona Zhang: All right. I know I am pushing on my time. With that, thank you so much for listening. Those were the five lessons that I personally learned around product strategy that I hold near and dear to my heart, and hopefully that can be helpful to your respective companies.

Angie Chang: Thank you, JZ. That was really an excellent product strategy talk. Thank you so much for sharing. I’m going to just really quickly remind everyone that we are taking questions for the speakers in the Q&A. So if you go below, there’s a little Q&A button that you should be seeing, and you can ask your question there, or you can ask it in the chat, and we’ll copy it over.

Angie Chang: Olena is a tech lead and staff software engineer at Webflow, and she loves react function programming and non-fiction books, and is currently focusing on creating value between engineering and product. Welcome Olena.

Olena Sovyn: Hi, I am Olena. I’m very happy to be, to be here today, and as an engineer, I would like to talk a little bit about how we can make code reviews even better experience for everyone. A little bit about myself before we start. So I’m Ukrainian who lives currently in London. For last nine years, I am working in software industry, and half of the time I am with Webflow. Currently I am a tech lead and staff software engineer at Webflow, and during my last four and a half years, I did more than a thousand blue requests and performed more than a thousand code reviews.

Olena Sovyn: Let’s get back to our topic for today. Let’s talk a little bit what we can do better in our code reviewers. First of all, why I choose actually to talk about code reviews? Why doing bad jobs with code reviews actually matter so much. I choose to talk today about this because I believe this is a unique opportunity as it is a win-win situation for a company, for you, and for your teammates.

Olena Sovyn: When we have code reviews, we have that unique process that can empower both your company, your teammates, and yourself, for you to develop your career, and for your teammates to develop theirs, and at the same time to company to sustainably grow. So typical cycle for code request, look like something like this. Request is ready for a code review, then code review is happening. And then you might think that today at my cloud will be concentrated mostly on this code review, but actually interesting thing with code review can start even before code review itself. And one of the things can be self review.

Olena Sovyn: What is self review? Self review is one full request also is going to his own change, his or her or them, on changes and leaving useful comments for code review body. For example, what these comments can be about. They can provide information why these changes were added to the code base, or they can provide information what specifically code review body should look for the most important part of the change.

Olena Sovyn: What else can happen before the code review and is happening actually? Every time when we are entering code review process? We are choosing code review body, and let’s look how typically this process look like. For example, we have a team with three people, Rose, Mark, and Boris. Rose is very experienced engineer. She is with a company for a very long time. Mark is with a company also for a long time, but he only recently switched to become an engineer. And Boris just joined a few weeks ago, but have been in the industry for a very long time.

Olena Sovyn: Whom would you choose to be your code review body? Looks like Rose is an obvious choice, but actually if you look wider on the code review process, we might want to choose Mark or Boris. Why? Because code review can be not always a place where other engineers provide. You feel bad, but this can be also a place where other engineers can learn from you and from the changes that you introduced to the code base. And if, for example, code review body would be someone like Boris, they can bring fresh perspective on the changes that you introduce into the code base.

Olena Sovyn: You can specifically enforce and empower them to see code review as aligning process, by reaching to them directly and asking them to specifically ask your questions in the code reviews.

Olena Sovyn: Use code reviews as a way to share a general and domain specific knowledge. So we talked a little bit about choosing code review body. What is happening next? I’m calling a magic moment in code review process. What is happening next? Next is actually code review body reading code for the first time. Why I’m calling this a magic moment? Be with me and listen carefully.

Olena Sovyn: Reading this code for the first time is something that you never will be able to do again. You can read this code for the second time, for the third time, for like end time, but never for the first time. Why is this such a unique opportunity and such a unique signal? Because this is a way where you can really evaluate. Is this code readable? Is this code obvious? Will be it easy to work with this code? And my advice for you, how to better capture this signal of following.

Olena Sovyn: How to read code for the first time. Take notes. If you are doing code reviews with GitHub, you actually can make a draft of the comments in the code review process. You don’t need to share them all in code review you when you submitted, but you can capture this your first thoughts when you first saw this code. And at this moment, what I am advising to pay attention to. Ask yourself a question. How easy code is to understand, do all variables make sense? Does code organization make sense? Is it obvious actually what code is doing? And remember that this first impression is your invaluable signal.

Olena Sovyn: After our first read, let’s deep dive in code review process itself. In many cases, code review process might end up being like request for change. Or if everything was straightforward, this can be like, look good to me and look good to marriage, but actually code review can be a great about sharing feedback, about changes that one engineer want to introduce to the code base, share this structure from other engineers. This feedback can be like anything. It can be that code review body lines today. Something that they found useful, something that something surprise them. How to make this sharing feedback on all the ways. This sharing of difference in the code review process be good is to talk from the place of respect.

Olena Sovyn: When I’m code reviewing changes, I trust it’s a person that did these changes did their best, and no matter where they are in their engineering journey, are they junior? Have they been in the in industry for 30 years? Have they just joined the company? Or been with the company for five years?

Olena Sovyn: I believe they did their best and with best intentions to also to come up all my, this understanding in the code reviews. What I try to do is if I’m not sure about anything, any change that happened in the changes I ask rather than state. When I review changes, I am trying to remember that I’m not reviewing a person. I’m reviewing changes. So I try to avoid using you word anywhere in my code review feedback. And also I’m trying to be as specific as possible.

Olena Sovyn: Let’s concentrate on this last point a little bit more. So what does it mean to be as specific as possible? Let’s look at this one example of the code review comment. It’s not the best approach to do this calculation. Yes, it is a feedback, but how can it be better? Maybe something like this? Why the second one is better than the first one? Because it’s contained why changes are needed as well as it is generous with example. The one blue request author will be seeing such code review comment. They will be easy. It’ll be easier for them to act on it.

Olena Sovyn: Make sure that your comments include everything that the blue request author might need. We talked a little bit about like that we need to include why we request some changes, examples, but there is one more thing that we would need to include in the code review comments. For example, be as specific as possible. Let’s look at another example. So this is an example of a code review comment. We have three version of this pattern in our code base now. We should have only one, okay. This is a feedback and it is information, but it is unclear and unspecified in this code review comment is what actually code request also should do business information. So make sure that in your code review comments, it is clear what changes are expected to happen after the code review was submitted.

Olena Sovyn: You can use, for example, for this emoji system. Well, by one emoji you can code changes that are requested before a request will be good to match. With other emoji you can mark with just a suggestion. And with third one with appreciation. This case remember two things. First, your teammates should understand what each emoji means. And second one is that this emoji can’t be only color coded because you want the system to be accessible.

Olena Sovyn: Also make sure that they have different shapes. And also remember to bring some positivity and praise in your code review feedback. It can be in the form of…

Olena Sovyn: One of the tips that one of the engineering manager in the Webflow that is really nice to place some humor in the general code review feedback. Like here is a memes that in some way is connected to the changes that was request also introduced.

Olena Sovyn: After the deep dive, what happened next is end of the code review. What can we do at the end of the code review? Already mentioned that we can in place here, some positivity, some praise, but what else can we do here? I think like in the end of the code review, it might be good also from time to times to connect full request also to the bigger picture, to the big aim, to which the changes have contributed. For example, let’s compare, let’s see this code review summary.

Olena Sovyn: This is really great explanatory documentation. It is a good, positive feedback, but how much better is this one? This is really great explanatory documentation. It’ll be such much easier with each to onboard new engineers in this code review comment. We are connecting changes with a bigger purpose of this change, but from these changes, it’ll be not easy to onboard.

Olena Sovyn: I today talked a little bit about what hidden gems I found in the code review processes during my times at Webflow, but I’m more than sure that between you are many skillful engineers and you also know much more hidden gems in the code review. Please share them in the chat, and I want you to thank you.

Sukrutha Bhadouria: Thank you so much. That was just absolutely insightful and wonderful. Let’s see real quick if there are any questions for any for you. I think there are a few that you can take on the chat. Let’s move on to the next speaker.

Sukrutha Bhadouria: The next speaker is Siobhan. Siobhan is the lead data engineer at Webflow. Getting to work with engineers, scientists, analyst, and everyone else who will talk data with her. That’s awesome. She’s been a teacher, a mentor, and a workplace culture, plus mental health advocate. Some tech topics she loves are functional programming, so less and coding, Fred Brooks on architecture. Her dream is to one day, be in the Kafka Four Comma Club. Welcome.

Siobhan Sabino: Thank you. Let me share my screen. I’m going to mix it up. I’m not going to ask if everyone can see my slides. I’m just going to hope for the best here. In this presentation, we’re going to talk about data engineering. To give you a little bit of background, I am a data engineer. We will talk about what that means. I’m Jersey born and raised. It is very late for me here. Apologies if I yawn.

Siobhan Sabino: I know here is where I would typically put a picture of me, but since we can all see me, here is a picture of my nephew Brody instead, because he has a very handsome cat. What we’re going to be doing and talking about the data engineering secrets is first we’ll go through my journey to become a data engineer, what the job entails, what I’ve learned about it, the glory or lack thereof in the job, the whys of being a data engineer, some of our secrets so you can take them and use them, and then the final slide to wrap it all up. The journey…

Siobhan Sabino: All data engineers tend to have very different ways we came to this job. So my journey is that out of college, with my CS degree, I got my first job in data warehousing and ETL. These are not flashy technologies, but they’re very stable things, very common in finance and healthcare, very well established.

Siobhan Sabino: From there, I moved on to a job where for whatever reason, technology was picked by our manager, seeing stuff on hacker news and going, that seems fun. For some reason, that was how our office picked tools like Kafka and Avro, and they needed an engineer who would be able to feel comfortable working with those. And my options were to learn those or learn JavaScript. And I didn’t want to learn JavaScript.

Siobhan Sabino: I learned those things would put me on this trajectory to becoming a data engineer where I finally moved into a job where I oversaw systems that had more than 2 billion messages a week coming through terabytes of data. And now I’m here at Webflow giving this presentation to you.

Siobhan Sabino: What does the job entail? If you ask a data engineer, what data engineering is, I think this subreddit from data engineering community really sums it up. Where the question was, tell you a data engineer that telling your data engineer and the community you voted the top answer to be, “I have no idea what my job actually is,” which does sum it up immensely.

Siobhan Sabino: If it’s hard to say what a data engineer is, let me sort of show you what a data engineer does to give you an idea of what those might entail. A day in my job might look something like this. I come in nice and early, ready for the day. Overnight, the transformer failed. Why? None of the 35 error we expected or why it failed. So we have to figure that out. No worries. Then someone shows up to say, but the numbers look wrong. I have to figure out, does that mean we’re missing data? Is there a bug upstream? Is there a bug downstream? Or have we accurately reflected what are, unfortunately, just numbers today?

Siobhan Sabino: Then it gets to about 9 o’clock. This is the point where someone tells me the iOS app has stopped sending events. It did that a couple weeks ago. No one noticed. And in looking into this, we realized the Android app, it’s sending events. All of them are wrong. No one knows why this is happening or how to fix it. There isn’t really a statement or call to action there. You just got to figure it out yourself.

Siobhan Sabino: About 10, 10:30, legal asked me to describe about 75 terabytes worth of data by the end of the day. Ironically, this will be the easiest thing I tackle in my day. Right before lunch, find out another team’s going to do a very risky production deployment, because that’s always fun. It might damage our data. We have a big report running the next day. I have three minutes to figure it out. Typical. Then when I finally get to lunch, the new manager tells me they’re excited to hear about my small data platform, which makes me cry on the inside.

Siobhan Sabino: After lunch is when the Postgres incident happens because we all know that’s when database incidents happen. My job doesn’t actually involve Postgres, but as someone who spends a lot of time thinking about databases, pitch in to help. As that wraps up finance of course hits me up on Slack to know why data systems are expensive. This week it’s about why moving billions of messages costs money, because last week it was about why storing terabytes of data costs money.

Siobhan Sabino: At this point, one of the data scientists asked me to explain what a container is. This will be the hardest thing I have to do. And I will hand them off to the junior engineer who is much better qualified to explaining this than I am. At that point, my manager is unfortunately told he gets to tell me that I don’t get to send metrics out for these systems because these are big systems and they’re expensive to monitor. We then get in an argument. I tell him that when someone gets paid, they’re not going to know what’s happening. He says, “We’re going to have to run that risk.” The joke will be on him. He will get paged on in the middle of the night, and I be asleep and not able to help him.

Siobhan Sabino: And at that point, there’s 17 minutes left in my day. So, that’s what being a data engineer is like. The sort of problems you face as a data engineer are not just the immediate, something’s on fire. It’s thinking about this long-term design and maintenance systems that will live for years. I inherited a Kafka cluster so old that the people who made Kafka couldn’t believe we were still running it. I don’t like that achievement. We should not have hit that. We have streaming systems now, which are exciting, but they’re also overwhelming. Because that means lots of things are happening at once, which is a different can of worms than batch problems, because there, you don’t know how you messed up for several hours or possibly days.

Siobhan Sabino: Pick your poison. There is data everywhere. It’s in databases. It’s in spreadsheets, it’s in people’s heads. None of it has schema. All of it looks slightly different from the other, because it’d be too easy if the field state always meant either New York or active and not both of them at the same time.

Siobhan Sabino: People also want answers, but they don’t know what the questions are, and you got to figure that out, which is exciting, but the main sort of problem as a data engineer I face is these problems of negative engineering. Writing defensive code, paying down tech debt, refactoring, updating, upgrading, as opposed to positive engineering, which is what we tend to think of, which is I write code for a new feature. It goes off and it’s a great time, which leads to the glory or lack thereof in data engineering.

Siobhan Sabino: When no one knows what you stop from happening, no one knows what you’re doing, right? So if all of your work is negative engineering, you’re not shifting new features. You’re not shipping new services. You don’t really have updates for the rest of the company to understand, and you tell them everything’s very niche and backend and people sort of nod, but they don’t understand what you’re talking about because you can’t show people very easily. Here’s the incidents that didn’t happen because we made the system resilient and self-healing. Here’s the data that was not lost. There was no loss of data or trust because we’ve been working on the tech debt, and the bugs, and the monitoring people can’t see how much work you’ve removed from their plate by really thinking about how can I make this as easy as pie.

Siobhan Sabino: …plate, by really thinking about how can I make this as easy as possible for engineers or scientists or analysts. When you’re doing your job, you’re invisible and the moment something fails, that’s when that’s all anyone can see and suddenly everyone just wants to know, what did you do and when are you going to fix it?

Siobhan Sabino: It might be obvious by now why a company would want a data engineer. When you have questions, like how do we get the data to answer our questions? How do we move it fast enough? How do we store it? How are we in compliance? How do we make sure that people who need it can use it. Those are the sorts of things a data engineer thinks about and can help you answer. But at the same time, why would someone become a data engineer?

Siobhan Sabino: Because you’re probably sitting at home thinking, Siobhan, you’re not really selling me on this and I get that.

Siobhan Sabino: If what your favorite part of being an engineer is, is producing new features or making visible work that end users can see and use, this probably isn’t for you and that’s okay. I don’t know how front-end engineers like Olena do it, I would not have the patience for it but I appreciate the apps and websites that are built.

Siobhan Sabino: What I like though – and what you might like – is when you have these really hard problems that have no easy or obvious solution. When you think at massive scales of time and space, this is a system that will live for at least five years and how many terabytes of data it will process monthly, weekly. When you’re exposed to every bit of technology in engineering.

Siobhan Sabino: I’ve made an Android pull request. I’ve never used an Android device in my life but I made an Android pull request. I’ve had to have the inner parts of Objective-C explained to me. There is no reason I could have been there but I’ve gotten to work with it because if your favorite part of engineering is getting to help others to do their job, then maybe you’d like being a data engineer.

Siobhan Sabino: I promised you in this presentation, I’d give you secrets. Here are the secrets. This is going to be a crash course, some tools, some ideas that you might need or use or want to look into. That way there is no gatekeeping, no one can make you feel like you don’t belong. These are the magic words to know, that way you can get involved.

Siobhan Sabino: When we talk about data, data is the raw representation of a thing. Information is the value extracted from data. People will often tell you that they want data, what they really want is value from that data.

Siobhan Sabino: A bounded data set is finite which is what we traditionally face. Infinite sets though, those unbounded data sets of constantly growing data, that’s what we’re faced with now in the world.

Siobhan Sabino: A data warehouse is a database that’s been specifically designed to hold all your data for analytical purposes. It’s expensive because it does its job. If it costs half a million dollars a year to run but it lets you make at least $2 million in decisions, it’s paid for itself.

Siobhan Sabino: On the other hand, the data lake is an application that has an actual purpose. Don’t worry about it, that’s not entry level. You will hear lots about data lakes because vendors love selling them to people and many data lakes go awry so often, that has a name, a data swamp. That’s not a helpful term to know but I just think it’s very pleasing.

Siobhan Sabino: What is probably more of interest to you is a data vault, a place where you can keep a copy of all of your data just in case someone deletes the production database, you find a bug and you want to re-run the code.

Siobhan Sabino: When you act on data, a batch engine is a way to process a bounded data set. A streaming engine is processing an unbounded data set. Just because one is newer, does not mean it’s better. They’re both tools that can be used correctly for the right problem.

Siobhan Sabino: ETL is the idea of extracting data from a source, transforming it and then loading it to its destination. This is not a new idea, it’s been around for decades and even if you don’t design your systems to reflect those steps, it’s a great logical way of thinking about acting on data.

Siobhan Sabino: Data cleansing is, heck grating to do but it’s super important. If you want to have opinions about data, start cleansing that data, you will have opinions really fast.

Siobhan Sabino: Data lineage tells you where your data’s been and who’s used it. That’s great for legal and compliance reasons, it’s great for debugging and for making live diagrams of what does the system look like.

Siobhan Sabino: Data tests compliment your code tests so that you know that things are right. This is not an area where the industry really has good examples of the way with code tests, we could talk about a meaningful unit test, integration tests, test driven development, we don’t really have that for data tests. So just do your best and know that, that’s all we can ask of you.

Siobhan Sabino: When you talk about data systems, a change data capture system pushes the events that happen inside of a database out to other systems so they know about it. If you want a CDC, you probably want one going to an append-only log.

Siobhan Sabino: A data pipeline allows large volumes of data to move around freely and quickly. It allows systems to come and go either producing data or consuming data without really needing to worry about each other, they don’t have to be the same language, they don’t have to share the same paradigm. If you want a data pipeline, you probably want to build one around an append-only log.

Siobhan Sabino: If you’re looking into an event driven architecture, you might think you want a message queue or a publish/subscribe system. In reality, you probably want an append-only log. I don’t know if you could tell but I like append-only logs which is great for you because I have a suggestion for one, Kafka.

Siobhan Sabino: Kafka is a great append-only log that can really scale. It’s written in Scala and Java, it has lots of support for non-JVM languages. So, if your backend is Node.js and Python, you’ll work great but for data engineering, JVM, especially Java and Scala are going to be the main languages you’re working with.

Siobhan Sabino: To go with Kafka, I’d suggest the library Avro. This allows you to find schemas about your data, part of that data cleansing and understanding what your data looks like. It also works beautifully with a system called Schema Registry. As this name suggests, it lets you register schemas there so you can see what they all are. That’s why answering a legal request about what terabytes of data looks like is easy. You just go tell Schema Registry, explain everything to me and then you send that to lawyers in a spreadsheet and it makes them happy.

Siobhan Sabino: Functional programming is a paradigm that works really well with data system because it lets you compose together these very small pieces that you can test and feel really confident about and then build them up for each use case.

Siobhan Sabino: Scala and functional programming, both get a bad rap. People will tell you they’re really hard to learn. The official language book for Scala does a really good job covering the basics of functional programming because a lot of people do a lot of crazy things, they get all over the place. Don’t worry about that. We’re looking for basic fundamentals here.

Siobhan Sabino: If you’re looking for setting up cloud storage, use your cloud-of-choice’s storage system for cold storage for that data vault. For the warehouse, just use what your cloud offers. So if you’re in AWS, for example, put your data vault in S3 and Glacier. If you want a warehouse, put it in Redshift. There’s vendors selling other products but your cloud-of-choice’s options, those are going to be super easy and they’re actually going to work really well.

Siobhan Sabino: To wrap this all up, this is my website and email. If you ever need me, if you’re a data engineer and you need someone to talk to or if you’re just having an absolutely terrible day and you want to talk to somebody, that’s my email. So long as you’ve spelt my name right, you hit me up.

Siobhan Sabino: Being a data engineer can be extremely thankless job but it’s still an incredible feeling to get to help others and see what they can do because you were there to support them right?

Siobhan Sabino: In the chart JZ showed, I work in the infrastructure pillar. I’m way at the bottom and I love that. Even if you don’t want to be a data engineer, use our tools and secrets but also if you work with data engineers, maybe be kind to them, they’re probably having a rough day and would appreciate it.

Siobhan Sabino: Again, even if you don’t want to be a data engineer, something like Kafka, that’s a massive ecosystem, it has lots of great community support, lots of great tools and articles. Maybe it’ll give you an idea for something you can do, just another tool in your toolbox for how to solve the problems you are working on.

Siobhan Sabino: Functional programming or Scala, like I said, people give them a bad rap. They’re not that hard and as a data engineer, they’re going to be your friends. Even if you don’t need to move terabytes of data or millions or billions of messages a week, you probably still work with data, so maybe figuring out what it is data engineers would suggest might make your job easier.

Siobhan Sabino: Because I had a manager who believed all presentations should end with a call to action, my call to action is to tweet my sister, that Brody is a handsome cat. Don’t worry about her, she’s in the chat. So she knows I’m doing this. And that is my presentation. Thank you.

Angie Chang: Thank you, Siobhan for that excellent talk on data engineering and for hanging out with us from the East Coast. So, our last speaker is Katie. If you have any questions at all, please do add them to the Q&A in Zoom or ask it in the chat and Arquay will be asking these questions to our speakers after Katie’s talk.

Angie Chang: I’ll do a quick intro to Katie. She is a software engineer on the Collaboration Team at Webflow and she co-leads the Asians at Webflow Affinity Group. Previously she founded and ran a tech meetup in Portland, focused on career development for people who are newer to tech. She’s passionate about helping people from underrepresented backgrounds get involved in tech and creating safe spaces for them to feel welcome. So welcome, Katie.

Katie Fujihara: Hello. I’m going to go ahead and share my screen, one second. Are you able to see that?

Angie Chang: Yes.

Katie Fujihara: Okay, perfect. Hello! Hi everyone, my name is Katie Fujihara. Today I’ll be giving a talk on how to be your biggest fan, a guide on how to self advocate. Just a little TLDR of who I am. Yes, I am Katie, said that already.

Katie Fujihara: I come from a non-traditional background, majored in marketing and Japanese, never thought I’d be a software engineer. In 2018, I attended a coding bootcamp and co-founded and ran a local tech meetup in Portland called Future Leaders in Tech.

Katie Fujihara: In 2019, I joined Webflow as an apprentice software engineer and currently I am a software engineer too, still at Webflow. That’s my baby dog Yochi who is quarantined right now so he doesn’t bark while I’m giving this talk.

Katie Fujihara: Just a little quick agenda breakdown – what we’re going to be talking about today, should be a fairly short presentation. First up, will be Glue work versus Glamour work and how that relates to unconscious bias. Next, will be personal concerns and challenges when it comes to promotions. And the last bit will be the meat of the presentation which is tips on how to advocate for yourself.

Katie Fujihara: First off, let’s go over glue work versus glamour work. You’ll notice that on the slides in the bottom corner, some of them will have these QR codes. If any of these particular topics interest you, feel free to hold your phone up or your camera up to the QR code and scan that, it’ll take you directly to the source of work I got all of this information from.

Katie Fujihara: What is glue work? According to Tanya Reilly, glue work is, “the less glamorous and often less-promotable work that needs to happen to make a company successful.” So examples of glue work can include writing docs, setting up team meetings, improving team process, establishing coding standards, mentoring and coaching, improving new member onboarding.

Katie Fujihara: What is glamour work? So, examples of glamour work on the other hand are what sounds like more glamorous, writing code, and shipping features. As software engineers, this type of work is often valued more than glue work because it signals technical competency. A poor manager may determine promotions and rewards based off of the false equivalence that more code written automatically means a stronger, more impactful engineer, which we know is not always the case.

Katie Fujihara: Next, I’m going to be going over a bit of the importance of glue work versus glamour work. A national study conducted by the Center for WorkLife Law and the Society of Women Engineers surveyed over 3000 engineers. It showed that women were 29% more likely than white men to report doing more office housework, and for the sake of this talk, we’ll call it glue work, than their colleagues.

Katie Fujihara: Prescriptive stereotypes show that women of color are under the most social pressure to volunteer for glue work and the unequal distribution of glue work and glamour work between women and men is evident when you see that men are more likely to be promoted to executive positions. This is an indicator of how much more impact glamour work has when it comes to promotions.

Katie Fujihara: A little bit about my experience with all of this. I’ve been at Webflow for two years now and in that time I’ve gotten two promotions, but I’ve also cycled through four different managers in that time. I’ve also found myself doing quite a bit of glue work and was nervous about it being invisible. As many of you who have worked in the startup world know, organizational change and uncertainty is common among early stage startups and startups going through hyper-growth.

Katie Fujihara: Therefore, it is important to learn how to navigate these spaces and to track your own personal growth. During all of this uncertainty and changes and not wanting my career to get stunted, I had these four major questions on my mind.

Katie Fujihara: How do I ensure my career does not stagnate? How do I ensure that my new manager understands my impact? How do I make the glue work I do visible? And how do I ensure organizational changes do not affect my promotion timeline?

Katie Fujihara: My solution, pretty simple, make all the work I do, glue work included, as visible as possible and to advocate for myself. Next up the meat of this presentation, tips on how to advocate for yourself or tips that I find helped me in getting promoted.

Katie Fujihara: A quick reminder, you can have good peers and a good manager but at the end of the day, you need to be your biggest advocate, your biggest fan.

Katie Fujihara: My first little tip would be to track your progress and wins. Seems pretty obvious but it needs to be stated. I would recommend creating a progress document that you update a regular cadence, whether that’s monthly, bi-weekly, whatever it takes for you to remember to do it, that’s the cadence for you.

Katie Fujihara: If you know what is required to get to the next level, organize your progress doc in a way that highlights how you are satisfying these requirements. Link to PRs, Slack conversations, code reviews, screenshots of public or private praise, anything that can serve as evidence of your impact.

Katie Fujihara: Lastly, share this doc with your managers. If you move managers, bring this doc with you and show it to your new one, that way they can have all the context of work that you’ve done.

Katie Fujihara: I have two examples below of how you could structure your progress doc. For example #1, I have it broken down in six month increments, so January to June, July to December and I’ll usually put bullet points of the things that I’ve done, the contributions and I’ll also link or take screenshots of Slack conversations, PRs, things that serve as evidence that support the contributions that are the things that I’m saying I contributed to in the progress doc. For example #2, this is if it’s structured by knowing the requirements to get promoted already.

Katie Fujihara: Say the requirements are, must write documentation, must write unit tests and provide thoughtful feedback and code reviews. The way I would organize my progress doc would be to have each of these requirements as the headers of each section and then put the contributions underneath each section that support this. That way, you have all of the proof you need to show that you’re operating at that next level.

Katie Fujihara: Another tip is to not lose promotion traction during manager handoffs. If you. #1, know you are close to being promoted and, #2, know you will be changing managers soon, push for your current manager to start the promotion process because there are so many unknowns when getting a new manager.

Katie Fujihara: I actually had to do this when I was going from apprentice software engineer to junior software engineer because I found out I was getting a new manager. Usually the amount of time they want you to be an apprentice is six months but I was at about my four month mark and I was worried that when I switched managers, my new manager wouldn’t have all the context that my current manager did. I really, really pushed for him to start that and luckily he did. I was able to transition before I got that new manager.

Katie Fujihara: The reason you want to do this is because your new manager will not have as much context as your current manager. They may not be as helpful when it comes to advocating for you and they maybe preoccupied with other things as they are onboarding.

Katie Fujihara: Consider that your new manager might be onboarding to your team or onboarding to your company, they have a lot on their plate. I don’t know if any of you are managers, but I’ve heard if you’re a manager, you have just a ton, a ton of things to do that most people don’t ever see.

Katie Fujihara: As a report, do your due diligence to get those things started and to advocate for yourself because who knows your promotion might be the bottom of their priority list at this time. Start it before if you can.

Katie Fujihara: Another piece of advice would be to push for 1:1:1s during manager handoffs. As Lara Hogan states, “Your new manager might not be familiar with all that you’ve done already, which could slow your career momentum.”

Katie Fujihara: What exactly are 1:1:1s? It’s when you, your current manager and your new manager all sit down and go over your previous work, strengths and areas for improvement. It’s a time for everyone to get aligned on your goals. This is a good time for everyone to get context around everything and your new manager to understand exactly your impact. I found these really helpful in the past.

Katie Fujihara: Last but definitely not least, make your work as visible as possible, be as loud as you can about wins, be transparent about what you’re working on.

Katie Fujihara: This could be through updating your Slack status, we could be like, “Oh, I’m working on a bug fix” or something along those lines, just so people know what you’re doing.

Katie Fujihara:Be transparent in stand-ups, just make sure everyone knows what you’re working on, don’t undersell anything that you’re doing and ask questions and be open about any blockers that you’re facing. You don’t need to suffer in silence if you are stuck on something.

Katie Fujihara: Find quantitative ways to measure the impact of your work on a business level. Talk to your product managers or talk to your engineering managers about how what you worked on is performing, so you can get those hard numbers that you can use to your advantage when it’s time for a promotion.

Katie Fujihara: Lastly, talk to teammates and mentors about your progress. The more they know about your work, the more they can help advocate for you when the time comes for a promotion.

Katie Fujihara: A little quick summary, I told you this talk was going to be quite short. You have to be your biggest advocate, track your progress and wins in a progress doc, try to kick off a promotion process before a manager handoff if you’re able to, push for those 1:1:1s and make your work as visible as possible.

Katie Fujihara: Thank you. Twitter is @KatieFujihara if you want to keep in touch or LinkedIn, whatever works for you but that is all I have for you today. Thanks for listening.

Arquay Harris: Once everyone joins, I’m going to lead into some Q&A. We actually have a question come through and since it’s a question that could apply for all of us, maybe I’ll just sort of facilitate for a couple of us to answer it.

Watch the Webflow Girl Geek Dinner Panel Discussion with Arquay Harris (Webflow VP of Engineering), Siobhan Sabino (Webflow Lead Data Engineer), Jiaona Zhang (Webflow VP of Product Management), Katie Fujihara (Webflow Software Engineer) and Olena Sovyn (Webflow Staff Software Engineer).

Arquay Harris: I’ll start with you, JZ. The question is, how did each speaker find their way to their current role at Webflow? The tactical stories help us understand if everyone applied got recruited, how they got to do something new, i.e. if their resume didn’t show the same job title. Effectively, what was your journey to Webflow?

Jiaona Zhang: Sure, happy I started. I joined Webflow at this point about a year and a half ago. I joined Webflow actually coming from WeWork. WeWork was definitely a very interesting journey. I joined after I left four years of Airbnb and what was really attractive about it was to be able to start in the tech organization from scratch in a place that didn’t exist.

Jiaona Zhang: That company did go through a lot, that was very unexpected. After hyper-growth for six months, I was really thinking about restructuring the team in the latter six and so when I was leaving WeWork, I really thought about what I wanted to do next. I actually had the time to think about it because I was pre-planning and working with my team to make sure I was landing in a really good place. I knew that I wasn’t going to be staying with the team long term.

Jiaona Zhang: I thought about smaller companies, I thought about larger companies and what really drew me to Webflow was a couple things. One, I think when you have… Personally for me, when I had the opportunity to lead all of product, was a very different experience than what I’ve done before which was large organizations but not necessarily the entirety of the product team and there’s something that I was really excited about to do that at Webflow.

Jiaona Zhang: The other thing is, why Webflow? There are other companies out there but I think it’s so rare to find these two things, the first one is a mission and a product that just gets me fired up every single day.

Jiaona Zhang: We talked about this before, which is getting to help the world be able to create something that is in 1% of the world today, which is right access to the web, being able to democratize that and make sure that everyone has access to it. I think that’s something that a lot of us here, it really resonates with. I talked about this earlier, where I’m not technical, that wasn’t my background and being able to build a tool that everyone can create, regardless of their… Do you have a CS degree? Do you have a coding background? That’s just something that really draws me.

Jiaona Zhang: The other piece is the people. I truly think that… I’ll say this as a product leader, the product leader and the CEO needs to have a very, very strong relationship and I can’t think of another person who is both a combination of the best chief vision officer and the kindest human but that combination is just so rare and finding both of those things in Vlad is something that was really appealing.

Jiaona Zhang: I, again, talked to a lot of different founders and it’s a really, like a one in a million opportunity to get to work with Vlad to bring that vision to life. Then the team, the team here is just so kind. Our mission statement is not just, this is a thing we want to accomplish but these are the things that we want to accomplish with the people here together. So, again, that is just very rare and that’s how I made my way to Webflow.

Arquay Harris: Great. Although JZ, it’s a little bit hurtful because I thought you told me that the most important relationship is between the VP of product and VP of Eng. I don’t know, this is all just, this is all…

Jiaona Zhang: You weren’t there when I joined.

Arquay Harris: Hurtful.

Jiaona Zhang: Now, Vlad has been replaced by Arquay.

Arquay Harris: That rings hollow. I don’t know. I’m actually going to go second because it’s a good segue into why I joined the company which is definitely the people. Every single person that I talked to… Vlad is maybe the nicest human ever. Everybody read the Steve Job’s book, so you get these kind of megalomaniac CEOs or founders of companies and it’s a little scary out there and he’s definitely nothing like that, has a really great vision.

Arquay Harris: Every single person that I met, I met Brian, I met some of the other people at the company and then, honestly, JZ, one of the things that really struck me, especially considering this demographic, Girl Geek Dinner, is I have literally racked my brain and in over 20 years of experience, I cannot think of another company where both the VP of product and the VP of Eng. are women of color.

Arquay Harris: If someone in chat can give me another example, I thought about it and we are not diversity hires or something like that. We are just women who are out in the world doing our jobs but that is something interesting that you don’t see every day and I think it’s just JZ’s vision and the way she could articulate, not just what we’re doing today but what we’re doing in the future and how she also values that part. Today, what we’re doing in the future and how she also values that partnership between product and engine design, and then just generally every person. I’m coming in…

Arquay Harris: I’ve only been at Webflow for about six months and I’m coming in, I’m kind of doing this operational Excel and stuff and no one is licking all the cookies being like, “No, this is the way you do it.” They’re like, “Great. That sounds awesome! You have a great idea. Let’s try it.” You know, everyone’s amenable to change and they know that we’re here and we are going to have to do different things to get us to a different point.

Arquay Harris: And it’s just a really like amazing place to be. Speaking of that, like maybe we transition to Olena. I’m curious about you ’cause it’s very, very late for you, so I’ll let you go and see what your answer is.

Olena Sovyn: I joined Webflow four and a half years ago when it was like a company of 40 people. And for me it was a company where I was able to solve very difficult challenges while still being on the front-end part of the application. It was very rare for me and also of all people that I met through my interview process were just awesome. As a first retreat, I just understood that these people are my people because they were so kind, so human, so humble, at the same time, so smart and so visionary, focused on our mission. It was just awesome.

Arquay Harris: I’m on mute, but what about you, Katie?

Katie Fujihara: I came to Webflow. Yeah, two and a half years ago actually through… Vlad found me on Twitter, but it was a, you know, I think that when I first joined Webflow, I was in a point of my career, I didn’t have a tech career yet, like I was looking for my first role. And it was at that point where I was just looking for someone to take a chance on me. I know a lot of people who are juniors in their engineering career, this resonates with you, where you just want someone to believe that you can do it ’cause you know, you can do it.

Katie Fujihara: It’s just you need to sell yourself. And luckily for me, I was given that opportunity at Webflow and you know, like everyone else was saying, what really sold me on the company was meeting the people and just like going through the interview process and being introduced to all of the engineers. I’ve always been kind of intimidated by what people consider like the engineering type or the stereotype of engineers.

Katie Fujihara: Everyone I worked with in the interview process was not that stereotype, and it was just really fun. And I could tell that they really value the personality of people here on top of their technical skill and everything like that. They just want people who can work with other people well, and for me to that was really attractive. So, and two and a half years later, here I am so… yay.

Arquay Harris: Yay. And then Siobhan, I’m really hoping there’s going to be a Brody cameo at some point. Your sister’s on there, I saw.

Siobhan Sabino: Well, she is. Oh, remind me tomorrow. I have so many pictures of Brody I’ve stolen from her, I have a whole album. I joined Webflow earlier this year, essentially how I came to here was my previous job I’d been at for a couple of years, we’d finished a couple big projects. My junior engineer that I had been training, I knew she was fine, that she could take this on her own.

Siobhan Sabino: I felt like my chapter there had come to its natural conclusion, of everything was sort of in a better place. And the day I realized maybe it’s time to look for something else, it ended up better. A recruiter I had worked with there, now works at Webflow, and she had reached out a couple days earlier and I felt like the universe was like, “Hey Siobhan, here’s the next thing.” Because as everyone said, it was really the people.

Siobhan Sabino: When I think back on my favorite projects technically I’ve worked on, those teams, it was awful. Whereas my favorite teams I’ve worked on, those have been some of the hardest projects technically, but we were all in it together because it was the right people trying to do the right thing. That’s what really drew me to Webflow, was feeling like, “Yes. We’re going to come in and we’re going to make a change and we’re going to do the best we can and we’re going to do it together.” And that’s what really drew me in.

Siobhan Sabino: Plus now, we have a cat channel and I can show people pictures of Brody all the time and they love it, because I also love them.

Arquay Harris: That’s a great answer. We have an answer that came in through the Q and A, and it is, “What’s the best advice that someone gave you early on in your career in software development engineering?” How are you Katie?

Katie Fujihara: Okay, so actually I will say that… not necessarily a piece of advice, but something that Olena did actually, because I worked with Olena before on a feature.

Katie Fujihara: One thing that she taught me is that even if you or a staff or senior level engineer, you have so much you can learn from junior engineers, and that has stuck with me since. I remember she would vocalize also when I would teach her something that she liked or she didn’t know about or something. She would always just like be so open and humble in saying that to me, and it helps reinforce my confidence that even though I don’t have the years upon years of experience like her, she still can take away something from me and that I have knowledge that is valuable to other people, so that’s what I would say.

Arquay Harris: That’s very sweet. I won’t make every single person answer but I’m curious about you, JZ, I know that you’ve had a very long and kind of storied career. What was some early advice that you got?

Jiaona Zhang: Yeah, I can’t exactly answer the, you know, what advice I get in software engineering since that’s not my background, but the two pieces of advice that really have just stuck with me in my career, and there’s one that was early career and then was one that’s later career. I’ll share both.

Jiaona Zhang: The early career one was optimized for learning. That’s a piece of advice I’d gotten and really resonated with me and something that guided my career for the first five, six years, where when I joined Pocket Gems, which is the mobile gaming company, I was like, “I don’t know what a PM is. I’m here to do it.” When I joined Dropbox, I actually worked on the most technical product I could find because I had that imposter syndrome about working in technology, so I joined the Dropbox developer platform team and I was, “What is an API? I’m here to learn.”

Jiaona Zhang: You know, when I joined Airbnb, I’d worked at engineering driven companies. I’d worked at almost like business product driven companies, but I never worked at design driven company. When I joined, I really wanted to learn what it would mean to build from first principles and design thinking. And again, so that’s really something that guided me early career.

Jiaona Zhang: Later in my career, a piece of advice that’s really stuck with me that might seem unintuitive is “Ask for help.” I think that when you get further and further in your career, it’s almost harder to feel like you can ask for help. And I think, especially in the industry that we work in, that we all work in. It’s like, “Oh, if I ask for help, are people going to think I’m not competent?” Or they think, “I look 20, are they going to think…” Like there’s just like a whole like thing where if you ask for help, is that going to be looked down upon?

Jiaona Zhang: The advice I’d gotten from a mentor was, ask for help because you should always feel like you’re failing. If you’re learning and you’re growing, you’re pushing and pushing on that impact, like there are always days where you’re like, “Man, that did not go the way I wanted to go.” Like you should be in some ways failing and learning from that. And in order to do it in a way where…

Jiaona Zhang: Because the more senior you are, you are responsible for a lot of people and for the impact on the company. And if you do not ask for help clearly and often, it’s just not human to be able to take it on your shoulders all the time. Ask for help is something that I think about every single day, something I push myself to do every single day.

Arquay Harris: Oh, that’s so good. I mean, this reminds me of… It’s a similar question that people ask, like, “What is the advice you give to your younger self?” And for me, there is an extra pressure like, to take you back on what Jay-Z just said, where there’s this famous XKCD comic where, the first pane is like, “Oh, Bob doesn’t understand math.” And the second one is like, “Women don’t understand math.” But for me it’s like, if I admit that, I’ll miss something. It’s like, “See, I told you black women can’t do SQL.” And it’s just like this.

Arquay Harris: I would rather like buy a book on Java Beans than ask for help. And I would have wished I would not done… I wish I would had not done that, because I could have gone farther faster, right? Had I had that support system.

Arquay Harris: We have another question in chat and I’m going to start with you, Siobhan, and the question is, “What advice would you give to someone who has no technical background, that’s starting this from scratch in a bootcamp?”

Siobhan Sabino: Oh, that’s an interesting one. I like that. I think oftentimes engineers, whether intentionally or not, do a lot of gatekeeping, especially in a company where engineers are asked to talk with non-engineers. We are not taught to think about how do we communicate that? How do we make sure we’re using language that everyone understands? We’re bringing everyone along.

Siobhan Sabino: One of the things I’ve really learned as a data engineer and I’ve really loved is getting to work with non-engineers and having to explain technical things to them. It is very hard to explain what is a Docker container to someone who’s not an engineer, but when you practice that, that’s really important.

Siobhan Sabino: I think oftentimes when you come from a non-technical background and suddenly you’re faced with engineers throwing these words around, you think, “I’m the problem.” It’s important to remember from a cultural perspective, “no, this is something where everyone should feel involved.”

Siobhan Sabino: You should feel comfortable saying, “I don’t understand. Can you explain that to me? Can you explain that in a less technical manner?” That way, we are bringing everyone along and sometimes engineers will say, “Well, I understand so why don’t you?” The point is, if we’re all on the same page, we’re all going to do better together, right? High tide raises all boats.

Siobhan Sabino: I think, especially if you’re coming from a non-technical background, I’ve done a lot of interviews. And let me tell you the non-technical people or people who switch careers, they’re always my favorite, because they think about things so differently. And that different perspective is so valuable. They’re thinking about how do I explain this to non-technical people? How do I approach the problem differently? As much as many people will tell you, this is a weakness, it’s not. It really gives you a different insight. And I think it will make you just a better person to work with all around

Arquay Harris: Totally agree. I’m curious about you Olena and then I feel like we should go to Katie as well after that, who actually lived through this experience?

Olena Sovyn: I would dabble on like advice about education. There are so many teachers out there, especially with everything available now online. If you have teacher at your boot camps that can’t explain something, look for another teacher on YouTube, because there are so many approach how to teach others and some approach might work for you better.

Olena Sovyn: Some approach might work for other people better. If you don’t understand something, this is not your problem or your limitation. This is just the way someone tries to educate you, and it doesn’t work for you, but there are so many other ways how you can be educated and learn about something. Maybe your way is to ask for examples, maybe by watching videos, maybe by creating talks, who knows.

Olena Sovyn: Explore how best you can learn, what you want to learn.

Arquay Harris: Oh yeah. Katie, go for it, if you wouldn’t mind.

Katie Fujihara: I think something that is really important when you’re coming from a non-technical, completely unrelated background, doing a bootcamp, the thing that people often don’t tell you while you’re in the bootcamp is how difficult it is once you’re out of the bootcamp to get a job and to get noticed.

Katie Fujihara: What I would recommend is to find a way to differentiate yourself early on, figure out what you like, really double down on your strengths and your weaknesses, like figure out what those are early on and just like really lean into your strengths.

Katie Fujihara: I know for me, personally, like one of my strengths is I’m very community driven. And so while I might not be the most technically savvy person or I definitely don’t consider myself that, I lean into the participating and building communities.

Katie Fujihara: That for me, that meant attending lots of conferences and networking at them or speaking at conferences or speaking at meetups, just getting myself out there so that I could network with people.

Katie Fujihara: Another way, if in-person type of things isn’t your jam, I’ve noticed that people were very excited when they found out that I contributed to open source projects. That shows that you’re engaged with like the open source community, you know? And you have proof that, of your technical skills, by like linking to PRs. It’s evidence. It’s very easy to just point things like, “Oh, I’m active in here and this is my work.”

Katie Fujihara: I found that this was a lot more useful than working on side projects that were never finished or anything like that. This one you just pick and choose what things you want to learn about in open source projects. Maybe it’s “Oh, I want to learn more about CSS.” So you take only those types of issues, and then you get to go through the code review process with maintainers, and so then you get to learn about that.

Katie Fujihara: That would be my recommendation, contribute to open source and join communities.

Arquay Harris: Oh, I love all of these answers and Katie, especially. I remember years and years ago, people used to say like, “Oh, I want to get into coding, like what should I do it?” I would always say, “Code.” Even if you make a website for your mom’s Canasta club, just do it, like just start doing stuff. And so it’s just like, “I love your attitude.”

Arquay Harris: And the other thing that I wanted to comment on is when Siobhan said that the people who are kind of career changers are her favorite. I have had that experience as well, where like, if you say you have a CS degree and you go to college and let’s say you learn something like data types, you might learn it over a quarter semester, like group projects, and they have like a lot of fillers, right?

Arquay Harris: If you go to a bootcamp and for 24 hours, you know, 20 hours a day for two weeks, you learn data types, you’d be pretty good at them. Right? Like there is this like practical thing. And so for me, I always like… It takes all types, right? Like there’s many different roads traveled. And I think that this very much feeds into the next question that we have, which is, what advice… Sorry. Oh, there was a question, and it was deleted, but I remember what the question was.

Arquay Harris: JZ had talked about imposter syndrome. Have any of you ever – has anyone else ever experienced that? And what, what are some ways that you tackled it?” How about we start with, how about you, Siobhan? And then JZ, I think, if you have anything more to add.

Siobhan Sabino: Yeah. I actually started studying computer science when I was 14. My high school had classes. The way I got into it was when I was 13, my mother made a passing comment of, “Oh, you’re always on the computer. Maybe you should go to the computer science academy.” I was like, “Well, yeah, maybe I will.” In that way, teenagers have, so that’s how I got into computer science. And at that point I don’t even think we had internet at home, my sister can correct me if I’m wrong. Like we didn’t have internet at home. Internet was something at the library.

Siobhan Sabino: It’d probably be like 10 years until I’d hear the phrase imposter syndrome. But I remember that was the first time I encountered this idea that because I was a girl, I wasn’t supposed to be good at math or like computers. I remember when other middle schoolers found out I was going into the program. They were like, “Well, why are you in it?” I was like, “You’re just mad, because you’re not as good of algebra as I am. Study harder.” But that was…

Siobhan Sabino: I remember getting to the classroom and there was three girls in the class, which in retrospect is three more than probably most people expected. I remember sitting at my terminal one day because all the boys were having a great time, goofing off, being friends, and I felt so alone crying to myself, thinking, “Either I can quit or I’m going to see this through.” And I look back at that now, and that was probably the closest I came of… I was exposed to that at a very precious age where 14 is a very hard age to do that, but it was in a controlled environment.

Siobhan Sabino: I had a mother who supported me, and I had teachers who were women and supported me, so I was able to make it through. By the time I got into college and then into my first jobs, it was absolutely normal that I was the only woman. And at that point it was what it was. When people say, “Don’t you feel like you don’t belong?” It’s like, that feels like a “You” problem.

Siobhan Sabino: I know I belong and I don’t need to prove that to anybody. That was very unique experience for me, but really having that moment of saying either, “I’m going to quit or I’m going to keep going”, and I loved it too much to quit.

Arquay Harris: That is great. I think this will be our final answer before we head into the break, the networking sessions. JZ, take us home. I’m curious to hear your thoughts on this.

Jiaona Zhang: I’ll share my thoughts, but I actually think Arquay, you should take us home, because I’m really curious also in your thoughts on imposter syndrome, but I’ll share mine. Yes, a hundred percent. Definitely something that I’ve struggled with, I still struggle with.

Jiaona Zhang:I think that the first piece is when… so the product has been a discipline that I think has taken lots of different twists and turns, and it really was rooted more in marketing. I think the modern day product management like… a lot of people actually harken it back to Google with APM program, right? They had program managers at Microsoft, so on, so forth, but Google was really where it was like, “You have to have a computer science degree and then like being a PM is awesome. You get to be part of like this club and you get to like make decisions.”

Jiaona Zhang: But the thing is like, it was such a closed door kind of environment, where it’s like you really did have to have a computer science degree to even be interviewed for Google. And so, one of the things that I thought was really important as a turning point, a product as a discipline is like, no, actually recognizing that by bringing in perspectives of like other perspectives that aren’t just the computer science one, in a fact that your role is to really understand the user. Like, do you need to be technical to understand the user? No.

Jiaona Zhang: In fact, the more the user… You were like, the more that you project yourself onto the user, the worse products you’re going to build, right? And so I think that like moving away from that, like you have to have that degree in order to get an interview, into actually the role is really to understand people. That requires empathy, that requires curiosity. It does not require a coding background. I think really where things are changing and have changed.

Jiaona Zhang: I think part of my personal journey with imposter syndrome, especially in the product management role, is really understanding better what is the role of product management. It’s actually a very different role than the role of an engineer, for example. I think that is a big piece, but Arquay, I’m really curious about you.

Arquay Harris: Yeah.

Jiaona Zhang: Yeah.

Arquay Harris: I definitely have struggled with imposter syndrome, my whole career.

Arquay Harris: I was thinking recently, there was this article and it talked about how if you track historically over time, women’s participation in computer science, it was mostly female dominated. You had like the hidden figures. They were literally called computers, Bletchley Circle, like all this stuff.

Arquay Harris: Then at certain point, there was a huge drop off. What people have attributed to is the eighties, because you got into this like revenge of the nerds or like weird science and like, “Computer science is for boys and nerds and white dudes”, and that sort of thing got into the culture. Women began to think that like, “They don’t belong here. This is not for us. Math is hard.” Like all of these things, right? Yeah, I definitely struggled with like, “Oh, do I fit?”

Arquay Harris: Add the race and the gender, like all the things, right? At a certain point, I really just like stopped comparing myself. I have this whole like… When I’m talking to my friends or whatever, this kind of joke, how like I never compare myself to other women, for example. I’m never like, I’m like, “Oh yeah, she looks great in that dress, but I bet she’s terrible at CSS specificity or whatever.” Right?

Arquay Harris: You can’t compare yourself, like it doesn’t work like that. There’s all like apples to apples kind of thing. You really need to think about what makes you special, whether it’s like whatever dress you’re wearing or how good you are at JavaScript or whatever.

Arquay Harris:And thinking about like knowledge is a circle, right? Like no one knows everything in the circle.

Arquay Harris: What part of it do you specialize in? I think we do in culture, you’re sort of ingraining people to focus more on your weaknesses than your strengths, right? Like if there is a thing that you struggle with, I bet there’s something that you’re really great at.

Arquay Harris: We all have value. The thing that I think is also interesting is like, we were all hired. There was all some spark that people saw in us. There’s some potential that we have and it’s up to us, whether or not we live up to that potential and we lean into it or we let others define who we are and how we behave.

Arquay Harris: On that note, this has been super great. I’m not exactly sure what happens, but somehow magically, we are going to go into breakout sessions. Oh, there’s Angie! Oh, save us.

Angie Chang: Well, thank you all so much for joining us. We are going to be wrapping up. I want to say that Webflow is hiring!

Angie Chang: We’re going to be sending out an email afterwards to ask about how this event went and then we’ll have some link to the jobs there, so check those out. If not, just go to the website and look at the jobs there as well.

Angie Chang: They’re hiring for engineers, engineering managers, product managers, design managers, a ton of jobs.

Angie Chang: Tell your friends this is an awesome place to work. You’ve met some amazing women who work there and yeah, spread the word.

Angie Chang: In the chat, there is a link to the Zoom meeting where we’ll be doing the breakout sessions. If you want to join us, I know it’s getting late, please do join us by clicking on that link. It’s also available in an email that will be sent- that’s already been sent to all attendees. There’s a link to the Zoom meeting.

Angie Chang: Thanks so much for everyone for speaking. We’ll see you on the Zoom meeting, if you want to have another 20 minutes of meeting each other face to face. See you on the other side.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!