Angie Chang: Welcoming Tasha Penwell, who is an educator and founder at Bytes and Bits. Did I get that right this time? Yes.
Tasha Penwell: <Laugh>
Angie Chang: She is passionate about scaling women in tech as an AWS educator, and will be sharing her insights for teaching training and certification for AWS Academy and at AWS re/Start. So, welcome, Tasha.
Tasha Penwell: Thank you. Thank you everybody for joining me, and thank you for having me. We are obviously on a tight schedule, so I’m just gonna go ahead and get started. My name is Tasha Penwell and the founder of Bites and Bits located out here in southeastern Ohio. A little bit about me. I’m an AWS educator, certified AWS solutions architect. I’m also an AWS community builder specializing in security cloud security. Google has a program called Women Tech Maker. I’m part of that and all of the thing, all the resources that you’re gonna be seeing today is available through this link. AWS is awesome because it is, and there’s also that QR code that you’re more than welcome to grab.
Tasha Penwell: And I will be taking any kind of questions that you have at the last few minutes of our time together. Please feel free to put ’em in the chat and I’ll be reviewing them as we go along or after we finish. Why AWS? AWS is, first of all, AWS stands for Amazon Web Services, and it is one of the most in-demand skills that’s needed in technology today. There’s a variety of different certifications, no matter the path or interests of computing that you may have.
Tasha Penwell: Maybe you’re interested in security like I am, or you have an interest in data analytics, gaming, IOT, high performance computing, such as AI, machine learning, and there’s a variety of different industries. And I always tell my learners that to find something that you’re the most interested in for me it’s security and just see how you can utilize that with AWS and you’re gonna have a, it is gonna help you build a really awesome career that is something you enjoy and would be help you live the life that you want to live.
Tasha Penwell: Whenever I have a new class, one of the first things I ask my learners, I is like, who’s a lifelong learner? And I told them, if they don’t raise their hands, they’re in the wrong class. Because working in technology or specifically in AWS, you need to have that mentality of being a lifelong learner. And that means trials and errors. That means some may be some failures along the way. And you have to have that mentality that every failure is going to lead you to a learning opportunity. And those, you know, those will stack up on top of each other and will give you a wonderful, wonderful experience being you know, continuing to inspire some curiosity in your growth in a really fun and challenging field.
Tasha Penwell: The three takeaways that I would like everybody to have from our time together today is to be able to identify two training and library sources that are available to anyone. And I’m going to share my personal methods for guided notes and note taking. These are things that I do whenever I’m teaching a class or whenever I’m, you know, studying for my own certifications or just my own, you know, continued knowledge and also explore some different paths and resources for learning, working towards certification. Certification is obviously one goal that you have, you know, earning their cloud practitioner certification, your solutions architect or maybe even a specialty certification. That’s one goal that you may have, but there’s other incremental goals that you can have that can continue to propel you and inspire you. And I’ll give you the, the proper motivation to kind of get over the hard, because it does get hard sometimes.
Tasha Penwell: The first thing we’re gonna talk about is training and lab resources. AWS Skill Builder and AWS Educate are two wonderful resources that are available free. At Skill Builder, there’s a free platform in the late last year, they also launched another kinda like a premium version of Skill Builders like $29 a month. But the free is still awesome, and that’s actually what I still use, and this is also what I encourage my students to use as well to supplement. There’s a variety of subjects and you don’t need to sign up for an AWS account. All you need is your amazon.com credentials. This is, there’s an assumption being made that everybody has purchased something from Amazon before. And whatever your login credentials were for that, that is the credentials that you need to use to log in to Skill Builder to create an account with Skill Builder.
Tasha Penwell: Now, because of the time that we have in our class today, I offer you, whenever you see this little link with Highlight, it says video that is actually going to take you to a platform I like to use called Loom, and it, you can play the video. These are short videos that will go into more details of how to use these platforms that just don’t have time to cover in our 20 minutes together today, and that’s available for anybody to access. One of the things that I really like about this platform for anybody who’s not familiar with it, is like you can hit play mm-hmm. <Affirmative>, and I’m just gonna go ahead and mute that. And if you get to a certain point where you have a question or you’re not understanding something in the instructions or maybe you don’t see something, there’s a little button here where you can comment and you can say, I don’t see that. Something along those lines. And I can, I will actually get a notification in the timestamp of when that, whenever you have that question, and I can respond to you very quickly. So even after the day’s over, or, you know, later on this weekend, maybe next week, next month, next year, if you have troubles on anything, feel free to just comment on there and I will get that notification and I’ll be able to respond to you directly. So with throughout these slides you’ll see those type of videos embedded in there.
Tasha Penwell: AWS Educate is also another great resource. And let’s see. So AWS Educate, it actually started for it was only intended for high school and college students. I was originally, I was one of the original AWS Educate Cloud ambassadors back in 2019, I think it was whenever it was coming around. And it has a lot of great training and labs that you can do on a variety of subjects. Again, it’s originally targeted at high school and college students but it’s now available to anybody. And the only, there are some kind of hiccups with the registration process, and I kind of go over that in the video link here. It just kind of tells you how to get over the hiccups because it used to be re pretty strict on who could access it. Now it’s open for everybody, but some of the signed up processes are kind of still a little, there’s still some hiccups in there.
Tasha Penwell: Okay. So there’s, you know, again, some videos, if we have time at the end of our class, we’ll go over one of these. But like I said, class at the time of our session, we’ll go over them, but I just wanna be mindful of our time. Te next thing that we’re gonna talk about is guided notes and note taking. Guided notes is something that I learned whenever I was working in higher ed. I worked in higher ed for about eight years. And guided notes are basically, they are prepared handouts that I provide to my learners. And the purpose of the guided notice is to give them some structure to their note-taking. I don’t know if anybody’s heard of the of the phrase, you know, trying to drink from a fire hose and, but basically if you’re brand new to technology, if you’re brand new to computing, if you’re brand new to AWS, there’s a lot of content just being, you know, coming at you.
Tasha Penwell: There’s a lot of variables, there’s a lot of things you gotta take in considerations and trying to make the right choice, trying to be cost optimized, think, you know, thinking about high availability, all these things that come into play whenever you’re trying to make a decision, and it can be overwhelming. The purpose of the guided notes is to try to give the, the the learners an opportunity to focus on terminology and scaffolding, scaffolding what they’re learning as an iterative process. And in the future, slides you’ll see that there’s a link that gives you an idea of what that looks like, but the feedback from the students ultimately has been very positive. They have admitted that they have poor note-taking skills and poor note-taking habits, and they said that this has helped them give them some structure to the note-taking as opposed to just trying to do it on their own.
Tasha Penwell: I’ll go back to the note-taking section here in a little bit, but I just wanna kind of follow up with the unguided notes. The guided notes, the learner, like I said, the, they have the option to use this. It’s not a requirement, it’s just meant to be a tool to help them with their studies. If they know they’re, they already have a learning style that works well for them, they’re more than welcome to use it. It’s just, that’s another tool for their toolbox if they need some assistance. After the guided notes, we review and discuss, one of the things that I tell my students, first thing is, I hate reading from slides. I’m not gonna read from slides. You guys know how to read, I am not gonna read to you. You guys are expected to review the content.
Tasha Penwell: And then we come in for discussion. We have a review and a discussion, and we also do some knowledge checks. It can be in a game format such as using Cahoot or just do the knowledge check that’s provided by AWS as part of their curriculum. In here, there’s another video link that explains it in a little bit more detail, but in here is just like a really simple sample copy of what guided notes look like. There says there’s a link to the skill builder, which again is available to anybody with an amazon.com account. And there’s three sections to it. The first section is focusing on terminology. All they need to worry about is just focusing on understanding the terms. And then their second section is a short answer. And again, they’re going through the content and going through and, you know, filling in the blanks, kind of like a scavenger hunt. And then the third section is the loan answer.
Tasha Penwell: The expectation is that the students, by the time they’ve gone through the first and second section, that for the loan answer that they, you know, or should be able to answer these without some students who actually take this as a way to quiz themselves. If they’re able to answer the questions without looking it up, then they know they, they have a good grasp on it. If they’re struggling with it, then that gives them a, a barometer way to check to see that that gives them something to look into so that they know that this is something they need to focus on to improve their skills or to improve their studies.
Tasha Penwell: Let’s see, and I’m gonna go back to this. Note-taking, I talk about learning styles a lot. Note-taking is more for like the self-learner. I was, I’m completely self-taught on AWS and I struggled actually to find my own learning style with AWS when learning AWS. Whenever I was in school, whenever I was in high school and college I didn’t really develop good study habits. I’m one of those people that I was talking about have not developed good study habits, and I had to, you know, improve on my study habits because a w s was just, you know, it was challenging to me and that’s why I still enjoy it. It still challenges me, but I, I had to find something that worked well for me and for what I’ve learned is note taking, like improved a lot more on the note taking.
Tasha Penwell: Sometimes I’ll do something more of creative, like using Canva or Figma to create some sort of infographic or even look a little book with characters. It just, it has a creative outlet when I feel like I need to be creative, like I cannot do anymore. Analytical. and the also listening to podcast I’ll give and the, and the slides. There’s some podcast links with aAWS and some other resources. I also encourage my students to make your own flashcards. I encourage you to make them, instead of buying them from Amazon, you can purchase them from Amazon, but with the act of writing out and creating your own flashcards, using the index cards you can get from a dollar store, that is one way that you can, you know, that’s a study that you are studying whenever you’re having to read and write and you put it on the flashcard.
Tasha Penwell: Then whenever you’re reviewing them on your own homemade flashcards, you’re again, you’re you’re improving your knowledge, you’re, you’re re improving your understanding in retention of the information you’re trying to learn. And this again, a video. Well this is one of the feedback that I’ve gotten from my learners about guided notes. And as you can see, she talks about, you know, guided notes have really helped her a lot. And she says that she would quickly lose patience reading the text and will lose track of what she was reading. And I’m the same way, like if I’m trying to read something, I have to make a conscientious effort to ignore, ignore any squirrels or shiny objects that’s coming in my way because I will get distracted. And she likes to fill in the blanks and it shows that the important point to note down and remember.
Tasha Penwell: It gives them, you know, some guidance, something very pointed to this is what they need to focus on, ignore and remove all the distractors and then showcasing the accomplishments. So AWS Educate again was one of those resources that I introduced earlier and one year you are I have my students actually doing this right now with their AWS Academy class. They have courses that you can take and you can earn a badge. So those badges is after, you know, storage or this, these two right here is compute and storage and you can, you know, finish a course in there and you’ll earn that digital badge that is a digital badge that you can share it on LinkedIn and other platforms. It’s a small accomplishment towards your larger goal, which is the certification. So if you go and AWS Educate, these are the different badges that you can do.
Tasha Penwell: And again, there’s a video on it with Skill Builder. You also get certificates of completion after you complete a course you’ll get a certificate of completion again. And that’s something that you can share on LinkedIn. When I worked in higher ed, whenever I completed it, I would actually send it to HR and my dean, who is my supervisor’s, like, Hey, I am doing this. Put it in my portfolio. So whenever there’s an opportunity, remember that I’m doing this. I go, I’m trying to, you know, make, be mindful for time. There’s also Skill Builder learning badges and these are just, just similar to the Educate badges. They’re just a little bit different and they are a lot more on the time. You can see like 62 hours, 11 hours, nine hours, et cetera. But these are also badges that you can learn again and shared on platforms such as LinkedIn.
Tasha Penwell: And there’s a video on that. Building a digital presence using LinkedIn this gives you, so if you’re struggling on how to fill, you know, build your LinkedIn profile, there is some tips on how you can do that if you are an AWS Academy, you know, learner or if you’re in some sort of similar program finding your path. So AWS has some resources, you know, if you’re looking, if you know what kind of role you’re wanting to be in cloud practitioner, developer DevOps, or if you have a particular type of specialty, like I said earlier, minus security, and you can go through here and and I know I’m going fast, but I’m just trying to let and be mindful of the time. They have a ramp up guide. And these ramp up guides can give you some specific pathway of specific resources with the links and the types in the amount of time.
Tasha Penwell: And it gives you some, because it’s a lot of different things if you just try to go into AWS and try to, you know, figure out what you need to know. This gives you a specific pathway based off whatever interest you want to have to your career in and again, minus security. This could give you some guidance on specific resources that you know to help build that career for you to meet your career goals. And let’s see a video on that. White papers, you know, saying awesome with AW s white papers, blogs, podcast. Victoria Seaman, she is she’s an advocate. She’s awesome. If you don’t follow her on LinkedIn yeah, she is just constantly sharing great resources, so I highly recommend checking her out and following her on LinkedIn. And then these are my how you can find me, <laugh>. Thank you.
Angie Chang: We think we’re at time, but thank you so much. I think this was very, very helpful and educational and a great resource. Thank you again for your time and we’re gonna hop to our next and last session of Elevate. Thank you.
Tasha Penwell: Thank you.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Sukrutha Bhadouria: Marilyn is the Senior Director of User Experience at BetterWorks. Her passion is designing and building experiences that delight users and maximizes productivity. She has worked at Mark Logic, Guidewire, Genentech, Intuit, Oracle, and Xerox. We are excited to hear her UX insights. Welcome.
Marilyn Hollinger: Thank you. Thank you so much. I’m very glad to see there are still people trickling in. So I’ll just but I have 20 minutes and those of you have been attending sessions. You know, that 20 minutes is <laugh> very short. I’m gonna go through quite a number of slides. I will talk fast. I’m from New York. I can talk very fast. And at the end there will hopefully be time for questions and if not, my email address will be on the last slide and you’ll be, I’m happy to take questions by email. Feel free to do that.
Marilyn Hollinger: I’m just gonna jump right into this background for me, I started as a developer and doing front end work, working very closely with UX people. Ended up getting doing transition into doing UX full-time many years ago and have been leading UX teams for long, long, long time.
Marilyn Hollinger: I’ve done this many times at different companies and I’ve kind of worked out what I think works best. Let’s talk about how UX teams evolve in a company. Generally the first thing that needs to happen is there needs to be an appetite for, for a team to exist. There are many, many companies out there, including the one I’m at right now, which start, start off with nobody do having the role of, of UX that usually product managers and developers kind of partner on this. If there is a product manager, sometimes there’s just a developer who kind of knows the front end, so really it’s about having people understand what it means to have a UX team. And we’re gonna go into detail on this in a minute. And then showing the value of such a team. Okay, now I understand it, but why do I, why is it important for me to spend headcount and time?
Marilyn Hollinger: Why is it important for me to invest? Cuz value is about your investment and what you get out of your investment. And then working on how a u so now you get to a point where a team can exist. How does that team collaborate with the rest of the organization? And then finally, how can the UX team become strategic partners in the organization? And my background is in companies that deliver software. And so I’m gonna really be talking about this in a software development environment, not so much in a marketing or anything like that, but very much in a software development environment. Although a lot of what I’m gonna talk about is very applicable to different environments. Let’s talk about this.
Marilyn Hollinger: The Nielsen Norman Group highly recommend this website if you’re doing anything in UX has this concept of stages of UX maturity. If we look at the evolution of how you build a UX, it really tracks to this, to these stages of maturity. You have to build the appetite. You generally, you’re building the appetite when you’re in these very early stages where there isn’t a UX team where it’s just people have no idea what UX is. And if you’re very lucky at this point, you might be able to get one person who would be a generalist. And I’ll talk a little bit more about that in just a minute. And really this is about the appetite can come from uncovering negative user experiences, so lemme talk about that. A lot of the appetite comes from people saying, well, they don’t even understand what it means to do. Ux, UX is often seen and I’ll talk about this in a minute as pretty pictures.
Marilyn Hollinger: You need to figure, make sure they know what UX is, understand the benefits of it and overcoming some fear. I wanna get into this a little bit. UX is usually seen as the, the pictures on the screens. Can you design me a dialogue box? Can you make a pretty image for this? Those are really the two things that most people think UX is, and I use UX as user experience as opposed to user interface design or anything like that, because I consider UX is a very all-encompassing profession where you have interaction design and that’s like what controls do you put on the screen? In what order do people click on things? And that’s interaction design. And then you have to make it look nice and that’s the visual design. And behind all that, you really have to do research to build software that is to build designs that match user expectations that match the environments they work in, that answer the questions they need, answers that get the jobs done, they need done, not just, this sounds like a good idea.
Marilyn Hollinger: Let me give you an example. I was doing a, a UX design for people who determine insurance rates. I know that sounds really dry, but it was great, great gig at Guidewire. And in fact, I got a a patent for this. And what we did was we talked to people who do that which are actuaries and actuaries. It turns out live and breathe Excel. So we and had we not talked to them and done the research, we wouldn’t have known that. So the interface that we designed, we made it as close to excel as we could, and we allowed important export to and from Excel so that actuaries could work in their native environment and then bring that into the tool at Guidewire, so really important to understand what people, where people live, and then what words do they use when you talk, when you talk to actuaries, they use various terms and we had to make sure that we knew what terms to use so that our design could, could parallel that how they think.
Marilyn Hollinger: The information architecture, so how we lay things out and what hierarchy could match what they were doing. And this is super important to understand all of these different aspects of UX and be able to explain that to people if you’re evangelizing. So the other thing is what is why is why should we do this? What’s the value? Okay, so there’s a couple of values. One is if you design things better, the company does better. And this is super important. This is you know, these statistics are a little old at this point, but it really shows that design centric companies do better because their products do better. And there’s, there’s a lot of literature out there about this and there’s a cost involved. So there’s be cost and benefits and that’s what value’s about. And there’s many, many iterations of this particular picture out there in the, in the universe.
Marilyn Hollinger: This is kind of a short one where, you know, customers can explain what they want and it’s often not what they end up with because we don’t actually do the research to watch them work. Well, I remember going into, again, at Guidewire going into insurance offices and listening in on people’s phone calls and watching how they worked to see what it was that they really needed. And or else you’ll end up with who knows what, because, you know, marketing people and salespeople will come up with a thousand different features and really the customer needs three of them. There’s cost of, if you don’t do that, user analysis and design. And this is where we really want to explain that, to help build that appetite to show, you know, if you don’t build what users want, your your first version is not gonna sell and you’re, and you may be dead at that point, your company may be dead.
Marilyn Hollinger: Here we’ve got a quote because there’s a fear involved. There’s a fear that, oh, if we, if we put this extra thing in our process, we won’t get our software out as in as much time. Well, if you think about the software engineering process in good software engineering, you do, you do architectural design upfront, you do your implementation, and then you do code reviews, right? And you wouldn’t think of skipping and then you do QA and you wouldn’t think of skipping any of those parts. It doesn’t make sense to skip any of that because your quality will go down. And it’s exactly the same argument with user experience. If you don’t take the time to design it right, then your development will have to be redone and you have to redo that whole cycle. When you talk about in the software engineering process doing the architecture right upfront saves you in the long run go back one step, even doing the user experience, right?
Marilyn Hollinger: Saves you in the architecture, which saves you in the development cost. I’s understanding that that’s doing making sure that you do it right or else you’ll, you know, it doesn’t matter how fast your product gets out there, it won’t succeed. Once you’ve got some kind of appetite we’ve also, we’ve talked a little bit about the value, but what, what usually I’ve seen is at the very beginning when you’re building the appetite, you’re, you’re pointing out bad user experiences and say, you know, we could have done this better. And then you say, and this is how it is better. Now you start looking at touching those user experiences to improve them and say, this is what we have. We’ve gotten a lot of complaints about it. What here is a redesign? And you see how much better it could be, and the little light bulbs start going on for people.
Marilyn Hollinger: Okay? And, and this is where you point out, look, we have one function that works away this way on this screen. And look, that same function works differently on another screen. It’s not consistent. Look, we can have one design and users will be able to use them better because they’ll, they’ll understand how to use that same function in both places. And this is where your team might grow to just be kind of generalist still, but having strengths in those three areas. Like one might be a slightly better visual designer, one might be an interaction designer and one might be a researcher, for example. But they all have to be able to cross Polly. When I joined Guidewire, I was the only UX person there and I, and I am in those three areas. My weakness is in visual design. I can do it, but I’m kind of the B team.
Marilyn Hollinger: When I got hired on there, I said, you know, I’m gonna have to spend some money to hire a visual designer to assist me when it comes to visual design issues. And they were fine with that so they understood what they were getting there. And then my next hire was very strong in visual design, although she could do interaction design and research. You build up the team with these different strengths and also you show over and over again, look how we can improve existing user experiences or you build new experiences that are people look at and go, oh, that’s great. And that it, it’s it feeds itself. As you design better experiences, you get more support, and you can build better experiences when you’ve got a team in place. That’s the time where you really have to think about structure.
Marilyn Hollinger: I’m gonna talk about that in just a minute. How you actually structure your team. This is where you really need to have processes where in place to make sure that you are part of the team collaboration. What I mean by that is that there is and we’re gonna talk about this in just a minute. There’s early on ideation, then there should be design and then architecture, and then development, and then QA. So there, it’s a linear process. I mean, it repeats itself in cycles and there’s, there’s agile work and things like that. But it’s basically that same, that same process and figuring out where UX fits in with each of those processes is very important. And that’s what allows you to design truly outstanding user interfaces. And at this point in a team, you start to have more specialists, start to have people who are like the UX research lead, okay?
Marilyn Hollinger: Or the visual design lead who owns your style guide, for example, okay? And then when the, the organization truly has maturity in terms of UX, you start to be a strategic partner, you start actually saying, here’s, here’s a whole initiative that’s going to drive the product. Here’s a whole area of the product that we’re not doing, but it will make the overall experience better. And this is where people wouldn’t even dream. Actually, hopefully in the, in the step four people wouldn’t even dream of not having the UX team. It’s very, very important to to sort of strive for this, where you end up having an architect who oversees everything. You have specialists and you’re really a strategic partner in the organization. And in organizations that are mature that I’ve worked in, I I sit, I help drive product content because that’s part of the strategy.
Marilyn Hollinger: User experience needs to be a core part of the strategy. Let’s dive into this a little bit more in terms of process. I mentioned this a little while ago. There’s, there’s the product visioning. What are we gonna build? What should we be building? Not just, I mean, it could be very broad. What product are we building? Or it could be what features do we need to add or what new areas or new ways of thinking? And then there’s the design before build process, which is before anybody starts coding anything except for proof of concept or anything, you do design work, you don’t finish it. But I always think about it as, as building a house, you would never bring a carpenter in until the architect is done. And it doesn’t need to be quite as baked as that. Because there is, there is agile processing that you can do.
Marilyn Hollinger: You can, but you have to have a pretty good idea of what the house is gonna look like. <Laugh> before you start, you know, hammering and nailing. And then during the build process, you need to have design come in because there are gonna be changes, there are gonna be misinterpretations of the design, there are gonna be use cases we didn’t think of. So there’s a partnership that happens there. And then there’s an acceptance time when you say, yes, this is the user experience I designed, you have built it the way I, I spec it, or we discussed it and let’s ship it. Let’s go through each of these. So in product visioning, this is really, it’s generally, now I’m gonna not say how it should be necessarily, but I’m saying how it generally works. Well, and this can be different at your company, there can be other people involved in this, but ascent, but how I’ve seen it be really successful is you have an owner for the product vision, generally product management and ux.
Marilyn Hollinger: And you do this together. And there are other people involved in this. Usually you have customer people, you have sales, you have marketing bring giving input, but the key people who decide where are we going with the product, 10, if you have a partnership between product management and UX, that works really well. And this is where you say, what are these cases? What problems are we trying to solve? What are the requirements there? And really develop that and, and spec it. Write it down because we all speak different languages, not just, not just actual com human communication languages, but I look at things from a user perspective and an artistic perspective, and a product manager looks at it from the functionality perspective and a customer perspective. And I do too look at it from a customer perspective, but we don’t often speak the same language.
Marilyn Hollinger: It’s important to write it down. You can’t, and and this is where like taking minutes and meetings where you have agreements, this good meeting management underlying all of this. And there’s tools that can help you. Often you’ll have there’s tools like aha, I’ve got some tools listed along the bottom of some of these slides. Putting together road maps to say what we’re gonna do over different quarters. For example, writing requirement specs. And by the way, good requirement specs don’t say we need a button to blah, blah, blah, blah, blah. It should say we need to do blah, blah, blah, blah, blah. So requirements are not about design. They’re not about what they’re about, the what we need, not the how we’re gonna do it. And there’s, there’s whole classes on doing requirement specs. I’m not gonna go into that. This is this type stage that you have, you haven’t already.
Marilyn Hollinger: You do the research, you go in and you talk to people and you do the things we talked about before. How do they speak? What do they need? What’s missing in our current product? And it’s really, really important for the UX team to get firsthand exposure to those customers, to be able to have a conversation with them, not have it filtered through a lens of, of customer people or product people or whatever. But to actually hear those people talk and be able to ask them questions, very, very important at the product visioning stage. And also that comes in later at the design stage. I can’t design for someone that I haven’t actually had a conversation with. It just doesn’t work very well. That’s product vision. In this decision making, often you when you decide about what you’re gonna put in the product, it comes from a lot of sources, it comes from the competition, it comes from, you know, there’s sort of just boxes I need to check off to make sure I can even get in the, in the game.
Marilyn Hollinger: We need to fix bugs. We, and then there’s this concept of, oh, this would be cool if we did blah. And all the use cases, all the things that people could think of about doing with the product tend to be created equal when you don’t have that, that UX perspective. Oka? Especially when it comes to engineers. Engineers can think of the 35 different use cases and, and they wanna build all of them because they tend to, and I’m, I’m doing a vast generalization here. I know, no offense to anybody, I used to be an engineer, but really it’s like, well, here’s all these different options. I, I need to code all of them. But what we really wanna get to is keeping track of that competitive advances, advanced, excuse me, advantages, making sure we are checking off the boxes, making sure we are fixing the bugs.
Marilyn Hollinger: But what UX brings to the table is what problems are we solving? What, how can we streamline the, the most important use cases to make those easy, the happy path. How can we keep different parts of the product consistent? And making sure that UX bugs are fixed too, not just functional bugs. Okay, so this is where we wanna make decisions about what goes in each release. Bringing both lenses to the table, take a breath, moving on. Design before build, this is about, then now it’s the UX team. The UX team. In the earlier stage, the product management team generally drives, in this stage, UX generally drives and it’s a partnership. Okay? It’s really important to have a design library to say, okay, all of our buttons are gonna look like this and all of our searches are gonna work like this.
Marilyn Hollinger: And all of our, our wizard interactions are gonna be like this so that you get consistency and you align out with development. You say, okay, you guys build the button once and everybody uses that same button, and we’re gonna talk about that in just a minute. It’s important to track this work to make sure it’s really, really common for UX person to be working on something and have a product manager go, I need you to add this other feature to this other thing and mock that up. <Laugh>, okay? And so they get sidetracked. UX tends to be very responsive in general. And so of course we say yes, right? <Laugh>, but it’s really important to track that. So when you get dis behind in your, in your design on this thing, because this other thing came on, it’s really important to track that and be able to say, Hey, this, and, and to ask for prioritization, okay, you want me to add this new feature?
Marilyn Hollinger: I can add that new feature, but this other feature that I’m working on is gonna slip. And what I often see happening with UX teams is that things just get, keep getting piled on. And so it’s really important to be able to say, you know, Susie is already working on these four UX tickets, which one of these is lower priority than this new thing you’re asking about. Is it more important and to get actual sign off to say we are done? We do that in, I’ve done this in several organizations where we do this by having UX tickets for all the work. We often using Jira all the work with links to the actual designs and something like Figma Sketch or InVision, okay? And having phases of it’s in progress, it’s being reviewed, I’m awaiting input. Because often you need like a developer to say, is this even implementable, for example, or is this a key feature?
Marilyn Hollinger: Or you need something about terminology from a customer. There’s a waiting input is another is another status. And then design complete. And that’s when everybody’s given a thumbs up. And so there’s a very visible way to say, I’m done with this, I’m putting this aside and moving on to something else. And then anyone who can look at the, who needs to look at the design can go to the design complete ticket look, which is now linked to the actual design and, and see everything. Very, very important that anybody outside the UX team needs a way to find all of your designs in a very structured manner and be able to see what state they’re in. I highly recommend Jira or a similar type of tool for this. The designs can be in whatever you want to use. I’ve, I’ve named Figma and Sketch and InVision.
Marilyn Hollinger: You can use whatever tool you want as long as there’s a way for people to get into the design, see them and comment on them. That’s really important. There needs to be ongoing commentary about your designs, preferably in the designs themselves. Figma, Sketch and InVision, they all allow that. And if you’re using a different tool, having that functionality is important. Making sure that they’re that you look at competition make. And there’s this concept of sprint zero.
Marilyn Hollinger: There’s a lot of folks who do agile who don’t wanna do waterfall, don’t do this in any particular order, but I’m sorry, you need to do design before you do development. And maybe that’s three sprint zeros depending on the side length of your sprints and the amount of design work to do. You have to be planning. I like to tell people you have to plan at least a quarter in advance to do your product and UX work before you start your development sprints, at least, if not more, depending on the size of what you’re working on.
Marilyn Hollinger: Okay, moving on. During the build process, again, now we went from product management, sort of owning the, the planning process to UX owning the design process. Dev owns the, the build process. They’re actually putting their fingers on the keyboard. And this is where we really highly recommend to get good UX, you have common base set of components that the development team built uses. Everyone’s using the same buttons, everyone’s using the same search search functionality, so UX is not designing custom buttons or custom dialogues or custom anything for standard components. So I really like to track the componentry with the design, with the development component library. And now you have to have dev tickets, right? And we’re, a lot of us are familiar with this. Often you use Jira for this, but what you need is a designer for every dev ticket that has UX involved.
Marilyn Hollinger: Obviously backend tickets, you don’t have to, but you need a designer who’s watching that ticket and tracking it. And it, there’s now a conversation, there’s now this teamwork where the designers attend the standups. There’s active conversations between the developer and the designer. If there’s any questions on the design that you, I’ve, I’ve seen organizations where the developer goes to the product manager who goes to the designer, who goes to the product manager, goes to the developer, and you end up with this great game of telephone. And, and that’s just not okay. You need to have this be a team environment. And often you’re working on agile sprints, and that’s perfectly fine. There’s nothing that that gets in the way in terms of UX around that, but UX needs to be a part of that conversation during the build process. Okay? And finally, there’s an acceptance criteria.
Marilyn Hollinger: And what I mean by this is a built-in process that tickets get stories, get built by developers, and then when they say they’re done, it’s an automated process for UX to, to jump in and do a review. What I’ve got on the bottom here is this is a suggested flow for Jira tickets where you create the ticket, you set UX required, yes or no, okay? And then the build happen after design <laugh>, the build happens again, conversation’s going on. And when the build, when the developer says, yep, I think I’m finished, then it, if there’s no UX required, it just goes into QA or whatever your process is. If there’s a UX required, then there’s a UX review. And I’ll talk about the SLA in just a minute. The UX review, the UX person has to have, so the, by the way, the classification, that’s the UX required, the UX person needs to have the authority to say no, to say no, it’s not done, and be able to send that back to build.
Marilyn Hollinger: In the current organization I’m in, the UX member gets a notification when it goes into, we have a, a design acceptance phase for the ticket, and they get to do one of two things with it. Well, first of all, they can have a conversation, ask questions, they have to be able to review the, the design, and then they have the authority to send it back to impart if they know you’re not done and they add a comment about why, and they, then they have the authority to send it on to QA.
Marilyn Hollinger: It’s really important that the UX person has that authority and that they have an environment that they can do a real time review of, of the, the whatever it is that they’re reviewing. And that can happen in multiple ways. That can be very, very fast. Sometimes we just jump on a quick zoom call, the developer shows, the, the designer designer asks questions. They’re done in five minutes. Sometimes it’s a, we need a whole test environment. And so we’ll do that and have a whole test environment. So, but you need to have some way to do that. I’m sorry, we’re at five minutes. Yeah,
Sukrutha Bhadouria: We’re, yeah, we’re five minutes fast.
Marilyn Hollinger: I’m good.
Sukrutha Bhadouria: Shall we wrap?
Marilyn Hollinger: I need, I think we have, I have four more minutes, don’t I? I’m almost done.
Sukrutha Bhadouria: Okay. Right.
Marilyn Hollinger: I’ll be done shortly.
Sukrutha Bhadouria: Okay.
Marilyn Hollinger: I will be done shortly. I thought I had to 1:50.
Sukrutha Bhadouria: We’re we’re past. Yeah, we’re past time. We’re overtime right now.
Marilyn Hollinger: I thought it was 50. It was 20 to 50. No, it’s 20 minutes. Okay. I’m gonna, there’s my summary, there’s my email address. We’re done. I’m sorry, I, I got my timing off. You’re right, it’s 20.
Sukrutha Bhadouria: Minutes. No worries at all.
Marilyn Hollinger: Very fast. No questions. I apologize. Please send me email if you have any questions. And, and thank you so much for, for your attention and your attendance. Sorry.
Sukrutha Bhadouria: About that. No, thank you, Marilyn. Please do reach out to Marilyn via her emails, take a quick screenshot and we’re going to move on to the next session. Thank you, Marilyn.
Marilyn Hollinger: Thank you. Apologies.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Angie Chang: I have with us today Sheri Byrne-Haber, who’s a Senior Staff, Accessibility Architect at VMware. She’s a prominent global subject matter expert in the field of disability and accessibility, and known for launching digital accessibility programs at McDonald’s, Albertsons and VMware. And she writes a popular blog called “This Week in Accessibility”. Welcome, Sheri.
Sheri Byrne-Haber: Well, thank you so much, Angie. I’m really excited to be here. I always like to drop the Sheri’s secret fun fact before I start events like this, which I was the first Girl Scout in the US to get a badge in Computer Science coming up on my 45th anniversary of that event this August. I’ve been doing tech for a long time.
Sheri Byrne-Haber: I did my first degree at Cal in computer science back when it was, you know, 90% guys, and I was basically the diversity in the room. Been doing this for a long time. Went and got a law degree 10 years after my computer science degree, then did an MBA 10 years after that. I’m here today to talk to you about why access. Yeah, Go Bears, Angie <laugh>. Why accessibility, what it is.. Okay, so the the brief 50,000 foot version and why it is a great career, especially for women.
Sheri Byrne-Haber: When most people think about accessibility, if they’ve heard that word, accessibility means making stuff work for people with disabilities, that’s kind of the TLDR version. They think about visible disability. You might think about somebody with a prosthetic arm.
Sheri Byrne-Haber: This is actually me practicing in my wheelchair on my Olympic range at home. I’m trying to qualify for the 2024 Paralympic games. People with service animals. People with hearing aids. Something that you can see. Accessibility has to take care of a lot more things than that.
Sheri Byrne-Haber: First of all, we have to deal with hidden disabilities, disabilities that aren’t obvious, that can’t be seen. That might be, I tell people all the time, you see me in a wheelchair, you assume, you know what my disability is, right?
Sheri Byrne-Haber: My real disability is type one diabetes kicks my ass on a daily basis. It interferes with everything I do. My wheelchair is just a way to get around. And I’ve been doing it for a very long time.
Sheri Byrne-Haber: You need to think about hidden disabilities. And some examples of hidden disabilities include Millie Bobby Brown, who’s deaf in one ear. Bono wears tinted sunglasses because he has a glaucoma. It’s not a rockstar affectation.
Sheri Byrne-Haber: Neurodiverse statuses. Mental health issues. The reason why all of the colors in Facebook are blue is that it’s the only color that Mark Zuckerberg sees. When you’re thinking about disability for starters, you really have to broaden the definition to make sure that you’re including both visible disabilities and invisible disabilities.
Sheri Byrne-Haber: Then you need to add two different types of disability. A permanent disability might be a limb difference, but if somebody tears their rotator cuff temporarily, they’re gonna have the same disability as somebody with a limb difference. They’re not gonna be able to use their arm or situationally you might be holding something that prevents you from using an arm.
Sheri Byrne-Haber: When you take permanent plus, temporary plus situational disabilities and, and look at it from both the visible and the invisible perspective, you’re talking about 30% of your potential users. And accessibility is about making technology work for that 30% of users.
Sheri Byrne-Haber: Okay, so what do accessibility testers need to learn? First of all accessibility testing is a lot about interacting with assistive technology. You may have heard from other people talking about software testing as a field that automating is the greatest thing ever because then you can just push a button and repeat all those tests and not have to do anything that requires manual intensive interaction. It’s not so easy to do with accessibility because only about 30% of the tests can be executed in an automated manner by inspecting the code.
Sheri Byrne-Haber: 70% actually require being able to interact with the assistive technology. And so that includes things like screen readers which is what the woman in the middle graphic is using. She’s listening to her iPhone, tell her what’s on the screen in front of her that she can’t see. Some other forms of assistive technology are not using a mouse. Using alternative input devices like keyboards touch pads you know, those graphics pens things of that nature captions, magnification.
Sheri Byrne-Haber: Then we get into a little bit more obscure, slightly less used assistive technology that would include things like sip and puff devices, which is how people who are quadriplegic interact with the internet. Obviously speech recognition is becoming more and more popular and, and actually better and cheaper than it used to be in the past. Once you know how to use assistive technology, you have to learn about the accessibility guidelines.
Sheri Byrne-Haber: There’s something called WCAG, which stands for Web Content Accessibility Guidelines. The version that’s just about to come out is version 2.2. And that is a standard WCAG that has been adopted pretty much globally. Anywhere that you have a law that requires inclusion of people with disabilities, usually it references one of the WCAG versions, not always the same version. That’s would make things too easy, right?
Sheri Byrne-Haber: The EU, Canada, Australia, the us India, some countries in Africa, they all use WCAG as the standard to determine whether or not you’ve made something accessible enough. That is, that, you know, the majority of people with disabilities would be able to use it just as if they didn’t have a disability. These are the two basic things that entry level accessibility testers focus on.
Sheri Byrne-Haber: What do they do once they know how to do all that stuff? Well, they participate in designing, building, and testing software, but a hundred percent through the lens of accessibility, not whether or not does it work which is the functional side of the fence, but does it work with assistive technology that people with disabilities are likely to use? And do those people, are they having an equal experience? Okay.
Sheri Byrne-Haber: Those are the two things that the lens of accessibility provides for an accessibility tester. Other than that you’re participating throughout the entire life cycle, just as if you were a, a designer, a builder, or a tester. You’re just looking at it with a very particular point of view, okay? Women are really well represented in the accessibility space. There’s five times as many women in accessibility as there are women in non-accessible roles, just traditional straight up software testing you know, analytics coding, program management, things like that.
Sheri Byrne-Haber: It’s actually a good place to be because there are other women that can help you support your careers who have been there and done that. And you, you may get a better level of, of understanding from getting mentored by other women than you might be by getting mentored by somebody who doesn’t have the lived experience that you do trying to survive in your career. Okay?
Sheri Byrne-Haber: There is a significant demand for accessibility testers. Unless you work for Elon Musk, chances are you are not gonna get laid off, and that’s because the demand for accessibility testing is being driven by regulations and litigation, especially in the us. So the Americans with Disabilities Act require it, it, the language of the law itself doesn’t require accessibility, but it requires equal access. And the litigation, and we have about 4,000 plus or minus cases per calendar year in the US is focusing on WCAG as that standard to determine whether or not something’s accessible enough.
Sheri Byrne-Haber: As long as there’s laws and there’s long as there’s litigation, there is going to be a demand for accessibility testers. And right now we’re in a place where colleges are not turning out a lot of people skilled in accessibility testing because it’s not required as part of the computer science program. You barely even touch on it if you’re in a graduate HCI program. This is something that’s very much self-taught, and to be honest with you, it’s also very much passion driven.
Sheri Byrne-Haber: A lot of people get involved in accessibility because they have a personal experience with the disability. Again, don’t make assumptions. People see the wheelchair and they’re like, ah, I know why Sheri got into accessibility. Now, I actually got into accessibility because I have a deaf daughter and my deaf daughter you know, experienced a lot of issues when captions weren’t made available to her.
Sheri Byrne-Haber: There a lot of times there’s this, like I said, personal connection that makes people passionate about being in this space. Keep in mind disability is the only dimension of DEI – diversity, equity, and inclusion – that everybody is guaranteed to experience at one point in time or another in their lives. Unless you die getting struck by lightning, never having broken a bone in your life, chances are at some point in time, if you’re not disabled right now, you are going to be disabled.
Sheri Byrne-Haber: When you’re working inaccessibility, you’re working to make the make the place better for your future self. That’s a, that’s another way to look at it if you don’t have a personal connection to disability currently. Okay. being disabled is actually a bonus when you’re working in the field of accessibility, because not only are you bringing the things that you learned about screen readers and, and other assistive technology and the things that you learned about WCAG, you’re bringing lived experience.
Sheri Byrne-Haber: And that’s something that’s very valuable for this type of work. The other thing is work from home has been a thing for people in the field of accessibility. Long before the pandemic 30% of people with disabilities can’t drive. And so work from home is critical, especially if their disabilities prevent them from being able to commute or make it harder or more expensive for them to commute.
Sheri Byrne-Haber: Other than the usual, you get paid well, it’s a fun job. You get to make the life lives of other people better. But this is, this is somewhere where we’re having a disability and being willing to talk about that disability actually helps. And if you need to work from home or if you would benefit from to work from home it’s something that the accessibility managers in the world are very accustomed to.
Sheri Byrne-Haber: There are a broad range of employment opportunities government and education anything attached to federal money, okay? Including money that passes through states and cities and counties has to be accessible. There are strings attached, and those strings are called Section 508. Universities have to make things accessible. Hospitals have to make things accessible. Courts, anything municipal, anything federal, all has to be accessible. The nonprofit space also wants to be accessible because they don’t wanna say, oh, we’re here to help out this group of people, but hey, you people with disabilities, you get in the back of the line. There is typically you know, NPR has somebody dedicated to accessibility. Washington Post, New York Times, they all have accessibility specialists. Those aren’t exactly nonprofits, but it’s places that you see accessibility thought about where you might otherwise think that it wouldn’t be addressed.
Sheri Byrne-Haber: There’s lots of accessibility consulting companies all the retail operations on the internet. If you’re selling in the us it has to be accessible or, or you’re probably going to get sued at one at some point in time. And then, as I mentioned, healthcare is another big field. For each one of these areas, you’re still taking the same domain knowledge that you have on assistive technology and the WCAG guidelines, you’re just applying it to one of these vertical markets. It does not take a whole lot to get started. It doesn’t, being in the field of accessibility does not require a college degree. There are apprenticeship programs for people who wanna get started in accessibility. There’s quite a few resources that are available for free or for low cost online.
Sheri Byrne-Haber: You don’t have to go out and get a college degree in accessibility. In fact, such a thing does not exist. What you have to do is you have to care enough to go learn about all this stuff yourself, invest time, go to meetups, talk to people who are already in the field. I think of accessibility today as where Quality Assurance (QA) was, you know 35 years ago when I had just graduated from Cal 35 years ago for QA, there were no degrees in QA. There was no Six Sigma. These things didn’t exist. You had to apprentice yourself basically to somebody who was really, really good and, and learned from them. And now you can get a degree in QA. You can get all kinds of certifications in QA.
Sheri Byrne-Haber: Accessibility today is where QA was 35 years ago in terms of how to, to, you know, get your foot in the door for the career, so to speak. You can easily evolve from accessibility into more senior careers. A lot of people who spend three to five years in accessibility will then move on to design or UX or UI and front end development, because you will learn a lot about these three things as you’re doing your accessibility testing work. And so if, if this is, if you’re interested in these three areas but don’t have the time to go back to get a degree or go to a boot camp or something else, you can use accessibility as a way to get into the door for some of these other careers, there are I’ve got here a list of some starting points if you’re interested in accessibility.
Sheri Byrne-Haber: Siri was actually invented for people with disabilities. And the iOS voiceover, which is kind of twined with Siri is the screen reader. If you don’t have an Apple platform, then NVDA is a free screen reader that you can use on Windows. Spend an hour not using your mouse. Lots of people can’t use mice. I can’t use a mouse because I’ve got pretty bad arthritis in my hands. That will give you a pretty quick perspective on what it’s like to be a keyboard only user.
Sheri Byrne-Haber: There’s a couple of places that you can register to be a crowdsourced accessibility tester that will help you learn more about how to find bugs, how to report bugs what is it that people are looking for. Most major city centers have a Lighthouse for the Blind, or Center for Independent Living. They usually have ways that they can point people to learn more things about accessibility. And we’ve got meetups all over the place.
Sheri Byrne-Haber: Thanks to the pandemic, most of the meetups are actually now hybrid. If the accessibility group in Orlando is meeting up, you know, it doesn’t matter you live in Portland, you can still go because they’re, like I said, largely hybrid these days. I wanted to give people my contact information.
Sheri Byrne-Haber: I have a website, which is sheribyrnehaber.com. It’s got a free archive of all the blogs that I’ve written over the years about accessibility. There’s probably 200 or 250 articles on there right now. If you don’t see something, ask me because I’m still writing, I’m not writing quite as much as I used to but I get a lot of my ideas from people pinging me and saying, well, what about, you know, how do you make a toast message accessible?
Sheri Byrne-Haber: One of my most popular articles I ever wrote came out of a question that somebody gave me on LinkedIn. And if you use that QR code, it should take you to my LinkedIn profile. I don’t use Twitter. LinkedIn is my only form of social media but I love to connect with people who are interested in accessibility, and you can always ask me questions.
Angie Chang: Thank you, Sheri. That was a really informative talk and accessibility. I love all the resources and the, the the knowledge you dropped on us today. This is the last talk of this career track. Thank you so much for being a part of ELEVATE and for everyone who’s still here with us after two days of nonstop talks, developer workshops, networking, meeting recruiters.
Angie Chang: Thank you again. Networking is gonna start. We’re gonna have some fluid networking, so if you’ve seen Everything Everywhere All At once, it’s gonna make some sense to you, or I think it makes sense to someone who’s in that movie. It might be something else to you. I’ll see you in networking. Thank you Sheri, for being so open and willing to connect on LinkedIn. And yeah, I’ll see you on the other side. Thank you.
Sheri Byrne-Haber: Much. Okay. And I just answered the one question that came in about IAAP certifications. Disclaimer, I’m on the certification committee. I actually help write the test. So yes, I believe that they’re worth worthwhile. They are standardized. They’ve been around for going on seven years now. But they’re not cheap, right? It’s $375 to take each of the tests. Plus, you know, if you wanna sign up for a membership, that’s another a hundred bucks.
Sheri Byrne-Haber: If you can’t afford the IAAP memberships, another path you can go is with the US Federal Government. It’s called Trusted Tester. It’s free, it requires a significant investment in time. Took me about 120 hours to complete it. Most people actually go faster. I had to struggle to unlearn everything that I knew and only respond in the way the government wanted me to respond. That was really hard to do. But if you’re new to accessibility, you should actually be able to get your Trusted Tester certification faster.
Angie Chang: Thank you.
Sheri Byrne-Haber: Thanks everybody.
Angie Chang: Thank you.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Sukrutha Bhadouria: I hope you’ve been having a good session so far. Good time so far in the conference and we are ready for the next session. Thank you all for waiting for us. Rashmy is here to give us our next talk.
Sukrutha Bhadouria: Rashmy is a manufacturing test engineering manager working at robotics and digital solutions at Johnson and Johnson. In her career over the last 13 years, she has worked on consumer products, trained signaling, and more recently robotic applications and medical devices. She’s passionate about making an impact on our society with technology and helping fellow women in tech in their journey. Welcome, Rashmy.
Rashmy Parimi: Thank you for the kind introduction. Hi everyone. I’m Rashmiy I am part of the robotics group in Johnson and Johnson currently working on the manufacturing side of one of our new robotics products soon to be released to the market. Through this talk “Dr Robot Will Now See You” I’d like to transport you to this future vision where this will be a more accessible reality for a lot of people. <Laugh>.
Rashmy Parimi: I want to go back a little in the history before I transport you to where we are today and what the future looks like. A lot of you must have seen this picture on the left of an early operating room where surgery was more of a spectator show. Antiseptics and anesthetics were not something of commonplace. There was no concept of sterilization and for a lot of, I would say decades back then, laughing gas was a commonly used anesthetic.
Rashmy Parimi: Even that was not highly recommended because there was mixed feelings either by the patients or the doctors to use it. A dentist came across ether being an effective anesthetic and he compelled the rest of the medical community to conduct a clinical trial to give more substantial data. And that was one of the starting milestones of making anesthesia a regular process of surgery.
Rashmy Parimi: I think the data convince people that one anesthetics are good. They’re not necessarily something that take you out of control. And also convince surgeons that they didn’t have to resort to methods like strapping down the patients, to help them go through the surgery, because without an anesthetic, the pain will make them move and that’s not something ideal. And they also felt that having a PA stable patient would give them more dexterity and stability to operate.
Rashmy Parimi: That was a very fast history of surgery back then. But from then to now, like there’s so much, you know, medicine has gone grown from deeps and bounds increasing human lifespan by at least 30 years.
Rashmy Parimi: And even today, I think the whole fascination with watching surgery has not gone away, but it’s a little more, I’d say refined from how it was in the photo depicted on the right towards, sorry, on the left to where it is on the right where there is more advanced rendering of the surgical procedure, either during to help other specialists participate in it or to a surgeon or a medical team in a far away location to help add more perspective to a complicated situation.
Rashmy Parimi: From a very low out like low outcome pain causing and a long recovery method to introduction of laparoscopy and endo, which has improved patient outcomes and reduced the recovery time and also improved the accessibility to a lot of people for complicated procedures. This is where I think with this is what most people are familiar with and laparoscopic was what sewed the seeds for the first ever use of robotic surgery.
Rashmy Parimi: This particular arm is maybe familiar to a lot of people as something used in, you know, large industrial assembly houses for large scale manufacturing, more like you know, car assembly facilities or other large equipment facilities.
Rashmy Parimi: But you’ll be surprised to learn was this was one of the first experimentations of whether robotic surgery can be used or not. And you will be even more surprised to learn that the area in which this was used was brain surgery. <Laugh>.
Rashmy Parimi: This was used to guide a percutaneous needle to do brain biopsy back about more than 25 years ago. And then this concept was further expanded to a colostomy and TransU urethral resection to further peak people’s re and research group’s interest to develop the concept of robotic surgery even more and work towards bringing it from a lab prototype to more of a reality.
Rashmy Parimi: In 2000, one of the pioneer companies of robotic surgery, Intuitive Surgical, they broke the ground finally when their system, the first ever Da Vinci system got FDA approval for general laparoscopic surgery.
Rashmy Parimi: It was this innovative device with robotic arms with visual systems and also they had help from nonprofit scientific research organization, SRI, to help them advance a lot of these initial prototypes. And that’s was how most people today, if they are familiar with robotic surgery, I think this is the one name they recognize instantly.
Rashmy Parimi: Let’s talk about what are the advantages of robotic surgery that makes it so attractive to use when evryone would admit that laparoscopy already takes us through a good bit of path onto, you know, smaller incisions and all of that.
Rashmy Parimi: We still get the same advantage as laparoscopy that is a smaller incision, which means quicker healing, lesser hospitalized time, which I’m sure all of you will, you know, relate to the expensive insurance bills and not having to deal with that. And also, it is cost saving and the body will recover faster through a smaller incision, since the amount of trauma is less.
Rashmy Parimi: The other advantage is the precision the instruments can reach into hard to reach places of the body without having a wide incision with accurate precision and stability, which makes a big difference in terms of your outcome of the surgery. And also with this precision al the comes with it, it adds an extra, I’d say boost to the surgeon’s abilities and gives them the confidence to tackle some really tricky procedures.
Rashmy Parimi: One of the important things of having a successful surgical outcome is good visualization. When you know you cut a part of the body, there is obviously going to be blood involved, and in typical surgery it could a lot of times block the view of what is going on there, but with the time your incision smaller cuts, that disadvantage can be overcome and it leads to a better outcome.
Rashmy Parimi: There’s a good example that I would like to use for how pressure virtualization improves the surgery. Having robotic vision is like if you want open surgery is like using a flashlight to look through a window into your house, while robotic surgery is like opening the door, turning on the lights, and then trying to look at your house. You can see it’s evident, which is a better way to look at your house.
Rashmy Parimi: And that advantage is offered to by the advanced imaging that comes with robotics surgery and with, in addition to all of these, the other advantage is exceptional dexterity. Everyone is familiar with how surgeons have these long schedules and if things do not go as planned, there is a lot of fatigue on them with the long hours and that can lead to that showing up on the surgery itself.
Rashmy Parimi: With robotic surgery, one of the things that can be controlled is to remove the tremor and other fatigue related impacts so we can reduce these inadvertent punctures or nicks which can cause unwanted bleeding into the body. Let’s look at few of the areas where today robotic surgery is used in one way of the other heart surgery where these very precise repairs that are needed is done using robotics stomach, though it looks like a big area, there is a lot of fine precise procedures that can be done in a better fashion using robotics.
Rashmy Parimi: General surgery of course, is another area where with a smaller incision and the precision offered, you can do a lot more compared to non robotic surgery. And same goes with the area of GY gynecological surgery where there is, you know, access issues and you want to make sure you don’t impact the healthy tissue or healthy organ parts.
Rashmy Parimi: Same thing goes to lungs where the access is extremely difficult and with kidneys where the, the areas so delicate important that you want to make sure you do not cause unwanted damage to the existing parts. In the area of orthopedic surgery, robotics have given an added advantage of very precise cuts and placement for implants and you know, it’s popularly used I think in hip replacement and knee replacements, which has become very common place in the society today. In the area of dental surgery, there is a product in the market today which help with dental implants and there’s, I’m sure there’s a lot more research going on.
Rashmy Parimi: And as I explained in my first example brain surgery, it started off <laugh>. The whole idea for this was sewn with brain surgery and it is still an area of widely researched today and they are trying to develop products in that area. So here I have some examples of some popular players in the market today. Roughly going over that, the first one is Johnson and Johnson’s robot Monarch, which is, which has FDA approval in the lung cancer and kidney stone management space.
Rashmy Parimi: Below that you have Medtronic’s robot Hugo, which has approvals in the general surgery space. And the picture below is Intuitives’ DaVinci. It’s a newer generation of it, which also has approvals in general surgery and a lot more areas on the right hand side. The first one is the Yumi robot, which is used in the dental surgery field. Their application right now is in the area of implants. The one below from Striker is the maker robot used for the orthopedic area. I don’t want to guess the wrong thing, but I think in the, a place of hip replacement probably. And the one below is from Siemens and this is a robot used in the cardiovascular area.
Rashmy Parimi: Now that I’ve peaked your interest on how, what are the advantages that come with this novel application? I’m sure all of you must be curious, how do you break it into this field? What are your pathways? Is it something very niche? Is it very small exclusive circle?
Rashmy Parimi: Well, I’d like to walk you through my own career path to kind of show you it’s really not all that difficult. And in the next slide, I will also kind of walk you through during the various stages in the life cycle of a product development, what are the different functions that interact and how different disciplines come together to successfully build a robotic surgical product. I started off by education as an electrical engineer, but using that as my foundation, I have worked on firmware for different products, electricity meters, crane systems, small devices which include wearables, thermostats.
Rashmy Parimi: I went into this not through either medicine or robotics. I started from a very normal field, which I’m sure most of you feel <laugh> a little easy to relate to. I did have a small ex in brush with medical devices early in my career where I was working as a part of a team on a prototype of a USB based ECG monitor.
Rashmy Parimi: If any of you have noticed the ECG monitor today used in the hospitals, it’s a big piece of equipment and it’s not portable. It’s used in a remote location and they want to share the data around for more opinions. It’s not easily done. There is that accessibility issue. But if it were in a USB form and the data can be collected wirelessly and shared across seamlessly without the boundary of a physical location, it it would be a great blessing to bringing healthcare to rural areas where accessibility is a big issue.
Rashmy Parimi: The proposition of that product was very interesting. And back then, I wanted to continue in that but then again it was just one research project. As I grew in my career, one of the chances I encountered was to be part of the startup Verb Surgical, which was working on a soft tissue surgical platform.
Rashmy Parimi: Verb Surgical has been acquired by Johnson and Johnson and that team is continuing the work on that platform. Hopefully soon that will be in the market helping people improve their quality of lives. And even if you notice through my career, the job duties I’ve done has varied from pure research projects to some integration to what I do today, which is manufacturing test. All of this is more about applying your skills, existing skills across different areas. I have not taken any new courses.
Rashmy Parimi: I have always maintained this curiosity to upskill myself on the job and try to read more on things I don’t much, that was how I was able to work through different domains within the same company.
Rashmy Parimi: Next, I want to talk about what are the various disciplines and roles that participate together during the development of a product. Initially when you want to establish the user needs and make sure a certain product is feasible from a regulatory perspective, the team that typically does the groundwork, the product managers who talk to the customers such as the physicians to make sure they understand what will help them. Then you have the systems engineers, who translate those customer needs into some kind of actionable product requirements. And then the clinical engineers, who also bridge the gap from a clinical perspective.
Rashmy Parimi: The regulatory affairs team helps trying to understand what, how the impact of that, you know, what is the burden of this product to make sure we are safe. And also how, how do we prove that this product is safe to use on human beings once the use case has been established And there is this clear requirements for the product.
Rashmy Parimi: Then comes a design phase where you have design engineers and various arenas. You have electrical design engineers, mechanical design engineers, UI engineers, UX engineers, all coming together to build different pieces of the system and of course test engineers to test all that has been built.
Rashmy Parimi: And for most large scale products, one of the things that has made big difference if the product moves forward in a given timeline or it does not launch off is the integration piece of it.
Rashmy Parimi: There is a lot of complex software and hardware coming together and integration plays a big role. We have the systems integration engineers trying to piece those puzzles, making sure two independent modules operate together as one big unit, and also clinical engineers from time to time to make sure what physically was decided in the beginning is still what the goal of it is towards the end.
Rashmy Parimi: And as the product goes into its future stages, the burden is to val validate and verify it so that we have the essential documentation for FDA approval. But before that, the manufacturing team and the supplier make sure they work with various vendors and internally and to build up these units that will provide the data for FDA to review and approve the device.
Rashmy Parimi: Once that is done during the commercialization phase, you have marketing team, the sales team, the service team to make sure the product is supported within the customers who are using it and also provide the feedback to support the next level of iteration of design and all of these resulting in a complete cycle.
Rashmy Parimi: As you can see, quality is something which is critically important through the whole process and weigh in in all of the design phases and the later validation and commercialization phases.
Rashmy Parimi: What is the future outlook for this field? This is an illustration from before the pandemic. Just few years ago, there’s been 77 companies and these are only the companies that are have gone public. There are a lot more stealth companies, who maybe close to finishing their product.
Rashmy Parimi: The number of companies have increased from a few million in the beginning of last decade to a lot more billions now. It’s a fast growing industry and there has been a lot of acceptance to make sure this field is supported.
Rashmy Parimi: And in general you’ll see these are the two areas where there has been a lot more progress in terms of adding new procedures and support in terms of surgeon’s interest and also success rates in the field.
Sukrutha Bhadouria: Rashmy, we can wrap up. It’ll be great.
Rashmy Parimi: Yeah, so I think this is my last slide, <laugh>. With this, I hope a lot of people have a lot of questions. I’m happy to answer that later. Please feel free to connect with me on LinkedIn. Thank you everyone for your time and thanks for having me here, <laugh>.
Sukrutha Bhadouria: Thank you so much Rashmy and thank you to everyone for attending and you know, posting all your comments and sharing your insights. Thank you.
Rashmy Parimi: Thank you.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
The 6th annual Girl Geek X: ELEVATE Conference and Career Fair on March 8-9, 2023 in celebration of International Women’s Day hosted over 3k women & allies globally, with 74% attendees interested in hearing about jobs, 40+ women speakers (with 63% women of color), 5 sponsors at Career Fair (virtual employer booths), and 2 developer workshops.
Thank You To ELEVATE 2023 Supporters – They’re Hiring!
Special thank you to our supporters at Autodesk, Cadence,United States Digital Service, CodeSee andDematic for recruiting from the Girl Geek X community of mid-to-senior level technical women. We can’t wait to help another girl geek get her next job in tech.
#1 – Feature women as speakers! From individual contributors to leadership onstage, Girl Geek Dinners speakers are women speaking about their expertise and sharing their career journeys along with key advice. We recommend providing a diverse speaker roster to showcase. The Girl Geek X team can also support new and experienced speakers with speaker prep.
#2 – Host a Girl Geek Dinner at your office, or outside! Many companies welcome Girl Geek Dinners into their all-hands space, a cafe, or outdoors. Some of our favorite Girl Geek Dinners in the Silicon Valley took place outdoors next to the swimming pool, or under a big white open-air tent outdoors. You can create more space and invite more attendees.
#3 – Make networking easy at Girl Geek Dinner! Before and after the programming, provide plenty of time for attendees to network. The company can provide demo stations, recruiting stations, food stations, drink stations, photo booth stations, dessert stations. Create cool experiences and photo ops along with an event hashtag.
#4 – Pro-tip for networking: Pre-print name tags with names, job title, and company in a large font size! Make it easy for the attendees to start a conversation. Create table topics or print little games for people to ask questions of each other while standing around to be completed for a chance to win a bigger swag item.
#5 – Get creative with swag! You can give away branded items, from socks to picnic blankets. Girl Geek X will also bring stickers as swag. We can also chat about making co-branded swag for the event, such as making laptop stickers!
#7 – Be generous with food and drink! Be sure to order plenty of food, with gluten-free and veg options. You can also provide dessert options and announce them and invite attendees to continue to hang out afterward chattering and connecting with recruiters and attendees.
#8 – Welcome your company staff as volunteers! Your ratio of company volunteers (wearing company t-shirts or something identifiable) is at least 1:8 (maybe even higher). Girl Geek Dinners are great for employee engagement and retention.
#9 – Be sure to plan and send an email to attendees after the event! The company should send an email after the event thanking attendees for coming and inviting them to apply, ask questions, get connected on LinkedIn.
#10 – Showcase your company’s women and underrepresented groups with your Girl Geek Dinner! Maximize your ROI by promoting to your existing candidate pipeline and talent pool. Be sure to write a blog post with the video embedded so people can replay the lightning talks and get a feel for your event in the post-event video.
Logistics: Budgeting
With a Girl Geek Dinner, the sponsoring company provides the venue, catering (food & drinks), speakers and sound system, in addition to sponsorship. An in-person Girl Geek Dinner brings attendees into your office where they can get a feel for your culture; event logistics requires resources, and attendance is limited to girl geeks locally. Girl Geek X provides sponsors a professional video production crew and video asset of the event.
With a sponsored virtual event, this provides an opportunity to reach a global audience, those who have obligations that make commuting to evening events difficult, your remote employees, etc.
Girl Geek X manages all promotion and registration, and provides guidance and consultations on planning your event and speaker preparation.
77% of girl geeks attending an event are open to hearing about new job opportunities. The video of your event will become your single best asset for recruiting diverse teams — providing potential candidates not only with a sense of your company’s culture and what type of team they would work with, but also demonstrates a clear commitment to creating opportunities for women to be visible within your organization.
Girl Geek Dinners are a fantastic way to team-build by connecting with women throughout your organization to put the event together, to put women onstage as speakers and role models.
Email sponsors@girlgeek.io to learn more about partnering with Girl Geek X. Thank you!
Logistics: Programming your Girl Geek Dinner
We recommend speakers giving lightning tech talks (7-10 minutes) – the speaker roster should showcase a variety of women working at your company in roles like engineering, product, operations, sales, marketing, finance, etc. Q&A via moderated panel discussion afterward is a great way to wrap up the programming.
Please ensure that your girl geek speakers are diverse across vectors like age, ethnicity, experience level, both management and individual contributor. The Girl Geek X community loves seeing diverse and inclusive women of color / underrepresented groups as speakers onstage! Sometimes, girl geeks at the company give talks offstage at demo stations around the venue, like a modern-day science fair. Cocktail tables are a great height for girl geeks demoing their work with posters and / monitors.
In the networking area, these cocktail tables can be helpful for encouraging easy connection while enjoying a plate of food and/or a beverage.
We gently discourage organizers from having male speakers except for the opening remarks (because most tech events are full of male speakers anyway) at Girl Geek X Dinners. We welcome all geeks at the hosting company to attend, network, recruit, listen to talks, etc. – and we recommend your company’s staff wear their company t-shirts at the event. Special shirts for the event are often worn by company staff/ERG (employee resource groups).
Here are guidelines from our CTO Sukrutha Bhadouria:
Sponsoring a Girl Geek Dinner is a way for attendees to get a sneak peak of what it will be like for them to work at the company — what differentiates from a product, design, and technology perspective.
1. Consider Your Audience!
Do:
Talk topics should cover a range of topics that map to the type of attendee you are trying to hire. Do you have an inspiring female exec or mid-to-senior career-level expert at your company? Have them give a talk like:
“Movin’ on Up: 10 Ways to Become an Engineering Leader” (talk topic by Kimber Lockhart, Engineering Leader) -Or- “No One Cares About Delighting the User” (talk topic by Cindy Alvarez, UX Researcher)
Don’t:
Stay away from topics that are about “what it’s like to be a woman in tech”, “work / life balance”, “bringing your authentic self to work”, etc. Our attendees want to hear from women onstage about the incredible successes accomplished women have achieved, and how they got there.
2. What Is Special About YOU As A Company?
Do:
Share with the audience why your company is different and innovative. Do you use Machine Learning to improve the experience of your customers? Do you use Functional Programming? Is there something unique you do to connect with your customers? Does your design team follow interesting / current design patterns?
Don’t:
Avoid repetitive topics, or themes that aren’t fresh and exciting. This means don’t talk about what it’s like being a woman in tech. With our events being as frequent as they are, we want the takeaways to stick with our attendees as long as possible! Talk about your professional expertise to your industry peers.
3. Give The Full Story!
Do:
Have your speakers cover topics that ultimately tie in to one final takeaway. Is there a big product announcement that could be the theme of the night? Include demos and examples in your story. For example, Netflix sponsored a Girl Geek Dinner and celebrated going global — the talk topics tied into the theme of how they had to build their platform, modify their recommendation algorithm, data centers, design their test environments and deployment strategy when they went global. The food had a global theme too!
Don’t:
Try not to have a panel of women with all the same job type or experience level in the company. Also, a diverse set of ethnic backgrounds on the speaker roster will be appreciated by attendees.
4. Definitely Prepare For The Presentation!
Do:
Set up time for all speakers to rehearse their talks onstage at least once for practice, with time. Slides are highly encouraged! If everyone has slides, have them all go into the same deck, so we don’t have to switch computers every time 🙂
Don’t:
Definitely don’t plan to wing it! Have fun with it and the audience will have fun too! The Girl Geek X team is happy to join the speaker prep and dry run calls to provide feedback and support.
The average Girl Geek Dinner in the San Francisco Bay Area hosts 150-200 women in tech; the range has been <100 to over 400 girl geeks hosted for an event, depending on company size / budget.
As the event is marketed as a dinner, attendees expect more than cheese and veggie platters! Heavy appetizers should include omnivore, vegetarian / vegan and gluten-free (GF) options.
Labeling dishes with ingredient list, or what is vegan and GF, as such would be extremely helpful during the dinner.
To avoid a very long line forming at the food table, we recommend placing food distributed across at least 2 food stations around your venue. Putting the vegan or GF food by the plates means it will be eaten quickly— and you may find hungry VG / GF girl geeks who won’t be able to eat anything at the event.
Drinks have varied from wine, beer and sodas, to specialty cocktails designed by organizing committee (themed to your company / industry and printed on menus at the bar).
The company’s marketing team may print out the co-branded creative onto posters for easels, or hang banners or decorations for the special event. Projectors are great as well for shining the co-branded creative, and having some lively music during networking hour playing on the speakers not-too-loudly is fun.
We recommend hosting the dinner event at your company, so attendees can get a feel for your office. Do you have open spaces for networking? A cafeteria or all-hands meeting area with seating can be used for talks.
We recommend renting cocktail tables so attendees can eat while networking easily with other attendees. If your company doesn’t have enough space to host, you may opt to rent a nearby bar, brewery, computer museum, or imaginarium. It’s not unusual for a company hosting to take advantage of their patio or other outdoor space, to set up a big white tent for shaded food and networking outdoors.
When calculating your capacity / desired attendee number, imagine a sold-out standing-room-only event. The girl geeks are excited to be in a crowded room of like-minded women. Take your upper range for capacity. We will aim to oversell tickets by 30%-40% to accommodate for cancellations and last-minute no-shows, and to ensure a packed room day-of and successful event.
Sponsoring companies may choose to create a piece of co-branded swag to give to all attendees. The Girl Geek X logo has no style guide or brand guidelines. We are thrilled when sponsoring companies remix our two logos together for unique event creatives, which we see projected on walls at the event or printed on company staff shirts, event signage and swag.
Participants have loved customized: socks, reusable grocery bags. A writeup with a pictures of Salesforce Girl Geek Dinner socks as swag are at Salesforce engineering’s blog. We welcome your blog posts!
You can also opt not to make swag and donate the money to a worthy cause instead.
On the week of your Girl Geek Dinner…
We recommend the hosting company to pre-print easy-to-read name badges. Ideally, the name badges are printing big and bold each attendee’s first name, last name, job title and company to help our attendees network at the event! Pre-printed name badges are super helpful alphabetically arranged at the door, so check-in folks can hand out badges at the door to save everyone time and avoid a long line at the door.
Networking can be facilitated by the hosting company by providing a fun networking “bingo” card for attendees to fill out, in exchange for a a bigger chance to win some cool door prizes. You can custom theme this “bingo” card to your company / event. Raffle prizes are also popular, as is providing a creative dessert after the programming to encourage women to stay and chatter.
Consider creating a photo booth / wall for this event — please feel encouraged to leverage the popularity of Instagram by creating “Instagram-worthy” places and activities.
If you’ve read this far down, thank you for your interest in helping women network and learn from each other – and we hope your company can sponsor a Girl Geek Dinner at your office!
Thank you to AWS volunteers for sharing valuable career insights with students who asked questions, in small groups of 5-15 students rotating around the room after AWS leader Trevor Moore gave an intro to Amazon as a business, with trivia and fun prizes. Special thank you to our executive sponsor Shannon Thoke and all the AWS volunteers!
AWS employees Gabe Pinar and Mae Reyes speaking with CCPA high school seniors.
Twitch employee Ashley Clark (and native Oakland resident!) joined by videoconference to talk about her career in technical program management with CCPA high school seniors.
Last school year, Girl Geek X volunteers helped CCPA staff prepare for school reopening in the pandemic, shared career insights for 11th and 12th graders in the school, gave feedback on senior capstone project presentations, and hosted a Teacher Appreciation luncheon with goodie bags for educators.
Field trips allow students to experience a variety of workplaces and gain invaluable insight on real-world job titles, professional skills, and modern workplaces – expanding their minds and STEM career options. Check out event photos at our Facebook page!
“I feel incredibly fortunate to work for a company that allows employees the freedom and means to lead these type of initiatives. I look forward to partnering again with Angie in the near future and continuing to drive change in our community.”
– Trevor Moore, Amazon Web Services Strategic Account Manager & CCPA CompSci Field Trip Organizer to AWS on December 9, 2022
“Everyone at Amazon has an inherent desire to make a positive impact, but sometimes we don’t know how we can add value. When Angie Chang from Girl Geek X approached us with the CCPA CompSci Field Trip concept, we jumped all over the opportunity.
I was immediately able to gather a diverse set of Amazonians, across multiple areas of the business, who shared the same passion for positively impacting where we work and live.
We worked with Angie and the CCPA teaching staff to build an impactful agenda which included a building tour, networking sessions, tech. talks and of course, free swag! Our events team did an incredible job accommodating a sizable group (70+) of visitors.”
Special thank you to Trevor Moore (AWS Strategic Account Manager) for organizing CCPA’s CompSci Field Trip to AWS on December 9, 2022.
If you are working at Google, LinkedIn, GitHub, Discord, Instagram, Reddit, Spotify or TikTok and want to host a student field trip, please email us at angie@girlgeek.io to chat about 2023 field trips.
Logistics and Playbook for Future Field Trips
Many companies have free and/or catered lunch at the workplace that field trips can take advantage of for “lunch and learn” field trips:
Company speakers hail from a broad representation of departments / teams, from data to accounting, from engineer to marketing, from project manager to support engineer, from sales to design.
Emphasize both traditional (higher education, vocational school) pathways to career success, in addition to “non-traditional” (coding bootcamps, self-learning with portfolio of work) ways to entering the tech workplace.
Employee resource groups (ERGs) may be interested in inviting their members to participate as volunteers and role models for the students. Girl Geek X may also source volunteers to join the students for conversation at lunch.
Sample Field Trip Agenda
10:00am – Students and educators arrive to office, check-in
10:30am – Tour of office & introduction to teams / departments / volunteers roles
11:30am – Lunch with volunteers (speed networking with a wide gamut of roles in the company)
1:00pm – Workshop(s) on coding, business, workplace skills, or speed networking (2-3 volunteers sit with a group of 5-15 students, students rotating chairs every 15 minutes)
1:45pm – Distribute company swag
2:00pm – Students and educators depart
Schedule for 2023-2024 School Year
Here are the times in the coming school year that field trips with CCPA can be scheduled, in order of preference:
Nationally-recognized annual call to action to inspire K-12 students to learn computer science, advocate for equity, and celebrate the contributions of students.
Late April 2024 (before AP tests / finals)
Any Friday during the school year – preferably adjacent to a long weekend or holiday.
Transportation
If the company is located within walking distance to a BART station, educators can walk with students to Coliseum Station to BART to the field trip by the BART public transportation system.
If the company is located in Silicon Valley, please provide a charter bus or Lyft/Uber codes for students to carpool.
Did you know mathematician Grace Hopper helped invent the programming language COBOL? Here she is pictured at the UNIVAC I console in 1960. Her birthday is December 9 – and why CS Ed Week is celebrated each December!
We’re excited to announce that we are launching Career Fairs! Registration will open in just a few weeks, and the first event is on Thursday, December 8, 2022.
Is your team is hiring for open technical roles? Now’s the perfect time tobuild your talent pipeline, attract passive candidates, shine a spotlight on your female leaders, and let us help you create evergreen talent branding and recruiting assets to support and highlight your organization’s DEI efforts!
We’re using a new event platform for Girl Geek X Career Fairs online with improved networking capabilities, and every sponsor will get:
a customizable virtual recruiting booth that can be staffed by members of your team, from hiring manager to recruiters,
networking tables within your virtual booth
opportunities to connect LIVE with attendees in 1:1 or small groups meetings
your open roles promoted to our community of over 40,000 women in tech
If your company wants to promote your talent brand and open roles to our community of over 40,000 women in tech, and give your female leaders a forum to share their experience, insights and excitement for their work, please check out our sponsorship prospectus and let’s talk!
Over 120 girl geeks joined networking and talks at the sold-out OpenAI Girl Geek Dinneron September 14, 2022 in San Francisco’s Mission district.
Hear lightning talks from OpenAI women working in AI with music and deep learning, sharing the power of trying and trying again, how to make language models useful, and much more at the OpenAI Girl Geek Dinner video on YouTube!
OpenAI Residency applications are open! OpenAI is looking for engineers and researchers who are interested in applying their skills to AI and machine learning. Please apply for OpenAI jobs here!
If you have an unconventional educational background, we encourage you to apply to OpenAI Residency (applications are open through September 30, 2022).
Table of Contents
Welcome – Elena Chatziathanasiadou, Talent Programs Leadat OpenAI, Recruiting & People –watch her talk or read her words
Multimodal Research: MuseNet & Jukebox – Christine McLeavey, Member of Technical Staff at OpenAI, Multimodal – watch her talk or read her words
If At First You Don’t Succeed, Try Try Again – Alethea Power, Member of Technical Staffat OpenAI – watch them talk or read their words
Making Language Models Useful – Tyna Eloundou, Member of Policy Staff at OpenAI, Policy Research – watch her talk or read her words
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Transcript of OpenAI Girl Geek Dinner – Lightning Talks:
Angie Chang: Hello. Thank you everyone for coming tonight. My name’s Angie Chang and I’m one of the founders of Girl Geek X. We started over a decade ago as, Bay Area Girl Geek Dinners, and we’re still going strong. Thank you to OpenAI for hosting us for a second time. We’re really excited to see the new office and invite a bunch of Girl Geeks over to hear these lightning talks on AI and policy and all these things that we’re so excited to learn about tonight!
Sukrutha Bhadouria: Hi. I know you all were still chatting when Angie introduced herself, but she’s Angie and Girl Geek X is basically her brainchild. It started off with Angie looking to bring women together, I’m doing your pitch, Angie for you because I have a louder voice. Some people, they ask me if I swallowed a mic as a child because I’m so loud and I don’t need a mic.
Sukrutha Bhadouria: Anyway, I’m Sukrutha, so Angie started Girl Geek and it was back then called Bay Area Girl Geek Dinners, this was over 10 years ago. And when I had just moved to the Bay Area, looking for ways to meet new people and I found out about Bay Area Girl Geek Dinners dot com at that time, and I tried really hard to meet with Angie, but she was a busy bee doing all sorts of cool things, trying to change the world. And this was way before ERGs existed, right? So people didn’t have a way to connect with the community until they went to meetups.
Sukrutha Bhadouria: And Girl Geek Dinners, at that time, was the one way you could also get an insight into what these sponsoring companies worked on, what life was like. And so it also allowed people to get an opportunity to speak and a lot of the speakers at Girl Geek Dinners were first time speakers. They were too afraid to sign up for conferences. If you go to our website (girlgeek.io), you’ll see all these amazing stats on how since Angie started, there’s been a real shift in the environment in how people are more willing to speak at conferences, due to some of the chances they’ve gotten as a result of speaking at an event sponsored by their company. This organization exists.
Sukrutha Bhadouria: I joined Angie and we tried to change the world together. I’m happy to report that I think we actually did. We rebranded to Girl Geek X, and that’s when the organization hit 10 years. It was a sizable number of people working on it, it was Angie and me and it was just the two of us. And then Angie had this idea to really evolving into a company and so that’s when she started to bring on contractors, more people such as somebody who could take video of our events to make us look a little bit more professional and somebody else to do our website besides me. And we started to do podcasts.
Sukrutha Bhadouria: We started to do virtual annual conferences and we really, really, really were always consistently sold out for our in-person events that would happen at various companies that we partnered with through the Bay Area. Then COVID hit and the good thing is that we had already started to have a global presence through the virtual conferences that we had and we’ve now had four? Five, yeah.
Sukrutha Bhadouria: We used to be carpooling all around the Bay Area together to these events after work and now we are moms. So it’s amazing. We would look up and see amazing people working at these sponsoring companies speak and we’d be like, “Wow, look at them managing their mom life and parent life and coming to these events.” But I just think that it’s now become such a common thing that it’s not as isolated anymore. And I’m hopeful that, you all can come back again and again, because this in person event has really made me really happy.
Sukrutha Bhadouria: I’ve been holed up in my home office today, which is basically a room which also has my… What’s it called? A bike that stays in one place, stationary bike, so it has too many things going on in the room, but I wanted to give a big thanks to OpenAI for hosting us for the second time, for sponsoring for the second time. And I hope that we can keep doing this. So please do get your companies to sponsor and encourage them to do it in person. That’s all I will say. I know I said a lot more than I had planned, but thank you again, and Angie.
Angie Chang: Thank you Sukrutha, for the intro. I guess I should talk up Sukrutha a little more. When I first met her, she was a software engineer in test, and now she is at Salesforce as a Senior Director of Engineering there, so I’m very proud of her. And over the years we… She mentioned we have a podcast, we have annual virtual conferences!
Angie Chang: We’ll be launching a career fair virtually as well, to be announced. And I don’t want to say too much. We have an amazing line up of speakers tonight and we’re going to invite up first, Elena, who is our host for the night from OpenAI.
Elena Chatziathanasiadou: Hi everyone, I’m Elena. I work here and I’m on the recruiting team, I’m leading the Residency program right now. I’m very excited that you’re all here and have joined us together. Really want to thank Angie and Girl Geek X. We’re very excited to deepen our partnership together and to be back in the office here all together, in the new space and to experience this tonight.
Elena Chatziathanasiadou: We’re very excited about having you here and in terms of what we’ll see tonight, we’ll have a series of lightning talks and then that will be followed by Q&A and then we’ll get some dessert in the area that we were before and then we’ll wrap up at 8:30. But before we get started, I did want to take a moment to make a quick plug and share that…
Elena Chatziathanasiadou: We’re actively hiring for our Residency program and that includes both research and engineering roles and the goal of it is really to help develop AI talent. The program, it offers a pathway to a full-time role at OpenAI for folks that are currently not focusing on AI and are already researchers or engineers in a different field.
Elena Chatziathanasiadou: We’re really excited to hear from you. If you do have an interest in making this career switch, come talk to me after. And we’ll also have full time recruiting team members and positions that we’re hiring for across research product and engineering that we can tell you more about. Please come find us and learn more about the interview process, but also what the program offers.
Elena Chatziathanasiadou: With that I wanted to introduce our first speaker, Christine, who’s currently managing our multimodal team and previously worked on music generation research, created MuseNet and was collaborating on Jukebox. And before that was a classical pianist who transitioned into a researcher as well. I’ll hand it over to Christine. Thank you so much.
Christine McLeavey: Thank you. So yes, it’s really an honor to be here tonight. Thank you all for being here. And this Residency program is near and dear to my own heart, because I first joined OpenAI through, what was then the Scholars Program and the Fellows Program and those are the programs which have since evolved into this Residency program. I’ll put a plug in for anyone who’s considering it.
Christine McLeavey: I want to talk this evening about my own path through OpenAI, but especially about the two music models that I worked on during the time here. I thought I’d start by just going ahead and playing an example of each of the models. The first one, this is the one I worked on when I was doing the Scholars and Fellows program. This is MuseNet, which works in the MIDI domain, so this is the model trying to generate in the style of jazz. Okay, I’ll cut that off and then after I joined full time, I was lucky enough to collaborate with some amazing researchers here to work on a model that was instead working in the raw audio domain. The fun of that is you get to imitate human voices. This is trying to do the style of Elvis with lyrics by Heewoo. Okay.
Christine McLeavey:Elena mentioned before being at OpenAI, I was actually working as a pianist, I had done some math and physics in college, but obviously it had been a long time and so I think I took a good year of self studying before I applied to anything. And I thought I would just give a shout out to three of the online programs that I particularly liked at that point. They’re all amazing. But then I was lucky enough to join the first cohort of scholars that we had here. And at that point I was just trying to do this process of learning about all these different models. And I had this feeling that instead of just copying a model or copying what someone else has done, let me just try to translate it into a field that I know well, which was music. And so what became MuseNet was really my attempt to take all of the stuff I was learning and then apply it to the music domain instead.
Christine McLeavey: MIDI format is this really nice representation of music. I think of it as the way that a composer thinks of music, so it’ll do things like it tells you what notes it plays when, the timing of it, the volume of it, things like that, which instrument is supposed to play. But it loses all the actual detail of when a human takes it and performs it. You don’t get a person’s voice, you don’t get the sound of a great cellist, anything like that.
Christine McLeavey: The nice thing is it’s what you trade in expressivity, you get in this nice really meaningful representation. It does sound pretty terrible when you try to render materials. As a musician, just thinking about the structure of music, this was a nice simplification for a scholars project. What I did is I took a bunch of MIDI files and I tried to pull them out and turned them into a sort of language to make them look as much the sort of thing that you could get in your own net to predict as possible.
Christine McLeavey: I did things like I would always tell the model which composer or which band was going to be first and then things like what tempo was going to be when notes would turn on and off, and a wait token, which would tell the model how long to wait, things like that. And then what you end up doing is you translate that tokenization into just a dictionary of numbers and the model sees something like this. Which I think that this is the first page of a Chopin bellade or something.
Christine McLeavey: What the model is faced with is this task of given the very first number, what number do you think is going to come next? And then given the first two numbers, what number is going to come next? And when you first look at the first thing and when the model first sees it’s like how do you do this? What does that even mean? It feels like an impossible task. But what happens is the model sees many, many, many examples of this.
Christine McLeavey: And over time it starts to pick up on, ah, if I see 4,006 somehow I tend to see 586 more often after that or something. It starts to pick up on these patterns, which we know because we know the tokenization was like, oh, if a piano plays the note G, then probably soon after it’s going to turn off the note G or something. It has real musical meaning to us. But the model is just seeing these numbers like that. The nice thing is the model gets really good at this job and then you can turn it into a generator just by sampling based on, I thinks there’s like a 20% chance this token’s going to come next, so 20% of the time take that.
Christine McLeavey: The other really fun thing you can do is you can then study the sort of mathematical representation you’ve gotten for these tokens. So I was always giving it the composer or band token in the beginning and now you can look at the vectors or the sort of embedding that it learns through these composers.
Christine McLeavey: And as a musician it’s really fun because I would clearly think that Da Vinci and Ravel, for all these French guys are related and the model just picked up on the same thing, which is cool. But the other really fun thing is that you can mix and match those [inaudible]. So here is the start of one of my very favorite Chopin, Nocturnes. So I actually just gave the model the first six notes of that and this is what the model thought, if instead it was being written by [inaudible] It was a bunch of VPs. It goes on for a while, but I’ll cut it off there. And that was MuseNet.
Christine McLeavey: And then I ended up joining full time after that and I was lucky enough to collaborate with Prafulla and Heewoo on taking music generation over to the raw audio domain. And so in a way this is a much harder problem because now whereas in MIDI world you have just nice tokens which are meaningful in a musical way, raw audio is just literally 22,000 or 44,000 times per second.
Christine McLeavey: You’re recording how loud the sound is at that moment in time and the nice thing about it is it gives you all this expressive freedom, right? Literally any sound you can imagine you can represent as a sound wave, just audio recording to that. The trouble is there are just so many ways for those waves to go wrong or those patterns to go wrong. If you mess up on the short scale, it’s just like crazy hissing noise. If you mess up on long scale, your piece sadly starts getting out of tune or the rhythm drifts or so many ways it can go wrong, it’s really an unforgiving sort of medium. And the problem is now in order to get a minute of music, it’s no longer maybe 3000 tokens you have to do, it’s maybe a million numbers that you have to get correct.
Christine McLeavey: We approached this by looking at ways that we could compress the music to make it more tractable because at that point a transformer could maybe deal well with the context of 4,000 tokens or something. We used an auto encoder to do three different layers or levels of compression and the sort of least compressed on the bottom. The nice thing about that is it’s very easy to translate it back to the regular raw audio. If you put some original song in and then back out, you don’t notice any loss at all. Whereas if you put it through the most compressed version, the nice thing is now it’s super compressed, like 3000 tokens might get you half a minute of music or something. But if you go through this simple just trying to reconstruct the raw audio, it sounds really bad. You can sort of tell that someone’s singing but you’ve lost most of the detail.
Christine McLeavey: The nice thing about it is when you work in that top layer of tokens, now this looks a lot like the MuseNet problem or even just a lot language problem where you’re just predicting tokens. So we train a transformer on that. We sort of added in the same which person was singing, which band was playing, and then we also added in where you can write the lyrics in, so the model conditions on the lyrics and then generates these tokens. And then I won’t get into the details, but we had to train extra transformers to do this upsampling process so that you could get back to raw audio without totally losing all the detail.
Christine McLeavey: The fun thing is you can do things like ask it to generate in the style of Sinatra singing Hot Tub Christmas and I have to put in a book, these were lyrics by at, that point, GPT-2. All right. It’s a Christmas classic now. And then last I wanted to wrap up by talking a little bit about the multimodal team, which is the team that I’m really excited to be managing these days. It’s this really, really great group of people. Unfortunately, our current projects are all internal and I can’t talk about them, although stay tuned, we’ll be publishing them to the blog when we can. You might recognize Clip, which was work done by Alec and Jong Wook both on our team. This is, I guess, nearly two years ago already, but made a really big impact on the image work at that point. And then just to put in a plug for the team, we’re about a group of 10 at this point and we will be hosting a resident in 2023.
Christine McLeavey: Please reach out if anyone’s interested to talk more. And then we’re doing all sorts of projects in the sort of image, audio and video domains both on the sort of understanding side and generation side. And we end up working really closely with algorithms, which is the other team that tends to do a lot of awesome multimodal projects. But then also anytime we get close to things that we’re looking at putting out tech customers, we end up working with applied through that and then also obviously scaling because at OpenAI we believe deeply in this, get a good pattern and then scale it up and it becomes awesome. So thank you so much for your attention.
Elena Chatziathanasiadou: Thank you so much, Christine. That was awesome. So now next we’ll have Alethea. Alethea has spent the last couple of years at OpenAI working on getting neural networks to do math. Before that, they built large infrastructure health system, studied math and philosophy and spent lots of time singing karaoke. Welcome, Alethea.
Alethea Power: Thank you. So this talk is called If At First You Don’t Succeed, Try Try Again. It’s been a wild few years. I decided I wanted to give an uplifting and encouraging talk. It’s a short talk so it doesn’t get too deep into technical details, but if you’re interested in it, please find me afterwards. I will talk your ear off about it.
Alethea Power: Okay, my name is Alethea Power and yes, Patience is actually my middle name, which will be very relevant for this talk. Okay, so about 10 years ago I was a software engineer and site reliability engineer and my dream was to get into artificial intelligence, but I didn’t know how to do it. I didn’t have a degree in AI, I didn’t have any background in AI, I didn’t have any idea how to break in. So I thought, ah, I probably need to take some time off to study this before I can get into the field.
Alethea Power: I started saving up some money so that I could take time off to study. But by the time I had enough money saved up, I realized I needed to handle my gender issues. So I took that time off to go through a gender transition instead of studying AI. Eventually though I was finally ready to try and break into AI in some form or fashion and that was about the time that OpenAI hosted their last Girl Geek Dinner, that was in 2019. And I came to that talk and I met one of the recruiters who stunned me by telling me I didn’t need to have a degree in AI and I didn’t need to have a background in AI to be able to work here.
Alethea Power: She introduced me to the Scholars Program, the same program that Christine went through, which today is called the Residency Program. And I applied to that and I got in and I had the best mentor in the entire program, Christine. I’m second generation scholar up here. But there were in addition to the obstacles before, there were obstacles after joining the program as well, about three weeks after I joined, there was a pandemic, you may have heard about it. But despite spending a lot of time fearing that I might die or people I love might die for some reason or another, health or political, Christine was very kind and understanding and supportive and she helped me get to the point where I had learned a ton about artificial intelligence and managed to do a great project and I ended up applying full-time and I got three offers here. Thank you. I wasn’t trying to brag, but thank you. This is more to encourage you.
Alethea Power: I ended up taking a job on a team that was trying to teach neural networks to reason and do math. And what I want to talk about here is about a year after I joined that team, I released my first research paper called Grokking: Generalization Beyond Overfitting on Small Datasets. I’m going to give you a very basic introduction to what all that jargon means. And like I said, if you want more technical details, come talk to me afterwards. So first I need to explain how training neural networks works. If you have a background in ML, this is going to be very basic 101. If you don’t, it’s going to be exciting.
Alethea Power: Okay, so usually when we’re trying to train a neural network, we’ve got some amount of data that captures a pattern that we want that neural network to recreate in the future. And often if we’re doing what’s called supervised training, we’ll break that data up into training data and evaluation data. And you can think of this, the training data is sort of what we actually teach the neural network, what it learns from. This is like classroom education and evaluation data is basically like pop quizzes to see how much the neural network learned. And neural networks have this nice property where you can pop quiz them. They don’t learn anything from the pop quiz, they just tell you how they did and then five minutes later you can pop quiz them again and the questions are all new again, they have no memory of them. Throughout the course of training, we measure the performance of the neural network on both the training data, the classroom instruction and the evaluation data, the pop quizzes.
Alethea Power: And there’s two main ways we measure this. One is called loss. I won’t go into details right now about what loss is, but the short version is it’s a differentiable function calculus derivatives that we use to actually figure out how to modify the network, so it learns, when loss goes down. The network is learning. Accuracy is exactly what you would think of being like a test score, so 0% accuracy means you got every question wrong. A hundred percent accuracy means you got every question right. This is what a very successful neural network training looks like. You can see, oh, the x axis here on both of these graphs is steps of training. You can see that as we train this neural network along the loss on both the training and evaluation go down. It’s learning what it’s supposed to learn from and it’s able to generalize that to the pop quizzes.
Alethea Power: It’s doing well on the tests as well and then this is what it’s actually scoring. So by the end of this training it gets up to 90% accuracy, so it’s got an A. Sometimes though, if you train a neural network for too long, it starts to do what’s called overfitting. You might remember the word overfitting from the title of the paper. In this case, the neural network learns too much detail from the training set that doesn’t really generalize to the rest of the world. And so its performance on the quizzes starts to get worse. So an example of this in this paper, I was training neural networks to do math, basic mathematical equations. For instance, if it happened to be the case that the training data had more even numbers than odd numbers, and if it was trying to learn addition, then it might learn that usually the answer is going to be even. Well, in reality that’s not true in addition.
Alethea Power: In reality, you want to actually know how to add and the number’s going to be whatever it is. So that would be an example where it learned some sort of incorrect, non-generalizable information from the training set and that made it start performing worse on the evaluation set. And you can see here in this situation, the accuracy on evaluation would go back down. Sometimes, and this is very common when you’re trying to get a neural network to do math, you have an even worse situation where the same thing happens with your loss, but it consistently fails the pop quiz every time. Gets to a 100% percent accuracy on the training data and fails the pop quiz. This means the network and we were using similar kinds of networks to the ones Christine was talking about, just math instead of music, this means the network never really understood what it was learning, it just memorized it.
Alethea Power: This is like the kid who knows that when you say six plus four, you’re supposed to respond with 10 but has no idea how to actually add. So this was a common scenario when training neural networks to do math. They’re really good at pattern recognition, but they’re not always good at understanding a deep analytical precise truth underneath the pattern. Well then one day we got lucky and by lucky I mean forgetful. So one of my coworkers was running an experiment like this and he went on vacation and forgot to stop it. And so a week later he came back and it had just kept studying and studying and studying and studying and studying and studying and studying and studying and studying. And it learned. So what happened here was, it went into this overfitting regime where usually we’d say, ah, it’s learned all it can learn from this training data.
Alethea Power: There’s no more to learn and see, it still had zero accuracy and it just kept getting worse and worse and worse. And then suddenly long after it memorized all of the training data, it had an ‘aha’ moment and it was like, oh, all this stuff that I memorized actually makes a pattern and the pattern is addition or division or S5 composition or whichever task we had it working on. And then the loss started coming back down on the pop quizzes and it went up and it got a 100%. This is weird, this never happens in neural networks. We dug in and recreated this many times, implemented it twice, saw the same behavior with two completely independent implementations on a wide variety of tasks and there’s all sorts of other interesting stuff about when this happens and when it doesn’t, ask me in the questions afterwards.
Alethea Power: The point here is at first the network didn’t succeed, but it just kept trying the same way I did when at first I couldn’t get into AI, but I just kept trying. We named this phenomenon where it finally figures it out Grokking, and we named this after Robert Heinlein’s novel Stranger in a Strange Land. It’s a science fiction book and Grok is a Martian word in that book, which means, “To understand so thoroughly that the observer becomes a part of the observed to merge, blend, intermarry, lose identity in group experience.” And it turns out this is exactly what these neural networks do. I’m going to let you take pictures before I change the slide.
Alethea Power: This network was trying to learn modular addition and modular addition you can think of is adding hours on a clock. Also, thank you to Christine for that analogy. If you have 11 and you add 3 to it, you don’t end up with 14, you end up with 2 because that’s what happens on the clock. The clock is modular 12, we were having it learn modular 97, and then we tore open the network that had grokked afterwards to see what was going on inside of it and it had actually built internally this circular structure of the numbers. It had created the mathematical structure we were trying to get it to learn that allowed it to actually solve the problem. Did this with all different kinds of problems, so we had one network learning to compose permutations and it found what are called subgroups and co-sets out of that, details later. But the point is, it worked so hard for so long through so much failure that it became the knowledge it was trying to get.
Alethea Power: The point here is, that if your dream is to get into AI, even if you have no background in AI or whatever your dream is, it doesn’t matter. Keep trying and keep trying and keep trying and keep trying and maybe you can get there eventually. And in particular, if your dream is to work at OpenAI, which I highly recommend because this place is fabulous, then try, even if it’s not the background you have already, even if you feel like you have a weird background or you’re not like the people here or like the people in this field.
Alethea Power: We’re a humanitarian organization. Our core mission embodied in our legal structure and our financial structure is to make sure that artificial intelligence benefits all of humanity instead of just a small number of rich people in Silicon Valley. And to be a humanitarian organization with a humanitarian mission, we need a wide diversity of perspectives here. If you have a different life story, a different path, different perspectives than we’ve seen before, that makes you more valuable here, not less, so please consider applying.
Elena Chatziathanasiadou: Thank you so much, Alethea, That was awesome. And now next we’ll have Tyna, who’s on the policy research team currently doing our rotation on applied research and she participated in the OpenAI Scholars Program, has spent some time researching economic impacts of our models, building safety evaluations, and collaborated on web GPT and moderation API. Let’s hear from Tyna.
Tyna Eloundou: Wow, so many of you. Let’s see. Okay, this works. Hi, everyone, thank you so much for coming. I’m Tyna Eloundou, I’ll be speaking to you today about making language models useful. A bit about myself, let’s see, wow, I’m also a former scholar. I can’t make the claim to third generation because Alethea was not my mentor, but they were super helpful in making my experience here amazing. And part of that culture and that welcoming environment was a reason I chose to stay on after the scholars program [now the Residency program].
Tyna Eloundou: Today we’re going to be talking about language models and by language model, I mean any model that has language as input and output. So that could mean GPT-3, CODE-X, or BigScience’s Bloom, what have you. Okay, this is going to be the only equation you see throughout this talk and it’s really not that important, but I think it gives us some context as to where we’re going.
Tyna Eloundou: Looking back at this, this is the training objective for GPT-3 and for all GPT like models. Given a corpus of tokens, right? We define the objective to maximize this likelihood, L, which is defined as a conditional log probability over a sequence of tokens that is modeled by a neural network with parameters data that is trained by gradient descent. Now you can forget everything I just said.
Tyna Eloundou: Essentially this optimization produces these models that are trained to predict tokens, but that in itself may not be that useful on its own. I don’t think I’m giving away any secret sauce by revealing this equation to you, but it is remarkable that somehow we go from this to models that can produce, oh sorry, that can do that, right? Write prose, write code or parse data and so on.
Tyna Eloundou: I’d like to talk a bit about the notion of usefulness itself. One way to think about whether language models are useful in the first place is in the pragmatic sense. In the ideal scenario, we would be able to succinctly communicate our goals and preferences to a language agent without having to laboriously explain and list what to do and what not to do.
Tyna Eloundou: How did we initially get usefulness out of language models? When these models were first being developed in research labs, some researchers came with some ideas about how to really get them to do what it is that you want them to do. And these are two of the most prominent ones. One was few shot prompting, which is a method by which you really tell the model what the task is and before putting it on the spot, so to speak, you give it some examples of what you like to do, some demonstrations, right? For translate English to French, you could have a pen to [foreign language], I’m hungry to [foreign language], et cetera. And the translation that you actually want, you say, I would like to eat ice cream and hopefully with that same formatting you get the model to translate to French.
Tyna Eloundou: The other method is supervised fine tuning, which involves essentially just having examples for the model and then kicking off another round of training so the model can become hyper focused on your task and hopefully improve its performance on that task. So as many of you probably know, OpenAI has since then adapted this iterative deployment approach, which helps us put models in the hands of real people and understand how they interact with them. At the time of GPT-3 release, it was doing great by research standards, right? And unfortunately a lot of these research metrics are designed around these methods that we’d spoke about before, which are to prompt with few shot prompting or perhaps to do supervised fine tuning. Once we deployed, we really quickly learned that people don’t like prompt engineering. In fact, they don’t really like to do a lot to communicate their goals to the model, which is fine. It’s a feature, not a bug.
Tyna Eloundou: At its most helpful, a language agent can infer what we want without lots of specification and carry out those inferred goals effectively and efficiently. Unlike researchers, people were using natural language instructions to ask GPT-3 for what they wanted. But because of the training objective that we saw previously, the model was really tempted to just pattern match, right? If you gave it a prompt of write a short poem about a wise frog, it would very helpfully give you similar types of prompts instead of following your intent. This spurred a research effort within our alignment team to teach the models how to follow direct instructions. They did this using two insights. The first is borrowing from the supervised fine tuning or supervised learning literature where you can train the model based on examples or demonstrations, right?
Tyna Eloundou: You have a prompt and you tell them what you would ideally like it to do. And the second insight came from the reinforcement learning literature where you have some humans compare outputs. And so this model learns to generate, that model learns to compare, right? That model learns to tell this is good, this is bad. And so now with these two things, you can kick off this joint training process where you have a model that’s generating and then a model that’s critiquing, and this is good, this is not so good.
Tyna Eloundou: Over the course of training, the model learns to get better at pursuing this objective, which is no longer the pure language model laying objective and now it’s the instruction following objective. So the resulting model was InstructGPT, which is presented here. Well, yeah, you can see the output. It’s a poem, it’s about a frog, mentions wisdom, and it’s pretty short. I feel like all the requirements were met for following instructions there.
Tyna Eloundou: This was a plot that was quite striking to me. This is one of the main results from the InstructGPT paper. When I first saw this, it didn’t make a ton of sense until I really understood the research behind it. But I think that you can think of the Y axis as a proxy for usefulness and the X axis. We have model size and conventional wisdom has it that… We’re at OpenAI as you scale things, things get in general better. But you can see that even at its smaller size, right here, if you can’t see it’s 1.5 billion parameters, even at its smallest size InstructGPT was deemed to be more useful than any permutation of the base GPT model. So I started this discussion by talking about how research based approaches were not pushing far enough in terms of getting us usefulness out of these models. There’s now this emerging literature focused on helping models be more effective in tasks.
Tyna Eloundou: Broadly speaking, this literature involves having models break big problems up into smaller problems or things step by step before coming up with a final answer. And this does not need to be at odds with our human alignment driven research. In fact, right here you see a result by Kojima et al. and although their results are great overall across the board, we do see that they make the Instruct models even greater. There’s such a huge gap, a huge gain that we see with the Instruct series of models.
Tyna Eloundou: I would like to conclude by thinking about the next steps in this line of research. We know that there can be some instructions that can be malicious or exploitative or deceptive. If language models were to pursue usefulness at all costs, they might become dangerous in the pursuit of dangerous instructions or dangerous intent. Could there be other objectives we pursue along with usefulness that get us helpful but not dangerous models, perhaps kindness or hopefulness?
Tyna Eloundou: And lastly, with instructions, we’re mainly in the driver’s seat and we initiate interactions. As language models become smarter, perhaps kinder, more capable, it may be appropriate to think of them as collaborators and they may be capable of initiating ideation, creation among other things. What are the different modes of interaction we would like to have with these models? Would we want them to advise us? Would we want them to inspire us? Perhaps at Girl Geek X 2042, it’ll be a language model presenting about something new. Thank you.
Elena Chatziathanasiadou: Thank you so much all for joining. I guess with that note, I did want to mention that we’ll kick off mingling time and dessert in the area that we were before and our speakers will be available for you to ask them questions. We also have some of our recruiting team members here tonight. If you all want to come up to the front to just quickly introduce yourself or just say hi so that people can see you and then you can all come find us.
Elena Chatziathanasiadou: As I mentioned in the beginning, I’m Elena, I’m also hiring for the Residency program, so come talk to me, come find me. And then we also have some demo stands of our Dolly product and also our GPT-3, if you want to check them out. Jessica and Natalie will be doing those demos. So yeah, go find them as well.
Elena Chatziathanasiadou: Thank you all for being here. I hope you enjoyed it. Thank you to our lovely speakers and to Girl Geek X, to Cory and to all of our ops team and everyone who helped put this together and let’s go enjoy some dessert!
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Our goal is to support students at Coliseum College Prep Academy in Oakland, California serving grades 6-12 with a computer science pathway, by providing access to volunteers and role models from the professional community for students. We partnered with Oakland Education Fund, a nonprofit coordinating volunteer activities with public schools in Oakland.
Thank you VOLUNTEERS for supporting back-to-school prep with CCPA teachers on Friday, August 5, 2022 in East Oakland!
Our Girl Geek X Community volunteers helped teachers with classroom projects to prepare for students return to campus in August for the new ’22-’23 school year.
Thank you to AWS volunteers for hosting CCPA high school seniors for a field trip to AWS offices in downtown San Francisco on December 9, 2022!
AWS employees volunteered valuable career insights with students, who asked questions in small groups of 5-15 students rotating around the room.
Thank you Girl Geek X community volunteers for attending Winter Expo Night (December 14, 2022 at CCPA) to give students feedback on their senior capstone projects!
May 25, 2023 Teacher Appreciation Luncheon at Coliseum College Prep Academy in East Oakland with Cafe Gabriela box lunches (pulled pork was the crowd favorite), Anthony’s Cookies, and Girl Geek X goodie bags.
