Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!

Angie Chang: I have with us today Sheri Byrne-Haber, who’s a Senior Staff, Accessibility Architect at VMware. She’s a prominent global subject matter expert in the field of disability and accessibility, and known for launching digital accessibility programs at McDonald’s, Albertsons and VMware. And she writes a popular blog called “This Week in Accessibility”. Welcome, Sheri.

Sheri Byrne-Haber: Well, thank you so much, Angie. I’m really excited to be here. I always like to drop the Sheri’s secret fun fact before I start events like this, which I was the first Girl Scout in the US to get a badge in Computer Science coming up on my 45th anniversary of that event this August. I’ve been doing tech for a long time.

Sheri Byrne-Haber: I did my first degree at Cal in computer science back when it was, you know, 90% guys, and I was basically the diversity in the room. Been doing this for a long time. Went and got a law degree 10 years after my computer science degree, then did an MBA 10 years after that. I’m here today to talk to you about why access. Yeah, Go Bears, Angie <laugh>. Why accessibility, what it is.. Okay, so the the brief 50,000 foot version and why it is a great career, especially for women.

Sheri Byrne-Haber: When most people think about accessibility, if they’ve heard that word, accessibility means making stuff work for people with disabilities, that’s kind of the TLDR version. They think about visible disability. You might think about somebody with a prosthetic arm.

Sheri Byrne-Haber: This is actually me practicing in my wheelchair on my Olympic range at home. I’m trying to qualify for the 2024 Paralympic games. People with service animals. People with hearing aids. Something that you can see. Accessibility has to take care of a lot more things than that.

Sheri Byrne-Haber: First of all, we have to deal with hidden disabilities, disabilities that aren’t obvious, that can’t be seen. That might be, I tell people all the time, you see me in a wheelchair, you assume, you know what my disability is, right?

Sheri Byrne-Haber: My real disability is type one diabetes kicks my ass on a daily basis. It interferes with everything I do. My wheelchair is just a way to get around. And I’ve been doing it for a very long time.

Sheri Byrne-Haber: You need to think about hidden disabilities. And some examples of hidden disabilities include Millie Bobby Brown, who’s deaf in one ear. Bono wears tinted sunglasses because he has a glaucoma. It’s not a rockstar affectation.

Sheri Byrne-Haber: Neurodiverse statuses. Mental health issues. The reason why all of the colors in Facebook are blue is that it’s the only color that Mark Zuckerberg sees. When you’re thinking about disability for starters, you really have to broaden the definition to make sure that you’re including both visible disabilities and invisible disabilities.

Sheri Byrne-Haber: Then you need to add two different types of disability. A permanent disability might be a limb difference, but if somebody tears their rotator cuff temporarily, they’re gonna have the same disability as somebody with a limb difference. They’re not gonna be able to use their arm or situationally you might be holding something that prevents you from using an arm.

Sheri Byrne-Haber: When you take permanent plus, temporary plus situational disabilities and, and look at it from both the visible and the invisible perspective, you’re talking about 30% of your potential users. And accessibility is about making technology work for that 30% of users.

Sheri Byrne-Haber: Okay, so what do accessibility testers need to learn? First of all accessibility testing is a lot about interacting with assistive technology. You may have heard from other people talking about software testing as a field that automating is the greatest thing ever because then you can just push a button and repeat all those tests and not have to do anything that requires manual intensive interaction. It’s not so easy to do with accessibility because only about 30% of the tests can be executed in an automated manner by inspecting the code.

Sheri Byrne-Haber: 70% actually require being able to interact with the assistive technology. And so that includes things like screen readers which is what the woman in the middle graphic is using. She’s listening to her iPhone, tell her what’s on the screen in front of her that she can’t see. Some other forms of assistive technology are not using a mouse. Using alternative input devices like keyboards touch pads you know, those graphics pens things of that nature captions, magnification.

Sheri Byrne-Haber: Then we get into a little bit more obscure, slightly less used assistive technology that would include things like sip and puff devices, which is how people who are quadriplegic interact with the internet. Obviously speech recognition is becoming more and more popular and, and actually better and cheaper than it used to be in the past. Once you know how to use assistive technology, you have to learn about the accessibility guidelines.

Sheri Byrne-Haber: There’s something called WCAG, which stands for Web Content Accessibility Guidelines. The version that’s just about to come out is version 2.2. And that is a standard WCAG that has been adopted pretty much globally. Anywhere that you have a law that requires inclusion of people with disabilities, usually it references one of the WCAG versions, not always the same version. That’s would make things too easy, right?

Sheri Byrne-Haber: The EU, Canada, Australia, the us India, some countries in Africa, they all use WCAG as the standard to determine whether or not you’ve made something accessible enough. That is, that, you know, the majority of people with disabilities would be able to use it just as if they didn’t have a disability. These are the two basic things that entry level accessibility testers focus on.

Sheri Byrne-Haber: What do they do once they know how to do all that stuff? Well, they participate in designing, building, and testing software, but a hundred percent through the lens of accessibility, not whether or not does it work which is the functional side of the fence, but does it work with assistive technology that people with disabilities are likely to use? And do those people, are they having an equal experience? Okay.

Sheri Byrne-Haber: Those are the two things that the lens of accessibility provides for an accessibility tester. Other than that you’re participating throughout the entire life cycle, just as if you were a, a designer, a builder, or a tester. You’re just looking at it with a very particular point of view, okay? Women are really well represented in the accessibility space. There’s five times as many women in accessibility as there are women in non-accessible roles, just traditional straight up software testing you know, analytics coding, program management, things like that.

Sheri Byrne-Haber: It’s actually a good place to be because there are other women that can help you support your careers who have been there and done that. And you, you may get a better level of, of understanding from getting mentored by other women than you might be by getting mentored by somebody who doesn’t have the lived experience that you do trying to survive in your career. Okay?

Sheri Byrne-Haber: There is a significant demand for accessibility testers. Unless you work for Elon Musk, chances are you are not gonna get laid off, and that’s because the demand for accessibility testing is being driven by regulations and litigation, especially in the us. So the Americans with Disabilities Act require it, it, the language of the law itself doesn’t require accessibility, but it requires equal access. And the litigation, and we have about 4,000 plus or minus cases per calendar year in the US is focusing on WCAG as that standard to determine whether or not something’s accessible enough.

Sheri Byrne-Haber: As long as there’s laws and there’s long as there’s litigation, there is going to be a demand for accessibility testers. And right now we’re in a place where colleges are not turning out a lot of people skilled in accessibility testing because it’s not required as part of the computer science program. You barely even touch on it if you’re in a graduate HCI program. This is something that’s very much self-taught, and to be honest with you, it’s also very much passion driven.

Sheri Byrne-Haber: A lot of people get involved in accessibility because they have a personal experience with the disability. Again, don’t make assumptions. People see the wheelchair and they’re like, ah, I know why Sheri got into accessibility. Now, I actually got into accessibility because I have a deaf daughter and my deaf daughter you know, experienced a lot of issues when captions weren’t made available to her.

Sheri Byrne-Haber: There a lot of times there’s this, like I said, personal connection that makes people passionate about being in this space. Keep in mind disability is the only dimension of DEI – diversity, equity, and inclusion – that everybody is guaranteed to experience at one point in time or another in their lives. Unless you die getting struck by lightning, never having broken a bone in your life, chances are at some point in time, if you’re not disabled right now, you are going to be disabled.

Sheri Byrne-Haber: When you’re working inaccessibility, you’re working to make the make the place better for your future self. That’s a, that’s another way to look at it if you don’t have a personal connection to disability currently. Okay. being disabled is actually a bonus when you’re working in the field of accessibility, because not only are you bringing the things that you learned about screen readers and, and other assistive technology and the things that you learned about WCAG, you’re bringing lived experience.

Sheri Byrne-Haber: And that’s something that’s very valuable for this type of work. The other thing is work from home has been a thing for people in the field of accessibility. Long before the pandemic 30% of people with disabilities can’t drive. And so work from home is critical, especially if their disabilities prevent them from being able to commute or make it harder or more expensive for them to commute.

Sheri Byrne-Haber: Other than the usual, you get paid well, it’s a fun job. You get to make the life lives of other people better. But this is, this is somewhere where we’re having a disability and being willing to talk about that disability actually helps. And if you need to work from home or if you would benefit from to work from home it’s something that the accessibility managers in the world are very accustomed to.

Sheri Byrne-Haber: There are a broad range of employment opportunities government and education anything attached to federal money, okay? Including money that passes through states and cities and counties has to be accessible. There are strings attached, and those strings are called Section 508. Universities have to make things accessible. Hospitals have to make things accessible. Courts, anything municipal, anything federal, all has to be accessible. The nonprofit space also wants to be accessible because they don’t wanna say, oh, we’re here to help out this group of people, but hey, you people with disabilities, you get in the back of the line. There is typically you know, NPR has somebody dedicated to accessibility. Washington Post, New York Times, they all have accessibility specialists. Those aren’t exactly nonprofits, but it’s places that you see accessibility thought about where you might otherwise think that it wouldn’t be addressed.

Sheri Byrne-Haber: There’s lots of accessibility consulting companies all the retail operations on the internet. If you’re selling in the us it has to be accessible or, or you’re probably going to get sued at one at some point in time. And then, as I mentioned, healthcare is another big field. For each one of these areas, you’re still taking the same domain knowledge that you have on assistive technology and the WCAG guidelines, you’re just applying it to one of these vertical markets. It does not take a whole lot to get started. It doesn’t, being in the field of accessibility does not require a college degree. There are apprenticeship programs for people who wanna get started in accessibility. There’s quite a few resources that are available for free or for low cost online.

Sheri Byrne-Haber: You don’t have to go out and get a college degree in accessibility. In fact, such a thing does not exist. What you have to do is you have to care enough to go learn about all this stuff yourself, invest time, go to meetups, talk to people who are already in the field. I think of accessibility today as where Quality Assurance (QA) was, you know 35 years ago when I had just graduated from Cal 35 years ago for QA, there were no degrees in QA. There was no Six Sigma. These things didn’t exist. You had to apprentice yourself basically to somebody who was really, really good and, and learned from them. And now you can get a degree in QA. You can get all kinds of certifications in QA.

Sheri Byrne-Haber: Accessibility today is where QA was 35 years ago in terms of how to, to, you know, get your foot in the door for the career, so to speak. You can easily evolve from accessibility into more senior careers. A lot of people who spend three to five years in accessibility will then move on to design or UX or UI and front end development, because you will learn a lot about these three things as you’re doing your accessibility testing work. And so if, if this is, if you’re interested in these three areas but don’t have the time to go back to get a degree or go to a boot camp or something else, you can use accessibility as a way to get into the door for some of these other careers, there are I’ve got here a list of some starting points if you’re interested in accessibility.

Sheri Byrne-Haber: Siri was actually invented for people with disabilities. And the iOS voiceover, which is kind of twined with Siri is the screen reader. If you don’t have an Apple platform, then NVDA is a free screen reader that you can use on Windows. Spend an hour not using your mouse. Lots of people can’t use mice. I can’t use a mouse because I’ve got pretty bad arthritis in my hands. That will give you a pretty quick perspective on what it’s like to be a keyboard only user.

Sheri Byrne-Haber: There’s a couple of places that you can register to be a crowdsourced accessibility tester that will help you learn more about how to find bugs, how to report bugs what is it that people are looking for. Most major city centers have a Lighthouse for the Blind, or Center for Independent Living. They usually have ways that they can point people to learn more things about accessibility. And we’ve got meetups all over the place.

Sheri Byrne-Haber: Thanks to the pandemic, most of the meetups are actually now hybrid. If the accessibility group in Orlando is meeting up, you know, it doesn’t matter you live in Portland, you can still go because they’re, like I said, largely hybrid these days. I wanted to give people my contact information.

Sheri Byrne-Haber: I have a website, which is sheribyrnehaber.com. It’s got a free archive of all the blogs that I’ve written over the years about accessibility. There’s probably 200 or 250 articles on there right now. If you don’t see something, ask me because I’m still writing, I’m not writing quite as much as I used to but I get a lot of my ideas from people pinging me and saying, well, what about, you know, how do you make a toast message accessible?

Sheri Byrne-Haber: One of my most popular articles I ever wrote came out of a question that somebody gave me on LinkedIn. And if you use that QR code, it should take you to my LinkedIn profile. I don’t use Twitter. LinkedIn is my only form of social media but I love to connect with people who are interested in accessibility, and you can always ask me questions.

Angie Chang: Thank you, Sheri. That was a really informative talk and accessibility. I love all the resources and the, the the knowledge you dropped on us today. This is the last talk of this career track. Thank you so much for being a part of ELEVATE and for everyone who’s still here with us after two days of nonstop talks, developer workshops, networking, meeting recruiters.

Angie Chang: Thank you again. Networking is gonna start. We’re gonna have some fluid networking, so if you’ve seen Everything Everywhere All At once, it’s gonna make some sense to you, or I think it makes sense to someone who’s in that movie. It might be something else to you. I’ll see you in networking. Thank you Sheri, for being so open and willing to connect on LinkedIn. And yeah, I’ll see you on the other side. Thank you.

Sheri Byrne-Haber: Much. Okay. And I just answered the one question that came in about IAAP certifications. Disclaimer, I’m on the certification committee. I actually help write the test. So yes, I believe that they’re worth worthwhile. They are standardized. They’ve been around for going on seven years now. But they’re not cheap, right? It’s $375 to take each of the tests. Plus, you know, if you wanna sign up for a membership, that’s another a hundred bucks.

Sheri Byrne-Haber: If you can’t afford the IAAP memberships, another path you can go is with the US Federal Government. It’s called Trusted Tester. It’s free, it requires a significant investment in time. Took me about 120 hours to complete it. Most people actually go faster. I had to struggle to unlearn everything that I knew and only respond in the way the government wanted me to respond. That was really hard to do. But if you’re new to accessibility, you should actually be able to get your Trusted Tester certification faster.

Angie Chang: Thank you.

Sheri Byrne-Haber: Thanks everybody.

Angie Chang: Thank you.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!

Sukrutha Bhadouria: I hope you’ve been having a good session so far. Good time so far in the conference and we are ready for the next session. Thank you all for waiting for us. Rashmy is here to give us our next talk.

Sukrutha Bhadouria: Rashmy is a manufacturing test engineering manager working at robotics and digital solutions at Johnson and Johnson. In her career over the last 13 years, she has worked on consumer products, trained signaling, and more recently robotic applications and medical devices. She’s passionate about making an impact on our society with technology and helping fellow women in tech in their journey. Welcome, Rashmy.

Rashmy Parimi: Thank you for the kind introduction. Hi everyone. I’m Rashmiy I am part of the robotics group in Johnson and Johnson currently working on the manufacturing side of one of our new robotics products soon to be released to the market. Through this talk “Dr Robot Will Now See You” I’d like to transport you to this future vision where this will be a more accessible reality for a lot of people. <Laugh>.

Rashmy Parimi: I want to go back a little in the history before I transport you to where we are today and what the future looks like. A lot of you must have seen this picture on the left of an early operating room where surgery was more of a spectator show. Antiseptics and anesthetics were not something of commonplace. There was no concept of sterilization and for a lot of, I would say decades back then, laughing gas was a commonly used anesthetic.

Rashmy Parimi: Even that was not highly recommended because there was mixed feelings either by the patients or the doctors to use it. A dentist came across ether being an effective anesthetic and he compelled the rest of the medical community to conduct a clinical trial to give more substantial data. And that was one of the starting milestones of making anesthesia a regular process of surgery.

Rashmy Parimi: I think the data convince people that one anesthetics are good. They’re not necessarily something that take you out of control. And also convince surgeons that they didn’t have to resort to methods like strapping down the patients, to help them go through the surgery, because without an anesthetic, the pain will make them move and that’s not something ideal. And they also felt that having a PA stable patient would give them more dexterity and stability to operate.

Rashmy Parimi: That was a very fast history of surgery back then. But from then to now, like there’s so much, you know, medicine has gone grown from deeps and bounds increasing human lifespan by at least 30 years.

Rashmy Parimi: And even today, I think the whole fascination with watching surgery has not gone away, but it’s a little more, I’d say refined from how it was in the photo depicted on the right towards, sorry, on the left to where it is on the right where there is more advanced rendering of the surgical procedure, either during to help other specialists participate in it or to a surgeon or a medical team in a far away location to help add more perspective to a complicated situation.

Rashmy Parimi: From a very low out like low outcome pain causing and a long recovery method to introduction of laparoscopy and endo, which has improved patient outcomes and reduced the recovery time and also improved the accessibility to a lot of people for complicated procedures. This is where I think with this is what most people are familiar with and laparoscopic was what sewed the seeds for the first ever use of robotic surgery.

Rashmy Parimi: This particular arm is maybe familiar to a lot of people as something used in, you know, large industrial assembly houses for large scale manufacturing, more like you know, car assembly facilities or other large equipment facilities.

Rashmy Parimi: But you’ll be surprised to learn was this was one of the first experimentations of whether robotic surgery can be used or not. And you will be even more surprised to learn that the area in which this was used was brain surgery. <Laugh>.

Rashmy Parimi: This was used to guide a percutaneous needle to do brain biopsy back about more than 25 years ago. And then this concept was further expanded to a colostomy and TransU urethral resection to further peak people’s re and research group’s interest to develop the concept of robotic surgery even more and work towards bringing it from a lab prototype to more of a reality.

Rashmy Parimi: In 2000, one of the pioneer companies of robotic surgery, Intuitive Surgical, they broke the ground finally when their system, the first ever Da Vinci system got FDA approval for general laparoscopic surgery.

Rashmy Parimi: It was this innovative device with robotic arms with visual systems and also they had help from nonprofit scientific research organization, SRI, to help them advance a lot of these initial prototypes. And that’s was how most people today, if they are familiar with robotic surgery, I think this is the one name they recognize instantly.

Rashmy Parimi: Let’s talk about what are the advantages of robotic surgery that makes it so attractive to use when evryone would admit that laparoscopy already takes us through a good bit of path onto, you know, smaller incisions and all of that.

Rashmy Parimi: We still get the same advantage as laparoscopy that is a smaller incision, which means quicker healing, lesser hospitalized time, which I’m sure all of you will, you know, relate to the expensive insurance bills and not having to deal with that. And also, it is cost saving and the body will recover faster through a smaller incision, since the amount of trauma is less.

Rashmy Parimi: The other advantage is the precision the instruments can reach into hard to reach places of the body without having a wide incision with accurate precision and stability, which makes a big difference in terms of your outcome of the surgery. And also with this precision al the comes with it, it adds an extra, I’d say boost to the surgeon’s abilities and gives them the confidence to tackle some really tricky procedures.

Rashmy Parimi: One of the important things of having a successful surgical outcome is good visualization. When you know you cut a part of the body, there is obviously going to be blood involved, and in typical surgery it could a lot of times block the view of what is going on there, but with the time your incision smaller cuts, that disadvantage can be overcome and it leads to a better outcome.

Rashmy Parimi: There’s a good example that I would like to use for how pressure virtualization improves the surgery. Having robotic vision is like if you want open surgery is like using a flashlight to look through a window into your house, while robotic surgery is like opening the door, turning on the lights, and then trying to look at your house. You can see it’s evident, which is a better way to look at your house.

Rashmy Parimi: And that advantage is offered to by the advanced imaging that comes with robotics surgery and with, in addition to all of these, the other advantage is exceptional dexterity. Everyone is familiar with how surgeons have these long schedules and if things do not go as planned, there is a lot of fatigue on them with the long hours and that can lead to that showing up on the surgery itself.

Rashmy Parimi: With robotic surgery, one of the things that can be controlled is to remove the tremor and other fatigue related impacts so we can reduce these inadvertent punctures or nicks which can cause unwanted bleeding into the body. Let’s look at few of the areas where today robotic surgery is used in one way of the other heart surgery where these very precise repairs that are needed is done using robotics stomach, though it looks like a big area, there is a lot of fine precise procedures that can be done in a better fashion using robotics.

Rashmy Parimi: General surgery of course, is another area where with a smaller incision and the precision offered, you can do a lot more compared to non robotic surgery. And same goes with the area of GY gynecological surgery where there is, you know, access issues and you want to make sure you don’t impact the healthy tissue or healthy organ parts.

Rashmy Parimi: Same thing goes to lungs where the access is extremely difficult and with kidneys where the, the areas so delicate important that you want to make sure you do not cause unwanted damage to the existing parts. In the area of orthopedic surgery, robotics have given an added advantage of very precise cuts and placement for implants and you know, it’s popularly used I think in hip replacement and knee replacements, which has become very common place in the society today. In the area of dental surgery, there is a product in the market today which help with dental implants and there’s, I’m sure there’s a lot more research going on.

Rashmy Parimi: And as I explained in my first example brain surgery, it started off <laugh>. The whole idea for this was sewn with brain surgery and it is still an area of widely researched today and they are trying to develop products in that area. So here I have some examples of some popular players in the market today. Roughly going over that, the first one is Johnson and Johnson’s robot Monarch, which is, which has FDA approval in the lung cancer and kidney stone management space.

Rashmy Parimi: Below that you have Medtronic’s robot Hugo, which has approvals in the general surgery space. And the picture below is Intuitives’ DaVinci. It’s a newer generation of it, which also has approvals in general surgery and a lot more areas on the right hand side. The first one is the Yumi robot, which is used in the dental surgery field. Their application right now is in the area of implants. The one below from Striker is the maker robot used for the orthopedic area. I don’t want to guess the wrong thing, but I think in the, a place of hip replacement probably. And the one below is from Siemens and this is a robot used in the cardiovascular area.

Rashmy Parimi: Now that I’ve peaked your interest on how, what are the advantages that come with this novel application? I’m sure all of you must be curious, how do you break it into this field? What are your pathways? Is it something very niche? Is it very small exclusive circle?

Rashmy Parimi: Well, I’d like to walk you through my own career path to kind of show you it’s really not all that difficult. And in the next slide, I will also kind of walk you through during the various stages in the life cycle of a product development, what are the different functions that interact and how different disciplines come together to successfully build a robotic surgical product. I started off by education as an electrical engineer, but using that as my foundation, I have worked on firmware for different products, electricity meters, crane systems, small devices which include wearables, thermostats.

Rashmy Parimi: I went into this not through either medicine or robotics. I started from a very normal field, which I’m sure most of you feel <laugh> a little easy to relate to. I did have a small ex in brush with medical devices early in my career where I was working as a part of a team on a prototype of a USB based ECG monitor.

Rashmy Parimi: If any of you have noticed the ECG monitor today used in the hospitals, it’s a big piece of equipment and it’s not portable. It’s used in a remote location and they want to share the data around for more opinions. It’s not easily done. There is that accessibility issue. But if it were in a USB form and the data can be collected wirelessly and shared across seamlessly without the boundary of a physical location, it it would be a great blessing to bringing healthcare to rural areas where accessibility is a big issue.

Rashmy Parimi: The proposition of that product was very interesting. And back then, I wanted to continue in that but then again it was just one research project. As I grew in my career, one of the chances I encountered was to be part of the startup Verb Surgical, which was working on a soft tissue surgical platform.

Rashmy Parimi: Verb Surgical has been acquired by Johnson and Johnson and that team is continuing the work on that platform. Hopefully soon that will be in the market helping people improve their quality of lives. And even if you notice through my career, the job duties I’ve done has varied from pure research projects to some integration to what I do today, which is manufacturing test. All of this is more about applying your skills, existing skills across different areas. I have not taken any new courses.

Rashmy Parimi: I have always maintained this curiosity to upskill myself on the job and try to read more on things I don’t much, that was how I was able to work through different domains within the same company.

Rashmy Parimi: Next, I want to talk about what are the various disciplines and roles that participate together during the development of a product. Initially when you want to establish the user needs and make sure a certain product is feasible from a regulatory perspective, the team that typically does the groundwork, the product managers who talk to the customers such as the physicians to make sure they understand what will help them. Then you have the systems engineers, who translate those customer needs into some kind of actionable product requirements. And then the clinical engineers, who also bridge the gap from a clinical perspective.

Rashmy Parimi: The regulatory affairs team helps trying to understand what, how the impact of that, you know, what is the burden of this product to make sure we are safe. And also how, how do we prove that this product is safe to use on human beings once the use case has been established And there is this clear requirements for the product.

Rashmy Parimi: Then comes a design phase where you have design engineers and various arenas. You have electrical design engineers, mechanical design engineers, UI engineers, UX engineers, all coming together to build different pieces of the system and of course test engineers to test all that has been built.

Rashmy Parimi: And for most large scale products, one of the things that has made big difference if the product moves forward in a given timeline or it does not launch off is the integration piece of it.

Rashmy Parimi: There is a lot of complex software and hardware coming together and integration plays a big role. We have the systems integration engineers trying to piece those puzzles, making sure two independent modules operate together as one big unit, and also clinical engineers from time to time to make sure what physically was decided in the beginning is still what the goal of it is towards the end.

Rashmy Parimi: And as the product goes into its future stages, the burden is to val validate and verify it so that we have the essential documentation for FDA approval. But before that, the manufacturing team and the supplier make sure they work with various vendors and internally and to build up these units that will provide the data for FDA to review and approve the device.

Rashmy Parimi: Once that is done during the commercialization phase, you have marketing team, the sales team, the service team to make sure the product is supported within the customers who are using it and also provide the feedback to support the next level of iteration of design and all of these resulting in a complete cycle.

Rashmy Parimi: As you can see, quality is something which is critically important through the whole process and weigh in in all of the design phases and the later validation and commercialization phases.

Rashmy Parimi: What is the future outlook for this field? This is an illustration from before the pandemic. Just few years ago, there’s been 77 companies and these are only the companies that are have gone public. There are a lot more stealth companies, who maybe close to finishing their product.

Rashmy Parimi: The number of companies have increased from a few million in the beginning of last decade to a lot more billions now. It’s a fast growing industry and there has been a lot of acceptance to make sure this field is supported.

Rashmy Parimi: And in general you’ll see these are the two areas where there has been a lot more progress in terms of adding new procedures and support in terms of surgeon’s interest and also success rates in the field.

Sukrutha Bhadouria: Rashmy, we can wrap up. It’ll be great.

Rashmy Parimi: Yeah, so I think this is my last slide, <laugh>. With this, I hope a lot of people have a lot of questions. I’m happy to answer that later. Please feel free to connect with me on LinkedIn. Thank you everyone for your time and thanks for having me here, <laugh>.

Sukrutha Bhadouria: Thank you so much Rashmy and thank you to everyone for attending and you know, posting all your comments and sharing your insights. Thank you.

Rashmy Parimi: Thank you.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!

Over 120 girl geeks joined networking and talks at the sold-out OpenAI Girl Geek Dinner on September 14, 2022 in San Francisco’s Mission district.

Hear lightning talks from OpenAI women working in AI with music and deep learning, sharing the power of trying and trying again, how to make language models useful, and much more at the OpenAI Girl Geek Dinner video on YouTube!

OpenAI Residency applications are open! OpenAI is looking for engineers and researchers who are interested in applying their skills to AI and machine learning. Please apply for OpenAI jobs here!

If you have an unconventional educational background, we encourage you to apply to OpenAI Residency (applications are open through September 30, 2022).

Table of Contents

1. Welcome – Elena Chatziathanasiadou, Talent Programs Lead at OpenAI, Recruiting & People – watch her talk or read her words
   
   
2. Multimodal Research: MuseNet & Jukebox – Christine McLeavey, Member of Technical Staff at OpenAI, Multimodal – watch her talk or read her words
   
   
3. If At First You Don’t Succeed, Try Try Again – Alethea Power, Member of Technical Staff at OpenAI – watch them talk or read their words
   
   
4. Making Language Models Useful – Tyna Eloundou, Member of Policy Staff at OpenAI, Policy Research – watch her talk or read her words

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• See open jobs at OpenAI and check out open jobs at our trusted partner companies.
• Find more OpenAI Girl Geek Dinner photos from the event – please tag yourselves!
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!

Transcript of OpenAI Girl Geek Dinner – Lightning Talks:

Angie Chang: Hello. Thank you everyone for coming tonight. My name’s Angie Chang and I’m one of the founders of Girl Geek X. We started over a decade ago as, Bay Area Girl Geek Dinners, and we’re still going strong. Thank you to OpenAI for hosting us for a second time. We’re really excited to see the new office and invite a bunch of Girl Geeks over to hear these lightning talks on AI and policy and all these things that we’re so excited to learn about tonight!

Sukrutha Bhadouria: Hi. I know you all were still chatting when Angie introduced herself, but she’s Angie and Girl Geek X is basically her brainchild. It started off with Angie looking to bring women together, I’m doing your pitch, Angie for you because I have a louder voice. Some people, they ask me if I swallowed a mic as a child because I’m so loud and I don’t need a mic.

Sukrutha Bhadouria: Anyway, I’m Sukrutha, so Angie started Girl Geek and it was back then called Bay Area Girl Geek Dinners, this was over 10 years ago. And when I had just moved to the Bay Area, looking for ways to meet new people and I found out about Bay Area Girl Geek Dinners dot com at that time, and I tried really hard to meet with Angie, but she was a busy bee doing all sorts of cool things, trying to change the world. And this was way before ERGs existed, right? So people didn’t have a way to connect with the community until they went to meetups.

Sukrutha Bhadouria: And Girl Geek Dinners, at that time, was the one way you could also get an insight into what these sponsoring companies worked on, what life was like. And so it also allowed people to get an opportunity to speak and a lot of the speakers at Girl Geek Dinners were first time speakers. They were too afraid to sign up for conferences. If you go to our website (girlgeek.io), you’ll see all these amazing stats on how since Angie started, there’s been a real shift in the environment in how people are more willing to speak at conferences, due to some of the chances they’ve gotten as a result of speaking at an event sponsored by their company. This organization exists.

Sukrutha Bhadouria: I joined Angie and we tried to change the world together. I’m happy to report that I think we actually did. We rebranded to Girl Geek X, and that’s when the organization hit 10 years. It was a sizable number of people working on it, it was Angie and me and it was just the two of us. And then Angie had this idea to really evolving into a company and so that’s when she started to bring on contractors, more people such as somebody who could take video of our events to make us look a little bit more professional and somebody else to do our website besides me. And we started to do podcasts.

Sukrutha Bhadouria: We started to do virtual annual conferences and we really, really, really were always consistently sold out for our in-person events that would happen at various companies that we partnered with through the Bay Area. Then COVID hit and the good thing is that we had already started to have a global presence through the virtual conferences that we had and we’ve now had four? Five, yeah.

Sukrutha Bhadouria: We used to be carpooling all around the Bay Area together to these events after work and now we are moms. So it’s amazing. We would look up and see amazing people working at these sponsoring companies speak and we’d be like, “Wow, look at them managing their mom life and parent life and coming to these events.” But I just think that it’s now become such a common thing that it’s not as isolated anymore. And I’m hopeful that, you all can come back again and again, because this in person event has really made me really happy.

Sukrutha Bhadouria: I’ve been holed up in my home office today, which is basically a room which also has my… What’s it called? A bike that stays in one place, stationary bike, so it has too many things going on in the room, but I wanted to give a big thanks to OpenAI for hosting us for the second time, for sponsoring for the second time. And I hope that we can keep doing this. So please do get your companies to sponsor and encourage them to do it in person. That’s all I will say. I know I said a lot more than I had planned, but thank you again, and Angie.

Angie Chang: Thank you Sukrutha, for the intro. I guess I should talk up Sukrutha a little more. When I first met her, she was a software engineer in test, and now she is at Salesforce as a Senior Director of Engineering there, so I’m very proud of her. And over the years we… She mentioned we have a podcast, we have annual virtual conferences!

Angie Chang: We’ll be launching a career fair virtually as well, to be announced. And I don’t want to say too much. We have an amazing line up of speakers tonight and we’re going to invite up first, Elena, who is our host for the night from OpenAI.

Elena Chatziathanasiadou: Hi everyone, I’m Elena. I work here and I’m on the recruiting team, I’m leading the Residency program right now. I’m very excited that you’re all here and have joined us together. Really want to thank Angie and Girl Geek X. We’re very excited to deepen our partnership together and to be back in the office here all together, in the new space and to experience this tonight.

Elena Chatziathanasiadou: We’re very excited about having you here and in terms of what we’ll see tonight, we’ll have a series of lightning talks and then that will be followed by Q&A and then we’ll get some dessert in the area that we were before and then we’ll wrap up at 8:30. But before we get started, I did want to take a moment to make a quick plug and share that…

Elena Chatziathanasiadou: We’re actively hiring for our Residency program and that includes both research and engineering roles and the goal of it is really to help develop AI talent. The program, it offers a pathway to a full-time role at OpenAI for folks that are currently not focusing on AI and are already researchers or engineers in a different field.

Elena Chatziathanasiadou: We’re really excited to hear from you. If you do have an interest in making this career switch, come talk to me after. And we’ll also have full time recruiting team members and positions that we’re hiring for across research product and engineering that we can tell you more about. Please come find us and learn more about the interview process, but also what the program offers.

Elena Chatziathanasiadou: With that I wanted to introduce our first speaker, Christine, who’s currently managing our multimodal team and previously worked on music generation research, created MuseNet and was collaborating on Jukebox. And before that was a classical pianist who transitioned into a researcher as well. I’ll hand it over to Christine. Thank you so much.

Christine McLeavey: Thank you. So yes, it’s really an honor to be here tonight. Thank you all for being here. And this Residency program is near and dear to my own heart, because I first joined OpenAI through, what was then the Scholars Program and the Fellows Program and those are the programs which have since evolved into this Residency program. I’ll put a plug in for anyone who’s considering it.

Christine McLeavey: I want to talk this evening about my own path through OpenAI, but especially about the two music models that I worked on during the time here. I thought I’d start by just going ahead and playing an example of each of the models. The first one, this is the one I worked on when I was doing the Scholars and Fellows program. This is MuseNet, which works in the MIDI domain, so this is the model trying to generate in the style of jazz. Okay, I’ll cut that off and then after I joined full time, I was lucky enough to collaborate with some amazing researchers here to work on a model that was instead working in the raw audio domain. The fun of that is you get to imitate human voices. This is trying to do the style of Elvis with lyrics by Heewoo. Okay.

Christine McLeavey: Elena mentioned before being at OpenAI, I was actually working as a pianist, I had done some math and physics in college, but obviously it had been a long time and so I think I took a good year of self studying before I applied to anything. And I thought I would just give a shout out to three of the online programs that I particularly liked at that point. They’re all amazing. But then I was lucky enough to join the first cohort of scholars that we had here. And at that point I was just trying to do this process of learning about all these different models. And I had this feeling that instead of just copying a model or copying what someone else has done, let me just try to translate it into a field that I know well, which was music. And so what became MuseNet was really my attempt to take all of the stuff I was learning and then apply it to the music domain instead.

Christine McLeavey: MIDI format is this really nice representation of music. I think of it as the way that a composer thinks of music, so it’ll do things like it tells you what notes it plays when, the timing of it, the volume of it, things like that, which instrument is supposed to play. But it loses all the actual detail of when a human takes it and performs it. You don’t get a person’s voice, you don’t get the sound of a great cellist, anything like that.

Christine McLeavey: The nice thing is it’s what you trade in expressivity, you get in this nice really meaningful representation. It does sound pretty terrible when you try to render materials. As a musician, just thinking about the structure of music, this was a nice simplification for a scholars project. What I did is I took a bunch of MIDI files and I tried to pull them out and turned them into a sort of language to make them look as much the sort of thing that you could get in your own net to predict as possible.

Christine McLeavey: I did things like I would always tell the model which composer or which band was going to be first and then things like what tempo was going to be when notes would turn on and off, and a wait token, which would tell the model how long to wait, things like that. And then what you end up doing is you translate that tokenization into just a dictionary of numbers and the model sees something like this. Which I think that this is the first page of a Chopin bellade or something.

Christine McLeavey: What the model is faced with is this task of given the very first number, what number do you think is going to come next? And then given the first two numbers, what number is going to come next? And when you first look at the first thing and when the model first sees it’s like how do you do this? What does that even mean? It feels like an impossible task. But what happens is the model sees many, many, many examples of this.

Christine McLeavey: And over time it starts to pick up on, ah, if I see 4,006 somehow I tend to see 586 more often after that or something. It starts to pick up on these patterns, which we know because we know the tokenization was like, oh, if a piano plays the note G, then probably soon after it’s going to turn off the note G or something. It has real musical meaning to us. But the model is just seeing these numbers like that. The nice thing is the model gets really good at this job and then you can turn it into a generator just by sampling based on, I thinks there’s like a 20% chance this token’s going to come next, so 20% of the time take that.

Christine McLeavey: The other really fun thing you can do is you can then study the sort of mathematical representation you’ve gotten for these tokens. So I was always giving it the composer or band token in the beginning and now you can look at the vectors or the sort of embedding that it learns through these composers.

Christine McLeavey: And as a musician it’s really fun because I would clearly think that Da Vinci and Ravel, for all these French guys are related and the model just picked up on the same thing, which is cool. But the other really fun thing is that you can mix and match those [inaudible]. So here is the start of one of my very favorite Chopin, Nocturnes. So I actually just gave the model the first six notes of that and this is what the model thought, if instead it was being written by [inaudible] It was a bunch of VPs. It goes on for a while, but I’ll cut it off there. And that was MuseNet.

Christine McLeavey: And then I ended up joining full time after that and I was lucky enough to collaborate with Prafulla and Heewoo on taking music generation over to the raw audio domain. And so in a way this is a much harder problem because now whereas in MIDI world you have just nice tokens which are meaningful in a musical way, raw audio is just literally 22,000 or 44,000 times per second.

Christine McLeavey: You’re recording how loud the sound is at that moment in time and the nice thing about it is it gives you all this expressive freedom, right? Literally any sound you can imagine you can represent as a sound wave, just audio recording to that. The trouble is there are just so many ways for those waves to go wrong or those patterns to go wrong. If you mess up on the short scale, it’s just like crazy hissing noise. If you mess up on long scale, your piece sadly starts getting out of tune or the rhythm drifts or so many ways it can go wrong, it’s really an unforgiving sort of medium. And the problem is now in order to get a minute of music, it’s no longer maybe 3000 tokens you have to do, it’s maybe a million numbers that you have to get correct.

Christine McLeavey: We approached this by looking at ways that we could compress the music to make it more tractable because at that point a transformer could maybe deal well with the context of 4,000 tokens or something. We used an auto encoder to do three different layers or levels of compression and the sort of least compressed on the bottom. The nice thing about that is it’s very easy to translate it back to the regular raw audio. If you put some original song in and then back out, you don’t notice any loss at all. Whereas if you put it through the most compressed version, the nice thing is now it’s super compressed, like 3000 tokens might get you half a minute of music or something. But if you go through this simple just trying to reconstruct the raw audio, it sounds really bad. You can sort of tell that someone’s singing but you’ve lost most of the detail.

Christine McLeavey: The nice thing about it is when you work in that top layer of tokens, now this looks a lot like the MuseNet problem or even just a lot language problem where you’re just predicting tokens. So we train a transformer on that. We sort of added in the same which person was singing, which band was playing, and then we also added in where you can write the lyrics in, so the model conditions on the lyrics and then generates these tokens. And then I won’t get into the details, but we had to train extra transformers to do this upsampling process so that you could get back to raw audio without totally losing all the detail.

Christine McLeavey: The fun thing is you can do things like ask it to generate in the style of Sinatra singing Hot Tub Christmas and I have to put in a book, these were lyrics by at, that point, GPT-2. All right. It’s a Christmas classic now. And then last I wanted to wrap up by talking a little bit about the multimodal team, which is the team that I’m really excited to be managing these days. It’s this really, really great group of people. Unfortunately, our current projects are all internal and I can’t talk about them, although stay tuned, we’ll be publishing them to the blog when we can. You might recognize Clip, which was work done by Alec and Jong Wook both on our team. This is, I guess, nearly two years ago already, but made a really big impact on the image work at that point. And then just to put in a plug for the team, we’re about a group of 10 at this point and we will be hosting a resident in 2023.

Christine McLeavey: Please reach out if anyone’s interested to talk more. And then we’re doing all sorts of projects in the sort of image, audio and video domains both on the sort of understanding side and generation side. And we end up working really closely with algorithms, which is the other team that tends to do a lot of awesome multimodal projects. But then also anytime we get close to things that we’re looking at putting out tech customers, we end up working with applied through that and then also obviously scaling because at OpenAI we believe deeply in this, get a good pattern and then scale it up and it becomes awesome. So thank you so much for your attention.

Elena Chatziathanasiadou: Thank you so much, Christine. That was awesome. So now next we’ll have Alethea. Alethea has spent the last couple of years at OpenAI working on getting neural networks to do math. Before that, they built large infrastructure health system, studied math and philosophy and spent lots of time singing karaoke. Welcome, Alethea.

Alethea Power: Thank you. So this talk is called If At First You Don’t Succeed, Try Try Again. It’s been a wild few years. I decided I wanted to give an uplifting and encouraging talk. It’s a short talk so it doesn’t get too deep into technical details, but if you’re interested in it, please find me afterwards. I will talk your ear off about it.

Alethea Power: Okay, my name is Alethea Power and yes, Patience is actually my middle name, which will be very relevant for this talk. Okay, so about 10 years ago I was a software engineer and site reliability engineer and my dream was to get into artificial intelligence, but I didn’t know how to do it. I didn’t have a degree in AI, I didn’t have any background in AI, I didn’t have any idea how to break in. So I thought, ah, I probably need to take some time off to study this before I can get into the field.

Alethea Power: I started saving up some money so that I could take time off to study. But by the time I had enough money saved up, I realized I needed to handle my gender issues. So I took that time off to go through a gender transition instead of studying AI. Eventually though I was finally ready to try and break into AI in some form or fashion and that was about the time that OpenAI hosted their last Girl Geek Dinner, that was in 2019. And I came to that talk and I met one of the recruiters who stunned me by telling me I didn’t need to have a degree in AI and I didn’t need to have a background in AI to be able to work here.

Alethea Power: She introduced me to the Scholars Program, the same program that Christine went through, which today is called the Residency Program. And I applied to that and I got in and I had the best mentor in the entire program, Christine. I’m second generation scholar up here. But there were in addition to the obstacles before, there were obstacles after joining the program as well, about three weeks after I joined, there was a pandemic, you may have heard about it. But despite spending a lot of time fearing that I might die or people I love might die for some reason or another, health or political, Christine was very kind and understanding and supportive and she helped me get to the point where I had learned a ton about artificial intelligence and managed to do a great project and I ended up applying full-time and I got three offers here. Thank you. I wasn’t trying to brag, but thank you. This is more to encourage you.

Alethea Power: I ended up taking a job on a team that was trying to teach neural networks to reason and do math. And what I want to talk about here is about a year after I joined that team, I released my first research paper called Grokking: Generalization Beyond Overfitting on Small Datasets. I’m going to give you a very basic introduction to what all that jargon means. And like I said, if you want more technical details, come talk to me afterwards. So first I need to explain how training neural networks works. If you have a background in ML, this is going to be very basic 101. If you don’t, it’s going to be exciting.

Alethea Power: Okay, so usually when we’re trying to train a neural network, we’ve got some amount of data that captures a pattern that we want that neural network to recreate in the future. And often if we’re doing what’s called supervised training, we’ll break that data up into training data and evaluation data. And you can think of this, the training data is sort of what we actually teach the neural network, what it learns from. This is like classroom education and evaluation data is basically like pop quizzes to see how much the neural network learned. And neural networks have this nice property where you can pop quiz them. They don’t learn anything from the pop quiz, they just tell you how they did and then five minutes later you can pop quiz them again and the questions are all new again, they have no memory of them. Throughout the course of training, we measure the performance of the neural network on both the training data, the classroom instruction and the evaluation data, the pop quizzes.

Alethea Power: And there’s two main ways we measure this. One is called loss. I won’t go into details right now about what loss is, but the short version is it’s a differentiable function calculus derivatives that we use to actually figure out how to modify the network, so it learns, when loss goes down. The network is learning. Accuracy is exactly what you would think of being like a test score, so 0% accuracy means you got every question wrong. A hundred percent accuracy means you got every question right. This is what a very successful neural network training looks like. You can see, oh, the x axis here on both of these graphs is steps of training. You can see that as we train this neural network along the loss on both the training and evaluation go down. It’s learning what it’s supposed to learn from and it’s able to generalize that to the pop quizzes.

Alethea Power: It’s doing well on the tests as well and then this is what it’s actually scoring. So by the end of this training it gets up to 90% accuracy, so it’s got an A. Sometimes though, if you train a neural network for too long, it starts to do what’s called overfitting. You might remember the word overfitting from the title of the paper. In this case, the neural network learns too much detail from the training set that doesn’t really generalize to the rest of the world. And so its performance on the quizzes starts to get worse. So an example of this in this paper, I was training neural networks to do math, basic mathematical equations. For instance, if it happened to be the case that the training data had more even numbers than odd numbers, and if it was trying to learn addition, then it might learn that usually the answer is going to be even. Well, in reality that’s not true in addition.

Alethea Power: In reality, you want to actually know how to add and the number’s going to be whatever it is. So that would be an example where it learned some sort of incorrect, non-generalizable information from the training set and that made it start performing worse on the evaluation set. And you can see here in this situation, the accuracy on evaluation would go back down. Sometimes, and this is very common when you’re trying to get a neural network to do math, you have an even worse situation where the same thing happens with your loss, but it consistently fails the pop quiz every time. Gets to a 100% percent accuracy on the training data and fails the pop quiz. This means the network and we were using similar kinds of networks to the ones Christine was talking about, just math instead of music, this means the network never really understood what it was learning, it just memorized it.

Alethea Power: This is like the kid who knows that when you say six plus four, you’re supposed to respond with 10 but has no idea how to actually add. So this was a common scenario when training neural networks to do math. They’re really good at pattern recognition, but they’re not always good at understanding a deep analytical precise truth underneath the pattern. Well then one day we got lucky and by lucky I mean forgetful. So one of my coworkers was running an experiment like this and he went on vacation and forgot to stop it. And so a week later he came back and it had just kept studying and studying and studying and studying and studying and studying and studying and studying and studying. And it learned. So what happened here was, it went into this overfitting regime where usually we’d say, ah, it’s learned all it can learn from this training data.

Alethea Power: There’s no more to learn and see, it still had zero accuracy and it just kept getting worse and worse and worse. And then suddenly long after it memorized all of the training data, it had an ‘aha’ moment and it was like, oh, all this stuff that I memorized actually makes a pattern and the pattern is addition or division or S5 composition or whichever task we had it working on. And then the loss started coming back down on the pop quizzes and it went up and it got a 100%. This is weird, this never happens in neural networks. We dug in and recreated this many times, implemented it twice, saw the same behavior with two completely independent implementations on a wide variety of tasks and there’s all sorts of other interesting stuff about when this happens and when it doesn’t, ask me in the questions afterwards.

Alethea Power: The point here is at first the network didn’t succeed, but it just kept trying the same way I did when at first I couldn’t get into AI, but I just kept trying. We named this phenomenon where it finally figures it out Grokking, and we named this after Robert Heinlein’s novel Stranger in a Strange Land. It’s a science fiction book and Grok is a Martian word in that book, which means, “To understand so thoroughly that the observer becomes a part of the observed to merge, blend, intermarry, lose identity in group experience.” And it turns out this is exactly what these neural networks do. I’m going to let you take pictures before I change the slide.

Alethea Power: This network was trying to learn modular addition and modular addition you can think of is adding hours on a clock. Also, thank you to Christine for that analogy. If you have 11 and you add 3 to it, you don’t end up with 14, you end up with 2 because that’s what happens on the clock. The clock is modular 12, we were having it learn modular 97, and then we tore open the network that had grokked afterwards to see what was going on inside of it and it had actually built internally this circular structure of the numbers. It had created the mathematical structure we were trying to get it to learn that allowed it to actually solve the problem. Did this with all different kinds of problems, so we had one network learning to compose permutations and it found what are called subgroups and co-sets out of that, details later. But the point is, it worked so hard for so long through so much failure that it became the knowledge it was trying to get.

Alethea Power: The point here is, that if your dream is to get into AI, even if you have no background in AI or whatever your dream is, it doesn’t matter. Keep trying and keep trying and keep trying and keep trying and maybe you can get there eventually. And in particular, if your dream is to work at OpenAI, which I highly recommend because this place is fabulous, then try, even if it’s not the background you have already, even if you feel like you have a weird background or you’re not like the people here or like the people in this field.

Alethea Power: We’re a humanitarian organization. Our core mission embodied in our legal structure and our financial structure is to make sure that artificial intelligence benefits all of humanity instead of just a small number of rich people in Silicon Valley. And to be a humanitarian organization with a humanitarian mission, we need a wide diversity of perspectives here. If you have a different life story, a different path, different perspectives than we’ve seen before, that makes you more valuable here, not less, so please consider applying.

Elena Chatziathanasiadou: Thank you so much, Alethea, That was awesome. And now next we’ll have Tyna, who’s on the policy research team currently doing our rotation on applied research and she participated in the OpenAI Scholars Program, has spent some time researching economic impacts of our models, building safety evaluations, and collaborated on web GPT and moderation API. Let’s hear from Tyna.

Tyna Eloundou: Wow, so many of you. Let’s see. Okay, this works. Hi, everyone, thank you so much for coming. I’m Tyna Eloundou, I’ll be speaking to you today about making language models useful. A bit about myself, let’s see, wow, I’m also a former scholar. I can’t make the claim to third generation because Alethea was not my mentor, but they were super helpful in making my experience here amazing. And part of that culture and that welcoming environment was a reason I chose to stay on after the scholars program [now the Residency program].

Tyna Eloundou: Today we’re going to be talking about language models and by language model, I mean any model that has language as input and output. So that could mean GPT-3, CODE-X, or BigScience’s Bloom, what have you. Okay, this is going to be the only equation you see throughout this talk and it’s really not that important, but I think it gives us some context as to where we’re going.

Tyna Eloundou: Looking back at this, this is the training objective for GPT-3 and for all GPT like models. Given a corpus of tokens, right? We define the objective to maximize this likelihood, L, which is defined as a conditional log probability over a sequence of tokens that is modeled by a neural network with parameters data that is trained by gradient descent. Now you can forget everything I just said.

Tyna Eloundou: Essentially this optimization produces these models that are trained to predict tokens, but that in itself may not be that useful on its own. I don’t think I’m giving away any secret sauce by revealing this equation to you, but it is remarkable that somehow we go from this to models that can produce, oh sorry, that can do that, right? Write prose, write code or parse data and so on.

Tyna Eloundou: I’d like to talk a bit about the notion of usefulness itself. One way to think about whether language models are useful in the first place is in the pragmatic sense. In the ideal scenario, we would be able to succinctly communicate our goals and preferences to a language agent without having to laboriously explain and list what to do and what not to do.

Tyna Eloundou: How did we initially get usefulness out of language models? When these models were first being developed in research labs, some researchers came with some ideas about how to really get them to do what it is that you want them to do. And these are two of the most prominent ones. One was few shot prompting, which is a method by which you really tell the model what the task is and before putting it on the spot, so to speak, you give it some examples of what you like to do, some demonstrations, right? For translate English to French, you could have a pen to [foreign language], I’m hungry to [foreign language], et cetera. And the translation that you actually want, you say, I would like to eat ice cream and hopefully with that same formatting you get the model to translate to French.

Tyna Eloundou: The other method is supervised fine tuning, which involves essentially just having examples for the model and then kicking off another round of training so the model can become hyper focused on your task and hopefully improve its performance on that task. So as many of you probably know, OpenAI has since then adapted this iterative deployment approach, which helps us put models in the hands of real people and understand how they interact with them. At the time of GPT-3 release, it was doing great by research standards, right? And unfortunately a lot of these research metrics are designed around these methods that we’d spoke about before, which are to prompt with few shot prompting or perhaps to do supervised fine tuning. Once we deployed, we really quickly learned that people don’t like prompt engineering. In fact, they don’t really like to do a lot to communicate their goals to the model, which is fine. It’s a feature, not a bug.

Tyna Eloundou: At its most helpful, a language agent can infer what we want without lots of specification and carry out those inferred goals effectively and efficiently. Unlike researchers, people were using natural language instructions to ask GPT-3 for what they wanted. But because of the training objective that we saw previously, the model was really tempted to just pattern match, right? If you gave it a prompt of write a short poem about a wise frog, it would very helpfully give you similar types of prompts instead of following your intent. This spurred a research effort within our alignment team to teach the models how to follow direct instructions. They did this using two insights. The first is borrowing from the supervised fine tuning or supervised learning literature where you can train the model based on examples or demonstrations, right?

Tyna Eloundou: You have a prompt and you tell them what you would ideally like it to do. And the second insight came from the reinforcement learning literature where you have some humans compare outputs. And so this model learns to generate, that model learns to compare, right? That model learns to tell this is good, this is bad. And so now with these two things, you can kick off this joint training process where you have a model that’s generating and then a model that’s critiquing, and this is good, this is not so good.

Tyna Eloundou: Over the course of training, the model learns to get better at pursuing this objective, which is no longer the pure language model laying objective and now it’s the instruction following objective. So the resulting model was InstructGPT, which is presented here. Well, yeah, you can see the output. It’s a poem, it’s about a frog, mentions wisdom, and it’s pretty short. I feel like all the requirements were met for following instructions there.

Tyna Eloundou: This was a plot that was quite striking to me. This is one of the main results from the InstructGPT paper. When I first saw this, it didn’t make a ton of sense until I really understood the research behind it. But I think that you can think of the Y axis as a proxy for usefulness and the X axis. We have model size and conventional wisdom has it that… We’re at OpenAI as you scale things, things get in general better. But you can see that even at its smaller size, right here, if you can’t see it’s 1.5 billion parameters, even at its smallest size InstructGPT was deemed to be more useful than any permutation of the base GPT model. So I started this discussion by talking about how research based approaches were not pushing far enough in terms of getting us usefulness out of these models. There’s now this emerging literature focused on helping models be more effective in tasks.

Tyna Eloundou: Broadly speaking, this literature involves having models break big problems up into smaller problems or things step by step before coming up with a final answer. And this does not need to be at odds with our human alignment driven research. In fact, right here you see a result by Kojima et al. and although their results are great overall across the board, we do see that they make the Instruct models even greater. There’s such a huge gap, a huge gain that we see with the Instruct series of models.

Tyna Eloundou: I would like to conclude by thinking about the next steps in this line of research. We know that there can be some instructions that can be malicious or exploitative or deceptive. If language models were to pursue usefulness at all costs, they might become dangerous in the pursuit of dangerous instructions or dangerous intent. Could there be other objectives we pursue along with usefulness that get us helpful but not dangerous models, perhaps kindness or hopefulness?

Tyna Eloundou: And lastly, with instructions, we’re mainly in the driver’s seat and we initiate interactions. As language models become smarter, perhaps kinder, more capable, it may be appropriate to think of them as collaborators and they may be capable of initiating ideation, creation among other things. What are the different modes of interaction we would like to have with these models? Would we want them to advise us? Would we want them to inspire us? Perhaps at Girl Geek X 2042, it’ll be a language model presenting about something new. Thank you.

Elena Chatziathanasiadou: Thank you so much all for joining. I guess with that note, I did want to mention that we’ll kick off mingling time and dessert in the area that we were before and our speakers will be available for you to ask them questions. We also have some of our recruiting team members here tonight. If you all want to come up to the front to just quickly introduce yourself or just say hi so that people can see you and then you can all come find us.

Elena Chatziathanasiadou: As I mentioned in the beginning, I’m Elena, I’m also hiring for the Residency program, so come talk to me, come find me. And then we also have some demo stands of our Dolly product and also our GPT-3, if you want to check them out. Jessica and Natalie will be doing those demos. So yeah, go find them as well.

Elena Chatziathanasiadou: Thank you all for being here. I hope you enjoyed it. Thank you to our lovely speakers and to Girl Geek X, to Cory and to all of our ops team and everyone who helped put this together and let’s go enjoy some dessert!

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• See open jobs at OpenAI and check out open jobs at our trusted partner companies.
• Find more OpenAI Girl Geek Dinner photos from the event – please tag yourselves!
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!

Women at New Relic discuss observability, metrics, monitoring, community, APIs, React, and leadership at the New Relic Girl Geek X event with over 190 girl geeks joining the lightning talks and leadership panel discussion online.

Table of Contents

1. What is Observability? – Padmaja Gohil, Senior Solutions Consultant at New Relic – watch her talk or read her words

2. Customer Success and Value Realization Through Value Metrics – Kate Kordnejad, Lead Principal Technical Account Manager at New Relic – watch her talk or read her words

3. How Browser Monitoring Can Be Used To Improve Website UX and UI? – Carolina Rotstein, Solutions Consultant at New Relic – watch her talk or read her words

4. DE&I – Finding a Community with New Relic ERGs – Solmaira Flores-Valadez, Senior Technical Account Manager at New Relic – watch her talk or read her words

5. Observability in the Age of Web3 – Nora Shannon Johnson, Solutions Consultant II – LATAM at New Relic – watch her talk or read her words

6. APIs: Get Your Data When You Want It and How You Want It – Sarah Hudspeth, Solutions Consultant at New Relic – watch her talk or read her words

7. The Power of React.js – Jo Ann de Leon, Senior Technical Account Manager at New Relic – watch her talk or read her words

8. Leadership Panel – Ariane Evans, DEI Manager at New Relic, Nada Da Veiga, GVP, Technical Solutions Sales at New Relic, Erin Dieterich, Senior Director, Social Impact & ESG at New Relic, Kim Camacho, Director, DE&I at New Relic, Tracy Ravenscraft, Director, Technical Account Management at New Relic, Stefanie Smith, Senior Manager, Talent Acquisition at New Relic – watch the panel or read their words

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• See open jobs at New Relic and check out open jobs at our trusted partner companies.
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!

Transcript of Girl Geek X New Relic – Lightning Talks:

Angie Chang: We’re going to give people a chance to join us, but in the meantime, I guess I’ll start with some introductions. Hi. My name is Angie Chang. My pronouns are she, her, hers. I wanted to say, thank you so much for joining us for our Girl Geek X New Relic event. I want to encourage us to connect with each other. If you can, I would invite you to put in the chat, your name, your location, your job title company, and your LinkedIn URL, so we can all get connected. Feel free you to connect with me. I wanted to introduce myself and give you some background as to what Girl Geek Dinners is about.

Angie Chang: I started Girl Geek Dinners in San Francisco when I started working in engineering, and I felt a bit lonely on the team as the only female engineer. And I go to all these tech events, but I wanted to go to tech events where the gender ratio was flipped. These didn’t exist in 2008. I decided to start my own series of Girl Geek Dinners. It turns out, after five days of posting about online, we had over 400 girl geeks that were interested in joining us for our first Girl Geek Dinner. And then the next one was sponsored by Facebook. And then we just snowball from there.

Angie Chang: And now today we have over 200… I think we’re at 300 Girl Geek events. We’ve also started things like a virtual conference every year, celebrating international women’s day. We really have also filled out our product portfolio of this podcast. You can go on YouTube. All the talks that you will hear today will also be on our YouTube channel. I invite you to subscribe to that. It’s at youtube.com/girlgeekx. And you can find all the videos from our previous events, and today’s event on there as well. [inaudible] chatting.

Angie Chang: I wanted to share how much I love learning from going to all these events over the years, because from listening to the women working in the diverse corners of male dominated industries, from engineering to sales, we have heard from people share their expertise. And we also learned things like, that job titles are constantly evolving. I remember thinking that this was a really interesting part of engineering and tech that we often don’t think about, of the first thought of big tech or tech companies.

Angie Chang: When I used to work at Hackbright Academy, a coding bootcamp for women, there was some women that I met at New Relic who were sales engineering leaders. And I thought they were so cool, because they not only knew engineering, but they were also very savvy on the business side. It’s because of the sales stuff. I remember thinking that this was a really interesting part of engineering and tech that we often don’t think about, of the first thought of big tech or tech companies.

Angie Chang: The sales engineering side is overlooked. I’m glad that we have heard from people like Tracy, and all the solutions consultants and technical account managers, who are interested in sharing the projects they’ve been working on and their passion for technology, today with us. We are excited to partner with New Relic, a company leading in full stack observability. We’ll hear from the solutions consultants. And they’re formerly called solutions engineers, sales engineers, and technical account managers. I think what I’ve learned is that solutions consultants are pre-sales, and technical account managers are post-sales, but that’s something that you can have a conversation with people about afterwards in networking.

Angie Chang: These lightning talks will be discussing observability metrics, ReactGraphQL for APIs and more.

Angie Chang: Now our first speaker on customer facing technical roles at New Relic is Padmaja Gohil. Padmaja is a senior solutions consultant at New Relic, and loves being a sales engineer, because it not only helps her stay at the cutting edge of technology, and she gets to work with a multitude of customers using these technologies. In her free time, she loves listening to music and adventure parks. Welcome, Padmaja.

Padmaja Gohil: Thank you, Angie. Hey, everyone. Very nice to you. Is everyone able to see my screen? Angie, can you just give me a thumbs up?

Angie Chang: Yay.

New Relic solutions engineer Padmaja Gohil talks about observability in software development, the phases of observability, and observability as code at Girl Geek X New Relic virtual event. (Watch the talk)

Padmaja Gohil: Okay. Awesome. I’m Padmaja Gohil. I’m currently a senior solutions engineer with New relic. Today we’re going to be talking about all things observability. Quick disclaimer, please not hold me accountable to any sort of overlooking statements. Before we dive into the presentation itself, I would like to give you guys a quick glimpse into my journey so far. Growing up, I’ve always wanted to be an engineer, but once I started my engineering degree, I realized that my interest lay somewhere at that nexus of tech and business, which led me to do my masters in engineering management, where I studied business concepts, but focused in high tech industry. I’ve also previously dabbled in consulting, venture capital and data privacy.

Padmaja Gohil: I’ve been a solutions engineer with New Relic for the last three years. I absolutely love what I do. New Relic is an observability platform, and because of which I’m going to be talking about observability today. But at the same time, in my day to day, I get to work with a lot of different customers. Understand how they’re using technology, and I help them achieve their goals using New Relic. If you guys have any questions about what solutions consulting, solutions engineering, sales engineering is all about, feel free to reach out to me on LinkedIn or my email address, and I would love to have a chat. The way I’ve structured the presentation today is, we’re going to talk about what were the changes that we saw in the software development space, that led to observability. Why do we really need it? What it is, and the different phases in which you can implement it.

Padmaja Gohil: And finally, we’re going to touch very quickly on observability as code. We’re going to be covering a lot of ground. Again, feel free to get in touch with me if you have more questions, or if you would like to learn more. Now let’s take a look at how has the model software industry evolved. If you look at the left, on the left side of this screen, you’re looking at our past. Our past was primarily Monoliths. They were stood up on on-premise servers. Usually scaled vertically, very static operations based scenario. We would release once or twice a year. I still remember the days when we would have to manually update our softwares. Now, fast forward to today. Today’s architectures are more microservice based. They’re open sourced. They’re more complicated. They’re usually hosted on Kubernetes cluster.

Padmaja Gohil: We went from releasing once or twice a year, to releasing maybe multiple times a day. This has been great in terms of the business. We’re able to push out new code, push out your releases and update our software faster, but it on-boards with it a level of complexity when it comes to troubleshooting, detecting issues and finding resolutions for it. This alongside other reasons is why we need observability. In the days of mainframes and static operations, when things went wrong, what would happen is, we would have maybe a couple of dashboards, that we would get alerted on. Usually these dashboards were static. We had run books for all of them, to figure out what’s going wrong and to fix issues. Now, typically, these systems would fail in the same manner over and over and again and again.

Padmaja Gohil: It was a little more simplistic than maybe today. Now, today if things were to go wrong, I’d be staring at my screen, wondering what’s going wrong. Is my cloud provider seeing an outage? Is someone deploying code? Is that the reason why I’m seeing some sort of an issue. Or I could be staring at the symptoms and not the root cause. There is so many ways in which things could break, that it’s really hard and complicated in how we do troubleshooting today. Also there has been an increased frequency of CodeDeploys. We went from once or twice a year, to multiple times a day, which can increase the chances of things going wrong. We no longer have discrete application owners.

Padmaja Gohil: We have distributed systems, but at the same time, we also have distributed teams working on things. There is a need for contextualized data in case of… if a person were to just come in blind, not knowing the history of the systems, they can quickly take a look at things and start fixing. These are just some of the reasons why we need observability today. But let’s take a look at what the definition is. There are a lot of definitions out there. The way I like to think about it is, how well do you understand your system from the work it does? It enables you to do a lot of things. For example, it enables you to collect and alert on the telemetry data types. There’s four telemetry data types, and these are the pillars of observability.

Padmaja Gohil: I’ll speak to those further in the presentation as well, but it’s metrics, events, logs, and traces. These are the four pillars of observability. Observability allows you to focus on your day to day. As software engineers, your job is to, let’s say, deploy code faster, come out with newer features. Your job is not to spend a lot of time in fixing issues. Observability also allows you to focus on that. It enables you to troubleshoot faster. It makes sure that you are ensuring up time and performance while you push out this newer code. It also gives you the confidence to push out new code, because let’s say if things were to go wrong when you were deploying, you have the confidence that yes, I have the system in place to fix those things. It builds that culture of innovation as well.

Padmaja Gohil: In real life scenario, there are so many different ways in which you can implement observability, but there are three phases, three broad phases in how we implement it. I would like to talk to you about it. The first phase is the reactive phase. All of us might have heard the saying that you cannot improve what you cannot measure. The first phase is where you start instrumenting your entire tech stack to collect data. You’re collecting metrics, events, logs and traces from all of the tech stack. You are then understanding how your applications are behaving. A lot of times you might not know what normal looks like for your applications. What does your normal response time look like? What does the normal error rate look like? The first phase is when you are establishing the normals and the baselines, and then you’re setting up foundational alerts on it.

Padmaja Gohil: That’s what the first phase is about. The second phase is now codifying your team’s work. Now, when I say that, what I mean is, you are setting up service level objectives for your application, because what happens is you’re seeing plethora signals coming at you. And you now need to understand how do you measure the success of your application? One of the ways to do that is by setting up service level objectives, and service level indicators, which are SLIs. Let me give you an example of what an SLO can look like. For a web application, an SLO could be that the videos should start playing within the two seconds, and 499% of the time during that one week period. That is your SLO. Now, the service level indicator, which is the SLI, measures the proportion of videos on the website that start playing in less than two seconds.

Padmaja Gohil: You start setting up these kinds of SLOs, SLIs. You measure them over time in the second phase. Now, lastly, the data driven phase. The ultimate aim of observability is to help teams within a company make data driven decisions. You are doing a lot of trend analysis of the SLOs and the SLIs that you set up. But at the same time, you’re evangelizing this to the teams beyond, let’s say, site reliability, DevOps, or application engineers. You’re pulling in folks from, let’s say, customer support, product. Everyone’s looking at the same data, and you’re making decisions. Eventually, you want to get to a stage where you can figure out, how is it that your digital operations are impacting business KPI. For example, if you were an eCommerce website, if the page load of that eCommerce website increases by, let’s say, 10%, are you seeing a drop in the number of users on the website?

Padmaja Gohil: Are you seeing lesser number of things in your card? These are the kinds of relationships you want to start visualizing and measuring. That’s the last phase of observability. One of the things of last phase, is also being able to automate processes. That’s where observability as code comes into the picture. Now, observability as code can again, mean a lot of things. It could mean that the way you are interacting with your observability platform, you’re automating it, but it can also mean Gitops, config as code, infrastructure as code, CICD. Whenever you hear these things, know that these are observability as code. Now, what we’re doing essentially here is that we’re taking some of the best practices from software development, and we are applying it to the operations world. Think reproducible builds, reproducible deployments.

Padmaja Gohil: You are automating processes, you are testing them. And you’re making sure that no matter how many times you run these processes, you’re getting the same result. There are a few things common as a part of observability as code. Firstly, observability as code, it’s literally code. So it does not have a UI. It is declarative. So you are specifying the exact state in which it should exist. For example, if you write a piece of code to create an alert in New Relic, you should be able to take that same code or a template, and then modify it slightly to create a thousand alerts. It’s also reproducible. You are reducing the amount of time you’re spending in managing your observability systems as well. The first thing is it’s declarative. Secondly, it’s versioned and immutable. Ideally, it should not reside in a shared drive.

Padmaja Gohil: Ideally, you should be using a get for it. You should be able to go back and figure out what were the changes made if things were going wrong. It should be versioned and immutable. And lastly, it’s pulled and reconciled automatically. Now, what I mean by this is that if you had created a dashboard in New Relic or in any other observability system, and let’s say one of your colleague comes to you and says that this is a great dashboard. I want to use it for my own needs. They can go ahead, take the dashboard, and maybe they modify it. Then you go into New Relic and you figure out that your dashboard is modified, and you won’t actually revert the changes. You can directly take the code, apply it, and you can get your original dashboard.

Padmaja Gohil: And now you can take the template that you used, or the code that you used, and you can give it your colleague, and they can use it to create their own dashboard. It’s usually pulled and reconcile automatically. There are a lot of solutions available for observability as code. I’ve mentioned some of these here. We also have our own templates for, let’s say, Terraform, in case if you guys are interested. Feel free to look at it in our docs page. But these are just some of the solutions that you can use to implement observability as code. This brings me to an end of my presentation. I know that we covered a lot of cloud. In case if you guys are interested in knowing more, feel free to reach out to me on LinkedIn or my email address. Thank you so much. I very much enjoyed speaking here.

Angie Chang: Thank you, Padmaja. That was really great. And thank you for leaving an email address so people can reach out to you with any questions. moving on to our next speaker. Kate is a lead principal technical account manager at New Relic. She comes with a background in helping customers thrive in their business with the latest software monitoring tools. In her current role, she partners with customers to help them with their full stack observability requirements. So welcome, Kate.

Kate Kordnejad: Hey, Angie. Hi, everyone. Thank you for hosting us. Give me a second to share my screen, and put it in slide mode. All right. I’ll be talking about customer success and value realization through value metrics. I’m just going to jump into a little bit of legal disclaimer, so don’t make any financial decisions based on our discussions today, and or any statements we make, and some proprietary copyright information. All right.

New Relic principal technical account manager Kate Kordnejad talks about the evolution of maturity, TAM goals, maturity journey, maturity metrics & more at Girl Geek X New Relic virtual event. (Watch the talk)

Kate Kordnejad: A little bit about me. My name is Kate, and I’m a principal technical account manager here at New Relic. As TAMs, we are an extension to our customers teams. We help them with their full stack observability requirements. We want to make sure they see value, and we basically help them get enabled, follow best practices. We work as a trusted advisor with them. A little data point about me; I love working out. I love yoga, especially Bikram yoga. I love to travel, and I’m a data nerd.

Kate Kordnejad: Okay. Our agenda for today is going to be evolution of maturity, goals for technical account management, our maturity journey, defining maturity metrics, and how can you define maturity in your organization? All right. Starting off with evolution of maturity. In our evolution and journey, we found ourselves improving efficiency from four to five hours to one minute by automating our solution. I’m going to explain how we did this. As things evolved over time, we found our defined metrics to be meaningful. And we did find out more about our customer’s maturity, and how we can help them improve stickiness. For example, are they using custom attributes, or do they have data instrumented for more visibility? With our help, they started getting more mature within the platform. And we were able to identify the gaps, improve upon them. We did soon realize to deliver an observability platform value for our customers.

Kate Kordnejad: We needed to recognize value drivers and use cases, that actually deliver those business outcomes for each and every customer. For example, to improve customer experience, quadrant you see on the left hand side. We had to understand our customer’s business needs. Card abandonment, any association with an operational gap like card crash rates, were stuff that we needed to figure out. We identified the steps to maturity, is basically summarized in alignment. What that means is we need to align customer priorities to the observability value drivers. And agree on prescribed observability use cases, and then enable based on an agreed upon description work streams with the customer, and then finally, value realization. Reflecting on the business and the operational KPIs that we agreed upon during and prior to going through maturity. We actually evolve from just collecting metrics to quantifying metrics into meaningful business values, with a growth mindset, of course. We realize without having a continuous growth mindset, we won’t be able to evolve and improve our solution.

Kate Kordnejad: Our next thing is the goals that are for technical account management. Having an involved automated way to quantify metrics into business values, provides us leverage as TAMs. TAMs, as in technical account managers. We now have data to analyze customer usage, to reduce overall churn, by identifying any sort of gaps we have in utilization, by providing enablement based on usage, and engage platform users and drive valuable engagement by meeting them where they’re at. And directly communicating with our customers and being a liaison internally and a voice for our customers. And essentially, we want to reach value realization with them.

Kate Kordnejad: The next I want to is our maturity journey. Our journey basically started at looking at our platform per customer account, and literally eyeballing metrics we had identified as crucial to understanding and analyzing customer data. It was really hard to assess the pieces of the product they were using by manually assessing their usage and engagement. The normal customer metrics success wasn’t really working for us anymore. For example, if they were building dashboard, this wasn’t showing us the full picture, or the reason behind it that’s looking at their user behavior. It was very one-dimensional, and we didn’t really know if they were getting value out of it. We basically had to look deeper into the metrics, and then identify and associated with value drivers.

Kate Kordnejad: How do we define maturity metrics to get to that point? As a team, we basically start asking ourselves, what results do we want to see from this? Ultimately, what does a good maturity look like? And what does it look like for each product? We needed KPIs to show actual investments. For example, if we looked at our alerting product, we wanted to drive an alerting strategy, or potentially set our customers up with anomaly detection. Next, we had to break each product into maturity metrics. Initially, this was done manually through APIs and us eyeballing accounts, but after we broke down our KPIs by product, we had to describe a desired performance level, and determine how data is interpreted. We had to set up thresholds, place and score for each one, each of the metrics that make upper and lower limits of a desired performance.

Kate Kordnejad: This basically allowed us to understand overall maturity for each customer product using a heat map, and really made maturity pop up the page for us. Now that we had our results defined, maturity metrics chosen by product, we had to basically come up with a way to automate this. Our internal teams were able to automate the process, build out an app using APIs, grab the required data from accounts, and assess maturity. Finally, the last piece of the puzzle was to ensure we documented every single steps, our definitions that are associated with each of the metrics collected for further analysis. Our document includes a breakdown of the products, the metrics associated with it, and each and every single step you need to take to improve your score. From all of this, we want to cover, how can you define maturity in your organization?

Kate Kordnejad: It really comes down to three pillars. Goals and baseline. You have to ask yourself, what does maturity look like for your organization? Describe those intended results. Do you understand the alternate measures for those intended results? Then you move on to data identification. Have you identified any composite indices as needed? And do you collect any of the data right now? Is it accessible to you? And finally, business alignment. Have you thought about targets? Thresholds? Do you have a baseline that you can work with. And then finally, have you tied your maturity metrics to business values that deliver value realization? That concludes my presentation. Thank you for having me.

Angie Chang: Thank you so much for that talk, Kate. Our next speaker is Carolina Rotstein. She is a solutions consultant at New Relic. She is also an economist and political scientist that fell in love with programming and data, and is passionate about untangling holistic customer journeys across complex stacks, which she’ll be speaking about today. So welcome, Carolina.

Carolina Rotstein: Can everybody see my screen?

Angie Chang: Perfect.

Carolina Rotstein: All right. Oh. Today we’re going to talk about browser monitoring, and how it can help us improve UX and UI. Some safe Harbor information, a bit of housekeeping, some proprietary information, and just please don’t use this to make any financial decisions.

New Relic solutions consultant Carolina Rotstein talks about improving website UX and UI with real user monitoring at Girl Geek X New Relic virtual event. (Watch the talk)

Carolina Rotstein: A bit about me. I’m a solutions consultant for New Relic in the commercial E-sales team. I’m an economist and a political scientist, but I fell in love with programming and big data. I’m passionate about untangling holistic customer journeys across complex stack, and my most previous role included optimizing UX and UI for the gaming industry. And yes, we did collect a lot of data.

Carolina Rotstein: Today’s agenda, we’re going to focus on improving the website’s UX and UI, and using real user monitoring for this. Also we’ll cover why we should focus on UX and UI optimizations, and some of the metrics that we can use to do this as well as the metrics that come out of the box for New Relic and some other tools. And then an approach towards optimizing customer experience, including UX and UI, the traditional way and the enhanced way using big data. My peers talked a bit about observability maturity. At New Relic, we focus on data driven decisions. We want to have an approach with this framework towards taking data driven decisions.

Carolina Rotstein: Now, in this part, customer experience is closer to product and support. While it does have a lot of positive impact into how customer support, user product, and just impact on their KPIs. It’s mostly geared around design and product and development. Experience optimization, and a big portion of that is user experience. And also user interface optimizations are closer to the revenue. Even though it’s at the bottom of the the funnel, any impact that we might have into optimizing the experience, will have a monetary increase for the companies that we’re in.

Carolina Rotstein: First I’d like to talk about browser versus synthetics. We talk a lot about the jungle versus the lab. The jungle would be empirical data. So just every browser, every device, every location, and what your customers are using. The lab will be how we are tracking the health of the site just as we mapped it. We eliminate all the variables to just understand performance and solve problems quickly. This is done by synthetics. The jungle piece or the real user monitoring is once we deploy that application into the wild. So the users might take pads that we just did not foresee. For that we use browser monitoring. It’s an essential tool for user experience. It has a couple of places that we would focus on. Product usage, front end performance issues, content strategy, and in this case, websites, UX and UI.

Carolina Rotstein: I’d like to talk a bit about the metrics that we can use for this. This are not all the metrics, but I strongly recommend this as a start. Just for front end performance and monitoring, we have core web vitals, the user time on site. And it’s just user centric health metrics, such as throughput chart. For New Relic, we can divide what the time that it takes to load is split by the front end versus the back end. But for product usage, which is getting closer to that UX and UI, we track that funnel.

Carolina Rotstein: These are those conversion funnel related metrics that map to the business. These are unique to every company and every website. Those are success events, which can be form fills, video watch, purchase, and business classifications. These are custom metrics that we would map. Then we have all these audience insights such as device and location and vanity metrics. The vanity metrics normally come out of every tool, but they’re a great place to just look at your application, sort of like the Canary in a coal mine. Then for content strategy, we can see how users are navigating through the site, such as in the metrics that we would use, are link positions, most popular, previous page, the next page. And we also have pages report, such as the most popular pages, the time spent on site, how long it takes to load an assets. But we also can track audience insights. This can come from your BI data.

Carolina Rotstein: New Relic can take just any sort of data, but with other tools, you can certainly integrate it. This can be things that are a bit more robust, such as persona development, even the VIP level of your users, or the user IDs. And then very targeted towards UX and UI, and specific real user monitoring. We have the time spent on task, which will be the time before a user completes a success event. The ease to perform a task, rage clicks, which is just a user frantically clicking. Marketing funnels is a good one. In New Relic, we have something called the apex score, which is just taking into account the error rate and the load time to proxy some of the survey based customer satisfaction, traditional UX and UI metrics.

Carolina Rotstein: Now, very related to UI in just design, we have AB test, popular device sizes, screen size, and size orientation and night mode. Those are a few ones in there. Finally, I would like to show you what comes out of the box of New Relic. This is browser monitoring. We have those core web vitals that user spent on site, initial page load throughput, and some other additional charts. This comes just out of loading a user agent. It’s as simple as adding a marketing tag, and this dashboards just magically appear. But if we go back to talking about UX and UI, why is this important? It’s like 68%. I’m talking here about eCommerce just because it’s the easiest, cleanest use case to see revenue changes when you deploy UX and UI changes.

Carolina Rotstein: We can see that a lot of eCommerce… 68% of them just had performance issues. And that translates into 40% of those issues resulting in revenue lost. For instance, most eCommerce retailers have reported that they would like to have a response time below two seconds. 90% of that website response time, on average occurs due to do the loading of front end resources. This is why it’s so important to start your optimization with your front end as well. Some core customer experience questions that real user monitoring will help you solve is, for instance, if your website is easy and friendly to use, that would be through the balance rate, for instance. Whether it’s easy to navigate or not, and that will be through the number of pages that your users take to get somewhere.

Carolina Rotstein: Ideally, you want to slice all those additional page views, just because you want a seamless interaction with your site. Just think about loading a YouTube video and having to click 20 times before you go to the music that you want. For instance, how easy it is to get in touch with a customer agent for your user. Now, not all sites want you to immediately get those agents, but those are done through custom events. You track that chat click, or that phone call on a mobile browser, as a success event. And just how comfortable your visitors are after landing in your site, can be done through a number of other metrics that we track on browser. Why is this important as well? Just to do it via RAM, is because it’s big data and data driven.

Carolina Rotstein: So if we look at traditional UX and UI optimization, it’s done through user research, such as interviews, focus groups, usability testing. And they would put a couple of people to see how easy it is to finish a task on their site, through surveys, AB testing sometimes, and session recording. Now, the size of this data tends to be, from my experience, to 100 people, to a thousand people. When we’re talking about big data, it’s millions of people. It helps us prioritize and not get narrow focus on the people for which we’re auto… well, for which we’re optimizing too. That is done to AV testing. Some companies that are very developed, they do multi-variate testing. So they have several versions of the same design, such as… Netflix is one of the big guys at the same time. They’re just running algorithms while they’re doing that.

Carolina Rotstein: The way they pass that data into their systems, is via an integration to their either browser or mobile tracking. Same session recording, conversion funnel. This is just big data that allows you to ultimately do persona building. In the particular case of New Relic, this is fully integrated with observability. Now, just because I’m coming from New Relic, I will like to show you how easy it is to implement custom events and attributes, which are those business events and additional metrics that I was talking to. This is just an example of a JavaScript snippet, and this is how you would pass it into New Relic. It gives you that full stack of observability that my peers for talking about. And that’s it.

Angie Chang: Thank you, Carolina. Our next speaker is Solmaira. She’s a technical account manager at New Relic, based out of Atlanta, serving as a technical advisor for enterprise customers in Latin America. She currently serves as chair of the Relics of Color ERG, which she’ll be speaking about today. Welcome, Solmaira.

New Relic technical account manager Solmaira Flores-Valadez talks about finding community with New Relic ERGs at Girl Geek X New Relic virtual event. (Watch the talk)

Solmaira Flores-Valadez: Hi, everyone. My name is Solmaira Flores-Valadez, and I’m a technical account manager at New Relic. I’ve been with New Relic for about over two and a half years. I serve as pretty much like a technical advisor to some of our larger enterprise customers within the Latin America region. I’m like a post sales resource to them, helping them get the most out of New Relic, and also providing trainings, things like that, to make sure that they are utilizing New Relics to the best of their abilities. Today I’m going to talk about diversity, equity and inclusion, and the part that it plays in my life. How I was able to find a community with New Relic ERGs, which are employee resource groups.

Solmaira Flores-Valadez: A little bit about me. My pronouns are she, her, hers. I live in Atlanta. I went to the University of Georgia. I am a first generation Latina. Mexican-American. First person in my family to go to college. I am a woman in tech, and I’m also a dog mom. First I wanted to start off with a few definitions around what diversity, equity, inclusion are. And then I’ll jump in and talk a little bit more about what it means to me, how I got involved, and all of that. Diversity is the presence of differences that may include race, gender, religion, sexual orientation, ethnicity, nationality, socioeconomic status, language, disability, age, religious commitment or political perspective. These populations have been and remain underrepresented within the broader society, and within practitioners in the field as well, within the workplace.

Solmaira Flores-Valadez: Equity is promoting justice, impartiality and fairness within the procedures, processes and distribution of resources by institutions or systems. Equity is really the approach to ensure that everybody has access to the same opportunity. In the context of the workplace, how is it that employees have access to the same levels of attraction, promotion and retention within the company?

Solmaira Flores-Valadez: And then lastly we have inclusion, which is an outcome to ensure that those that are from these diverse backgrounds actually feel and or are welcome. It pretty much boils down to people with different identities, feeling or being valued. All right. So why is it important to me? I actually started doing D and I type of work long before I even knew it was D and I work. As I mentioned, I went to the University of Georgia. UGA is a predominantly white institution. So there was very little people that looked like me. I was always looking for my community, people that looked like me, that share common backgrounds, but then at the same time, got involved with certain organizations such as Students for Latino Empowerment, that not only helped build that community, gave me that social aspect in college, but also we were doing D and I work [inaudible] students into the campus. We have various events throughout the year, where we would pretty much show them the ropes, let them know, if I can do it, you can do it. They’ll be able to tour campus.

Solmaira Flores-Valadez: We would give them workshops around financial aid, how to get started, the college journey and all of that. That’s what sparked I guess that interest in being involved within D and I type of efforts. As I have here, it’s important to me, for me to lift as I climb to, to be that change that I wish to see in the world. And also D and I has been very important in my life, not only because I’m able to in a way give back, but also it’s helped me in my professional, in my personal growth. Being able to develop certain leadership skills.

Solmaira Flores-Valadez: A little bit about how I got involved at New Relic. I was involved in college with those type of organizations. As I left college and I moved on to my professional career, my first job I worked at a big accounting firm. I got there similar to when I joined UGA. Not a lot of people that looked like me. They had an Hispanic network. I joined that. We did a lot of social events, but at the same time, we also did a lot of also lifting as you climb, bringing in students. We also did events for students and things like that. I loved being plugged in, being that person, going to recruiting events and seeing others like me, and then being able to see that they could also do it.

Solmaira Flores-Valadez: And then after that I switched careers. I came over to New Relic. As soon as I joined, I let my manager know that I was interested in still being a part of something like this. I asked if he had a Hispanic network. He told me he wasn’t familiar if there was an Hispanic network, but there were employee resource groups, and got me connected to the person that was the D and I manager at the time. Met with her. I talked about my experiences. She got me connected to the Relics of Color, which is the employee resource groups for our POC at New Relic. I got to meet them. Loved what they were doing, got to participate in some of their events. When I joined, it was right around Hispanic heritage month.

Solmaira Flores-Valadez: I asked if there was anything I could help with. At the moment, the Atlanta office was pretty new. There wasn’t a lot of representation there. I told them that I wanted to host an event. I was brand new. I didn’t know how people were going to take it, but I knew that I wanted to do this. Since it was a little bit later in the Hispanic heritage timeframe, I decided to give a twist. We did a day of the dead event, which I have some pictures here. I put this together. We painted skulls. And then we watched Coco, and we also ordered Tamales and we had a really good time doing this event. That was pretty much my golden ticket into not only being a member of the Relics of Color, but becoming an executive, one of the ROC execs. After I did that, the leader of the Relics of Color reached out and was like, “Oh, I want you to be part of the exec board.” And then that’s how I got plugged in my golden ticket.

Solmaira Flores-Valadez: I loved it ever since. I’ve been able to help with a lot. Currently, I’m one of the co-chairs of the Relics of Color here. Have our exec board of our offsite that we had earlier this year, where we got together to build out our strategy, the events. We host and we celebrate different events throughout the year, black history month, Hispanic heritage month, Asian Pacific Islander month. Putting together content around that. And then on the right hand side, that was us at the sales kickoff. We managed to get people together before 8:00 AM or at breakfast. It was great. It was intimate. We had our relative color sponsor. Tracy Williams, she’s our chief diversity equity and inclusion officer, as well as our chief people officer there. Being part of the Relics of Color, being part of the exec board, has been, like I said, great.

Solmaira Flores-Valadez: I’ve been able to learn a lot, gain exposure to different things. For example, we meet with our C-suite on a quarterly basis. Being able to have visibility into the C-suite. And not only that, but be able to represent the Relics of Color as a whole, be able to communicate some of our challenges, what we’re doing, where we want to get, our goals. And then listening to us and our needs and seeing where and what they can do to help. It’s been great. On the social aspect, I’ve made really good friends, but also helped me grow professionally.

Solmaira Flores-Valadez: I talked about the Relics of Color, but we do have other employee resource groups. We have the women at New Relic. We have the veterans at New Relic, and we have access at New Relic, which encompasses neurodiversity, mental health and disability. Relics of Color, which is the ERG that I’m a part of. And then we also have our Rainbow Relics for LGBTQ plus relics. These are the different ERGs that you can get involved with. At Relic, we are working towards a more perfect New Relic. These are some of the initiatives that we have going on. We definitely believe that inclusion means everyone. We want to make sure that we’re having some progress. We understand that there isn’t always… or there is always more work that needs to be done, but we do value the progress over the perfection.

Solmaira Flores-Valadez: These are some of the initiatives that we have at New Relic. To help us accomplish that we have, for example, the Mikey rule, named in honor of our departed team, VP of engineering, who was the executive sponsor of our first employee resource group, which was the Relics of Color. This Mikey role focuses on sourcing and hiring more relics from underrepresented groups. Whenever we have an opening, this Mikey rule kicks in. We also have leader-led action plans. These were started in 2020 by our founder, Lew Cirne. He challenged the company to level up with D and I leader-led action thoughts, and maximize the recruitment retention and career growth for underrepresented groups. And now it’s one of the top level organizational priorities across the board for every single part of our business.

Solmaira Flores-Valadez: We also have D and I working groups. Our company leaders, like I said, sit down with us, with the ERG executives, to ensure that our commitment to diversity, equity, inclusion, is put into practice around the globe. Just wanted to call out some of our progress that we’ve seen with these initiatives. We’ve definitely increased our BIPOC engagement. We’ve also helped reduce bias. There’s different trainings that our managers have to take, every year or every so often around bias. We’ve also reached pay equity. There was an analysis that was made a couple years ago, that took a look at the pay, and made sure that everyone’s pay was equal. There’s been a lot of progress lately around career mobility, where we’ve built a lot of mentorship groups throughout the different businesses, to be able to help the career mobility of our underrepresented groups. And then as I mentioned, you also have the Mikey role, which focuses on the recruiting efforts. All right. Well, that’s all that I have for today. Thank you all for joining. Have a great day.

Angie Chang: Our next speaker is Nora, who is a solutions engineer at New Relic, where she advises enterprise clients on their observability engineering practices to answer the what, how and why of system performance. Her research focuses on application of blockchain, and she speaks Portuguese, Spanish and French, and resides in Florida. So welcome, Nora.

Nora Shannon Johnson: Hi, everybody. How are you all doing? Well, I’ll assume you’re doing fine because I can’t hear you, but can you see my screen?

Angie Chang: Yes.

Nora Shannon Johnson: Awesome. Okay. Cool. Like everyone else said, I don’t know enough to make any predictions that you guys could invest in, but anyway, welcome.

New Relic solutions consultant Nora Shannon Johnson talks about observability in the age of web3 at Girl Geek X New Relic virtual event. (Watch the talk)

Nora Shannon Johnson: Today I’m going to talk about observability in the context of Web3. A little bit about me. Like Angie said, my name is Nora Shannon Johnson. I’m a solutions consultant, which basically means that I help customers answer the what, how and why of system performance. Outside of work, I love languages and linguistics. I love planting things, but everything I’ve ever planted has died, unfortunately. So still working on that. And skateboarding. Today we’re going to talk about applying the principles of observability to Web3, and what the specifics of monitoring blockchain technologies looks like. I took an interest in this because I work with a lot of financial services and eCommerce organizations in Latin America.

Nora Shannon Johnson: The integration of blockchain into their existing business operations is a big question for them right now, for reasons that I’ll get into in a few minutes. This is not New Relic’s main use case, but as a solutions consultant, a lot of times customers come to you with their data, their technology, and their business requirements, and say, “Make it work.” Which is my favorite part of the job, when somebody says, “How do you do this?” And I say, “I don’t know. Let’s figure it out together.” So this is an example of doing that. Over the next nine or 10 minutes, we’re going to talk at a very, very high level about what Web3 is, why we would care about monitoring it, what specifically we would be monitoring what we want to look at. And then a quick example of what that might look like.

Nora Shannon Johnson: To get started, what is Web3? Web3 is the name given to the idea, and idea is a very keyword here, of a new sort of internet that is built using decentralized blockchains. As a disclaimer, throughout this entire presentation, I’m describing the idea, not the reality of what may come to fruition. Again, Web3 is powered by the concept of… or by blockchain technology. Blockchain is a relatively new method of storing data online. It’s built around two core concepts, those being decentralized computing and encryption. The fact that it is decentralized, means that files or data is shared across many computers or servers, rather than centralized in a single server or group of servers. You might hear it referred to as a peer to peer network for that reason. The fact that it’s decentralized also means that it’s immutable. You can’t change data on the blockchain, because in order to do so, you would’ve to corrupt data on every single machine that’s participating in the network, which is just really not feasible when you’re looking at large scale blockchain like Ethereum, which is the example we’re going to use.

Nora Shannon Johnson: And then the fact that it’s encrypted, means that people can’t access it unless they have permission to do so, and you can give and rescind access as you choose. So why would we want to monitor Web3? Frankly, for a lot of the same reasons that we already monitor the existing technology, the web two technology, so to speak, in the same industries. for financial services, that’s eCommerce integrating with blockchain for payments. It’s important to know that this isn’t just like cryptocurrency exchanges. This is brands like Gucci and the Dallas Mavericks and Microsoft, Whole Foods, even Save the Children. They all accept one or more cryptocurrencies for payments. Across a ton of different industries, this is an important aspect of their technology stack. We’ve also got healthcare. One of the driving or the driving use case for applying blockchain technology to healthcare, is to restore the rights of data back to users or patients in this case.

Nora Shannon Johnson: You would be able to give or rescind access to your health records to a healthcare professional, organization at will. Whereas right now your test records or health records are held in a database owned by maybe some company like Quest. And you wouldn’t really necessarily be able to remove it if you wanted to. And then finally supply chain. Supply chain is arguably at the enterprise level, the most interesting use case, the most sought after use case for blockchain. Specifically the validation of providence or origin and authenticity. Using a public ledger like Ethereum, you could actually trace the roots of a product that you purchased, to ensure that it is in fact organic or fair trade, or even from a location that you believe it to be from, which is pretty interesting use case. There’s many more, but in all of these use cases, we’re talking about people’s privacy, their security, their wellbeing. Obviously, their financial assets.

Nora Shannon Johnson: The fact that data on blockchain is immutable and that it’s decentralized, doesn’t mean that it’s immune to failure or to attack. It’s simply is creating this new monitoring paradigm. What might we monitor on the blockchain? I’m not going to explain all these, but like people before may have said, the slide decks will be shared out. But you might monitor something like a decentralized application or dApp. Decentralized autonomous organization or DAO. Decentralized finance exchanges or DeFi. And then non fungible tokens, which I think everybody’s probably familiar with. The infamous apes, or NFTs. Lots of acronyms here, cause it’s a mouthful. And then of course, you can monitor the blockchain itself too. Which is the example that we’re going to look at. We’re going to look at the Ethereum blockchain. If you’re not familiar, Ethereum is, you guess it, a blockchain platform with its own cryptocurrency. Ether, shortened to ETH. It’s also got a programming language called Solidity, which you can use to write smart contracts, and decentralized applications so that you can actually interact with the blockchain.

Nora Shannon Johnson: This is by far, especially for DeFi, the most popular blockchain, but there are a lot of alternatives that are gaining popularity, things like Cardano and Solana, because they’re faster and cheaper than working with Ethereum. Monitoring a blockchain or assets that are deployed to a blockchain, is going to include both the typical metrics and data types that we’d be used to seeing, as well as some that are specific to this realm. There’s three example categories here. We’ve got system performance, security events and business metrics. If you go from left to right, this is like more familiar to less familiar. Something like system performance is something that we’re very seeing.

Nora Shannon Johnson: When Netflix is… well, all the time it’s up and running, they want to know how quickly transactions are executing, the rate of error, as well as resource utilization. The only difference being in a world of Web3, this might be the number of nodes, but very similar to what we see today in terms of the number of idle versus busy workers. For security, this is very important. We’ve all heard about many attacks made to different blockchain or cryptocurrency exchanges. Things like changes to access controls, when there’s a lot of failed login attempts from especially specific IP address or geographic location where you don’t normally have those. And then finally, unusual transaction patterns. So there being a lot of transaction outside of your normal business operating hours.

Nora Shannon Johnson: And then all the way to the right. And this is where we see things that are more specific to the use cases that I described earlier. Things like measuring the gas fee. When you interact with blockchain, you have to pay a transaction fee. And it’s a dynamic transaction fee. It changes throughout the day. We’ll take a look at that in a minute when we get into New Relic. But that’s something you want to pay attention to, because whether you are paying that or receiving that, that affects your bottom line. You’d also want to pay attention to things like, the number of active users or wallet holders, the number of active connections. And then of course, the number of minors that are mining. And then finally something like the rate of currency being paid out. As you probably know, miners mine, because they get paid for it. But there’s a lot of blockchain platforms, especially ones that are oriented towards the arts and culture, where they will actually pay you for posting your content to their platform.

Nora Shannon Johnson: They pay the content creators. You want to know, again, whether you’re paying or receiving or you’re somewhere in the middle. As an integrator, you want to know what the rate of payout is. How might we do this? We know what we want to look at. We know the importance of it, but how might we actually do that? We’re going to go through very quickly, the two pieces that fit together, and then we’re going to look at what that looks like in an observability platform. In any situation, there’s two parts to monitoring. There’s the data and the platform. There’s, how can we get it, it being the data. And then the second part is, how do we make sense of it? Because just having the data is not super helpful to anyone unless you’re a computer.

Nora Shannon Johnson: We might use something like Web3.py, which is a Python library for interacting with Ethereum. You can do all kinds of cool stuff. You can read data, you can send transaction, you can even set the gas price if you are the owner. Not the owner, but you’re responsible for the operation of the blockchain. And so we see on the right hand side, we can import the Web3 library, confirm that we’re connected to Web3. And then as the last example shows, we can read block data or look at different people’s wallets. Here I’m pulling Snoop Dogg and Paris’s wallet balance. Which again, we’ll take a look at in a minute. This is the part of how we’re getting it. if you’re more of a fan of JavaScript, there’s also a Web3.js library that you can use instead. I’m just a Python loyalist.

Nora Shannon Johnson: And then the second part, which is, how do we make sense of it? Using a wonderful observability platform like New Relic, we can use an API suite to pull in the metrics, events, logs and traces that are important to us. And then you can look at all of the block statistics for the last hour, week or month or whatever the case may be. Let’s take a look at what that might look like if we were actually quoting it over to… oops. [inaudible]. I didn’t need to unshare. Let me reshare. Quoting it over to an observability platform. This is a simple use case, but this is just basically a dashboard that I put together. We’ve got our cool little… I don’t own. This guy, unfortunately, no Ethereum funds for that.

Nora Shannon Johnson: But we can look at changes to the different whales. Here I’m tracking people like Snoop Dogg. He’s buying and selling all over the place. We’ve got Lindsay Lohan, Mark Cuban. We can see the top miners for the period of time. The gas price, that’s what I mentioned earlier. And then again, the black activity, which is interesting to see. Whether you are responsible, you’re part of some exchange that is leveraging the Ethereum on blockchain, or you are a decentralized application developer, or maybe you’re just somebody that is posting their content or their NFTs to a blockchain. This is all going to be relevant for you. You can also make actionable insights based on the data that you poured over. If we take a look at, for example, logs. Logs are just like step by step data coming from either applications or servers or what have you.

Nora Shannon Johnson: And in this case, it’s coming from the Ethereum blockchain. I’m going to filter on a certain block number. Let’s do this guy. We can actually see. I’m going to shut this down. We can actually see the step by step of what it looks like. We can see that transaction was requested, that the block creation was initiated. Blocks were then sent to nodes in the network. It got validated. Transaction is now complete. And then they update that to… or they send that update to the network. The block gets added to it, and the proof of work is dispersed. People get paid on it. But as we know, things do not always work as we anticipate that they will in technology. So if we take a look at this other block, we can see that in this example, the transaction is requested, the block creation is initiated. The block is sent to the nodes in the network, but then the transaction is pending validation several times.

Nora Shannon Johnson: We might do something like create an automated remediation workflow here. Based on maybe the messages, the strings in this message, or repeated data, or examples of the same messages over a long period of time, we could actually set it up such that it automatically triggers external events based on what we see in the log messages. Again, this has been a very quick and very high level example of what you might see if you wanted to monitor something on the blockchain, and how you could make use of that information in a wonderful observability platform like New Relic. I hope you enjoyed it. Thank you very much for your time, and I hope to talk to you all soon.

Angie Chang: Awesome. Thank you, Nora. So Sarah is a solutions consultant at New Relic. She loves working with data, and in a previous life was a math teacher. She uses her skills to help customers use their own data to improve their uptime, performance resilience, reliability, and customer experience. Welcome, Sarah.

Sarah Hudspeth: Okay. Hopefully you can see me and my slide.

Angie Chang: Yes.

Sarah Hudspeth: Are we good? Okay. Hi. All right. Hi, all. I’m excited to talk to you all about APIs, and getting your data when you want it and how you want it. It’s a very common theme here at New Relic. We love data. We’re data nerds. And we have a safe Harbor just for legal purposes.

New Relic solutions consultant Sarah Hudspeth talks about REST APIs and GraphQL queries at Girl Geek X New Relic virtual event. (Watch the talk)

Sarah Hudspeth: A quick bio about me. Yes, I’ve probably been in tech for three and a half, four years. Before that I was a math teacher. I taught middle school and high school math. I did attend Hackbright Academy, and so I’m a boot camp graduate. If you have any questions about that, please reach out. I’m a mom of two, plus I have a puppy, a lab mix, and then you can see the hamster in the background.

Sarah Hudspeth: I’m a huge reader. And you’ll see one of the projects I walk you through is all about books. The last book I read was Stalingrad, super interesting. The best part of my job is working with customers and helping them solve their problems. And yes, we are all about data and using data. Feel free to put stuff in the chat or follow up with me afterwards. Here are my objectives for my talk. I am still a teacher at heart. I want you guys to understand what REST APIs are, how they’re used, what is GraphQL, and what are some interesting trends in APIs today? I want you to understand the difference between your REST APIs and GraphQL APIs, and possibly articulate use cases for each. Also we’re going to be talking about GraphQLs, query and mutations.

Sarah Hudspeth: I’m just making sure you understand the difference between that. And then I am giving you all homework. After this session, if you haven’t played with API calls, go find some APIs, play with them. Go play with GraphQL, do some queries and mutations. If you need a GraphQL API Explorer, New Relic, you can sign up for a free account and play with our GraphQL API, which we call NerdGraph. Feel free to do that. APIs. API stands for application programming interface. It’s basically a way for clients and servers to talk to each other. It’s a set of protocols and it’s called a REST. I’m going to be talking about REST APIs, because those are usually the ones I would say they’re the most popular. And REST is short for RESTful, meaning stateless.

Sarah Hudspeth: The state of the client doesn’t affect the state of the server. They should be able to talk no matter what’s going on within their own environments. I like to think of API calls as programs, throwing Frisbees back and forth. Even though the Frisbee is actually data. But a client will make a call, throw the Frisbee to a server. The server gets the Frisbees if there are any instructions, and throws the Frisbee back as a response. If all goes well, you get a 200 response. If it doesn’t, you’ll get one of the four hundreds or five hundreds based on whatever the errors are. Let’s take a look at what an API call looks like. This is code from my virtual bookshelf project that I did at Hackbright. I allowed folks to build out this visual bookshelf of the books they were reading.

Sarah Hudspeth: The main API I used was Google’s Books API, where I could get a thumbnail of a picture of the cover of the book, and a lot of information. When I was feeding my database, I had a list of titles and authors, and then I made a call to the Google API, Book API, using my Google API key. I used the Python HTDP request library to get that information. And then I stored the response in a dictionary in JSON form. so that I could fill out my database with all sorts of interesting things. Hold on. I was going to say, there’s a few things. We have a URL. I have some variables in here, parameters that changed that I had to go through in a for loop.

Sarah Hudspeth: And then I also needed permission to have access to APIs. Those are the key components of an API call. This was one book. This was one response I got back from Google’s API. There’s a lot going on here. I would say, there’s a lot of information here, some of which I needed, a lot of which I didn’t need, and I had to sort through it and figure out, what is going to be helpful to me in my project, and then get rid of the rest, which if you notice, is a lot of waste. My code was not optimized. This was the slow part of my program, which if I go back, I would focus on this and try to do this in a better way, just because it ate up so much [inaudible].

Sarah Hudspeth: To summarize, I showed you what the components of the REST API and the results are. You have to have a URL. You have to call to someplace. You can send parameters on variables. I did title and author. You usually need a key to access the APIs, so you have permission to get the information. You need some HTTP requests. I use the Python’s request, but I’ll show you a cURL snippet when I do the GraphQL. The other interesting thing to note is that each API, you can call various APIs, will have their own way of formatting the data. Google Books API – just sent me everything I could possibly need about a book. And it was up to me to go through and figure out the structure of it.

Sarah Hudspeth: I showed you a get REST API call, but there are also posts where you can actually post data to the API. You can update data or you can delete it. I said, “This is kind of ugly data.” There was a nested JSON. I found out the hard way that sometimes some of the things I wanted were empty, and I had to find workarounds. I had to go and I had to clean up and structure the data. There even updates and the data would get restructured and I’d have to go back and figure out how to do that. I’m glad to now transition to a new way to get data, called GraphQL. It is also an API, but it is a very structured way we can access data. This is an example of New Relic’s NerdGraph API Explorer.

Sarah Hudspeth: And if you notice, I have to my left, a query builder with very specific key value pairs that I can build out for a query. Here I’m going to query an account and get the name and ID, and here I’m going to do an entity search. You all have been hearing us talk about observability, and learning about applications and performance and getting metrics and events. This is a way you can go in. I’m just going to get the name of things I want to monitor, the type. I’m going to get a special GUI. And then I’m just actually going to get the tags that I’ve tagged with my entity. It’ll pop up here in a very nice structured JSON. I know exactly how many levels I need to go in to get specific information. And then here’s how you could do it and build in a program.

Sarah Hudspeth: We talk about automation and observability as code. It’s really easy to take these GraphQL calls, and build in structures and processes to get the information that you can then take action on. Again, here’s just the API link. I have some headers with my key. And then here, I’m sending this query that’s going to go to the GraphQL server, and pass back all this information about this application, name, box, that’s in development. All right. Let me quickly summarize what we did or what I just showed you with GraphQL. Instead of posting or getting data, we’re going to query data and mutate data in order to update it. You might see that you can use GraphQL iteratively. I had that GUI ID that I could query for and then use it to change of I needed to update the application, add it to an alert policy, add it to a dashboard.

Sarah Hudspeth: It’s nice that you can just build off each other. I know exactly what data I’m going to get, and I’m only going to get that data. It’s going to be nice and structured. It’s going to be fast. I’ll tell you right now, New Relic is powered by this NerdGraph which you saw. That data that we accessed, we inside our platform also use it to access… or to build out all the dashboards and charts. I should say that GraphQL was developed by Facebook in 2012. Obviously when you’re processing that much amount of data, you want to be specific about the data you get, and get it as quickly as possible. The one downside is it does require a lot of upfront work. You have to build out that data schema so that folks can get the access.

Sarah Hudspeth: But once you have it built, you have a very powerful GraphQL engine. There’s some other cool things. I was going to say, with my API call, I had to call it many times in that for loop, because I could only get one book at a time. In GraphQL, you can make multiple calls even to multiple servers to get multiple data requests. It’s just a lot more robust and flexible. I’m quickly going to go through this slide. I think from the other talks, you’ve seen how we use data and how we want access to data and how we want to build it out programmatically, and automate and really be able to empower our data to… or empower our customers to use their data in a lot of different ways. Some of those are alerts. Getting alerted on any issues, updating with microservices and Kubernetes. You can spin things up, spin things down. You need to add them to alert policies or delete them.

Sarah Hudspeth: I also work with customers a lot about either storing or dropping data they don’t need. Sometimes companies need to store their logs to be in compliance with certain data rules. And so we can export data rules and NerdGraph to AWS buckets so they meet that requirement. We did talk about dashboards and S… or others talked about dashboards and SLOs. You can update dashboards with GraphQL. You can add things, you can subtract things. You can actually have a call to get a PDF. So if you need to email it to your superior and be like, “Hey, look at our application performance for the week.” You’re able to do that with a GraphQL API call, and then synthetics as well. If you want to check on Ping Checks if anything’s failing, or if you need to update, add end points, you can all do that in GraphQL.

Sarah Hudspeth: I think I’m good on time. I was just going to quickly show you how you can build out the query in the query builder. Let’s see. Maybe I’ll get the synthetic monitor. If I just wanted a list of synthetic monitors, I could just click whatever I wanted to see. I could add here. And when I press play, it just comes up to the right. I did add a permalink. So if maybe there was something I noticed, it was a critical learner. When I wanted to go check it out, I could quickly copy and paste or build out a script to go into New Relic and see what was happening. Looks like this check is okay, but I can go in and get that view. If I wanted to mutate, I could just continue to build out.

Sarah Hudspeth: Let’s say I wanted to create a workload. I could build out a workload using whatever data here. You can use the cURL up here. You could use our New Relic command line interface. It’s really flexible and robust. For all the data nerds out there, it’s just really fun to use. That was my talk. Hopefully you picked up a lot or a little about REST APIs and GraphQL and the differences. Just wanted to let you know, my team is hiring, so please reach out. Tap me up if you have questions, but thank you for listening.

Angie Chang: Thank you, Sarah, for the talk and demo on GraphQL. It’s very informative. I’m sure people have lots of questions will like to connect with you. So thank you so much. Our next speaker, we’re going to try Jo Ann again.

Angie Chang: Jo Ann is a senior technical account manager at New Relic. Has been working directly with customers, helping them use and implement the New Relic platform, including best practices. Prior to that, she was a solutions architect at Delta Airlines in Atlanta. So welcome, Jo Ann.

New Relic senior technical account manager Jo Ann de Leon talks about programmability, React, Nerdpacks and much more at Girl Geek X New Relic virtual event. (Watch the talk)

Jo Ann de Leon: Thank you, Angie. All right. Hello, everyone. I am Jo Ann de Leon, and I will be talking about the power of ReactJS and how it transformed the New Relic platform to be an open connected and programmable platform. Before I get started, I’d like to share some tidbits about myself. I am a senior technical account manager. I have been with New Relic for three and a half years, working directly with customers, acting as a technical advisor and solutions architect, to help them implement their observability use cases. I was born and raised in the Philippines. I graduated with a math degree, but never really thought I’d work in the IT industry. But in the past 20 something years, I have worn a lot of different IT hats, including a software developer, a designer, architect and project manager. Outside of work, my wife and I enjoy traveling, playing bocce, and cuddling with our two adorable orange tabbies.

Jo Ann de Leon: For this talk, I will introduce the concept of programmability. Show where you can find some of the open source apps and custom visualizations. And finally do a quick demo of how you can build your own. In a nutshell, programmability is about giving engineers full access to the New Relic database engine, and the building blocks they need to consume data in ways that solve their unique business problems. It also means giving our engineer users and customers the same set of tools our own engineers use to build our platform key rated experiences. What does this look like?

Jo Ann de Leon: At its foundation, is the telemetry data platform, that is able to ingest not just the data from the New Relic agents, but also from integrations that support open standards such as open telemetry. On top of this data platform, is a series of scalable services such as GraphQL APIs, as well as the developer tools, such as the software development kit or SDK for short, and the command line interface, or CLI for short, that allow you to access and interact with the data. Finally, a user interface is built on open source React JavaScript, with a flexibility to support the development of custom applications and visualizations. If you’re like me, I find it really helpful to look at what others have already created before I try to write a piece of code. It’s a good idea to explore what is already available in the open source community, as it may help inspire you to build your own New Relic custom application and visualization.

Jo Ann de Leon: The first place to explore is the New Relic Instant Observability or IO, which you can find via the apps icon in the New Relic toolbar. It contains a catalog of public apps and visualizations that are maintained by New Relic, and can be managed via the UI. The catalog also allows you to manage your own custom apps. You can find a number of other open source apps and visualizations in the New Relic open source website. The great thing about open source is that these apps are extensible, meaning you can customize them to fit your needs, and you can easily install them via the CLI.

Jo Ann de Leon: Here are a couple of examples that I wanted to showcase. The first one is a cloud optimized application, which analyzes your cloud environment, figures out where you’re wasting money on excess cloud capacity. The application compares the size of your instances to their utilization, finds resources that are sized larger than needed, and estimates how much you could save by optimizing the resource size. The browser analyzer app displays an analysis of performance, and forecast how improving the performance of your website can impact your key performance indicators, such as bounce rate or traffic. It also figures out which individual site pages have the worst impact on performance, so you know where to start making fixes and improvements.

Jo Ann de Leon: A popular visualization is the status widget pack, which contains three types of visualizations. One of those three is this status timeline widget, that allows you to display how your services are performing over time using traffic lights as visual indicators. Now it’s time to build our own app.

Jo Ann de Leon: I will show you how to build a Nerdpack, which is the deployable package of an application containing all the source code and resources required to run it. It is basically a collection of React components, including launchers, nerdlets and virtualizations, all structured into a JavaScript app bundle. A launcher is a declarative file. It allows you to configure your application’s name and description, as well as which nerdlet within the nerdpack to run when it is clicked. An application is made up of one or more nerdlets, which are renderable views or windows. So they can link to each other or be launched by launchers. And finally, a visualization is a custom view or widget that can be added to a dashboard. Similar to nerdlets, it can display data whether it’s from New Relic or an external data source. You can find them via the custom visualizations app.

Jo Ann de Leon: All right. In an alternate universe, I have open a number of cat cafes around the country, where I serve coffee and cute cats or lunging around to entertain my customers, who may then fall in love with them, and decide to adopt them. In order to achieve my goal of helping these cats find their forever home, I need to keep track of how many have been adopted, and how many are still up for adoption. I went ahead and sent this data to New Relic, but how do I visualize all my data since I have so many cat cafes around the country. Luckily, I can build an awesome nerdpack. So let’s go ahead and create it.

Jo Ann de Leon: I am in the New Relic homepage. I hope you can still see it. In the New Relic homepage, you can go to the apps and click on build your own app. You can follow these instructions in the quick start. If you haven’t already done so, you can create an API key in your New Relic account, or select an existing API key. This is where you can download and install the NR One CLI, and make sure that it is up and running. And then the last step before you build your nerdpack, is to save your credentials. Let me copy this, and we’ll go ahead and create the package and run it. I am going to name my nerdlet as cat café tracker, and launcher as cat café launcher.

Jo Ann de Leon: Install the dependencies and create all the different components needed for my app. And then I can go to that NerdPack and let me open this in my Visual Studio Code. All right. Let me open the Terminal here, and then I can run my server through the New Relic One CLI, with this command: nr1 nerdpack:serve. All right. You will notice that now you can run one.newrelic.com with nerdpacks=local. This means that any local development you make can be tested in the New Relic platform. You’re also given a shortcut to the launcher, which will open your Nerdlet directly. So let’s go ahead and copy that. And let’s go back to the browser here, and let’s close this prompt.

Jo Ann de Leon: And now we have our Hello World version for our cat cafe tracker, but it’s not really very exciting. Let’s go back to our code. For the sake of time, I will be copying and pasting my code, including index.js. This code will contain the logic to retrieve the data from the database and display it in two views, a table view and a map view. Let me go ahead and do that.

Jo Ann de Leon: And then I also need to update my styles.css. This will contain styling elements for my custom UI. All right. Third one. I need to update my package.json dependencies, because we will be using the leaflet package to create a map. All right. And then finally, I need the webpack config, which we will need to support the use of map tiling information data from leaflet. This will be copied at the root folder of our package. All right. Let me save all of that. I have to restart my server. Let me clear that. I have to do an npm install first, since I had to update my package.json. And now I’m going to restart my server and relaunch my app. All right. Let’s copy that new link. Go back to our browser.

Jo Ann de Leon: Hopefully this will work. There we go. All right. So now I can view all my cat cafes around the country. I created my visualization such that the size of the circles indicates how many cats are available for adoption in that area. The bigger the circle, the more cats are available. The green color means more cats have been adopted, while those that are yellow or dark orange means we have some work to do to get more cats adopted. Finally, I have also displayed my data in a table view to the side of my map. All right. I hope you have enjoyed this quick demo on how programmability through the use of ReactJS can help you create visualizations that focus on solving business problems. Please feel free to connect with me through my email or LinkedIn. Thank you.

Angie Chang: Thank you, Jo Ann, for that talk and demo, and we’ll be sure to connect with you on LinkedIn. Our leadership panel will talk about New Relic culture, inclusion, career development, and successful interview prep.

Angie Chang: Our moderator today is Ariane Evans. She’s a diversity equity and inclusion manager at New Relic, working with the talent acquisition, hiring managers, employees, and external organizations to recruit, engage, develop underrepresented communities. And she co-leads the Relics of Color ERG. Welcome, Ariane.

Ariane Evans moderates New Relic leadership panel with Nada Da Veiga, Erin Dieterich, Kim Camacho, Tracy Ravenscraft, and Stefanie Smith. (Watch on YouTube)

Ariane Evans: Thanks, Angie. Hi, everyone. My name is Ariane Evans. And as Angie mentioned, I’m a [inaudible] manager at New Relic. I love that I get to spend a little time with you and facilitate a conversation with some of our incredible leaders at New Relic. All of them, women. It’s so inspiring to have leaders that are not only passionate about their work, but the communities that they work within. Before I dive into the questions to know more about New Relic and the areas of expertise of each of these leaders, let’s go through a quick lightning round of introductions. Please give me your name, title, and a sweet little fun fact about you. Let’s start with Kim.

Kim Camacho: Hi. Hi, everyone. Happy pride month. My name’s Kim Camacho, and my pronouns are she and her. I’ve been the director of DE&I at New Relic for about a year, and have also about 20 years of DE and I and HR experience. A fun fact about me is, I met Barack Obama right after he announced his candidacy for presidency a long time ago. So that is fun fact

Ariane Evans: Very cool, and also now very jealous. Let’s go ahead and hear from Erin.

Erin Dieterich: Very jealous of that fun fact. Hi, I’m Erin Dietrich. I lead the social impact and environmental, social and governance organizations at New Relic. My pronouns are she and her. I’ve been at New Relic for about four and a half years, and I’m based in Portland. My fun fact is that I have two small children, a one and a half year old little girl and a five and a half year old little boy. And they keep me incredibly busy, and very tired all the time. I don’t think I’ve slept well in five years. Fun fact.

Ariane Evans: Well, you look great even on little sleep Erin. Thanks for joining. Let’s hear from Tracy next.

Tracy Ravenscraft: Hi, my name is Tracy Ravenscraft. I’ve been here at New Relic for about five and a half years. I run a technical account manager team in central. My fun fact is I have two dogs, one Pomeranian, one Pomsky, and they have names like Friends characters, so their names are Phoebe and Ross. Thank you.

Ariane Evans: Love a good Friends joke. Let’s hear from Nada next.

Nada Da Veiga: Hi, everybody. I lead customer adoption organization. America’s customer adoption organization here at New Relic. Been here for five years. If you’re wondering what customer adoption is, basically, all engineers that work closely with our customers, helping them learn how to use our platform to solve their technical and business problem, basically. Fun fact: throughout my life, I have had five different passports. So no, I’m not a female version of James Bond, but that’s what my husband likes to think.

Ariane Evans: I might also think of a reference to Carmen Sandiego. Where in the world is Debeka? Where is she going next? Next let’s have Stephanie.

Stefanie Smith: Hi. Thanks, Ariane. I’m Stephanie Smith. I’m based in Massachusetts, I’ve been with New Relic for six years. Currently senior manager of talent acquisition. My team supports go to market customer adoption. Let’s see. Fun fact about me is I have two teenage daughters, one of which just graduated high school last weekend, which is very hard to believe, and a younger one. She’s a sophomore, she’ll be a junior. Erin, the exhaustion doesn’t stop. It only gets different. It’s bigger problems with bigger kids, but it’s all worth it. Fun ride for sure. Excited to be here.

Ariane Evans: Thanks, Stephanie. And thank you all. We all just listened to quite a few talks learning about why observability is important. What is monitoring? How do we implement these different products and technology? And also this happens inside of a company where the people work together. There’s a culture that allows us to do that work at our best and highest potential. I’d love to hear from each of you on how you are not only living those in practices, but working that out in your teams and your strategies at New Relic. I will start with a bit of our culture and understanding how is New Relic creating a culture where people from all backgrounds feel included.

Kim Camacho: All right. I could take a step at that, Ariane. First and foremost, I think we are very clear about our commitment to diversity, equity and inclusion. We communicate our vision, mission and objectives annually as we build out our short term annual plans and our long term strategy. All new employees and interns hear about our strategy as well as our organization when they onboard. We measure regularly how employees are feeling. The extent to which they feel belonging and respect to the company. So important to do that. I think also for our employees, one of the big things that’s really important is having communities of people that you can bond with, that are recognizable to you and have the same interests and backgrounds that you have. We have employee resource groups at New Relic. They’re fully funded and have leadership organizations as well as executive sponsors.

Kim Camacho: It’s through these organizations that we hope that people are building relationships, bonding, getting to know each other outside of their regular roles. In addition to our ERGs, we have other slack channels based on whatever people want to connect with. Whether it’s dogs, bunnies. There was one that was just started on crime channels, which I’m in love with, so you can bond. The last thing I’ll say, as it relates to really creating a culture where people feel connected is, the importance of managers. I think as our audience will know, your manager makes a big difference. Here at New Relic, it’s really important that we support train, help our managers really understand cultural competency, how to build a diverse and equitable workplace. Everyone I think on this call knows, because they’ve been through some of our trainings and are actively involved in these efforts. It’s just really important that we’re working with our managers so that they understand their role in helping create a nurturing environment for our employees. That’s a little bit about from that perspective.

Ariane Evans: That is all really cool. I know that there are also more things that New Relic is engaging. Erin, maybe you can tell us, what is New Relic focused on, or engaging our employees and social impact.

Erin Dieterich: Yeah, thanks Ariane. Newrelic.org is the name of our social impact work. We started it in early 2019, and really committed at that point to this mission of, how do we as a company, continue to push for more equitable access to technology? We really believe that accessing not just physical technology, having the best computer, having the best SaaS tools, but having the access to understanding what technology careers actually look like, what kinds of roles there are within technology. That is such a critical piece to creating this more equitable future for the industry, and to thinking about, how do we help people all along their learning journey? Whether they’re somebody who’s had a couple careers already, and are starting a career in technology, or a student who’s early in life thinking about what they want to be when they grow up. How do we give all of those people access to our incredible employees, so that they can hear the stories about how we all got where we are, and be able to start seeing themselves on this whole rainbow of pathways.

Erin Dieterich: It is not just one clear, point A to point B gets you a tech career. There are so many different ways to get where you’re going, and so many different destinations along the way. And so we’re just really passionate about infusing that into everything we do in social impact, and thinking about how we take the 2000 plus employees around the world with us on that journey. Some of the ways that we do that, we have a bunch of benefits that all of our employees get access to. They get to have 16 hours of paid time off to volunteer a year, plus we now have a set global day of service every winter. That’s three full days of volunteering, and you can slice and dice that however you want throughout the year. We incentivize our employees using that volunteering, by actually giving them dollars that they can push towards their fair charities every time they log their hours of volunteering.

Erin Dieterich: We have a $200 a year matching program. Employees can get up to $200 a year matched to any number of global charities. I think there’s 20,000 charities that they can pick from. And then we do a bunch of special campaigns. And so some of the things I really love that we’ve been building and you’ve actually been a big part of building these with us, Ariane, are some of our partnerships with our employee resource groups. Where we’re really going to our employee resource groups and helping them give us the understanding of where they want to impact in their communities, what organizations they want to work with. And then working together to make sure that that information is accessible to our employees, to incentivize and point them towards making really smart decisions with their wealth of how they can build this more equitable future.

Erin Dieterich: A great example of that is, since it’s June and it’s pride month, we are working with our rainbow relics ERG and just launched a $25,000 additional matching campaigns. In addition to those $200 employees have, they now can also put additional dollars towards this matching campaign, that goes to five different organizations that our rainbow [inaudible] helped us identify and pick in their communities. Organizations that they really care about that are helping the LGBTQ community with all of the different things going on, both in the US and abroad. Being able to be a part of understanding what that ERG community wants employees to support, and then helping employees understand how they can use their dollars to support their fellow relics, and the things they care about, is something that just makes me so excited.

Erin Dieterich: I just love seeing the way our employees are supporting each other through those special campaigns. I think I’m almost out of time, but I’ll tell one other very quick story, which is, since we have so many technical and inspiring folks on this call, I always like to take the opportunity to just pause and remind folks how valuable your skills are. Technology skills are so incredible. There’s a myriad of ways you could apply those to social good. Something we love to do is partnering our employees up with our nonprofit customers who get to use expanded access to New Relic for free. But we know that they need help with enablement. And so we partner them up with employees and the employees take on pro bono volunteering projects, where they’re using their technical skills to really support observability in nonprofits.

Erin Dieterich: And so you don’t have to be a New Relic employee to do something like that. You can really step back and say, “What causes are super important to me? What organizations do I love?” Reach out to them and say, “I want to talk to whoever’s running your technology, and see how I can be of support. I have X, Y, Z skillset that I’m really proud of. Is there a project I could help you on pro bono, and volunteer and support your organization, building your digital environment?” Because that is what every organization needs in order to power their mission. Every person with technology skills has just so much that they can give back. And so we love to do that at New Relic, but I also just love to encourage anyone anytime I can, to think about how you can use your skills out there in your community to power the charities and the causes that you care about.

Ariane Evans: Thanks, Erin. It sounds like New Relic is really building out a culture for people to live a life fully as they’d like, both internally and their communities. The things that they care about, but also themselves wholly. I’d love to hear from you, Tracy. Describing to us, which areas of your life would you like to spend more quality time when you think about work life balance.

Tracy Ravenscraft: That’s a great question. Thank you. When I think about where I like to spend my time outside of work, definitely with family and friends. Everybody wants that more family, friend time. But not only just spending more time with them, being present. Not checking my phone for slack messages, going on vacations and being able to completely disconnect. That’s what New Relic has brought to my life.

Tracy Ravenscraft: I’ve been at New Relic for five and a half years. I did site reliability in the past, network administration, network engineering. I never realized how I wasn’t there. I’m always looking for the next page. When I have time off, I’m bringing my laptop, I’m bringing my phone. I feel like New Relic, with our recharge week, which the summer that we all get off at the same time. FTO, so it’s flex time off. There’s really no limit to my vacation. Just some of the applications we have, like Ginger, that helps with mental health. I really feel connected when I’m using my own personal time and being with my family. So yeah, that’s how I like to recharge, if you will.

Ariane Evans: Yeah. So important. When you are moving on to the next project, or you are trying to get to the deadline of a particular thing, you can’t do that if you’re empty, and you don’t have the energy within you. And so I guess moving on, switching gears a little bit, want to talk with you, Nada, about navigating careers and career challenges. Career journeys can vary person to person. As Tracy just described, she’s been across the board of different kinds of engineers, and now a customer adoption leader, but how might you recommend navigating a career journey, and even a career journey into leadership?

Nada Da Veiga: Yeah. I mean, I think that’s an excellent question really. What I, or what we in general try to encourage folks in my organization, is to own their career, and be really proactive about it. And so a lot of people early in their careers think that they should somehow just wait for their manager to have these types of conversations. I would say quite the opposite. Be proactive about it, ask questions, share, what do you want for yourself? Where do you want to be three years from now, five years from now? Ask your manager, “What do I need to do to get there?” Because if you are informed and you know what this person expects from you, what three, five things they want to see from you in order for you to actually make it there, guess what? You have a lot higher chance of getting there, than if you’re just sitting and waiting for them to tell you, because they may or may not tell you actually.

Nada Da Veiga: They may or may not understand that you want to get from this role to some other role. That is what we see a lot with our teams. At New Relic, we are very much committed to our employee’s career progression. These are proactive conversations that are happening continuously. We encourage our employees to put together their career plans, to share those with their managers. And then some of them just want to go, “Hey, how do I go from this role that I’m in today, maybe to a senior role or a principal role?” Others want to move maybe from one org to a different org, so they want help with that path.

Nada Da Veiga: Third group will say, “Well, I want to get to leadership.” But I think how you approach it really doesn’t matter. New Relic specifically, if you are interested in leadership, we have about 14 different management classes that we recommend to folks that are setting you on that leadership path at New Relic. But whether you’re at New Relic or somewhere else, show your manager, show your leadership what you really are interested in, where your heart is at, and be proactive about it. That’s probably the best advice I can give.

Ariane Evans: Yeah. I love that. I will say that I think my career journey at New Relic is a testament to that. Starting in talent acquisition and getting to be a partner to Stephanie, but then moving into social impact and getting to learn from Erin, and now today, being a part of the DE and I team, and getting to work very closely with Kim, and that has all been championed by New Relic and the leaders within… I just said, “I’m interested in this thing, and I’m not really sure where to go from here.” But it did start with an interview. It started with a conversation with my manager. And so I’d love to kick it over to you, Stephanie, and think about, for a lot of people, getting started in your career, or looking for new opportunities, it starts with that interview process. You’ve interviewed hundreds of people in your career. And now as a recruiting leader, what is the best advice you have for anyone that is preparing to interview or in the process of interviewing currently?

Stefanie Smith: That’s a good question. I do want to just talk about just quickly, Ariane, your career progression. There’s so many people at New Relic that have had career progression, me included. I started off as a recruiter, and promoted along the way to senior manager. So there’s so much opportunity. But yes, there is an interview that’s involved. Interviewing with the company really, it’s your first impression, but it’s also our first impression to you as well. I always tell people that it’s your interview as much as it is ours. Make sure you qualify. Know what the company does. Really know what the company does. Do some research, do your homework. There’s a wealth of information about companies on the internet. It’s incredible. Link in with people on LinkedIn. Understand the roles and responsibilities, what people are.

Stefanie Smith: And then when you are talking to someone, likely it’s going to be a recruiter first, it’s a conversation. Like I said, really, you’re qualifying us, we’re qualifying you. Part of our core values is being authentic. I think that you’ve probably seen a lot of authenticity throughout this entire panel, and previous to the speakers. Be authentic during the interview, be yourself. Find some common ground. Look at it as just a conversation. Working, we spend more time than anywhere else. New Relic encourages everyone to be their best authentic self. When you’re in the interview, just really be yourself and ask good questions, and talk about career pathing and all the things that are important to you.

Stefanie Smith: Realize, if this is the right company, position, and so forth. And also even ask for guidance along the way. Your recruiter’s going to be the first step, and the recruiters are going to send you on for the next interview. Connect with the recruiter as often as possible. Even connect with the people that you’re interviewing with. We have multiple steps of roles when we interview here at New Relic. People are always going to be available to help guide you through the process. Ultimately, like I said, it’s your interview as much as it’s ours.

Ariane Evans: Yeah. I totally agree with that. Since we’ve also wrapped up this time with all of our leaders, I want to thank Girl Geek, thank New Relic for also putting this together, and everybody for listening in. I hope that you’ve gotten to pull out some nuggets of advice that are beneficial to you. If you are interested in learning more about New Relic or careers or opportunities, there are some things that Kim dropped into about our ERGs and our benefits. Please take a look at newrelic.com/culture. It will take you to our careers page and the opportunities that are currently live across the world. There are many.

Angie Chang: Thank you, Ariane, for moderating the panel, and to all the panelists for joining us. So now is time for our networking session. If you can click on the link at the bottom of the chat to our Zoom meeting, we can go into a Zoom meeting and have some breakout rooms where we can meet each other in person, and chat a little with our remaining 15 minutes that we have today. So if you can click on that link in chat that Amy has added, I’ll see you over at Zoom meeting and talk to you there. Thanks for coming.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• See open jobs at New Relic and check out open jobs at our trusted partner companies.
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!