Autodesk makes technology that millions of people use to design and make the world around us. Our solutions are used to create all kinds of amazing things—from the greenest buildings to the cleanest cars, from the smartest factories to the biggest blockbusters. And our platform helps our customers combine technologies to solve their challenges, whatever their industry. Together, we help our customers turn their ideas into new realities that shape a better future for everyone.
Cadence is a pivotal leader in electronic systems design, building upon more than 30 years of computational software expertise. The company applies its underlying Intelligent System Design strategy to deliver software, hardware, and IP that turn design concepts into reality. Cadence customers are the world’s most innovative companies, delivering extraordinary electronic products from chips to boards to complete systems for the most dynamic market applications, including hyperscale computing, 5G communications, automotive, mobile, aerospace, consumer, industrial, and healthcare. For eight years in a row, Fortune magazine has named Cadence one of the 100 Best Companies to Work For.
At USDS, we are mission-driven professionals who are passionate about applying our work and lived experiences to public service. We come from a range of cultural, geographical, and ethnic backgrounds, and we represent a myriad of intersecting identities, just like the people we serve. We are curious about understanding the needs of people and are excited to use our short tours of service to make a positive impact.
We collaborate with public servants throughout the government to help untangle complex challenges and ultimately deliver a better government experience to people. We work across multiple agencies and bring best practices from our various disciplines, which include engineering, product, design, procurement, data science, operations, talent, and communications.
USDS operates on a tour-of-service model with a maximum term of four years. While most people serve for one or two years, we’ll consider some shorter tours as well. Time commitments are not binding.
To learn about the developer, product, design, data science, and acquisition skills we’re hiring for, see “How we work”. We are always hiring for a number of different roles, so if you are an expert in your field and are interested in working at USDS, you should apply here: www.usds.gov/apply
* To work at USDS, you must be a U.S. citizen, and pass a background check and a drug test. As a federal employee, you will need to be vaccinated against COVID-19 or receive a legal exemption prior to employment.
Like GPS for your code, CodeSee is on a mission to help you build the applications you love without the guesswork. Instantly visualize a map of your app’s services, directories, and file dependencies. CodeSee ensures your team stays code synced.
Don’t miss CodeSee CEO and Co-Founder Shanea Leven speaking at ELEVATE 2023 Conference & Career Fair online – it’s FREE to register & attend!
Thank you to AWS volunteers for sharing valuable career insights with students who asked questions, in small groups of 5-15 students rotating around the room after AWS leader Trevor Moore gave an intro to Amazon as a business, with trivia and fun prizes. Special thank you to our executive sponsor Shannon Thoke and all the AWS volunteers!
Twitch employee Ashley Clark (and native Oakland resident!) joined by videoconference to talk about her career in technical program management with CCPA high school seniors.
Last school year, Girl Geek X volunteers helped CCPA staff prepare for school reopening in the pandemic, shared career insights for 11th and 12th graders in the school, gave feedback on senior capstone project presentations, and hosted a Teacher Appreciation luncheon with goodie bags for educators.
Field trips allow students to experience a variety of workplaces and gain invaluable insight on real-world job titles, professional skills, and modern workplaces – expanding their minds and STEM career options. Check out event photos at our Facebook page!
“Everyone at Amazon has an inherent desire to make a positive impact, but sometimes we don’t know how we can add value. When Angie Chang from Girl Geek X approached us with the CCPA CompSci Field Trip concept, we jumped all over the opportunity.
I was immediately able to gather a diverse set of Amazonians, across multiple areas of the business, who shared the same passion for positively impacting where we work and live.
We worked with Angie and the CCPA teaching staff to build an impactful agenda which included a building tour, networking sessions, tech. talks and of course, free swag! Our events team did an incredible job accommodating a sizable group (70+) of visitors.”
Special thank you to Trevor Moore (AWS Strategic Account Manager) for organizing CCPA’s CompSci Field Trip to AWS on December 9, 2022.
More Student Field Trips in 2023 – Get Involved!
If you are working at Google, LinkedIn, GitHub, Discord, Instagram, Reddit, Spotify or TikTok and want to host a student field trip, please email us at firstname.lastname@example.org to chat about 2023 field trips.
Logistics and Playbook for Future Field Trips
Many companies have free and/or catered lunch at the workplace that field trips can take advantage of for “lunch and learn” field trips:
Company speakers hail from a broad representation of departments / teams, from data to accounting, from engineer to marketing, from project manager to support engineer, from sales to design.
Emphasize both traditional (higher education, vocational school) pathways to career success, in addition to “non-traditional” (coding bootcamps, self-learning with portfolio of work) ways to entering the tech workplace.
Employee resource groups (ERGs) may be interested in inviting their members to participate as volunteers and role models for the students. Girl Geek X may also source volunteers to join the students for conversation at lunch.
Sample Field Trip Agenda
10:00am – Students and educators arrive to office, check-in
10:30am – Tour of office & introduction to teams / departments / volunteers roles
11:30am – Lunch with volunteers (speed networking with a wide gamut of roles in the company)
1:00pm – Workshop(s) on coding, business, workplace skills, or speed networking (2-3 volunteers sit with a group of 5-15 students, students rotating chairs every 15 minutes)/li>
1:45pm – Distribute company swag
2:00pm – Students and educators depart
Schedule for 2022-2023 School Year
Here are the times in the coming school year that field trips with CCPA can be scheduled, in order of preference:
This is a time for CCPA students to leave the campus and participate in a range of activities like backcountry backpacking, or participating in a course (e.g. art, STEM). The majority of students will choose among Art and PE electives. Postsession participation is required, so students are encouraged to pick an activity or course most aligned to their interests and schedule.
Any Friday during the 2022-2023 school year – preferably adjacent to a long weekend or holiday.
Semester 1: August 8 – December 16, 2022
Semester 2: January 4 – April 27, 2023 (before AP tests / finals)
If the company is located within walking distance to a BART station, educators can walk with students to Coliseum Station to BART to the field trip by the BART public transportation system.
If the company is located in Silicon Valley, please provide a charter bus or Lyft/Uber codes for students to carpool.
Did you know mathematician Grace Hopper helped invent the programming language COBOL? Here she is pictured at the UNIVAC I console in 1960. Her birthday is December 9 – and why CS Ed Week is celebrated each December!
Elevate brings together thousands of women technologists, innovators and tech leaders from around the world to share the latest in tech and leadership with fellow mid-and-senior level professional women.
This virtual conference is FREE for attendees – last year, over 4,000 women signed up to attend – tuning in from 42 countries around the world – to be inspired by speakers on the latest in tech trends and leadership on International Women’s Day.
Sessions content typically covers the following topics:
Lightning Tech Talks – Dive deep into an area that’s unique / critical to your business or role, from engineer to product manager.
Technical Skills & Tactics – Tutorials, walkthroughs, or deep dives into a skillset or tactical approach to how you solved a real-world challenge.
Learning & Development – Topics include negotiation, mid-career job searches, interviewing tips, managing up, self-awareness, ageism / return to work bias, mental health, etc.
We are seeking session proposals for the 6th annual ELEVATE 2023 Virtual Conference to be held March 8th, 2023.
Girl Geek X invites women technologists, innovators and tech leaders from around the world to participate in ELEVATE Virtual Conference to share the latest in tech and leadership with fellow mid-and-senior level women in technology.
Work on a unique technical project or have interesting insights you’d love to share? We want to hear from you! Both first-time and experienced speakers are welcome to apply.
Speaker submissions for ELEVATE 2023 are now open!
Don’t see your team / department represented? That’s OK – Tell us about your expertise, from TPM to customer success, operations to lab research, etc. Please do submit a talk about your unique domain expertise!
ETC, ETC – tell us what you are excited to geek out about in 2023!
How to write a speaker submission, from our friends at Autodesk:
Speaker Bio Template:
[name] is [job title] at [company]. In this role, she is responsible for [key activities]. Previously, she was [role] at [company] -OR- She has worked in this industry for [number of years]. She is passionate about [what motivates you]. She volunteers / leads [organizations and/or employee resource groups]. She studied [focus area] at [school].
Talk Title / Abstract Tips:
There are three parts to writing a talk title and abstract. Structure your thoughts around them to tell a short and complete story.
Talk Title – Keep it simple and straightforward. Use terms that others might use to search for it.
Problem Statement – Explain briefly the challenge you will help others address and the different perspective or experience that you can share with them.
Benefits / Takeaways – Tell others clearly how they will benefit by spending time with you (e.g. the insights or skills they will learn). This can be a simple list of takeaways for conference attendees.
We’re excited to announce that we are launching Career Fairs! Registration will open in just a few weeks, and the first event is on Thursday, December 8, 2022.
Is your team is hiring for open technical roles? Now’s the perfect time tobuild your talent pipeline, attract passive candidates, shine a spotlight on your female leaders, and let us help you create evergreen talent branding and recruiting assets to support and highlight your organization’s DEI efforts!
We’re using a new event platform for Girl Geek X Career Fairs online with improved networking capabilities, and every sponsor will get:
a customizable virtual recruiting booth that can be staffed by members of your team, from hiring manager to recruiters,
networking tables within your virtual booth
opportunities to connect LIVE with attendees in 1:1 or small groups meetings
your open roles promoted to our community of over 40,000 women in tech
If your company wants to promote your talent brand and open roles to our community of over 40,000 women in tech, and give your female leaders a forum to share their experience, insights and excitement for their work, please check out our sponsorship prospectus and let’s talk!
Over 120 girl geeks joined networking and talks at the sold-out OpenAI Girl Geek Dinneron September 14, 2022 in San Francisco’s Mission district.
Hear lightning talks from OpenAI women working in AI with music and deep learning, sharing the power of trying and trying again, how to make language models useful, and much more at the OpenAI Girl Geek Dinner video on YouTube!
OpenAI Residency applications are open! OpenAI is looking for engineers and researchers who are interested in applying their skills to AI and machine learning. Please apply for OpenAI jobs here!
If you have an unconventional educational background, we encourage you to apply to OpenAI Residency (applications are open through September 30, 2022).
Transcript of OpenAI Girl Geek Dinner – Lightning Talks:
Angie Chang: Hello. Thank you everyone for coming tonight. My name’s Angie Chang and I’m one of the founders of Girl Geek X. We started over a decade ago as, Bay Area Girl Geek Dinners, and we’re still going strong. Thank you to OpenAI for hosting us for a second time. We’re really excited to see the new office and invite a bunch of Girl Geeks over to hear these lightning talks on AI and policy and all these things that we’re so excited to learn about tonight!
Sukrutha Bhadouria: Hi. I know you all were still chatting when Angie introduced herself, but she’s Angie and Girl Geek X is basically her brainchild. It started off with Angie looking to bring women together, I’m doing your pitch, Angie for you because I have a louder voice. Some people, they ask me if I swallowed a mic as a child because I’m so loud and I don’t need a mic.
Sukrutha Bhadouria: Anyway, I’m Sukrutha, so Angie started Girl Geek and it was back then called Bay Area Girl Geek Dinners, this was over 10 years ago. And when I had just moved to the Bay Area, looking for ways to meet new people and I found out about Bay Area Girl Geek Dinners dot com at that time, and I tried really hard to meet with Angie, but she was a busy bee doing all sorts of cool things, trying to change the world. And this was way before ERGs existed, right? So people didn’t have a way to connect with the community until they went to meetups.
Sukrutha Bhadouria: And Girl Geek Dinners, at that time, was the one way you could also get an insight into what these sponsoring companies worked on, what life was like. And so it also allowed people to get an opportunity to speak and a lot of the speakers at Girl Geek Dinners were first time speakers. They were too afraid to sign up for conferences. If you go to our website (girlgeek.io), you’ll see all these amazing stats on how since Angie started, there’s been a real shift in the environment in how people are more willing to speak at conferences, due to some of the chances they’ve gotten as a result of speaking at an event sponsored by their company. This organization exists.
Sukrutha Bhadouria: I joined Angie and we tried to change the world together. I’m happy to report that I think we actually did. We rebranded to Girl Geek X, and that’s when the organization hit 10 years. It was a sizable number of people working on it, it was Angie and me and it was just the two of us. And then Angie had this idea to really evolving into a company and so that’s when she started to bring on contractors, more people such as somebody who could take video of our events to make us look a little bit more professional and somebody else to do our website besides me. And we started to do podcasts.
Sukrutha Bhadouria: We started to do virtual annual conferences and we really, really, really were always consistently sold out for our in-person events that would happen at various companies that we partnered with through the Bay Area. Then COVID hit and the good thing is that we had already started to have a global presence through the virtual conferences that we had and we’ve now had four? Five, yeah.
Sukrutha Bhadouria: We used to be carpooling all around the Bay Area together to these events after work and now we are moms. So it’s amazing. We would look up and see amazing people working at these sponsoring companies speak and we’d be like, “Wow, look at them managing their mom life and parent life and coming to these events.” But I just think that it’s now become such a common thing that it’s not as isolated anymore. And I’m hopeful that, you all can come back again and again, because this in person event has really made me really happy.
Sukrutha Bhadouria: I’ve been holed up in my home office today, which is basically a room which also has my… What’s it called? A bike that stays in one place, stationary bike, so it has too many things going on in the room, but I wanted to give a big thanks to OpenAI for hosting us for the second time, for sponsoring for the second time. And I hope that we can keep doing this. So please do get your companies to sponsor and encourage them to do it in person. That’s all I will say. I know I said a lot more than I had planned, but thank you again, and Angie.
Angie Chang: Thank you Sukrutha, for the intro. I guess I should talk up Sukrutha a little more. When I first met her, she was a software engineer in test, and now she is at Salesforce as a Senior Director of Engineering there, so I’m very proud of her. And over the years we… She mentioned we have a podcast, we have annual virtual conferences!
Angie Chang: We’ll be launching a career fair virtually as well, to be announced. And I don’t want to say too much. We have an amazing line up of speakers tonight and we’re going to invite up first, Elena, who is our host for the night from OpenAI.
Elena Chatziathanasiadou: Hi everyone, I’m Elena. I work here and I’m on the recruiting team, I’m leading the Residency program right now. I’m very excited that you’re all here and have joined us together. Really want to thank Angie and Girl Geek X. We’re very excited to deepen our partnership together and to be back in the office here all together, in the new space and to experience this tonight.
Elena Chatziathanasiadou: We’re very excited about having you here and in terms of what we’ll see tonight, we’ll have a series of lightning talks and then that will be followed by Q&A and then we’ll get some dessert in the area that we were before and then we’ll wrap up at 8:30. But before we get started, I did want to take a moment to make a quick plug and share that…
Elena Chatziathanasiadou: We’re actively hiring for our Residency program and that includes both research and engineering roles and the goal of it is really to help develop AI talent. The program, it offers a pathway to a full-time role at OpenAI for folks that are currently not focusing on AI and are already researchers or engineers in a different field.
Elena Chatziathanasiadou: We’re really excited to hear from you. If you do have an interest in making this career switch, come talk to me after. And we’ll also have full time recruiting team members and positions that we’re hiring for across research product and engineering that we can tell you more about. Please come find us and learn more about the interview process, but also what the program offers.
Elena Chatziathanasiadou: With that I wanted to introduce our first speaker, Christine, who’s currently managing our multimodal team and previously worked on music generation research, created MuseNet and was collaborating on Jukebox. And before that was a classical pianist who transitioned into a researcher as well. I’ll hand it over to Christine. Thank you so much.
Christine McLeavey: Thank you. So yes, it’s really an honor to be here tonight. Thank you all for being here. And this Residency program is near and dear to my own heart, because I first joined OpenAI through, what was then the Scholars Program and the Fellows Program and those are the programs which have since evolved into this Residency program. I’ll put a plug in for anyone who’s considering it.
Christine McLeavey: I want to talk this evening about my own path through OpenAI, but especially about the two music models that I worked on during the time here. I thought I’d start by just going ahead and playing an example of each of the models. The first one, this is the one I worked on when I was doing the Scholars and Fellows program. This is MuseNet, which works in the MIDI domain, so this is the model trying to generate in the style of jazz. Okay, I’ll cut that off and then after I joined full time, I was lucky enough to collaborate with some amazing researchers here to work on a model that was instead working in the raw audio domain. The fun of that is you get to imitate human voices. This is trying to do the style of Elvis with lyrics by Heewoo. Okay.
Christine McLeavey:Elena mentioned before being at OpenAI, I was actually working as a pianist, I had done some math and physics in college, but obviously it had been a long time and so I think I took a good year of self studying before I applied to anything. And I thought I would just give a shout out to three of the online programs that I particularly liked at that point. They’re all amazing. But then I was lucky enough to join the first cohort of scholars that we had here. And at that point I was just trying to do this process of learning about all these different models. And I had this feeling that instead of just copying a model or copying what someone else has done, let me just try to translate it into a field that I know well, which was music. And so what became MuseNet was really my attempt to take all of the stuff I was learning and then apply it to the music domain instead.
Christine McLeavey: MIDI format is this really nice representation of music. I think of it as the way that a composer thinks of music, so it’ll do things like it tells you what notes it plays when, the timing of it, the volume of it, things like that, which instrument is supposed to play. But it loses all the actual detail of when a human takes it and performs it. You don’t get a person’s voice, you don’t get the sound of a great cellist, anything like that.
Christine McLeavey: The nice thing is it’s what you trade in expressivity, you get in this nice really meaningful representation. It does sound pretty terrible when you try to render materials. As a musician, just thinking about the structure of music, this was a nice simplification for a scholars project. What I did is I took a bunch of MIDI files and I tried to pull them out and turned them into a sort of language to make them look as much the sort of thing that you could get in your own net to predict as possible.
Christine McLeavey: I did things like I would always tell the model which composer or which band was going to be first and then things like what tempo was going to be when notes would turn on and off, and a wait token, which would tell the model how long to wait, things like that. And then what you end up doing is you translate that tokenization into just a dictionary of numbers and the model sees something like this. Which I think that this is the first page of a Chopin bellade or something.
Christine McLeavey: What the model is faced with is this task of given the very first number, what number do you think is going to come next? And then given the first two numbers, what number is going to come next? And when you first look at the first thing and when the model first sees it’s like how do you do this? What does that even mean? It feels like an impossible task. But what happens is the model sees many, many, many examples of this.
Christine McLeavey: And over time it starts to pick up on, ah, if I see 4,006 somehow I tend to see 586 more often after that or something. It starts to pick up on these patterns, which we know because we know the tokenization was like, oh, if a piano plays the note G, then probably soon after it’s going to turn off the note G or something. It has real musical meaning to us. But the model is just seeing these numbers like that. The nice thing is the model gets really good at this job and then you can turn it into a generator just by sampling based on, I thinks there’s like a 20% chance this token’s going to come next, so 20% of the time take that.
Christine McLeavey: The other really fun thing you can do is you can then study the sort of mathematical representation you’ve gotten for these tokens. So I was always giving it the composer or band token in the beginning and now you can look at the vectors or the sort of embedding that it learns through these composers.
Christine McLeavey: And as a musician it’s really fun because I would clearly think that Da Vinci and Ravel, for all these French guys are related and the model just picked up on the same thing, which is cool. But the other really fun thing is that you can mix and match those [inaudible]. So here is the start of one of my very favorite Chopin, Nocturnes. So I actually just gave the model the first six notes of that and this is what the model thought, if instead it was being written by [inaudible] It was a bunch of VPs. It goes on for a while, but I’ll cut it off there. And that was MuseNet.
Christine McLeavey: And then I ended up joining full time after that and I was lucky enough to collaborate with Prafulla and Heewoo on taking music generation over to the raw audio domain. And so in a way this is a much harder problem because now whereas in MIDI world you have just nice tokens which are meaningful in a musical way, raw audio is just literally 22,000 or 44,000 times per second.
Christine McLeavey: You’re recording how loud the sound is at that moment in time and the nice thing about it is it gives you all this expressive freedom, right? Literally any sound you can imagine you can represent as a sound wave, just audio recording to that. The trouble is there are just so many ways for those waves to go wrong or those patterns to go wrong. If you mess up on the short scale, it’s just like crazy hissing noise. If you mess up on long scale, your piece sadly starts getting out of tune or the rhythm drifts or so many ways it can go wrong, it’s really an unforgiving sort of medium. And the problem is now in order to get a minute of music, it’s no longer maybe 3000 tokens you have to do, it’s maybe a million numbers that you have to get correct.
Christine McLeavey: We approached this by looking at ways that we could compress the music to make it more tractable because at that point a transformer could maybe deal well with the context of 4,000 tokens or something. We used an auto encoder to do three different layers or levels of compression and the sort of least compressed on the bottom. The nice thing about that is it’s very easy to translate it back to the regular raw audio. If you put some original song in and then back out, you don’t notice any loss at all. Whereas if you put it through the most compressed version, the nice thing is now it’s super compressed, like 3000 tokens might get you half a minute of music or something. But if you go through this simple just trying to reconstruct the raw audio, it sounds really bad. You can sort of tell that someone’s singing but you’ve lost most of the detail.
Christine McLeavey: The nice thing about it is when you work in that top layer of tokens, now this looks a lot like the MuseNet problem or even just a lot language problem where you’re just predicting tokens. So we train a transformer on that. We sort of added in the same which person was singing, which band was playing, and then we also added in where you can write the lyrics in, so the model conditions on the lyrics and then generates these tokens. And then I won’t get into the details, but we had to train extra transformers to do this upsampling process so that you could get back to raw audio without totally losing all the detail.
Christine McLeavey: The fun thing is you can do things like ask it to generate in the style of Sinatra singing Hot Tub Christmas and I have to put in a book, these were lyrics by at, that point, GPT-2. All right. It’s a Christmas classic now. And then last I wanted to wrap up by talking a little bit about the multimodal team, which is the team that I’m really excited to be managing these days. It’s this really, really great group of people. Unfortunately, our current projects are all internal and I can’t talk about them, although stay tuned, we’ll be publishing them to the blog when we can. You might recognize Clip, which was work done by Alec and Jong Wook both on our team. This is, I guess, nearly two years ago already, but made a really big impact on the image work at that point. And then just to put in a plug for the team, we’re about a group of 10 at this point and we will be hosting a resident in 2023.
Christine McLeavey: Please reach out if anyone’s interested to talk more. And then we’re doing all sorts of projects in the sort of image, audio and video domains both on the sort of understanding side and generation side. And we end up working really closely with algorithms, which is the other team that tends to do a lot of awesome multimodal projects. But then also anytime we get close to things that we’re looking at putting out tech customers, we end up working with applied through that and then also obviously scaling because at OpenAI we believe deeply in this, get a good pattern and then scale it up and it becomes awesome. So thank you so much for your attention.
Elena Chatziathanasiadou: Thank you so much, Christine. That was awesome. So now next we’ll have Alethea. Alethea has spent the last couple of years at OpenAI working on getting neural networks to do math. Before that, they built large infrastructure health system, studied math and philosophy and spent lots of time singing karaoke. Welcome, Alethea.
Alethea Power: Thank you. So this talk is called If At First You Don’t Succeed, Try Try Again. It’s been a wild few years. I decided I wanted to give an uplifting and encouraging talk. It’s a short talk so it doesn’t get too deep into technical details, but if you’re interested in it, please find me afterwards. I will talk your ear off about it.
Alethea Power: Okay, my name is Alethea Power and yes, Patience is actually my middle name, which will be very relevant for this talk. Okay, so about 10 years ago I was a software engineer and site reliability engineer and my dream was to get into artificial intelligence, but I didn’t know how to do it. I didn’t have a degree in AI, I didn’t have any background in AI, I didn’t have any idea how to break in. So I thought, ah, I probably need to take some time off to study this before I can get into the field.
Alethea Power: I started saving up some money so that I could take time off to study. But by the time I had enough money saved up, I realized I needed to handle my gender issues. So I took that time off to go through a gender transition instead of studying AI. Eventually though I was finally ready to try and break into AI in some form or fashion and that was about the time that OpenAI hosted their last Girl Geek Dinner, that was in 2019. And I came to that talk and I met one of the recruiters who stunned me by telling me I didn’t need to have a degree in AI and I didn’t need to have a background in AI to be able to work here.
Alethea Power: She introduced me to the Scholars Program, the same program that Christine went through, which today is called the Residency Program. And I applied to that and I got in and I had the best mentor in the entire program, Christine. I’m second generation scholar up here. But there were in addition to the obstacles before, there were obstacles after joining the program as well, about three weeks after I joined, there was a pandemic, you may have heard about it. But despite spending a lot of time fearing that I might die or people I love might die for some reason or another, health or political, Christine was very kind and understanding and supportive and she helped me get to the point where I had learned a ton about artificial intelligence and managed to do a great project and I ended up applying full-time and I got three offers here. Thank you. I wasn’t trying to brag, but thank you. This is more to encourage you.
Alethea Power: I ended up taking a job on a team that was trying to teach neural networks to reason and do math. And what I want to talk about here is about a year after I joined that team, I released my first research paper called Grokking: Generalization Beyond Overfitting on Small Datasets. I’m going to give you a very basic introduction to what all that jargon means. And like I said, if you want more technical details, come talk to me afterwards. So first I need to explain how training neural networks works. If you have a background in ML, this is going to be very basic 101. If you don’t, it’s going to be exciting.
Alethea Power: Okay, so usually when we’re trying to train a neural network, we’ve got some amount of data that captures a pattern that we want that neural network to recreate in the future. And often if we’re doing what’s called supervised training, we’ll break that data up into training data and evaluation data. And you can think of this, the training data is sort of what we actually teach the neural network, what it learns from. This is like classroom education and evaluation data is basically like pop quizzes to see how much the neural network learned. And neural networks have this nice property where you can pop quiz them. They don’t learn anything from the pop quiz, they just tell you how they did and then five minutes later you can pop quiz them again and the questions are all new again, they have no memory of them. Throughout the course of training, we measure the performance of the neural network on both the training data, the classroom instruction and the evaluation data, the pop quizzes.
Alethea Power: And there’s two main ways we measure this. One is called loss. I won’t go into details right now about what loss is, but the short version is it’s a differentiable function calculus derivatives that we use to actually figure out how to modify the network, so it learns, when loss goes down. The network is learning. Accuracy is exactly what you would think of being like a test score, so 0% accuracy means you got every question wrong. A hundred percent accuracy means you got every question right. This is what a very successful neural network training looks like. You can see, oh, the x axis here on both of these graphs is steps of training. You can see that as we train this neural network along the loss on both the training and evaluation go down. It’s learning what it’s supposed to learn from and it’s able to generalize that to the pop quizzes.
Alethea Power: It’s doing well on the tests as well and then this is what it’s actually scoring. So by the end of this training it gets up to 90% accuracy, so it’s got an A. Sometimes though, if you train a neural network for too long, it starts to do what’s called overfitting. You might remember the word overfitting from the title of the paper. In this case, the neural network learns too much detail from the training set that doesn’t really generalize to the rest of the world. And so its performance on the quizzes starts to get worse. So an example of this in this paper, I was training neural networks to do math, basic mathematical equations. For instance, if it happened to be the case that the training data had more even numbers than odd numbers, and if it was trying to learn addition, then it might learn that usually the answer is going to be even. Well, in reality that’s not true in addition.
Alethea Power: In reality, you want to actually know how to add and the number’s going to be whatever it is. So that would be an example where it learned some sort of incorrect, non-generalizable information from the training set and that made it start performing worse on the evaluation set. And you can see here in this situation, the accuracy on evaluation would go back down. Sometimes, and this is very common when you’re trying to get a neural network to do math, you have an even worse situation where the same thing happens with your loss, but it consistently fails the pop quiz every time. Gets to a 100% percent accuracy on the training data and fails the pop quiz. This means the network and we were using similar kinds of networks to the ones Christine was talking about, just math instead of music, this means the network never really understood what it was learning, it just memorized it.
Alethea Power: This is like the kid who knows that when you say six plus four, you’re supposed to respond with 10 but has no idea how to actually add. So this was a common scenario when training neural networks to do math. They’re really good at pattern recognition, but they’re not always good at understanding a deep analytical precise truth underneath the pattern. Well then one day we got lucky and by lucky I mean forgetful. So one of my coworkers was running an experiment like this and he went on vacation and forgot to stop it. And so a week later he came back and it had just kept studying and studying and studying and studying and studying and studying and studying and studying and studying. And it learned. So what happened here was, it went into this overfitting regime where usually we’d say, ah, it’s learned all it can learn from this training data.
Alethea Power: There’s no more to learn and see, it still had zero accuracy and it just kept getting worse and worse and worse. And then suddenly long after it memorized all of the training data, it had an ‘aha’ moment and it was like, oh, all this stuff that I memorized actually makes a pattern and the pattern is addition or division or S5 composition or whichever task we had it working on. And then the loss started coming back down on the pop quizzes and it went up and it got a 100%. This is weird, this never happens in neural networks. We dug in and recreated this many times, implemented it twice, saw the same behavior with two completely independent implementations on a wide variety of tasks and there’s all sorts of other interesting stuff about when this happens and when it doesn’t, ask me in the questions afterwards.
Alethea Power: The point here is at first the network didn’t succeed, but it just kept trying the same way I did when at first I couldn’t get into AI, but I just kept trying. We named this phenomenon where it finally figures it out Grokking, and we named this after Robert Heinlein’s novel Stranger in a Strange Land. It’s a science fiction book and Grok is a Martian word in that book, which means, “To understand so thoroughly that the observer becomes a part of the observed to merge, blend, intermarry, lose identity in group experience.” And it turns out this is exactly what these neural networks do. I’m going to let you take pictures before I change the slide.
Alethea Power: This network was trying to learn modular addition and modular addition you can think of is adding hours on a clock. Also, thank you to Christine for that analogy. If you have 11 and you add 3 to it, you don’t end up with 14, you end up with 2 because that’s what happens on the clock. The clock is modular 12, we were having it learn modular 97, and then we tore open the network that had grokked afterwards to see what was going on inside of it and it had actually built internally this circular structure of the numbers. It had created the mathematical structure we were trying to get it to learn that allowed it to actually solve the problem. Did this with all different kinds of problems, so we had one network learning to compose permutations and it found what are called subgroups and co-sets out of that, details later. But the point is, it worked so hard for so long through so much failure that it became the knowledge it was trying to get.
Alethea Power: The point here is, that if your dream is to get into AI, even if you have no background in AI or whatever your dream is, it doesn’t matter. Keep trying and keep trying and keep trying and keep trying and maybe you can get there eventually. And in particular, if your dream is to work at OpenAI, which I highly recommend because this place is fabulous, then try, even if it’s not the background you have already, even if you feel like you have a weird background or you’re not like the people here or like the people in this field.
Alethea Power: We’re a humanitarian organization. Our core mission embodied in our legal structure and our financial structure is to make sure that artificial intelligence benefits all of humanity instead of just a small number of rich people in Silicon Valley. And to be a humanitarian organization with a humanitarian mission, we need a wide diversity of perspectives here. If you have a different life story, a different path, different perspectives than we’ve seen before, that makes you more valuable here, not less, so please consider applying.
Elena Chatziathanasiadou: Thank you so much, Alethea, That was awesome. And now next we’ll have Tyna, who’s on the policy research team currently doing our rotation on applied research and she participated in the OpenAI Scholars Program, has spent some time researching economic impacts of our models, building safety evaluations, and collaborated on web GPT and moderation API. Let’s hear from Tyna.
Tyna Eloundou: Wow, so many of you. Let’s see. Okay, this works. Hi, everyone, thank you so much for coming. I’m Tyna Eloundou, I’ll be speaking to you today about making language models useful. A bit about myself, let’s see, wow, I’m also a former scholar. I can’t make the claim to third generation because Alethea was not my mentor, but they were super helpful in making my experience here amazing. And part of that culture and that welcoming environment was a reason I chose to stay on after the scholars program [now the Residency program].
Tyna Eloundou: Today we’re going to be talking about language models and by language model, I mean any model that has language as input and output. So that could mean GPT-3, CODE-X, or BigScience’s Bloom, what have you. Okay, this is going to be the only equation you see throughout this talk and it’s really not that important, but I think it gives us some context as to where we’re going.
Tyna Eloundou: Looking back at this, this is the training objective for GPT-3 and for all GPT like models. Given a corpus of tokens, right? We define the objective to maximize this likelihood, L, which is defined as a conditional log probability over a sequence of tokens that is modeled by a neural network with parameters data that is trained by gradient descent. Now you can forget everything I just said.
Tyna Eloundou: Essentially this optimization produces these models that are trained to predict tokens, but that in itself may not be that useful on its own. I don’t think I’m giving away any secret sauce by revealing this equation to you, but it is remarkable that somehow we go from this to models that can produce, oh sorry, that can do that, right? Write prose, write code or parse data and so on.
Tyna Eloundou: I’d like to talk a bit about the notion of usefulness itself. One way to think about whether language models are useful in the first place is in the pragmatic sense. In the ideal scenario, we would be able to succinctly communicate our goals and preferences to a language agent without having to laboriously explain and list what to do and what not to do.
Tyna Eloundou: How did we initially get usefulness out of language models? When these models were first being developed in research labs, some researchers came with some ideas about how to really get them to do what it is that you want them to do. And these are two of the most prominent ones. One was few shot prompting, which is a method by which you really tell the model what the task is and before putting it on the spot, so to speak, you give it some examples of what you like to do, some demonstrations, right? For translate English to French, you could have a pen to [foreign language], I’m hungry to [foreign language], et cetera. And the translation that you actually want, you say, I would like to eat ice cream and hopefully with that same formatting you get the model to translate to French.
Tyna Eloundou: The other method is supervised fine tuning, which involves essentially just having examples for the model and then kicking off another round of training so the model can become hyper focused on your task and hopefully improve its performance on that task. So as many of you probably know, OpenAI has since then adapted this iterative deployment approach, which helps us put models in the hands of real people and understand how they interact with them. At the time of GPT-3 release, it was doing great by research standards, right? And unfortunately a lot of these research metrics are designed around these methods that we’d spoke about before, which are to prompt with few shot prompting or perhaps to do supervised fine tuning. Once we deployed, we really quickly learned that people don’t like prompt engineering. In fact, they don’t really like to do a lot to communicate their goals to the model, which is fine. It’s a feature, not a bug.
Tyna Eloundou: At its most helpful, a language agent can infer what we want without lots of specification and carry out those inferred goals effectively and efficiently. Unlike researchers, people were using natural language instructions to ask GPT-3 for what they wanted. But because of the training objective that we saw previously, the model was really tempted to just pattern match, right? If you gave it a prompt of write a short poem about a wise frog, it would very helpfully give you similar types of prompts instead of following your intent. This spurred a research effort within our alignment team to teach the models how to follow direct instructions. They did this using two insights. The first is borrowing from the supervised fine tuning or supervised learning literature where you can train the model based on examples or demonstrations, right?
Tyna Eloundou: You have a prompt and you tell them what you would ideally like it to do. And the second insight came from the reinforcement learning literature where you have some humans compare outputs. And so this model learns to generate, that model learns to compare, right? That model learns to tell this is good, this is bad. And so now with these two things, you can kick off this joint training process where you have a model that’s generating and then a model that’s critiquing, and this is good, this is not so good.
Tyna Eloundou: Over the course of training, the model learns to get better at pursuing this objective, which is no longer the pure language model laying objective and now it’s the instruction following objective. So the resulting model was InstructGPT, which is presented here. Well, yeah, you can see the output. It’s a poem, it’s about a frog, mentions wisdom, and it’s pretty short. I feel like all the requirements were met for following instructions there.
Tyna Eloundou: This was a plot that was quite striking to me. This is one of the main results from the InstructGPT paper. When I first saw this, it didn’t make a ton of sense until I really understood the research behind it. But I think that you can think of the Y axis as a proxy for usefulness and the X axis. We have model size and conventional wisdom has it that… We’re at OpenAI as you scale things, things get in general better. But you can see that even at its smaller size, right here, if you can’t see it’s 1.5 billion parameters, even at its smallest size InstructGPT was deemed to be more useful than any permutation of the base GPT model. So I started this discussion by talking about how research based approaches were not pushing far enough in terms of getting us usefulness out of these models. There’s now this emerging literature focused on helping models be more effective in tasks.
Tyna Eloundou: Broadly speaking, this literature involves having models break big problems up into smaller problems or things step by step before coming up with a final answer. And this does not need to be at odds with our human alignment driven research. In fact, right here you see a result by Kojima et al. and although their results are great overall across the board, we do see that they make the Instruct models even greater. There’s such a huge gap, a huge gain that we see with the Instruct series of models.
Tyna Eloundou: I would like to conclude by thinking about the next steps in this line of research. We know that there can be some instructions that can be malicious or exploitative or deceptive. If language models were to pursue usefulness at all costs, they might become dangerous in the pursuit of dangerous instructions or dangerous intent. Could there be other objectives we pursue along with usefulness that get us helpful but not dangerous models, perhaps kindness or hopefulness?
Tyna Eloundou: And lastly, with instructions, we’re mainly in the driver’s seat and we initiate interactions. As language models become smarter, perhaps kinder, more capable, it may be appropriate to think of them as collaborators and they may be capable of initiating ideation, creation among other things. What are the different modes of interaction we would like to have with these models? Would we want them to advise us? Would we want them to inspire us? Perhaps at Girl Geek X 2042, it’ll be a language model presenting about something new. Thank you.
Elena Chatziathanasiadou: Thank you so much all for joining. I guess with that note, I did want to mention that we’ll kick off mingling time and dessert in the area that we were before and our speakers will be available for you to ask them questions. We also have some of our recruiting team members here tonight. If you all want to come up to the front to just quickly introduce yourself or just say hi so that people can see you and then you can all come find us.
Elena Chatziathanasiadou: As I mentioned in the beginning, I’m Elena, I’m also hiring for the Residency program, so come talk to me, come find me. And then we also have some demo stands of our Dolly product and also our GPT-3, if you want to check them out. Jessica and Natalie will be doing those demos. So yeah, go find them as well.
Elena Chatziathanasiadou: Thank you all for being here. I hope you enjoyed it. Thank you to our lovely speakers and to Girl Geek X, to Cory and to all of our ops team and everyone who helped put this together and let’s go enjoy some dessert!
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Our goal is to support students at Coliseum College Prep Academy in Oakland, California serving grades 6-12 with a computer science pathway, by providing access to volunteers and role models from the professional community for students. We partnered with Oakland Education Fund, a nonprofit coordinating volunteer activities with public schools in Oakland.
Women at New Relic discuss observability, metrics, monitoring, community, APIs, React, and leadership at the New Relic Girl Geek X event with over 190 girl geeks joining the lightning talks and leadership panel discussion online.
Leadership Panel – Ariane Evans, DEI Manager at New Relic, Nada Da Veiga, GVP, Technical Solutions Sales at New Relic, Erin Dieterich, Senior Director, Social Impact & ESG at New Relic, Kim Camacho,Director, DE&I at New Relic, Tracy Ravenscraft, Director, Technical Account Management at New Relic, Stefanie Smith, Senior Manager, Talent Acquisition at New Relic – watch the panel or read their words
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Transcript of Girl Geek X New Relic – Lightning Talks:
Angie Chang: We’re going to give people a chance to join us, but in the meantime, I guess I’ll start with some introductions. Hi. My name is Angie Chang. My pronouns are she, her, hers. I wanted to say, thank you so much for joining us for our Girl Geek X New Relic event. I want to encourage us to connect with each other. If you can, I would invite you to put in the chat, your name, your location, your job title company, and your LinkedIn URL, so we can all get connected. Feel free you to connect with me. I wanted to introduce myself and give you some background as to what Girl Geek Dinners is about.
Angie Chang: I started Girl Geek Dinners in San Francisco when I started working in engineering, and I felt a bit lonely on the team as the only female engineer. And I go to all these tech events, but I wanted to go to tech events where the gender ratio was flipped. These didn’t exist in 2008. I decided to start my own series of Girl Geek Dinners. It turns out, after five days of posting about online, we had over 400 girl geeks that were interested in joining us for our first Girl Geek Dinner. And then the next one was sponsored by Facebook. And then we just snowball from there.
Angie Chang: And now today we have over 200… I think we’re at 300 Girl Geek events. We’ve also started things like a virtual conference every year, celebrating international women’s day. We really have also filled out our product portfolio of this podcast. You can go on YouTube. All the talks that you will hear today will also be on our YouTube channel. I invite you to subscribe to that. It’s at youtube.com/girlgeekx. And you can find all the videos from our previous events, and today’s event on there as well. [inaudible] chatting.
Angie Chang: I wanted to share how much I love learning from going to all these events over the years, because from listening to the women working in the diverse corners of male dominated industries, from engineering to sales, we have heard from people share their expertise. And we also learned things like, that job titles are constantly evolving. I remember thinking that this was a really interesting part of engineering and tech that we often don’t think about, of the first thought of big tech or tech companies.
Angie Chang: When I used to work at Hackbright Academy, a coding bootcamp for women, there was some women that I met at New Relic who were sales engineering leaders. And I thought they were so cool, because they not only knew engineering, but they were also very savvy on the business side. It’s because of the sales stuff. I remember thinking that this was a really interesting part of engineering and tech that we often don’t think about, of the first thought of big tech or tech companies.
Angie Chang: The sales engineering side is overlooked. I’m glad that we have heard from people like Tracy, and all the solutions consultants and technical account managers, who are interested in sharing the projects they’ve been working on and their passion for technology, today with us. We are excited to partner with New Relic, a company leading in full stack observability. We’ll hear from the solutions consultants. And they’re formerly called solutions engineers, sales engineers, and technical account managers. I think what I’ve learned is that solutions consultants are pre-sales, and technical account managers are post-sales, but that’s something that you can have a conversation with people about afterwards in networking.
Angie Chang: These lightning talks will be discussing observability metrics, ReactGraphQL for APIs and more.
Angie Chang: Now our first speaker on customer facing technical roles at New Relic is Padmaja Gohil. Padmaja is a senior solutions consultant at New Relic, and loves being a sales engineer, because it not only helps her stay at the cutting edge of technology, and she gets to work with a multitude of customers using these technologies. In her free time, she loves listening to music and adventure parks. Welcome, Padmaja.
Padmaja Gohil: Thank you, Angie. Hey, everyone. Very nice to you. Is everyone able to see my screen? Angie, can you just give me a thumbs up?
Angie Chang: Yay.
New Relic solutions engineer Padmaja Gohil talks about observability in software development, the phases of observability, and observability as code at Girl Geek X New Relic virtual event. (Watch the talk)
Padmaja Gohil: Okay. Awesome. I’m Padmaja Gohil. I’m currently a senior solutions engineer with New relic. Today we’re going to be talking about all things observability. Quick disclaimer, please not hold me accountable to any sort of overlooking statements. Before we dive into the presentation itself, I would like to give you guys a quick glimpse into my journey so far. Growing up, I’ve always wanted to be an engineer, but once I started my engineering degree, I realized that my interest lay somewhere at that nexus of tech and business, which led me to do my masters in engineering management, where I studied business concepts, but focused in high tech industry. I’ve also previously dabbled in consulting, venture capital and data privacy.
Padmaja Gohil: I’ve been a solutions engineer with New Relic for the last three years. I absolutely love what I do. New Relic is an observability platform, and because of which I’m going to be talking about observability today. But at the same time, in my day to day, I get to work with a lot of different customers. Understand how they’re using technology, and I help them achieve their goals using New Relic. If you guys have any questions about what solutions consulting, solutions engineering, sales engineering is all about, feel free to reach out to me on LinkedIn or my email address, and I would love to have a chat. The way I’ve structured the presentation today is, we’re going to talk about what were the changes that we saw in the software development space, that led to observability. Why do we really need it? What it is, and the different phases in which you can implement it.
Padmaja Gohil: And finally, we’re going to touch very quickly on observability as code. We’re going to be covering a lot of ground. Again, feel free to get in touch with me if you have more questions, or if you would like to learn more. Now let’s take a look at how has the model software industry evolved. If you look at the left, on the left side of this screen, you’re looking at our past. Our past was primarily Monoliths. They were stood up on on-premise servers. Usually scaled vertically, very static operations based scenario. We would release once or twice a year. I still remember the days when we would have to manually update our softwares. Now, fast forward to today. Today’s architectures are more microservice based. They’re open sourced. They’re more complicated. They’re usually hosted on Kubernetes cluster.
Padmaja Gohil: We went from releasing once or twice a year, to releasing maybe multiple times a day. This has been great in terms of the business. We’re able to push out new code, push out your releases and update our software faster, but it on-boards with it a level of complexity when it comes to troubleshooting, detecting issues and finding resolutions for it. This alongside other reasons is why we need observability. In the days of mainframes and static operations, when things went wrong, what would happen is, we would have maybe a couple of dashboards, that we would get alerted on. Usually these dashboards were static. We had run books for all of them, to figure out what’s going wrong and to fix issues. Now, typically, these systems would fail in the same manner over and over and again and again.
Padmaja Gohil: It was a little more simplistic than maybe today. Now, today if things were to go wrong, I’d be staring at my screen, wondering what’s going wrong. Is my cloud provider seeing an outage? Is someone deploying code? Is that the reason why I’m seeing some sort of an issue. Or I could be staring at the symptoms and not the root cause. There is so many ways in which things could break, that it’s really hard and complicated in how we do troubleshooting today. Also there has been an increased frequency of CodeDeploys. We went from once or twice a year, to multiple times a day, which can increase the chances of things going wrong. We no longer have discrete application owners.
Padmaja Gohil: We have distributed systems, but at the same time, we also have distributed teams working on things. There is a need for contextualized data in case of… if a person were to just come in blind, not knowing the history of the systems, they can quickly take a look at things and start fixing. These are just some of the reasons why we need observability today. But let’s take a look at what the definition is. There are a lot of definitions out there. The way I like to think about it is, how well do you understand your system from the work it does? It enables you to do a lot of things. For example, it enables you to collect and alert on the telemetry data types. There’s four telemetry data types, and these are the pillars of observability.
Padmaja Gohil: I’ll speak to those further in the presentation as well, but it’s metrics, events, logs, and traces. These are the four pillars of observability. Observability allows you to focus on your day to day. As software engineers, your job is to, let’s say, deploy code faster, come out with newer features. Your job is not to spend a lot of time in fixing issues. Observability also allows you to focus on that. It enables you to troubleshoot faster. It makes sure that you are ensuring up time and performance while you push out this newer code. It also gives you the confidence to push out new code, because let’s say if things were to go wrong when you were deploying, you have the confidence that yes, I have the system in place to fix those things. It builds that culture of innovation as well.
Padmaja Gohil: In real life scenario, there are so many different ways in which you can implement observability, but there are three phases, three broad phases in how we implement it. I would like to talk to you about it. The first phase is the reactive phase. All of us might have heard the saying that you cannot improve what you cannot measure. The first phase is where you start instrumenting your entire tech stack to collect data. You’re collecting metrics, events, logs and traces from all of the tech stack. You are then understanding how your applications are behaving. A lot of times you might not know what normal looks like for your applications. What does your normal response time look like? What does the normal error rate look like? The first phase is when you are establishing the normals and the baselines, and then you’re setting up foundational alerts on it.
Padmaja Gohil: That’s what the first phase is about. The second phase is now codifying your team’s work. Now, when I say that, what I mean is, you are setting up service level objectives for your application, because what happens is you’re seeing plethora signals coming at you. And you now need to understand how do you measure the success of your application? One of the ways to do that is by setting up service level objectives, and service level indicators, which are SLIs. Let me give you an example of what an SLO can look like. For a web application, an SLO could be that the videos should start playing within the two seconds, and 499% of the time during that one week period. That is your SLO. Now, the service level indicator, which is the SLI, measures the proportion of videos on the website that start playing in less than two seconds.
Padmaja Gohil: You start setting up these kinds of SLOs, SLIs. You measure them over time in the second phase. Now, lastly, the data driven phase. The ultimate aim of observability is to help teams within a company make data driven decisions. You are doing a lot of trend analysis of the SLOs and the SLIs that you set up. But at the same time, you’re evangelizing this to the teams beyond, let’s say, site reliability, DevOps, or application engineers. You’re pulling in folks from, let’s say, customer support, product. Everyone’s looking at the same data, and you’re making decisions. Eventually, you want to get to a stage where you can figure out, how is it that your digital operations are impacting business KPI. For example, if you were an eCommerce website, if the page load of that eCommerce website increases by, let’s say, 10%, are you seeing a drop in the number of users on the website?
Padmaja Gohil: Are you seeing lesser number of things in your card? These are the kinds of relationships you want to start visualizing and measuring. That’s the last phase of observability. One of the things of last phase, is also being able to automate processes. That’s where observability as code comes into the picture. Now, observability as code can again, mean a lot of things. It could mean that the way you are interacting with your observability platform, you’re automating it, but it can also mean Gitops, config as code, infrastructure as code, CICD. Whenever you hear these things, know that these are observability as code. Now, what we’re doing essentially here is that we’re taking some of the best practices from software development, and we are applying it to the operations world. Think reproducible builds, reproducible deployments.
Padmaja Gohil: You are automating processes, you are testing them. And you’re making sure that no matter how many times you run these processes, you’re getting the same result. There are a few things common as a part of observability as code. Firstly, observability as code, it’s literally code. So it does not have a UI. It is declarative. So you are specifying the exact state in which it should exist. For example, if you write a piece of code to create an alert in New Relic, you should be able to take that same code or a template, and then modify it slightly to create a thousand alerts. It’s also reproducible. You are reducing the amount of time you’re spending in managing your observability systems as well. The first thing is it’s declarative. Secondly, it’s versioned and immutable. Ideally, it should not reside in a shared drive.
Padmaja Gohil: Ideally, you should be using a get for it. You should be able to go back and figure out what were the changes made if things were going wrong. It should be versioned and immutable. And lastly, it’s pulled and reconciled automatically. Now, what I mean by this is that if you had created a dashboard in New Relic or in any other observability system, and let’s say one of your colleague comes to you and says that this is a great dashboard. I want to use it for my own needs. They can go ahead, take the dashboard, and maybe they modify it. Then you go into New Relic and you figure out that your dashboard is modified, and you won’t actually revert the changes. You can directly take the code, apply it, and you can get your original dashboard.
Padmaja Gohil: And now you can take the template that you used, or the code that you used, and you can give it your colleague, and they can use it to create their own dashboard. It’s usually pulled and reconcile automatically. There are a lot of solutions available for observability as code. I’ve mentioned some of these here. We also have our own templates for, let’s say, Terraform, in case if you guys are interested. Feel free to look at it in our docs page. But these are just some of the solutions that you can use to implement observability as code. This brings me to an end of my presentation. I know that we covered a lot of cloud. In case if you guys are interested in knowing more, feel free to reach out to me on LinkedIn or my email address. Thank you so much. I very much enjoyed speaking here.
Angie Chang: Thank you, Padmaja. That was really great. And thank you for leaving an email address so people can reach out to you with any questions. moving on to our next speaker. Kate is a lead principal technical account manager at New Relic. She comes with a background in helping customers thrive in their business with the latest software monitoring tools. In her current role, she partners with customers to help them with their full stack observability requirements. So welcome, Kate.
Kate Kordnejad: Hey, Angie. Hi, everyone. Thank you for hosting us. Give me a second to share my screen, and put it in slide mode. All right. I’ll be talking about customer success and value realization through value metrics. I’m just going to jump into a little bit of legal disclaimer, so don’t make any financial decisions based on our discussions today, and or any statements we make, and some proprietary copyright information. All right.
New Relic principal technical account manager Kate Kordnejad talks about the evolution of maturity, TAM goals, maturity journey, maturity metrics & more at Girl Geek X New Relic virtual event. (Watch the talk)
Kate Kordnejad: A little bit about me. My name is Kate, and I’m a principal technical account manager here at New Relic. As TAMs, we are an extension to our customers teams. We help them with their full stack observability requirements. We want to make sure they see value, and we basically help them get enabled, follow best practices. We work as a trusted advisor with them. A little data point about me; I love working out. I love yoga, especially Bikram yoga. I love to travel, and I’m a data nerd.
Kate Kordnejad: Okay. Our agenda for today is going to be evolution of maturity, goals for technical account management, our maturity journey, defining maturity metrics, and how can you define maturity in your organization? All right. Starting off with evolution of maturity. In our evolution and journey, we found ourselves improving efficiency from four to five hours to one minute by automating our solution. I’m going to explain how we did this. As things evolved over time, we found our defined metrics to be meaningful. And we did find out more about our customer’s maturity, and how we can help them improve stickiness. For example, are they using custom attributes, or do they have data instrumented for more visibility? With our help, they started getting more mature within the platform. And we were able to identify the gaps, improve upon them. We did soon realize to deliver an observability platform value for our customers.
Kate Kordnejad: We needed to recognize value drivers and use cases, that actually deliver those business outcomes for each and every customer. For example, to improve customer experience, quadrant you see on the left hand side. We had to understand our customer’s business needs. Card abandonment, any association with an operational gap like card crash rates, were stuff that we needed to figure out. We identified the steps to maturity, is basically summarized in alignment. What that means is we need to align customer priorities to the observability value drivers. And agree on prescribed observability use cases, and then enable based on an agreed upon description work streams with the customer, and then finally, value realization. Reflecting on the business and the operational KPIs that we agreed upon during and prior to going through maturity. We actually evolve from just collecting metrics to quantifying metrics into meaningful business values, with a growth mindset, of course. We realize without having a continuous growth mindset, we won’t be able to evolve and improve our solution.
Kate Kordnejad: Our next thing is the goals that are for technical account management. Having an involved automated way to quantify metrics into business values, provides us leverage as TAMs. TAMs, as in technical account managers. We now have data to analyze customer usage, to reduce overall churn, by identifying any sort of gaps we have in utilization, by providing enablement based on usage, and engage platform users and drive valuable engagement by meeting them where they’re at. And directly communicating with our customers and being a liaison internally and a voice for our customers. And essentially, we want to reach value realization with them.
Kate Kordnejad: The next I want to is our maturity journey. Our journey basically started at looking at our platform per customer account, and literally eyeballing metrics we had identified as crucial to understanding and analyzing customer data. It was really hard to assess the pieces of the product they were using by manually assessing their usage and engagement. The normal customer metrics success wasn’t really working for us anymore. For example, if they were building dashboard, this wasn’t showing us the full picture, or the reason behind it that’s looking at their user behavior. It was very one-dimensional, and we didn’t really know if they were getting value out of it. We basically had to look deeper into the metrics, and then identify and associated with value drivers.
Kate Kordnejad: How do we define maturity metrics to get to that point? As a team, we basically start asking ourselves, what results do we want to see from this? Ultimately, what does a good maturity look like? And what does it look like for each product? We needed KPIs to show actual investments. For example, if we looked at our alerting product, we wanted to drive an alerting strategy, or potentially set our customers up with anomaly detection. Next, we had to break each product into maturity metrics. Initially, this was done manually through APIs and us eyeballing accounts, but after we broke down our KPIs by product, we had to describe a desired performance level, and determine how data is interpreted. We had to set up thresholds, place and score for each one, each of the metrics that make upper and lower limits of a desired performance.
Kate Kordnejad: This basically allowed us to understand overall maturity for each customer product using a heat map, and really made maturity pop up the page for us. Now that we had our results defined, maturity metrics chosen by product, we had to basically come up with a way to automate this. Our internal teams were able to automate the process, build out an app using APIs, grab the required data from accounts, and assess maturity. Finally, the last piece of the puzzle was to ensure we documented every single steps, our definitions that are associated with each of the metrics collected for further analysis. Our document includes a breakdown of the products, the metrics associated with it, and each and every single step you need to take to improve your score. From all of this, we want to cover, how can you define maturity in your organization?
Kate Kordnejad: It really comes down to three pillars. Goals and baseline. You have to ask yourself, what does maturity look like for your organization? Describe those intended results. Do you understand the alternate measures for those intended results? Then you move on to data identification. Have you identified any composite indices as needed? And do you collect any of the data right now? Is it accessible to you? And finally, business alignment. Have you thought about targets? Thresholds? Do you have a baseline that you can work with. And then finally, have you tied your maturity metrics to business values that deliver value realization? That concludes my presentation. Thank you for having me.
Angie Chang: Thank you so much for that talk, Kate. Our next speaker is Carolina Rotstein. She is a solutions consultant at New Relic. She is also an economist and political scientist that fell in love with programming and data, and is passionate about untangling holistic customer journeys across complex stacks, which she’ll be speaking about today. So welcome, Carolina.
Carolina Rotstein: Can everybody see my screen?
Angie Chang: Perfect.
Carolina Rotstein: All right. Oh. Today we’re going to talk about browser monitoring, and how it can help us improve UX and UI. Some safe Harbor information, a bit of housekeeping, some proprietary information, and just please don’t use this to make any financial decisions.
New Relic solutions consultant Carolina Rotstein talks about improving website UX and UI with real user monitoring at Girl Geek X New Relic virtual event. (Watch the talk)
Carolina Rotstein: A bit about me. I’m a solutions consultant for New Relic in the commercial E-sales team. I’m an economist and a political scientist, but I fell in love with programming and big data. I’m passionate about untangling holistic customer journeys across complex stack, and my most previous role included optimizing UX and UI for the gaming industry. And yes, we did collect a lot of data.
Carolina Rotstein: Today’s agenda, we’re going to focus on improving the website’s UX and UI, and using real user monitoring for this. Also we’ll cover why we should focus on UX and UI optimizations, and some of the metrics that we can use to do this as well as the metrics that come out of the box for New Relic and some other tools. And then an approach towards optimizing customer experience, including UX and UI, the traditional way and the enhanced way using big data. My peers talked a bit about observability maturity. At New Relic, we focus on data driven decisions. We want to have an approach with this framework towards taking data driven decisions.
Carolina Rotstein: Now, in this part, customer experience is closer to product and support. While it does have a lot of positive impact into how customer support, user product, and just impact on their KPIs. It’s mostly geared around design and product and development. Experience optimization, and a big portion of that is user experience. And also user interface optimizations are closer to the revenue. Even though it’s at the bottom of the the funnel, any impact that we might have into optimizing the experience, will have a monetary increase for the companies that we’re in.
Carolina Rotstein: First I’d like to talk about browser versus synthetics. We talk a lot about the jungle versus the lab. The jungle would be empirical data. So just every browser, every device, every location, and what your customers are using. The lab will be how we are tracking the health of the site just as we mapped it. We eliminate all the variables to just understand performance and solve problems quickly. This is done by synthetics. The jungle piece or the real user monitoring is once we deploy that application into the wild. So the users might take pads that we just did not foresee. For that we use browser monitoring. It’s an essential tool for user experience. It has a couple of places that we would focus on. Product usage, front end performance issues, content strategy, and in this case, websites, UX and UI.
Carolina Rotstein: I’d like to talk a bit about the metrics that we can use for this. This are not all the metrics, but I strongly recommend this as a start. Just for front end performance and monitoring, we have core web vitals, the user time on site. And it’s just user centric health metrics, such as throughput chart. For New Relic, we can divide what the time that it takes to load is split by the front end versus the back end. But for product usage, which is getting closer to that UX and UI, we track that funnel.
Carolina Rotstein: These are those conversion funnel related metrics that map to the business. These are unique to every company and every website. Those are success events, which can be form fills, video watch, purchase, and business classifications. These are custom metrics that we would map. Then we have all these audience insights such as device and location and vanity metrics. The vanity metrics normally come out of every tool, but they’re a great place to just look at your application, sort of like the Canary in a coal mine. Then for content strategy, we can see how users are navigating through the site, such as in the metrics that we would use, are link positions, most popular, previous page, the next page. And we also have pages report, such as the most popular pages, the time spent on site, how long it takes to load an assets. But we also can track audience insights. This can come from your BI data.
Carolina Rotstein: New Relic can take just any sort of data, but with other tools, you can certainly integrate it. This can be things that are a bit more robust, such as persona development, even the VIP level of your users, or the user IDs. And then very targeted towards UX and UI, and specific real user monitoring. We have the time spent on task, which will be the time before a user completes a success event. The ease to perform a task, rage clicks, which is just a user frantically clicking. Marketing funnels is a good one. In New Relic, we have something called the apex score, which is just taking into account the error rate and the load time to proxy some of the survey based customer satisfaction, traditional UX and UI metrics.
Carolina Rotstein: Now, very related to UI in just design, we have AB test, popular device sizes, screen size, and size orientation and night mode. Those are a few ones in there. Finally, I would like to show you what comes out of the box of New Relic. This is browser monitoring. We have those core web vitals that user spent on site, initial page load throughput, and some other additional charts. This comes just out of loading a user agent. It’s as simple as adding a marketing tag, and this dashboards just magically appear. But if we go back to talking about UX and UI, why is this important? It’s like 68%. I’m talking here about eCommerce just because it’s the easiest, cleanest use case to see revenue changes when you deploy UX and UI changes.
Carolina Rotstein: We can see that a lot of eCommerce… 68% of them just had performance issues. And that translates into 40% of those issues resulting in revenue lost. For instance, most eCommerce retailers have reported that they would like to have a response time below two seconds. 90% of that website response time, on average occurs due to do the loading of front end resources. This is why it’s so important to start your optimization with your front end as well. Some core customer experience questions that real user monitoring will help you solve is, for instance, if your website is easy and friendly to use, that would be through the balance rate, for instance. Whether it’s easy to navigate or not, and that will be through the number of pages that your users take to get somewhere.
Carolina Rotstein: Ideally, you want to slice all those additional page views, just because you want a seamless interaction with your site. Just think about loading a YouTube video and having to click 20 times before you go to the music that you want. For instance, how easy it is to get in touch with a customer agent for your user. Now, not all sites want you to immediately get those agents, but those are done through custom events. You track that chat click, or that phone call on a mobile browser, as a success event. And just how comfortable your visitors are after landing in your site, can be done through a number of other metrics that we track on browser. Why is this important as well? Just to do it via RAM, is because it’s big data and data driven.
Carolina Rotstein: So if we look at traditional UX and UI optimization, it’s done through user research, such as interviews, focus groups, usability testing. And they would put a couple of people to see how easy it is to finish a task on their site, through surveys, AB testing sometimes, and session recording. Now, the size of this data tends to be, from my experience, to 100 people, to a thousand people. When we’re talking about big data, it’s millions of people. It helps us prioritize and not get narrow focus on the people for which we’re auto… well, for which we’re optimizing too. That is done to AV testing. Some companies that are very developed, they do multi-variate testing. So they have several versions of the same design, such as… Netflix is one of the big guys at the same time. They’re just running algorithms while they’re doing that.
Angie Chang: Thank you, Carolina. Our next speaker is Solmaira. She’s a technical account manager at New Relic, based out of Atlanta, serving as a technical advisor for enterprise customers in Latin America. She currently serves as chair of the Relics of Color ERG, which she’ll be speaking about today. Welcome, Solmaira.
New Relic technical account manager Solmaira Flores-Valadez talks about finding community with New Relic ERGs at Girl Geek X New Relic virtual event. (Watch the talk)
Solmaira Flores-Valadez: Hi, everyone. My name is Solmaira Flores-Valadez, and I’m a technical account manager at New Relic. I’ve been with New Relic for about over two and a half years. I serve as pretty much like a technical advisor to some of our larger enterprise customers within the Latin America region. I’m like a post sales resource to them, helping them get the most out of New Relic, and also providing trainings, things like that, to make sure that they are utilizing New Relics to the best of their abilities. Today I’m going to talk about diversity, equity and inclusion, and the part that it plays in my life. How I was able to find a community with New Relic ERGs, which are employee resource groups.
Solmaira Flores-Valadez: A little bit about me. My pronouns are she, her, hers. I live in Atlanta. I went to the University of Georgia. I am a first generation Latina. Mexican-American. First person in my family to go to college. I am a woman in tech, and I’m also a dog mom. First I wanted to start off with a few definitions around what diversity, equity, inclusion are. And then I’ll jump in and talk a little bit more about what it means to me, how I got involved, and all of that. Diversity is the presence of differences that may include race, gender, religion, sexual orientation, ethnicity, nationality, socioeconomic status, language, disability, age, religious commitment or political perspective. These populations have been and remain underrepresented within the broader society, and within practitioners in the field as well, within the workplace.
Solmaira Flores-Valadez: Equity is promoting justice, impartiality and fairness within the procedures, processes and distribution of resources by institutions or systems. Equity is really the approach to ensure that everybody has access to the same opportunity. In the context of the workplace, how is it that employees have access to the same levels of attraction, promotion and retention within the company?
Solmaira Flores-Valadez: And then lastly we have inclusion, which is an outcome to ensure that those that are from these diverse backgrounds actually feel and or are welcome. It pretty much boils down to people with different identities, feeling or being valued. All right. So why is it important to me? I actually started doing D and I type of work long before I even knew it was D and I work. As I mentioned, I went to the University of Georgia. UGA is a predominantly white institution. So there was very little people that looked like me. I was always looking for my community, people that looked like me, that share common backgrounds, but then at the same time, got involved with certain organizations such as Students for Latino Empowerment, that not only helped build that community, gave me that social aspect in college, but also we were doing D and I work [inaudible] students into the campus. We have various events throughout the year, where we would pretty much show them the ropes, let them know, if I can do it, you can do it. They’ll be able to tour campus.
Solmaira Flores-Valadez: We would give them workshops around financial aid, how to get started, the college journey and all of that. That’s what sparked I guess that interest in being involved within D and I type of efforts. As I have here, it’s important to me, for me to lift as I climb to, to be that change that I wish to see in the world. And also D and I has been very important in my life, not only because I’m able to in a way give back, but also it’s helped me in my professional, in my personal growth. Being able to develop certain leadership skills.
Solmaira Flores-Valadez: A little bit about how I got involved at New Relic. I was involved in college with those type of organizations. As I left college and I moved on to my professional career, my first job I worked at a big accounting firm. I got there similar to when I joined UGA. Not a lot of people that looked like me. They had an Hispanic network. I joined that. We did a lot of social events, but at the same time, we also did a lot of also lifting as you climb, bringing in students. We also did events for students and things like that. I loved being plugged in, being that person, going to recruiting events and seeing others like me, and then being able to see that they could also do it.
Solmaira Flores-Valadez: And then after that I switched careers. I came over to New Relic. As soon as I joined, I let my manager know that I was interested in still being a part of something like this. I asked if he had a Hispanic network. He told me he wasn’t familiar if there was an Hispanic network, but there were employee resource groups, and got me connected to the person that was the D and I manager at the time. Met with her. I talked about my experiences. She got me connected to the Relics of Color, which is the employee resource groups for our POC at New Relic. I got to meet them. Loved what they were doing, got to participate in some of their events. When I joined, it was right around Hispanic heritage month.
Solmaira Flores-Valadez: I asked if there was anything I could help with. At the moment, the Atlanta office was pretty new. There wasn’t a lot of representation there. I told them that I wanted to host an event. I was brand new. I didn’t know how people were going to take it, but I knew that I wanted to do this. Since it was a little bit later in the Hispanic heritage timeframe, I decided to give a twist. We did a day of the dead event, which I have some pictures here. I put this together. We painted skulls. And then we watched Coco, and we also ordered Tamales and we had a really good time doing this event. That was pretty much my golden ticket into not only being a member of the Relics of Color, but becoming an executive, one of the ROC execs. After I did that, the leader of the Relics of Color reached out and was like, “Oh, I want you to be part of the exec board.” And then that’s how I got plugged in my golden ticket.
Solmaira Flores-Valadez: I loved it ever since. I’ve been able to help with a lot. Currently, I’m one of the co-chairs of the Relics of Color here. Have our exec board of our offsite that we had earlier this year, where we got together to build out our strategy, the events. We host and we celebrate different events throughout the year, black history month, Hispanic heritage month, Asian Pacific Islander month. Putting together content around that. And then on the right hand side, that was us at the sales kickoff. We managed to get people together before 8:00 AM or at breakfast. It was great. It was intimate. We had our relative color sponsor. Tracy Williams, she’s our chief diversity equity and inclusion officer, as well as our chief people officer there. Being part of the Relics of Color, being part of the exec board, has been, like I said, great.
Solmaira Flores-Valadez: I’ve been able to learn a lot, gain exposure to different things. For example, we meet with our C-suite on a quarterly basis. Being able to have visibility into the C-suite. And not only that, but be able to represent the Relics of Color as a whole, be able to communicate some of our challenges, what we’re doing, where we want to get, our goals. And then listening to us and our needs and seeing where and what they can do to help. It’s been great. On the social aspect, I’ve made really good friends, but also helped me grow professionally.
Solmaira Flores-Valadez: I talked about the Relics of Color, but we do have other employee resource groups. We have the women at New Relic. We have the veterans at New Relic, and we have access at New Relic, which encompasses neurodiversity, mental health and disability. Relics of Color, which is the ERG that I’m a part of. And then we also have our Rainbow Relics for LGBTQ plus relics. These are the different ERGs that you can get involved with. At Relic, we are working towards a more perfect New Relic. These are some of the initiatives that we have going on. We definitely believe that inclusion means everyone. We want to make sure that we’re having some progress. We understand that there isn’t always… or there is always more work that needs to be done, but we do value the progress over the perfection.
Solmaira Flores-Valadez: These are some of the initiatives that we have at New Relic. To help us accomplish that we have, for example, the Mikey rule, named in honor of our departed team, VP of engineering, who was the executive sponsor of our first employee resource group, which was the Relics of Color. This Mikey role focuses on sourcing and hiring more relics from underrepresented groups. Whenever we have an opening, this Mikey rule kicks in. We also have leader-led action plans. These were started in 2020 by our founder, Lew Cirne. He challenged the company to level up with D and I leader-led action thoughts, and maximize the recruitment retention and career growth for underrepresented groups. And now it’s one of the top level organizational priorities across the board for every single part of our business.
Solmaira Flores-Valadez: We also have D and I working groups. Our company leaders, like I said, sit down with us, with the ERG executives, to ensure that our commitment to diversity, equity, inclusion, is put into practice around the globe. Just wanted to call out some of our progress that we’ve seen with these initiatives. We’ve definitely increased our BIPOC engagement. We’ve also helped reduce bias. There’s different trainings that our managers have to take, every year or every so often around bias. We’ve also reached pay equity. There was an analysis that was made a couple years ago, that took a look at the pay, and made sure that everyone’s pay was equal. There’s been a lot of progress lately around career mobility, where we’ve built a lot of mentorship groups throughout the different businesses, to be able to help the career mobility of our underrepresented groups. And then as I mentioned, you also have the Mikey role, which focuses on the recruiting efforts. All right. Well, that’s all that I have for today. Thank you all for joining. Have a great day.
Angie Chang: Our next speaker is Nora, who is a solutions engineer at New Relic, where she advises enterprise clients on their observability engineering practices to answer the what, how and why of system performance. Her research focuses on application of blockchain, and she speaks Portuguese, Spanish and French, and resides in Florida. So welcome, Nora.
Nora Shannon Johnson: Hi, everybody. How are you all doing? Well, I’ll assume you’re doing fine because I can’t hear you, but can you see my screen?
Angie Chang: Yes.
Nora Shannon Johnson: Awesome. Okay. Cool. Like everyone else said, I don’t know enough to make any predictions that you guys could invest in, but anyway, welcome.
New Relic solutions consultant Nora Shannon Johnson talks about observability in the age of web3 at Girl Geek X New Relic virtual event. (Watch the talk)
Nora Shannon Johnson: Today I’m going to talk about observability in the context of Web3. A little bit about me. Like Angie said, my name is Nora Shannon Johnson. I’m a solutions consultant, which basically means that I help customers answer the what, how and why of system performance. Outside of work, I love languages and linguistics. I love planting things, but everything I’ve ever planted has died, unfortunately. So still working on that. And skateboarding. Today we’re going to talk about applying the principles of observability to Web3, and what the specifics of monitoring blockchain technologies looks like. I took an interest in this because I work with a lot of financial services and eCommerce organizations in Latin America.
Nora Shannon Johnson: The integration of blockchain into their existing business operations is a big question for them right now, for reasons that I’ll get into in a few minutes. This is not New Relic’s main use case, but as a solutions consultant, a lot of times customers come to you with their data, their technology, and their business requirements, and say, “Make it work.” Which is my favorite part of the job, when somebody says, “How do you do this?” And I say, “I don’t know. Let’s figure it out together.” So this is an example of doing that. Over the next nine or 10 minutes, we’re going to talk at a very, very high level about what Web3 is, why we would care about monitoring it, what specifically we would be monitoring what we want to look at. And then a quick example of what that might look like.
Nora Shannon Johnson: To get started, what is Web3? Web3 is the name given to the idea, and idea is a very keyword here, of a new sort of internet that is built using decentralized blockchains. As a disclaimer, throughout this entire presentation, I’m describing the idea, not the reality of what may come to fruition. Again, Web3 is powered by the concept of… or by blockchain technology. Blockchain is a relatively new method of storing data online. It’s built around two core concepts, those being decentralized computing and encryption. The fact that it is decentralized, means that files or data is shared across many computers or servers, rather than centralized in a single server or group of servers. You might hear it referred to as a peer to peer network for that reason. The fact that it’s decentralized also means that it’s immutable. You can’t change data on the blockchain, because in order to do so, you would’ve to corrupt data on every single machine that’s participating in the network, which is just really not feasible when you’re looking at large scale blockchain like Ethereum, which is the example we’re going to use.
Nora Shannon Johnson: And then the fact that it’s encrypted, means that people can’t access it unless they have permission to do so, and you can give and rescind access as you choose. So why would we want to monitor Web3? Frankly, for a lot of the same reasons that we already monitor the existing technology, the web two technology, so to speak, in the same industries. for financial services, that’s eCommerce integrating with blockchain for payments. It’s important to know that this isn’t just like cryptocurrency exchanges. This is brands like Gucci and the Dallas Mavericks and Microsoft, Whole Foods, even Save the Children. They all accept one or more cryptocurrencies for payments. Across a ton of different industries, this is an important aspect of their technology stack. We’ve also got healthcare. One of the driving or the driving use case for applying blockchain technology to healthcare, is to restore the rights of data back to users or patients in this case.
Nora Shannon Johnson: You would be able to give or rescind access to your health records to a healthcare professional, organization at will. Whereas right now your test records or health records are held in a database owned by maybe some company like Quest. And you wouldn’t really necessarily be able to remove it if you wanted to. And then finally supply chain. Supply chain is arguably at the enterprise level, the most interesting use case, the most sought after use case for blockchain. Specifically the validation of providence or origin and authenticity. Using a public ledger like Ethereum, you could actually trace the roots of a product that you purchased, to ensure that it is in fact organic or fair trade, or even from a location that you believe it to be from, which is pretty interesting use case. There’s many more, but in all of these use cases, we’re talking about people’s privacy, their security, their wellbeing. Obviously, their financial assets.
Nora Shannon Johnson: The fact that data on blockchain is immutable and that it’s decentralized, doesn’t mean that it’s immune to failure or to attack. It’s simply is creating this new monitoring paradigm. What might we monitor on the blockchain? I’m not going to explain all these, but like people before may have said, the slide decks will be shared out. But you might monitor something like a decentralized application or dApp. Decentralized autonomous organization or DAO. Decentralized finance exchanges or DeFi. And then non fungible tokens, which I think everybody’s probably familiar with. The infamous apes, or NFTs. Lots of acronyms here, cause it’s a mouthful. And then of course, you can monitor the blockchain itself too. Which is the example that we’re going to look at. We’re going to look at the Ethereum blockchain. If you’re not familiar, Ethereum is, you guess it, a blockchain platform with its own cryptocurrency. Ether, shortened to ETH. It’s also got a programming language called Solidity, which you can use to write smart contracts, and decentralized applications so that you can actually interact with the blockchain.
Nora Shannon Johnson: This is by far, especially for DeFi, the most popular blockchain, but there are a lot of alternatives that are gaining popularity, things like Cardano and Solana, because they’re faster and cheaper than working with Ethereum. Monitoring a blockchain or assets that are deployed to a blockchain, is going to include both the typical metrics and data types that we’d be used to seeing, as well as some that are specific to this realm. There’s three example categories here. We’ve got system performance, security events and business metrics. If you go from left to right, this is like more familiar to less familiar. Something like system performance is something that we’re very seeing.
Nora Shannon Johnson: When Netflix is… well, all the time it’s up and running, they want to know how quickly transactions are executing, the rate of error, as well as resource utilization. The only difference being in a world of Web3, this might be the number of nodes, but very similar to what we see today in terms of the number of idle versus busy workers. For security, this is very important. We’ve all heard about many attacks made to different blockchain or cryptocurrency exchanges. Things like changes to access controls, when there’s a lot of failed login attempts from especially specific IP address or geographic location where you don’t normally have those. And then finally, unusual transaction patterns. So there being a lot of transaction outside of your normal business operating hours.
Nora Shannon Johnson: And then all the way to the right. And this is where we see things that are more specific to the use cases that I described earlier. Things like measuring the gas fee. When you interact with blockchain, you have to pay a transaction fee. And it’s a dynamic transaction fee. It changes throughout the day. We’ll take a look at that in a minute when we get into New Relic. But that’s something you want to pay attention to, because whether you are paying that or receiving that, that affects your bottom line. You’d also want to pay attention to things like, the number of active users or wallet holders, the number of active connections. And then of course, the number of minors that are mining. And then finally something like the rate of currency being paid out. As you probably know, miners mine, because they get paid for it. But there’s a lot of blockchain platforms, especially ones that are oriented towards the arts and culture, where they will actually pay you for posting your content to their platform.
Nora Shannon Johnson: They pay the content creators. You want to know, again, whether you’re paying or receiving or you’re somewhere in the middle. As an integrator, you want to know what the rate of payout is. How might we do this? We know what we want to look at. We know the importance of it, but how might we actually do that? We’re going to go through very quickly, the two pieces that fit together, and then we’re going to look at what that looks like in an observability platform. In any situation, there’s two parts to monitoring. There’s the data and the platform. There’s, how can we get it, it being the data. And then the second part is, how do we make sense of it? Because just having the data is not super helpful to anyone unless you’re a computer.
Nora Shannon Johnson: And then the second part, which is, how do we make sense of it? Using a wonderful observability platform like New Relic, we can use an API suite to pull in the metrics, events, logs and traces that are important to us. And then you can look at all of the block statistics for the last hour, week or month or whatever the case may be. Let’s take a look at what that might look like if we were actually quoting it over to… oops. [inaudible]. I didn’t need to unshare. Let me reshare. Quoting it over to an observability platform. This is a simple use case, but this is just basically a dashboard that I put together. We’ve got our cool little… I don’t own. This guy, unfortunately, no Ethereum funds for that.
Nora Shannon Johnson: But we can look at changes to the different whales. Here I’m tracking people like Snoop Dogg. He’s buying and selling all over the place. We’ve got Lindsay Lohan, Mark Cuban. We can see the top miners for the period of time. The gas price, that’s what I mentioned earlier. And then again, the black activity, which is interesting to see. Whether you are responsible, you’re part of some exchange that is leveraging the Ethereum on blockchain, or you are a decentralized application developer, or maybe you’re just somebody that is posting their content or their NFTs to a blockchain. This is all going to be relevant for you. You can also make actionable insights based on the data that you poured over. If we take a look at, for example, logs. Logs are just like step by step data coming from either applications or servers or what have you.
Nora Shannon Johnson: And in this case, it’s coming from the Ethereum blockchain. I’m going to filter on a certain block number. Let’s do this guy. We can actually see. I’m going to shut this down. We can actually see the step by step of what it looks like. We can see that transaction was requested, that the block creation was initiated. Blocks were then sent to nodes in the network. It got validated. Transaction is now complete. And then they update that to… or they send that update to the network. The block gets added to it, and the proof of work is dispersed. People get paid on it. But as we know, things do not always work as we anticipate that they will in technology. So if we take a look at this other block, we can see that in this example, the transaction is requested, the block creation is initiated. The block is sent to the nodes in the network, but then the transaction is pending validation several times.
Nora Shannon Johnson: We might do something like create an automated remediation workflow here. Based on maybe the messages, the strings in this message, or repeated data, or examples of the same messages over a long period of time, we could actually set it up such that it automatically triggers external events based on what we see in the log messages. Again, this has been a very quick and very high level example of what you might see if you wanted to monitor something on the blockchain, and how you could make use of that information in a wonderful observability platform like New Relic. I hope you enjoyed it. Thank you very much for your time, and I hope to talk to you all soon.
Angie Chang: Awesome. Thank you, Nora. So Sarah is a solutions consultant at New Relic. She loves working with data, and in a previous life was a math teacher. She uses her skills to help customers use their own data to improve their uptime, performance resilience, reliability, and customer experience. Welcome, Sarah.
Sarah Hudspeth: Okay. Hopefully you can see me and my slide.
Angie Chang: Yes.
Sarah Hudspeth: Are we good? Okay. Hi. All right. Hi, all. I’m excited to talk to you all about APIs, and getting your data when you want it and how you want it. It’s a very common theme here at New Relic. We love data. We’re data nerds. And we have a safe Harbor just for legal purposes.
New Relic solutions consultant Sarah Hudspeth talks about REST APIs and GraphQL queries at Girl Geek X New Relic virtual event. (Watch the talk)
Sarah Hudspeth: A quick bio about me. Yes, I’ve probably been in tech for three and a half, four years. Before that I was a math teacher. I taught middle school and high school math. I did attend Hackbright Academy, and so I’m a boot camp graduate. If you have any questions about that, please reach out. I’m a mom of two, plus I have a puppy, a lab mix, and then you can see the hamster in the background.
Sarah Hudspeth: I’m a huge reader. And you’ll see one of the projects I walk you through is all about books. The last book I read was Stalingrad, super interesting. The best part of my job is working with customers and helping them solve their problems. And yes, we are all about data and using data. Feel free to put stuff in the chat or follow up with me afterwards. Here are my objectives for my talk. I am still a teacher at heart. I want you guys to understand what REST APIs are, how they’re used, what is GraphQL, and what are some interesting trends in APIs today? I want you to understand the difference between your REST APIs and GraphQL APIs, and possibly articulate use cases for each. Also we’re going to be talking about GraphQLs, query and mutations.
Sarah Hudspeth: I’m just making sure you understand the difference between that. And then I am giving you all homework. After this session, if you haven’t played with API calls, go find some APIs, play with them. Go play with GraphQL, do some queries and mutations. If you need a GraphQL API Explorer, New Relic, you can sign up for a free account and play with our GraphQL API, which we call NerdGraph. Feel free to do that. APIs. API stands for application programming interface. It’s basically a way for clients and servers to talk to each other. It’s a set of protocols and it’s called a REST. I’m going to be talking about REST APIs, because those are usually the ones I would say they’re the most popular. And REST is short for RESTful, meaning stateless.
Sarah Hudspeth: The state of the client doesn’t affect the state of the server. They should be able to talk no matter what’s going on within their own environments. I like to think of API calls as programs, throwing Frisbees back and forth. Even though the Frisbee is actually data. But a client will make a call, throw the Frisbee to a server. The server gets the Frisbees if there are any instructions, and throws the Frisbee back as a response. If all goes well, you get a 200 response. If it doesn’t, you’ll get one of the four hundreds or five hundreds based on whatever the errors are. Let’s take a look at what an API call looks like. This is code from my virtual bookshelf project that I did at Hackbright. I allowed folks to build out this visual bookshelf of the books they were reading.
Sarah Hudspeth: The main API I used was Google’s Books API, where I could get a thumbnail of a picture of the cover of the book, and a lot of information. When I was feeding my database, I had a list of titles and authors, and then I made a call to the Google API, Book API, using my Google API key. I used the Python HTDP request library to get that information. And then I stored the response in a dictionary in JSON form. so that I could fill out my database with all sorts of interesting things. Hold on. I was going to say, there’s a few things. We have a URL. I have some variables in here, parameters that changed that I had to go through in a for loop.
Sarah Hudspeth: And then I also needed permission to have access to APIs. Those are the key components of an API call. This was one book. This was one response I got back from Google’s API. There’s a lot going on here. I would say, there’s a lot of information here, some of which I needed, a lot of which I didn’t need, and I had to sort through it and figure out, what is going to be helpful to me in my project, and then get rid of the rest, which if you notice, is a lot of waste. My code was not optimized. This was the slow part of my program, which if I go back, I would focus on this and try to do this in a better way, just because it ate up so much [inaudible].
Sarah Hudspeth: To summarize, I showed you what the components of the REST API and the results are. You have to have a URL. You have to call to someplace. You can send parameters on variables. I did title and author. You usually need a key to access the APIs, so you have permission to get the information. You need some HTTP requests. I use the Python’s request, but I’ll show you a cURL snippet when I do the GraphQL. The other interesting thing to note is that each API, you can call various APIs, will have their own way of formatting the data. Google Books API – just sent me everything I could possibly need about a book. And it was up to me to go through and figure out the structure of it.
Sarah Hudspeth: I showed you a get REST API call, but there are also posts where you can actually post data to the API. You can update data or you can delete it. I said, “This is kind of ugly data.” There was a nested JSON. I found out the hard way that sometimes some of the things I wanted were empty, and I had to find workarounds. I had to go and I had to clean up and structure the data. There even updates and the data would get restructured and I’d have to go back and figure out how to do that. I’m glad to now transition to a new way to get data, called GraphQL. It is also an API, but it is a very structured way we can access data. This is an example of New Relic’s NerdGraph API Explorer.
Sarah Hudspeth: And if you notice, I have to my left, a query builder with very specific key value pairs that I can build out for a query. Here I’m going to query an account and get the name and ID, and here I’m going to do an entity search. You all have been hearing us talk about observability, and learning about applications and performance and getting metrics and events. This is a way you can go in. I’m just going to get the name of things I want to monitor, the type. I’m going to get a special GUI. And then I’m just actually going to get the tags that I’ve tagged with my entity. It’ll pop up here in a very nice structured JSON. I know exactly how many levels I need to go in to get specific information. And then here’s how you could do it and build in a program.
Sarah Hudspeth: We talk about automation and observability as code. It’s really easy to take these GraphQL calls, and build in structures and processes to get the information that you can then take action on. Again, here’s just the API link. I have some headers with my key. And then here, I’m sending this query that’s going to go to the GraphQL server, and pass back all this information about this application, name, box, that’s in development. All right. Let me quickly summarize what we did or what I just showed you with GraphQL. Instead of posting or getting data, we’re going to query data and mutate data in order to update it. You might see that you can use GraphQL iteratively. I had that GUI ID that I could query for and then use it to change of I needed to update the application, add it to an alert policy, add it to a dashboard.
Sarah Hudspeth: It’s nice that you can just build off each other. I know exactly what data I’m going to get, and I’m only going to get that data. It’s going to be nice and structured. It’s going to be fast. I’ll tell you right now, New Relic is powered by this NerdGraph which you saw. That data that we accessed, we inside our platform also use it to access… or to build out all the dashboards and charts. I should say that GraphQL was developed by Facebook in 2012. Obviously when you’re processing that much amount of data, you want to be specific about the data you get, and get it as quickly as possible. The one downside is it does require a lot of upfront work. You have to build out that data schema so that folks can get the access.
Sarah Hudspeth: But once you have it built, you have a very powerful GraphQL engine. There’s some other cool things. I was going to say, with my API call, I had to call it many times in that for loop, because I could only get one book at a time. In GraphQL, you can make multiple calls even to multiple servers to get multiple data requests. It’s just a lot more robust and flexible. I’m quickly going to go through this slide. I think from the other talks, you’ve seen how we use data and how we want access to data and how we want to build it out programmatically, and automate and really be able to empower our data to… or empower our customers to use their data in a lot of different ways. Some of those are alerts. Getting alerted on any issues, updating with microservices and Kubernetes. You can spin things up, spin things down. You need to add them to alert policies or delete them.
Sarah Hudspeth: I also work with customers a lot about either storing or dropping data they don’t need. Sometimes companies need to store their logs to be in compliance with certain data rules. And so we can export data rules and NerdGraph to AWS buckets so they meet that requirement. We did talk about dashboards and S… or others talked about dashboards and SLOs. You can update dashboards with GraphQL. You can add things, you can subtract things. You can actually have a call to get a PDF. So if you need to email it to your superior and be like, “Hey, look at our application performance for the week.” You’re able to do that with a GraphQL API call, and then synthetics as well. If you want to check on Ping Checks if anything’s failing, or if you need to update, add end points, you can all do that in GraphQL.
Sarah Hudspeth: I think I’m good on time. I was just going to quickly show you how you can build out the query in the query builder. Let’s see. Maybe I’ll get the synthetic monitor. If I just wanted a list of synthetic monitors, I could just click whatever I wanted to see. I could add here. And when I press play, it just comes up to the right. I did add a permalink. So if maybe there was something I noticed, it was a critical learner. When I wanted to go check it out, I could quickly copy and paste or build out a script to go into New Relic and see what was happening. Looks like this check is okay, but I can go in and get that view. If I wanted to mutate, I could just continue to build out.
Sarah Hudspeth: Let’s say I wanted to create a workload. I could build out a workload using whatever data here. You can use the cURL up here. You could use our New Relic command line interface. It’s really flexible and robust. For all the data nerds out there, it’s just really fun to use. That was my talk. Hopefully you picked up a lot or a little about REST APIs and GraphQL and the differences. Just wanted to let you know, my team is hiring, so please reach out. Tap me up if you have questions, but thank you for listening.
Angie Chang: Thank you, Sarah, for the talk and demo on GraphQL. It’s very informative. I’m sure people have lots of questions will like to connect with you. So thank you so much. Our next speaker, we’re going to try Jo Ann again.
Angie Chang: Jo Ann is a senior technical account manager at New Relic. Has been working directly with customers, helping them use and implement the New Relic platform, including best practices. Prior to that, she was a solutions architect at Delta Airlines in Atlanta. So welcome, Jo Ann.
New Relic senior technical account manager Jo Ann de Leon talks about programmability, React, Nerdpacks and much more at Girl Geek X New Relic virtual event. (Watch the talk)
Jo Ann de Leon: Thank you, Angie. All right. Hello, everyone. I am Jo Ann de Leon, and I will be talking about the power of ReactJS and how it transformed the New Relic platform to be an open connected and programmable platform. Before I get started, I’d like to share some tidbits about myself. I am a senior technical account manager. I have been with New Relic for three and a half years, working directly with customers, acting as a technical advisor and solutions architect, to help them implement their observability use cases. I was born and raised in the Philippines. I graduated with a math degree, but never really thought I’d work in the IT industry. But in the past 20 something years, I have worn a lot of different IT hats, including a software developer, a designer, architect and project manager. Outside of work, my wife and I enjoy traveling, playing bocce, and cuddling with our two adorable orange tabbies.
Jo Ann de Leon: For this talk, I will introduce the concept of programmability. Show where you can find some of the open source apps and custom visualizations. And finally do a quick demo of how you can build your own. In a nutshell, programmability is about giving engineers full access to the New Relic database engine, and the building blocks they need to consume data in ways that solve their unique business problems. It also means giving our engineer users and customers the same set of tools our own engineers use to build our platform key rated experiences. What does this look like?
Jo Ann de Leon: The first place to explore is the New Relic Instant Observability or IO, which you can find via the apps icon in the New Relic toolbar. It contains a catalog of public apps and visualizations that are maintained by New Relic, and can be managed via the UI. The catalog also allows you to manage your own custom apps. You can find a number of other open source apps and visualizations in the New Relic open source website. The great thing about open source is that these apps are extensible, meaning you can customize them to fit your needs, and you can easily install them via the CLI.
Jo Ann de Leon: Here are a couple of examples that I wanted to showcase. The first one is a cloud optimized application, which analyzes your cloud environment, figures out where you’re wasting money on excess cloud capacity. The application compares the size of your instances to their utilization, finds resources that are sized larger than needed, and estimates how much you could save by optimizing the resource size. The browser analyzer app displays an analysis of performance, and forecast how improving the performance of your website can impact your key performance indicators, such as bounce rate or traffic. It also figures out which individual site pages have the worst impact on performance, so you know where to start making fixes and improvements.
Jo Ann de Leon: A popular visualization is the status widget pack, which contains three types of visualizations. One of those three is this status timeline widget, that allows you to display how your services are performing over time using traffic lights as visual indicators. Now it’s time to build our own app.
Jo Ann de Leon: All right. In an alternate universe, I have open a number of cat cafes around the country, where I serve coffee and cute cats or lunging around to entertain my customers, who may then fall in love with them, and decide to adopt them. In order to achieve my goal of helping these cats find their forever home, I need to keep track of how many have been adopted, and how many are still up for adoption. I went ahead and sent this data to New Relic, but how do I visualize all my data since I have so many cat cafes around the country. Luckily, I can build an awesome nerdpack. So let’s go ahead and create it.
Jo Ann de Leon: I am in the New Relic homepage. I hope you can still see it. In the New Relic homepage, you can go to the apps and click on build your own app. You can follow these instructions in the quick start. If you haven’t already done so, you can create an API key in your New Relic account, or select an existing API key. This is where you can download and install the NR One CLI, and make sure that it is up and running. And then the last step before you build your nerdpack, is to save your credentials. Let me copy this, and we’ll go ahead and create the package and run it. I am going to name my nerdlet as cat café tracker, and launcher as cat café launcher.
Jo Ann de Leon: Install the dependencies and create all the different components needed for my app. And then I can go to that NerdPack and let me open this in my Visual Studio Code. All right. Let me open the Terminal here, and then I can run my server through the New Relic One CLI, with this command: nr1 nerdpack:serve. All right. You will notice that now you can run one.newrelic.com with nerdpacks=local. This means that any local development you make can be tested in the New Relic platform. You’re also given a shortcut to the launcher, which will open your Nerdlet directly. So let’s go ahead and copy that. And let’s go back to the browser here, and let’s close this prompt.
Jo Ann de Leon: And now we have our Hello World version for our cat cafe tracker, but it’s not really very exciting. Let’s go back to our code. For the sake of time, I will be copying and pasting my code, including index.js. This code will contain the logic to retrieve the data from the database and display it in two views, a table view and a map view. Let me go ahead and do that.
Jo Ann de Leon: And then I also need to update my styles.css. This will contain styling elements for my custom UI. All right. Third one. I need to update my package.json dependencies, because we will be using the leaflet package to create a map. All right. And then finally, I need the webpack config, which we will need to support the use of map tiling information data from leaflet. This will be copied at the root folder of our package. All right. Let me save all of that. I have to restart my server. Let me clear that. I have to do an npm install first, since I had to update my package.json. And now I’m going to restart my server and relaunch my app. All right. Let’s copy that new link. Go back to our browser.
Jo Ann de Leon: Hopefully this will work. There we go. All right. So now I can view all my cat cafes around the country. I created my visualization such that the size of the circles indicates how many cats are available for adoption in that area. The bigger the circle, the more cats are available. The green color means more cats have been adopted, while those that are yellow or dark orange means we have some work to do to get more cats adopted. Finally, I have also displayed my data in a table view to the side of my map. All right. I hope you have enjoyed this quick demo on how programmability through the use of ReactJS can help you create visualizations that focus on solving business problems. Please feel free to connect with me through my email or LinkedIn. Thank you.
Angie Chang: Thank you, Jo Ann, for that talk and demo, and we’ll be sure to connect with you on LinkedIn. Our leadership panel will talk about New Relic culture, inclusion, career development, and successful interview prep.
Angie Chang: Our moderator today is Ariane Evans. She’s a diversity equity and inclusion manager at New Relic, working with the talent acquisition, hiring managers, employees, and external organizations to recruit, engage, develop underrepresented communities. And she co-leads the Relics of Color ERG. Welcome, Ariane.
Ariane Evansmoderates New Relic leadership panel with Nada Da Veiga, Erin Dieterich, Kim Camacho, Tracy Ravenscraft, and Stefanie Smith. (Watch on YouTube)
Ariane Evans: Thanks, Angie. Hi, everyone. My name is Ariane Evans. And as Angie mentioned, I’m a [inaudible] manager at New Relic. I love that I get to spend a little time with you and facilitate a conversation with some of our incredible leaders at New Relic. All of them, women. It’s so inspiring to have leaders that are not only passionate about their work, but the communities that they work within. Before I dive into the questions to know more about New Relic and the areas of expertise of each of these leaders, let’s go through a quick lightning round of introductions. Please give me your name, title, and a sweet little fun fact about you. Let’s start with Kim.
Kim Camacho: Hi. Hi, everyone. Happy pride month. My name’s Kim Camacho, and my pronouns are she and her. I’ve been the director of DE&I at New Relic for about a year, and have also about 20 years of DE and I and HR experience. A fun fact about me is, I met Barack Obama right after he announced his candidacy for presidency a long time ago. So that is fun fact
Ariane Evans: Very cool, and also now very jealous. Let’s go ahead and hear from Erin.
Erin Dieterich: Very jealous of that fun fact. Hi, I’m Erin Dietrich. I lead the social impact and environmental, social and governance organizations at New Relic. My pronouns are she and her. I’ve been at New Relic for about four and a half years, and I’m based in Portland. My fun fact is that I have two small children, a one and a half year old little girl and a five and a half year old little boy. And they keep me incredibly busy, and very tired all the time. I don’t think I’ve slept well in five years. Fun fact.
Ariane Evans: Well, you look great even on little sleep Erin. Thanks for joining. Let’s hear from Tracy next.
Tracy Ravenscraft: Hi, my name is Tracy Ravenscraft. I’ve been here at New Relic for about five and a half years. I run a technical account manager team in central. My fun fact is I have two dogs, one Pomeranian, one Pomsky, and they have names like Friends characters, so their names are Phoebe and Ross. Thank you.
Ariane Evans: Love a good Friends joke. Let’s hear from Nada next.
Nada Da Veiga: Hi, everybody. I lead customer adoption organization. America’s customer adoption organization here at New Relic. Been here for five years. If you’re wondering what customer adoption is, basically, all engineers that work closely with our customers, helping them learn how to use our platform to solve their technical and business problem, basically. Fun fact: throughout my life, I have had five different passports. So no, I’m not a female version of James Bond, but that’s what my husband likes to think.
Ariane Evans: I might also think of a reference to Carmen Sandiego. Where in the world is Debeka? Where is she going next? Next let’s have Stephanie.
Stefanie Smith: Hi. Thanks, Ariane. I’m Stephanie Smith. I’m based in Massachusetts, I’ve been with New Relic for six years. Currently senior manager of talent acquisition. My team supports go to market customer adoption. Let’s see. Fun fact about me is I have two teenage daughters, one of which just graduated high school last weekend, which is very hard to believe, and a younger one. She’s a sophomore, she’ll be a junior. Erin, the exhaustion doesn’t stop. It only gets different. It’s bigger problems with bigger kids, but it’s all worth it. Fun ride for sure. Excited to be here.
Ariane Evans: Thanks, Stephanie. And thank you all. We all just listened to quite a few talks learning about why observability is important. What is monitoring? How do we implement these different products and technology? And also this happens inside of a company where the people work together. There’s a culture that allows us to do that work at our best and highest potential. I’d love to hear from each of you on how you are not only living those in practices, but working that out in your teams and your strategies at New Relic. I will start with a bit of our culture and understanding how is New Relic creating a culture where people from all backgrounds feel included.
Kim Camacho: All right. I could take a step at that, Ariane. First and foremost, I think we are very clear about our commitment to diversity, equity and inclusion. We communicate our vision, mission and objectives annually as we build out our short term annual plans and our long term strategy. All new employees and interns hear about our strategy as well as our organization when they onboard. We measure regularly how employees are feeling. The extent to which they feel belonging and respect to the company. So important to do that. I think also for our employees, one of the big things that’s really important is having communities of people that you can bond with, that are recognizable to you and have the same interests and backgrounds that you have. We have employee resource groups at New Relic. They’re fully funded and have leadership organizations as well as executive sponsors.
Kim Camacho: It’s through these organizations that we hope that people are building relationships, bonding, getting to know each other outside of their regular roles. In addition to our ERGs, we have other slack channels based on whatever people want to connect with. Whether it’s dogs, bunnies. There was one that was just started on crime channels, which I’m in love with, so you can bond. The last thing I’ll say, as it relates to really creating a culture where people feel connected is, the importance of managers. I think as our audience will know, your manager makes a big difference. Here at New Relic, it’s really important that we support train, help our managers really understand cultural competency, how to build a diverse and equitable workplace. Everyone I think on this call knows, because they’ve been through some of our trainings and are actively involved in these efforts. It’s just really important that we’re working with our managers so that they understand their role in helping create a nurturing environment for our employees. That’s a little bit about from that perspective.
Ariane Evans: That is all really cool. I know that there are also more things that New Relic is engaging. Erin, maybe you can tell us, what is New Relic focused on, or engaging our employees and social impact.
Erin Dieterich: Yeah, thanks Ariane. Newrelic.org is the name of our social impact work. We started it in early 2019, and really committed at that point to this mission of, how do we as a company, continue to push for more equitable access to technology? We really believe that accessing not just physical technology, having the best computer, having the best SaaS tools, but having the access to understanding what technology careers actually look like, what kinds of roles there are within technology. That is such a critical piece to creating this more equitable future for the industry, and to thinking about, how do we help people all along their learning journey? Whether they’re somebody who’s had a couple careers already, and are starting a career in technology, or a student who’s early in life thinking about what they want to be when they grow up. How do we give all of those people access to our incredible employees, so that they can hear the stories about how we all got where we are, and be able to start seeing themselves on this whole rainbow of pathways.
Erin Dieterich: It is not just one clear, point A to point B gets you a tech career. There are so many different ways to get where you’re going, and so many different destinations along the way. And so we’re just really passionate about infusing that into everything we do in social impact, and thinking about how we take the 2000 plus employees around the world with us on that journey. Some of the ways that we do that, we have a bunch of benefits that all of our employees get access to. They get to have 16 hours of paid time off to volunteer a year, plus we now have a set global day of service every winter. That’s three full days of volunteering, and you can slice and dice that however you want throughout the year. We incentivize our employees using that volunteering, by actually giving them dollars that they can push towards their fair charities every time they log their hours of volunteering.
Erin Dieterich: We have a $200 a year matching program. Employees can get up to $200 a year matched to any number of global charities. I think there’s 20,000 charities that they can pick from. And then we do a bunch of special campaigns. And so some of the things I really love that we’ve been building and you’ve actually been a big part of building these with us, Ariane, are some of our partnerships with our employee resource groups. Where we’re really going to our employee resource groups and helping them give us the understanding of where they want to impact in their communities, what organizations they want to work with. And then working together to make sure that that information is accessible to our employees, to incentivize and point them towards making really smart decisions with their wealth of how they can build this more equitable future.
Erin Dieterich: A great example of that is, since it’s June and it’s pride month, we are working with our rainbow relics ERG and just launched a $25,000 additional matching campaigns. In addition to those $200 employees have, they now can also put additional dollars towards this matching campaign, that goes to five different organizations that our rainbow [inaudible] helped us identify and pick in their communities. Organizations that they really care about that are helping the LGBTQ community with all of the different things going on, both in the US and abroad. Being able to be a part of understanding what that ERG community wants employees to support, and then helping employees understand how they can use their dollars to support their fellow relics, and the things they care about, is something that just makes me so excited.
Erin Dieterich: I just love seeing the way our employees are supporting each other through those special campaigns. I think I’m almost out of time, but I’ll tell one other very quick story, which is, since we have so many technical and inspiring folks on this call, I always like to take the opportunity to just pause and remind folks how valuable your skills are. Technology skills are so incredible. There’s a myriad of ways you could apply those to social good. Something we love to do is partnering our employees up with our nonprofit customers who get to use expanded access to New Relic for free. But we know that they need help with enablement. And so we partner them up with employees and the employees take on pro bono volunteering projects, where they’re using their technical skills to really support observability in nonprofits.
Erin Dieterich: And so you don’t have to be a New Relic employee to do something like that. You can really step back and say, “What causes are super important to me? What organizations do I love?” Reach out to them and say, “I want to talk to whoever’s running your technology, and see how I can be of support. I have X, Y, Z skillset that I’m really proud of. Is there a project I could help you on pro bono, and volunteer and support your organization, building your digital environment?” Because that is what every organization needs in order to power their mission. Every person with technology skills has just so much that they can give back. And so we love to do that at New Relic, but I also just love to encourage anyone anytime I can, to think about how you can use your skills out there in your community to power the charities and the causes that you care about.
Ariane Evans: Thanks, Erin. It sounds like New Relic is really building out a culture for people to live a life fully as they’d like, both internally and their communities. The things that they care about, but also themselves wholly. I’d love to hear from you, Tracy. Describing to us, which areas of your life would you like to spend more quality time when you think about work life balance.
Tracy Ravenscraft: That’s a great question. Thank you. When I think about where I like to spend my time outside of work, definitely with family and friends. Everybody wants that more family, friend time. But not only just spending more time with them, being present. Not checking my phone for slack messages, going on vacations and being able to completely disconnect. That’s what New Relic has brought to my life.
Tracy Ravenscraft: I’ve been at New Relic for five and a half years. I did site reliability in the past, network administration, network engineering. I never realized how I wasn’t there. I’m always looking for the next page. When I have time off, I’m bringing my laptop, I’m bringing my phone. I feel like New Relic, with our recharge week, which the summer that we all get off at the same time. FTO, so it’s flex time off. There’s really no limit to my vacation. Just some of the applications we have, like Ginger, that helps with mental health. I really feel connected when I’m using my own personal time and being with my family. So yeah, that’s how I like to recharge, if you will.
Ariane Evans: Yeah. So important. When you are moving on to the next project, or you are trying to get to the deadline of a particular thing, you can’t do that if you’re empty, and you don’t have the energy within you. And so I guess moving on, switching gears a little bit, want to talk with you, Nada, about navigating careers and career challenges. Career journeys can vary person to person. As Tracy just described, she’s been across the board of different kinds of engineers, and now a customer adoption leader, but how might you recommend navigating a career journey, and even a career journey into leadership?
Nada Da Veiga: Yeah. I mean, I think that’s an excellent question really. What I, or what we in general try to encourage folks in my organization, is to own their career, and be really proactive about it. And so a lot of people early in their careers think that they should somehow just wait for their manager to have these types of conversations. I would say quite the opposite. Be proactive about it, ask questions, share, what do you want for yourself? Where do you want to be three years from now, five years from now? Ask your manager, “What do I need to do to get there?” Because if you are informed and you know what this person expects from you, what three, five things they want to see from you in order for you to actually make it there, guess what? You have a lot higher chance of getting there, than if you’re just sitting and waiting for them to tell you, because they may or may not tell you actually.
Nada Da Veiga: They may or may not understand that you want to get from this role to some other role. That is what we see a lot with our teams. At New Relic, we are very much committed to our employee’s career progression. These are proactive conversations that are happening continuously. We encourage our employees to put together their career plans, to share those with their managers. And then some of them just want to go, “Hey, how do I go from this role that I’m in today, maybe to a senior role or a principal role?” Others want to move maybe from one org to a different org, so they want help with that path.
Nada Da Veiga: Third group will say, “Well, I want to get to leadership.” But I think how you approach it really doesn’t matter. New Relic specifically, if you are interested in leadership, we have about 14 different management classes that we recommend to folks that are setting you on that leadership path at New Relic. But whether you’re at New Relic or somewhere else, show your manager, show your leadership what you really are interested in, where your heart is at, and be proactive about it. That’s probably the best advice I can give.
Ariane Evans: Yeah. I love that. I will say that I think my career journey at New Relic is a testament to that. Starting in talent acquisition and getting to be a partner to Stephanie, but then moving into social impact and getting to learn from Erin, and now today, being a part of the DE and I team, and getting to work very closely with Kim, and that has all been championed by New Relic and the leaders within… I just said, “I’m interested in this thing, and I’m not really sure where to go from here.” But it did start with an interview. It started with a conversation with my manager. And so I’d love to kick it over to you, Stephanie, and think about, for a lot of people, getting started in your career, or looking for new opportunities, it starts with that interview process. You’ve interviewed hundreds of people in your career. And now as a recruiting leader, what is the best advice you have for anyone that is preparing to interview or in the process of interviewing currently?
Stefanie Smith: That’s a good question. I do want to just talk about just quickly, Ariane, your career progression. There’s so many people at New Relic that have had career progression, me included. I started off as a recruiter, and promoted along the way to senior manager. So there’s so much opportunity. But yes, there is an interview that’s involved. Interviewing with the company really, it’s your first impression, but it’s also our first impression to you as well. I always tell people that it’s your interview as much as it is ours. Make sure you qualify. Know what the company does. Really know what the company does. Do some research, do your homework. There’s a wealth of information about companies on the internet. It’s incredible. Link in with people on LinkedIn. Understand the roles and responsibilities, what people are.
Stefanie Smith: And then when you are talking to someone, likely it’s going to be a recruiter first, it’s a conversation. Like I said, really, you’re qualifying us, we’re qualifying you. Part of our core values is being authentic. I think that you’ve probably seen a lot of authenticity throughout this entire panel, and previous to the speakers. Be authentic during the interview, be yourself. Find some common ground. Look at it as just a conversation. Working, we spend more time than anywhere else. New Relic encourages everyone to be their best authentic self. When you’re in the interview, just really be yourself and ask good questions, and talk about career pathing and all the things that are important to you.
Stefanie Smith: Realize, if this is the right company, position, and so forth. And also even ask for guidance along the way. Your recruiter’s going to be the first step, and the recruiters are going to send you on for the next interview. Connect with the recruiter as often as possible. Even connect with the people that you’re interviewing with. We have multiple steps of roles when we interview here at New Relic. People are always going to be available to help guide you through the process. Ultimately, like I said, it’s your interview as much as it’s ours.
Ariane Evans: Yeah. I totally agree with that. Since we’ve also wrapped up this time with all of our leaders, I want to thank Girl Geek, thank New Relic for also putting this together, and everybody for listening in. I hope that you’ve gotten to pull out some nuggets of advice that are beneficial to you. If you are interested in learning more about New Relic or careers or opportunities, there are some things that Kim dropped into about our ERGs and our benefits. Please take a look at newrelic.com/culture. It will take you to our careers page and the opportunities that are currently live across the world. There are many.
Angie Chang: Thank you, Ariane, for moderating the panel, and to all the panelists for joining us. So now is time for our networking session. If you can click on the link at the bottom of the chat to our Zoom meeting, we can go into a Zoom meeting and have some breakout rooms where we can meet each other in person, and chat a little with our remaining 15 minutes that we have today. So if you can click on that link in chat that Amy has added, I’ll see you over at Zoom meeting and talk to you there. Thanks for coming.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
Over 120 girl geeks joined networking and talks at the sold-out MosaicML Girl Geek Dinner from women working in machine learning at MosaicML, Meta AI, Atomwise, Salesforce Research, OpenAI, Amazon, and Hala Systems.
Speakers discuss efficient machine learning training with MosaicML, reinforcement learning, ML-based drug discovery with AtomNet, evaluating recommendation robustness with RGRecSys, turning generative models into products at OpenAI, seeking the bigger picture at AWS, and more.
Transcript of MosaicML Girl Geek Dinner – Lightning Talks:
Angie Chang: Thank you so much for coming out. I’m so glad you’re here. My name is Angie Chang and I’m the founder of Girl Geek X. We’ve been doing Girl Geek Dinners in the San Francisco Bay Area for, if you can believe it, almost 15 years now. It’s the first Girl Geek Dinner in over two years, because Julie is a rock star and wanted to do a Girl Geek Dinner in person in the pandemic and we’re like, “Yes!” It was postponed and now in May, we’re finally doing this event. I’m so glad that we have a sold out event of amazing women in machine learning that we’re going to be hearing from tonight!
Girl Geek X founder Angie Chang welcomes the sold-out crowd to our first IRL Girl Geek Dinner in over two years during the pandemic! (Watch on YouTube)
Angie Chang: I don’t want to steal too much of the time, but I wanted to do a quick raffle of a tote bag that I have. I’m going to ask, who has been to the first Girl Geek Dinner and can name a speaker from that event? Sometimes I meet people who have. Are we making this really hard? Okay, the first Girl Geek Dinner was at Google. We had over 400 women show up for a panel of inspiring women. I just wanted to see because that’s a lot of people. Who thinks they’ve been to the most Girl Geek Dinners in the room? Okay.
Audience Member: It’s actually not me, but like, this clutch was designed by somebody who hates women because its super heavy – and I see that that [Girl Geek X tote] has handles.
Angie Chang: Okay, so we have a full agenda of machine learning lightning talks and I’m going to introduce you to our host for tonight. Julie Choi is the Chief Growth Officer of MosaicML and she is an amazing supporter of women and I would like to invite her up.
Julie Choi: Oh, thank you. Hi everyone. Thank you so much for coming out to the MosaicML Girl Geek Dinner. I am Julie Choi and I actually did go to the first Girl Geek Dinner. I want to thank Angie Chang and the Girl Geek Dinner organization. Angie has just been a pioneer and truly, just a very special person, bringing us together ever since, was that 2010 or something, I don’t know. When was that?
MosaicML VP and Chief Growth Officer Julie Choi welcomes the audience. She emcees the evening at MosaicML Girl Geek Dinner. (Watch on YouTube)
Angie Chang: The first event was 2008.
Julie Choi: 2008. Yes. And tonight we have amazing speakers to share with us about machine learning, about engineering, about diversity, and how that can really supercharge productivity in high growth organizations and machine learning research just from some of the world’s best AI companies and organizations.
Julie Choi: Our first speaker of the evening is my dear colleague at MosaicML, Laura Florescu. Laura and I met a little over a year ago. You greeted me at the front door. You were the first face I saw at MosaicML. She has just been an inspiration to me as an ML researcher at our company. Prior to joining MosaicML, Laura actually worked at several unicorn AI hardware startups. Then prior to that, she got her PhD from NYU in mathematics and is just a brilliant lady. Laura, can you come up and tell us about this amazing topic?
Laura Florescu: Thank you.
Julie Choi: Let’s give her a hand.
Laura Florescu: Can you hear me? Hi, thank you so much everybody for being here. Thanks to Julie, Sarah, Angie, Playground, for having this event. Very honored to be here.
MosaicML AI Researcher Laura Florescu talks about making ML training faster, algorithmically with Composer and Compute, MosaicML’s latest offerings for efficient ML. (Watch on YouTube)
Laura Florescu: As Julie said, I’m a researcher at MosaicML. A little bit about myself. I’m originally from Romania. I came to the states, did my undergrad in math, did a PhD at NYU, and then I got the kind of Silicon Valley bug. And now I’m at MosaicML. And so what we do is develop algorithms and infrastructure to train neural networks efficiently.
Laura Florescu: Basically for the people in the audience who are not into ML, training neural networks is at the core of artificial intelligence. It uses a lot of data. It’s applied to many different fields with image, language, speech, and kind of like a takeaway from this is that, for large powerful models, the training costs for one single run can get in the millions of dollars, to train one such model. And in order to get to a really good model, you need to do several such runs. So the cost can get extremely, extremely expensive.
Laura Florescu: Our belief at MosaicML is that state of the art, large, powerful models should not be limited to just the top companies. As we have seen over the last few years, the costs are getting larger and larger due to both the size of the models, also the data that the models ingest is just exploding.
Laura Florescu: A couple of years ago, a state-of-the-art model, Megatron, actually cost $15 million to train. As you can imagine, startups probably cannot really train models like that. And at MosaicML, this is our belief, that this kind of training should be accessible to other partners as well.
Laura Florescu: That’s where we come in. We want the state-of-the-art efficient ML training. Our co-founders are Naveen and Hanlin from Nirvana and Intel AI. Also, founding advisors from MIT and also founding engineers from leading AI companies. All of us have the same kind of goal to train machine learning basically faster and better and cheaper.
Laura Florescu: Our thesis is, the core of Mosaic is that algorithmic and system optimizations are the key to ML efficiency, right? And then the proof to that so far, is that we are working with enterprises to train ML models efficiently. And we want to enable our organizations to train the best ML models, the cheapest and the most efficient.
Laura Florescu: Some results that we already have. As I said, we want to be agnostic about the kind of models, data that we ingest. For some image classification tasks, we have shown 6x speed ups, like 6x cheaper, faster than like regular training. About 3x faster for image segmentation, about 2x for language models, language generation, and 1.5x for language understanding. We have been around as Julie said for about a year, a year and a few months, and these are already some of the results that we have achieved.
Laura Florescu: A use case is to train NLP models, such as BERT, for those of you who know that, and on our specific platform and without algorithms. Use case is for example, to increase sales productivity. If you see there, on our MosaicML 4-node, with our speed ups, which I’m going to discuss in a little bit, we can see up to 2x speed ups of training such models. And also about 60% training costs reduced by training with both our algorithms and on our platform, on our cloud.
Laura Florescu: The MosaicML cloud, we want it to be the first AI optimized cloud designed specifically for AI and directly to reduce training cost at any layer of the stack. For example, in the training flow, we want to reuse data from past runs. In the models that we’re using, so that’s where the kind of optimized model definitions come in, Composer, which is our open source library. We are doing the algorithmic speed ups, the training. And kind of like at the lower level, we want to also be able to choose the best hardware in order to get to the lowest training cost. At each layer, we are optimizing all of these system optimizations and composing all of those basically leads to the best training runs.
Laura Florescu: As I mentioned, Composer is our open source library. We have a QR code there if you want to check it out. We have about 25+ algorithms that we have worked on and given the name, they compose together, and that’s how we achieve basically the best, 2x to 5x speed ups. And as an example, for a BERT model, we have seen 2.5x speed up for pre-training, which as you can see, goes from nine days to about three days. That’s a huge win. Check it out if you would like. As I said, open source, so we’re always looking for feedback and contributions.
Laura Florescu: We’re open to partnering with any kind of corporate users, for anybody who has vision or language tasks, and then also industry, we want to be industry agnostic and global. Again, we want to optimize basically any kind of models. And as I said, the open source Composer speed up, we’re open to feedback and partnerships for that as well. And of course we are hiring. Yeah, it’s a really great team, really fun, really ambitious. And I’m so honored to work there and we’re looking for all kind of builders, engineers, researchers, products. And thank you very much.
Julie Choi: Thank you so much, Laura. Okay, that was wonderful. Let me just go to the next talk. Our next speaker is Amy Zhang, and Amy comes to us from, she’s currently a research scientist at Meta AI research, and a postdoc I think, you’re not a postdoc anymore, are you?
Amy Zhang: It’s kind of like a part-time thing.
Julie Choi: It’s like, never ending, huh? But you’re on your way to your assistant professorship at UT Austin, amazing, in Spring 2023. And Amy’s work focuses on improving generalization and reinforcement learning through bridging theory and practice. And her work, she was on the board most recently of women in machine learning for the past two years. And she got her PhD at McGill University and the Mila Institute and prior to that obtained her M.Eng. in EECS and dual Bachelors of Science degrees in math and EECS at MIT. Let’s welcome Amy Zhang to the stage. Thank you.
Amy Zhang: Thank you Julie for the really kind introduction and for planning this amazing event. It’s so nice to see people in real life. This is my first large in-person talk in over two years so please bear with me.
Meta AI Research Scientist Amy Zhang speaks about her career journey in reinforcement learning, from academia to industry, at MosaicML Girl Geek Dinner. (Watch on YouTube)
Amy Zhang: Today I’ll be talking about my research which is in reinforcement learning, but I first wanted to just give a little bit of introduction of myself. To me, I feel like I’ve had a fairly meandering journey through academia and industry and research in general. I wanted to give a little bit of introduction of what I’ve done so that if any of you feel like you’re going through something similar, please reach out and I’m happy to chat and give more details. Like Julie said, I did my undergrad at MIT, a year after I finished my masters, I started a PhD at UCSD. It did not go well, through no one’s fault, really. I just felt really isolated and so after about a year, I quit my PhD, meandered through a couple of startups, and then eventually found my way to Facebook, which is now Meta.
Amy Zhang: At Facebook, I initially started on the core data science team. I was a data scientist, but I was working on computer vision and deep learning. This was 2015, still fairly early days in terms of deep learning and everyone is really excited about the gains that it had shown for computer vision at the time. I was working on population density, so we were taking satellite images and doing building detection to find houses, to figure out mostly in third world countries, what the population density was so that we could provide internet to people and figure out what was the best way to provide that internet.
Amy Zhang: After about a year of that, I ended up joining the Facebook AI research team FAIR, as a research engineer. And after about a year of that, I happened to meet the person who became my PhD advisor, Joelle Pineau, who is now director of FAIR, and I got to do my PhD with her at McGill University while still staying in FAIR. Fast forward a few more years and I defended my PhD remotely in the middle of the pandemic last year and am now back in the Bay Area as a research scientist at FAIR and like I said, was part-time postdoc-ing at UC Berkeley.
Amy Zhang: After spending all this time in industry, and having a really great time in industry, and getting all of these amazing opportunities to do my PhD while in industry, with all of the nice resources that that provides, I decided to go back into academia. And, last year I was on the faculty market, on the academic faculty market, and I accepted a position at UT Austin. I will be starting there next year as an assistant professor.
Amy Zhang: With that, I’m just going to jump straight into my research. And this is going to be, again like I said, a very whirlwind, high level overview. I’m passionate about reinforcement learning. I love the idea of agents that can interact with the world, with us, and it can grow and learn from that interaction.
Amy Zhang: Okay, thinking a little bit about what reinforcement learning (RL) can do. I’m personally really excited about the idea of applying reinforcement learning to solve real world problems. To me, this is personalized household robots, having a robot that will do your dishes, clean the house, make your bed, learned autonomous driving so you can just drive in a car without having to actually drive and pay attention to the road, and personalize healthcare, so having like a robo-doctor who knows everything about you and can personalize healthcare and give recommendations for you specifically.
Amy Zhang: Unfortunately we are not there yet, as I think maybe all of us can tell. Deep reinforcement learning has had a lot of successes in the last two years. Maybe some of these things are familiar to you. AlphaGo, where we have an RL agent that was able to beat the best human experts. OpenAI with playing video games, which I’m not very familiar with, but like Dota and StarCraft. These are the things that have been hitting the news in terms of what Deep RL is capable of. But there are still a lot of disappointing failures and I think none of these videos are going to show, but imagine this robot trying to kick this ball and then just falling flat on its face. That’s what that video is. And in the other little cheetah looking thing is supposed to be like tripping and falling. Anyway, this is where we are currently with RL.
Amy Zhang: Why do we still see this discrepancy? How are we getting these amazing gains but still seeing these failures? And what we’re really seeing is that Deep RL works really well in these single task settings, in simulation, when you have access to tons and tons of training data, but it works less well in visually complex and natural settings. Basically we’re not seeing the same type of generalization performance that we’ve been getting out of deep learning in computer vision and natural language processing. My research agenda is mainly about how can we achieve RL in the real world? How can we solve these problems? And, to me, it seems that abstraction is one key to generalization. And I use this type of idea to develop algorithms that have theory-backed guarantees.
Amy Zhang: I’m going to not really talk a little bit about this math, but I’m particularly interested in being able to train reinforcement learning agents that can solve tasks from pixels. Imagine that you have this household robot or this autonomous driving car that is receiving information about the world through a camera, through RGB input, and that’s a big part of how we also perceive the world. There are things in this image that are relevant for the autonomous driver here and there are things that are not. And we want to figure out how can we determine from just a reward signal, what things are relevant versus irrelevant.
Amy Zhang: I’m just going to, as part of this project, we developed this representation learning method using this idea by simulation, and showed that in this type of simulation driving task, which is done in this platform called Carla, so we have just this figure eight simulation environment where this car is just driving along this highway, and there’s lots of other cars in the road and basically, it’s designed to try and drive as quickly as possible. Using the break as little as possible and maximizing throttle while also not hitting anything. And we find that our method which can basically ignore these kind of irrelevant details and figure out what things are irrelevant does much better compared to a lot of existing methods.
Amy Zhang: One really cool thing is that when we look at the representation that we actually learn here, and we look at what kind of observations get mapped to be close together in this representation space, so what information is actually getting captured by this representation, we see that… this is the agent’s point of view. We see that, in these three examples, you’re always on this straight road where you have an obstacle on the right hand side. It doesn’t matter what the obstacle is, but the representation just captures that something is there, which means that you can’t turn right. This is just kind of an example of what our algorithm can do. And unfortunately I’m going to skip over this, because these were just some videos showing what our agent can do.
Amy Zhang: I just wanted to end on talking about some open problems that I’m particularly excited about. I’m particularly excited about compositionality. How can we solve combinatorially difficult problems. And these correspond to a lot of real world tasks that we should care about. When we think about really simple versions of problems like this, you can have a block stacking task. You can have any number of blocks or any combination of blocks and so you can always have new environments that you give to your agent to try and solve.
Amy Zhang: Similarly, moving more towards actual real world problems that we care about. Again, going back to the dishwasher example or an agent that is trying to move boxes around in a warehouse. These are all settings where the exact environment, the exact state that the agent has to deal with, is constantly different. The objects that you want to place in your dishwasher on a day to day basis is always going to look different. How can we get agents that can actually generalize to all of these new states?
Amy Zhang: I think one really exciting direction to go when trying to solve this type of problem is to think about factorization. How can we break down a problem into smaller, easier building blocks? So if we understand how one block, the dynamics of one block moves, so creating a stack of two blocks, right? Babies play around with this sort of thing and then as they get older, they automatically can extrapolate to building like gigantic towers and castles. So how can we take that idea and give it to reinforcement learning? So this is something that I’m particularly excited by.
Amy Zhang: I just wanted to end talking about my sort of wider research agenda. Now that I’m starting as faculty, I have to start recruiting a group. If any of you are interested in doing a PhD at UT Austin or know anybody who is, please send them my way. But when I think about what my research group does, I’m particularly interested in these three applications of reinforcement learning.
Amy Zhang: The first is in robotics, trying to solve manipulation tasks. Going back to that block stacking example, trying to solve locomotion and navigation tasks. How can we build an autonomous driving system purely from first principles, like end to end machine learning? Reinforcement learning has to be a part of that.
Amy Zhang: Natural language processing, so using RL for text generation, being able to extract knowledge from text, when you build an interactive agent, how do you give it information about the world? We learn from textbooks, we don’t want an agent just deployed in the real world with no basic information. Healthcare, how do we build RL agents that can help out with diagnosis and treatment and tackle a lot of the problems that we have there? That’s basically it, very grandiose. I probably won’t make much progress on a lot of these fronts, but this is the dream. And thanks for listening.
Julie Choi: Thank you so much. I think this is very grandiose and amazing that you’re working on it. Amazing. Thank you, Amy, okay.
Julie Choi: We’re going to just clean some of this up. Okay. And we’re going to open up our next talk, which I’m extremely excited about. Let me introduce our next speaker. Our next speaker journeyed probably the furthest to join us tonight… all the way from the east coast, North Carolina, to be here today to deliver this talk. I want to thank you. Thank you.
Julie Choi: Tiffany Williams is a Staff Software Engineer at Atomwise working on AI-powered drug discovery. Prior to Atomwise, Tiffany was a Staff Software Engineer at Project Ronin where she was developing cancer intelligence software. Tiffany earned her PhD in cancer biology from Stanford University and her Bachelor’s in biology from the University of Maryland. Let’s give a warm welcome to Tiffany Williams.
Tiffany Williams: All right. Hello, everyone. I’ll admit, I have some notes here so I can stay on track, but I’m really glad to be here this evening. I was searching through my inbox and realized that I attended my first Girl Geek Dinner in April of 2015. I was fresh out of grad school and a coding boot camp. Eager to form connections, acquired some cool swag, and to be honest, eat some free food and have some drinks.
Atomwise Staff Software Engineer Tiffany Williams discusses the drug discovery process with AtomNet at MosaicML Girl Geek Dinner. (Watch on YouTube)
Tiffany Williams: I went to grad school to study cancer biology. My research was from the perspective of a molecular biologist exploring the role of a protein as a target for therapy, and skin cancer. And my interests have generally been at that sweet spot, leveraging data, and technology to improve human health. Coming from a biology research background, in my current role as a software engineer at Atomwise, I’m able to look further down that drug discovery process. I’m happy to share with you all today, what that looks like.
Tiffany Williams: Now, I’ll start by giving a very brief primer on the current state of the drug discovery process, as well as a very brief primer on biochemistry. From there, I’ll talk about what we’re doing at Atomwise to make a significant impact in human health, and some interesting challenges ahead of us. I do want to give a disclaimer that I am merely scratching the surface of what could be discussed in drug design, drug development, and even applying ML on top of that. But what I hope you all take away from my time with you all this evening is just general excitement about the possibility of improving human health.
Tiffany Williams: Beyond that, I hope you feel empowered to do even more digging into the drug discovery process, and maybe you’ll feel empowered to even… to find opportunities to make an impact in that space. For me personally, I’m coming from the east coast, but I have a personal attachment to improving the drug discovery process. I’m on the east coast serving as a caregiver now – my mom has endometrial cancer and it’s at a point where there’s only one treatment option. If any of you have ever been in, unfortunately, in a predicament like that, it sucks.
Tiffany Williams: There’s a lot of data out there, technology’s improving, and it would be great if we can leverage that to improve health, right? This diagram depicts the drug discovery process from the basic research to identify a potential drug target all the way to FDA approval. On the right, I’ve noted the average number of years of the different stages in the drug discovery process. In the middle, what it shows are like basically in these different stages, there are certain types of experiments that are done that basically kind of knock out the potential candidates in that step.
Tiffany Williams: Initially you might have like a candidate pool of over like 10,000 drugs, but in an early stage, which is known as like kid identification, computer simulation is used to predict potential drugs ability to bind to a target of interest. These subsequent steps will test for other characteristics like potential toxicity, or efficacy in cell cultures, and animal models. Eventually, hopefully, we end up with a few candidates that reach the human clinical trial phase to verify safety, and any other side effects along with a few other things.
Tiffany Williams: What I hope that you take away from this slide really is that the current basic research to FDA approval drug process takes a really long time. It takes on average, it’s estimated to take about 15 years, and it’s also really expensive for each drug that goes to market, it’s estimated that 2.6 billion is spent.It’s the case that not all of these trials have a happy ending. For every drug that makes it to the market, millions may have been screened, and discarded. We have to improve this process, right? But in order to appreciate how this process can be improved, let me first give that very, very, very brief biochem primer. And I’ll focus specifically on protein interactions.
Tiffany Williams: Up to this point, and then even later in this talk, I’m going to be using some words interchangeably, and I just want to make sure I bring some clarity to what I’m actually talking about. When I say like ligand, or ligand, I’m specifically referring to any molecule that combined to a receiving protein, and that receiving protein is also known as a receptor. And in this presentation, when I use the word or phrase drug candidate, I’m actually referring to that ligand and the receptor would be that the drug target.
Tiffany Williams: One model for how proteins interact is this lock and key model. And the gist of this is that the ligand and the receptor have these somewhat complimentary sites. And the ligand combined to this complimentary site or active site, and basically alter the shape, and or activity of the receptor. And one more thing is that this binding is also referred to as docking. If we know our bodies are made up of proteins, and they have a diversity of functions within the body, but in a disease state, a normal process can become dysregulated. This image on the far right, is taken from a classic cancer biology paper called The Hallmarks of Cancer. The premise of this is that there about 10 biological capabilities that cells take on as they like morph into this more cancerous state. And I’m not going to go over all of these, but I want to highlight that two of these biological capabilities would be like cell growth and like cell motility.
Tiffany Williams: Those are normal functions within the body. But I guess within a diseased state you’ll have overgrowth, or you may have cells that are primarily concentrated in one area, develop the capability to invade into other tissues or metastasize. I say all this to say that like in small molecule drug discovery, what we want to do is actually figure out what sort of structure is needed for a ligand to bind to this problematic protein or proteins, and counteract that like not so great behavior. If this is where Atomwise comes in, right? Or it does, if you don’t know. We want to know how can we efficiently explore this space of all potential chemical compounds to better identify small molecule drugs faster.
Tiffany Williams: Atomwise actually developed AtomNet the first deep convolution neural network for structure based drug design, so that we can actually make better predictions for potential drug candidates earlier in the drug discovery process and faster. I’m highlighting this paper, again, just trying to give high level overviews. Feel free to check this paper out, but what I want you to take away from this is that the AtomNet technology is currently being used in real drug research, in cancer, neurological diseases, various spaces, on the right is, it’s actually a GIF, but since it’s a PDF, it’s not showing up as a GIF, but basically this GIF would show the AtomNet model. It would simulate the AtomNet model, predicting candidate treatments for Ebola. And this prediction that AtomNet made is actually has led to candidate molecules that are now being studied in animal models.
Tiffany Williams: One, despite everything I’ve said up till now, you actually don’t need a background in wet lab research, or chemistry to appreciate what’s happening here. The power of convolute or the power of convolution neural networks, or CNN, is that it allows us to take these complex concepts as a combination of smaller bits of information. And I think if you’re familiar with CNN, or even if you don’t like one area that is really popular, has been computer vision.
Tiffany Williams: I’ll briefly go over like an example of image recognition, and then kind of like try to tie it into how AtomNet works. An image is essentially represented as a 2D grid with three channels, you have red, green, and blue. And this network learns images of objects or faces, for example, by first learning a set of basic features in an image like edges. Then, from there, by combining those edges, the model can then learn to identify different parts of that object. In the case of a face that might be ears, eyes, nose, et cetera. With AtomNet that it’s working in a similar fashion, that receptor ligand pair is represented in a 3D grid, and the channels are essentially the elements that make up protein like carbon oxygen, nitrogen, et cetera. In the case of AtomNet, that the learning of edges is actually the learning of the types of, or predicting the types of bonds between those elements.
Tiffany Williams: Then from there, the compliment to the ear eye detection would be actually identifying more complex molecular bonds. You could say, essentially, that this network is like learning organic chemistry 101. This is powerful because we can then train these models to make predictions about different aspects of the drug discovery process. Like what ligands, or what type of structures are most likely to bind to a certain target? At what strength? What are the additional modifications that we can make to a potential drug candidate to strengthen that bond? Beyond that, it’s not enough just for the ligand to bind to the receptor, it also has to be like biologically relevant in that, let’s say, if we’re looking for something that is treating some neurological disease. We need that ligand to be small enough to cross the blood brain barrier. Or we may need to take into account toxicity or other effects that may happen in the body. Metabolism. We don’t want that small molecule to become quickly metabolized in the body before it has the opportunity to have the intended effect.
Tiffany Williams: These are exciting times, and I’m really, really passionate about the work that we’re doing at Atomwise and any work that is being done at the intersection of health and technology. I wanted to briefly go over some of the projects that my team is currently working on. I work on the drug discovery team at Atomwise, it’s within the engineering team. I think we’re working on some pretty interesting issues. One of my, my teammates Shinji, he recently has been working on bringing best engineering practices, and improving performance of some of our ML tools. Adrian, my teammate, is working on optimizing algorithms to explore a three trillion chemical space. He is also been working on, or has been able to create simulating mocular… mocular, I’m combining words… molecular docking on GPUs.
Tiffany Williams: Then another thing that we’re working on that I actually have more of a hand in, is building a research platform to better enable drug discovery. Oh, I didn’t mention the third person Shabbir, who’s our manager, and he has his hands in a bit of everything. What I hope you take away from this is that I think we’re at an exciting time in today to like really leverage data, and technology to make a major impact in human health. I think there are a lot of challenges, interesting challenges in drug discovery. I hope that you may have been convinced that you actually don’t need a background in chemistry to contribute. There are a lot of transferable skills. If you just know software engineering, or you know ML, or if you’re in product or marketing, there is a place for you in this space.
Tiffany Williams: Finally, really important takeaway is that we are hiring. My team alone, please, if you have any front end experience, if you have backend experience, or if you have a background in computational chemistry, those are some of the positions that are open right now on my team, but then outside of my team, we also are hiring. Definitely check out our careers page. If you have any more questions or are interested in chatting, feel free to reach out to me. I have my LinkedIn handle as well as my Twitter handle, posted here. Then finally, I think I mentioned that I had some references to share. Again, I’m only scratching the surface. There’s so much information out there. I wanted to highlight two Medium articles that were written by former Atom, which is what we call people that work at Atomwise, machine learning for drug discovery, in a nutshell, I highly recommend starting there if you want to do a deeper dive. That’s it.
Julie Choi: Thank you so much, Tiffany. Yes. I think drug discovery is an incredible application domain for deep learning. Really appreciate your talk. Okay, let me introduce our next speaker. Our next speaker is Shelby Heinecke. Shelby is a senior research scientist. Again, I did not touch. Okay.
Julie Choi: Shelby is a Senior Research Scientist at Salesforce Research, developing new AI methods for research and products. Her work spans from theory-driven, robust ML algorithms to practical methods, and toolkits for addressing robustness in applied NLP and recommendation systems. She has a PhD in Mathematics from the University of Illinois at Chicago, and a Master’s as well in Math from Northwestern and her bachelor’s is from MIT. Let’s welcome Shelby.
Shelby Heinecke: Thanks so much. Awesome. I have to give a thank you to Julie for including me in this event, inviting me. This is my first Girl Geek, and not my last Girl Geek event. I’m super excited to be here. Yeah, let me get started.
Salesforce Research Senior Research Scientist Shelby Heinecke speaks about how to evaluate recommendation system robustness with RGRecSys at MosaicML Girl Geek Dinner. (Watch on YouTube)
Shelby Heinecke: Today I’m going to be talking about evaluating recommendation system robustness, but first I feel kind of like the new kid on the block. Let me just give a little bit of background about myself. I moved here to the Bay Area about a year and a half ago in the middle of the pandemic. Super excited to be here in person to meet people.
Shelby Heinecke: Currently I’m a Senior Research Scientist at Salesforce Research. As Julie mentioned, I work on both research and product. It’s pretty awesome to develop prototypes, and see them in production. Before I was here at Salesforce in the Bay Area, I was doing my PhD in math in Chicago. There, I focused on creating new ML algorithms that were robust. I worked on problems in the space of network resilience. Before that I got my master’s in math, kind of focusing on pure math at Northwestern. Originally I hail from MIT Math, focusing on pure math there. That’s my background. Today’s talk recommendation systems and robustness.
Shelby Heinecke: Let’s get started. A crash course and recommendation systems. So, what is a recommendation system? Well, it consists of models that learn to recommend items based on user interaction histories, user attributes, and or item attributes. Let me give you an example, say we want to build a recommender system that recommends movies to users. Well, what kind of data can we use? We’re going to use the users, previous movies. They viewed we’re going to use the ratings that they rated those movies.
Shelby Heinecke: We’re probably going to take a look at the user’s age. We’re going to look at the user’s location. We’re also going to take into account item attributes. The movie attributes, like the movie title, the movie genre, we’re going to take all that. That’s all of our data, and we’re going to train models. The models can build the recommendation system. As you can imagine, recommendation systems influence our daily lives. We’re all exposed to recommendation systems every day. Just think about purchases. Think about movies.
Shelby Heinecke: Think about songs you’re recommended. Think about the ads that you see every day, people, news, information. We are at the mercy of these recommendation systems and a lot of our decisions are highly influenced by what the recommendation systems decide to show us. Let’s think about the models that we see in recommendation systems. Models can range from simple heuristic approaches, like a rules based approach or co-sign similarity approach to complex deep learning approaches, think neural collaborative filtering, or even now we’re seeing transformer based approaches coming to light.
Shelby Heinecke: With the vast range of models available and how greatly they impact our daily lives, understanding the vulnerabilities of these models of each of these types of models is super critical. Let me get started about recommendation model robustness. As we all know, machine learning models are trained on data, and ultimately deployed to production. And in that process, there are some hidden sources of vulnerabilities that I want to bring to light.
Shelby Heinecke: One of the big issues is that training data may not reflect the real world data. In many cases, we’re training on data that’s been highly curated. That’s been cleaned up. And yeah, and so because of that, when we train a model on that very clean, highly curated data, it’s not going to necessarily perform well when it’s exposed to the messy data of the real world. Real world data has noise. Real world data is just can be unpredictable. As a result, sometimes we train model, we train a recommendation model, but we see poor performance and production. That’s definitely one type of vulnerability we need to watch out for.
Audience Member: Woo! So true.
Shelby Heinecke: What? Okay. Another type of vulnerability. I love the enthusiasm, okay? Another type of vulnerability. As you can imagine, recommendation systems are closely tied to revenue for a lot of different parties for companies, for sellers. There’s an issue that participants may intentionally manipulate the model. Think about creating fake accounts, trying to do things, to get your item higher on the list for customers, things like that. That is a reality and that’s something we have to take into account.
Shelby Heinecke: The last thing I want to bring to light is poor performance on subpopulations. This is a well known issue across all of ML, but I just wanted to bring it to light, to recommendation too, that when we train models and we test on the evaluation set, usually the basic evaluation methods think precision recall F1. We’re computing that on the entire test set, so we’re averaging overall users. And in that average, sometimes we’re hiding…
Shelby Heinecke: Sometimes there’s poor performance on subpopulations that are hidden. For example, a subpopulation that you may care about could be new users, or maybe a users of a certain gender. That’s something that we just need to keep an eye on. We don’t want poor performance on key subpopulations I’ve told you about all these different types of vulnerability.
Shelby Heinecke: What is a robust model? Well, a robust model you can think of it intuitively as a model that will retain great performance in light of all of these potential perturbations, or in light of all of these scenarios. The question is how can we assess the robustness of models? That is where one of our contributions comes in.
Shelby Heinecke: I want to introduce one of our open source repos called RGRecSys, and it stands for robustness gym for recommendation systems. Our library kind of automates stress testing for recommendation models. By stress testing, I mean, you can pick a model, you can pick a data set, and you can stress test it in the sense that you can manipulate the data set.
Shelby Heinecke: You can add in attacks, you can add in noise and so on. I’ll actually go into more detail about that. And you can see in a really simple way how your model, the robustness of your model. As I mentioned, RGRecSys, is a software, is a software toolkit. It’s on GitHub. I’ll talk about that in a second and it’s going to help you assess the robustness of your models.
Shelby Heinecke: One thing that we contribute is that RGRecSys provides a holistic evaluation of recommendation models across five dimensions of robustness. When I say I told you about various types of vulnerabilities, there’s various types of robustness. Our library helps you to quickly and easily test all these different types of robustness for your model. Using our API, you just simply select a model for testing, and then you specify the robustness test that you’re interested in trying along with the robustness test parameters. What I’ll do is I’ll go over the types of tests that are in the library.
Shelby Heinecke: Let’s talk about the different tests that we have in our library. First is around subpopulations and this kind of goes back to what I mentioned in the previous slides. This will allow you the test. What is the model performance on subgroup A versus subgroup B? And this is going to be very useful because, as I mentioned, most of the time, we’re just computing precision recall these usual metrics on the test set, but this gives you an easy way to test performance on specific subgroups. This could be useful, for example, if you want to test you want to test performance on gender A versus gender B, or new users versus old versus like users that have been in the system for longer time. Just some examples there. That is one type of test you can run in our library. The second type of test is around sparsity. If you think about recommendation systems and you think about the items available like a movie recommender, or a purchasing recommender there’s millions of items, and each user really only interacts with a handful of those items.
Shelby Heinecke: Each user is only clicking, only purchasing, only viewing a handful. This is a source of data sparsity. Data sparsity is a huge problem in recommendation systems. It will be good to test the degree to which your model is sensitive to sparse data. That’s one thing you can test with our library. The third test is around transformations. There are a lot of ways that data can be perturbed when you’re training a recommendation system model. For example, maybe you’re gathering data about your users and maybe that data is erroneous in some ways. And because of that, you might want to test if a recommendation model will be robust, if user features, for example, are perturbed. The fourth test that you can test is around attacks. As I mentioned, there’s a lot of reasons why people would try to attack a recommendation system it’s tied to revenue ultimately.
Shelby Heinecke: What you can do with our library is implement some attacks and test how your model performance would change under those manipulations. And finally distribution shift. What we see is that the training data that you train on is often different from the data that you’ll see in production. It’s super important to be able to know, get a sense of how was my model going to perform if it’s exposed to data from a slightly different distribution? You can go ahead and test that with our library., I definitely encourage you to check out our library on GitHub, and feel free to check out the paper for way more details about the capabilities. And with that, thank you so much for listening. It was great to share it. Feel free to reach out.
Julie Choi: Thank you, Shelby. That was great. It’s very important to be robust when we’re doing model deployment. Okay. Angela Jiang, thank you so much for joining us tonight.
Julie Choi: Angela is currently on the product team at OpenAI. Previously, she was a product manager at Determined AI building, deep learning training, software and hardware, deep learning… I think Determined AI was recently acquired by HPE. And Angela graduated with a PhD in machine learning systems from the CS department at Carnegie Mellon University. And we actually have built some of our own speed up methods on your research. It’s an honor to have you here today.
Angela Jiang: Thank you so much, Julie, for inviting me, for the introduction, as well as Sarah, and Angie for organizing the event and bringing us all together too. Yeah. Like Julie mentioned, I’m Angela. I work on the product team at OpenAI.
OpenAI Product Manager Angela Jiang speaks about turning generative models from research into products at MosaicML Girl Geek Dinner. (Watch on YouTube)
Angela Jiang: I work on our Applied team where we really focus on essentially bringing all of the awesome research coming out of the org and turning them into products that are hopefully useful, safe, and easy to use.
Angela Jiang: Most of my day, I’m thinking about how to turn generative models from research into products. I thought I’d make the talk about that as well. This might as well be a list of things that keep me up at night, think about it that way. But in particular, what I really wanted to highlight is just some of the observations I’ve had about things that make these generative models unique and tend to have large implications on how we actually deploy them into the market.
Angela Jiang: To start, I want to share a little bit about what OpenAI’s products are. OpenAI does a lot of AI research. In particular, we do a lot of generative models. Over the last two years, we’ve really started to work to bring those generative models into real products. Three examples here are GPT-3, that does text generation; Codex, that does code generation; and most recently, DALL-E, which does image generation. These were all made by DALL-E.
Angela Jiang: To get a little bit more concrete, what these products are is that we expose these models like GPT-3 and Codex via our APIs. As a user, you can go to the OpenAI website, sign up, and then hit these endpoints and start using these models. And here’s an example of how you might use the GPT-3 model. Here we have an example of your input in the gray box. It might say something like, “Convert my shorthand into a firsthand account of the meeting and have some meeting notes.” And you submit this to the model, and then the model will do its very best to return you an output that completes this text. And in this case, the response is essentially a summarization of your notes.
Angela Jiang: I think GPT-3, Codex, and DALL-E are all really good examples of taking research and seeing that there’s a really big market need for the kinds of capabilities that they expose. Then working as an Applied team to actually transform that research into a productionized product by essentially figuring out where that user value is, designing the product so that we can deliver that user value in a way where users are set up for success, making sure it’s deployed responsibly, et cetera, et cetera.
Angela Jiang: To date, there’s hundreds of applications that have been submitted for production review that is built upon this API. Those are applications like writing assistants like CopyAI if you’ve heard of that, coding assistants like GitHub’s Copilot is built on top of Codex, as well as a lot of other applications like chatbots for video games, question answering bots, etc. This is pretty good validation that these models are not only really exciting research, but are also solving problems in the market.
Angela Jiang: Now, when we think about bringing these to market and turning them into real products, I think there are two interesting properties of these generative models that really change who uses them as well as how we actually deploy them. I’ll talk about how these models are stochastic and how they’re very general. With typical products, you might expect that the results are very predictable and deterministic. For instance, if you’re coding up a payments API, every time you use that payments API, you’d expect it to react in a similar way. This is not how our models work at all.
Angela Jiang: Our models are probabilistic. Every time you submit a prompt, you could get a different response back. If you tweak that prompt ever slightly, then you could get a very different response back. That really changes how you interact with a product and what kind of applications can be built on top of it. Another difference is I think that typical software products that I think about are typically focused on solving a need or a problem very directly, whereas, in these generative models, they’re very, very general. They’re capable of doing a lot of things all simultaneously.
Angela Jiang: For instance, you would think that GPT-3 would be able to summarize legal texts and also simultaneously write poems or maybe even code. I think this is super exciting from a science perspective as we get more and more powerful models, closer and closer to something that’s super general. But from a product perspective, this comes with some challenges because we have to now take one model and one product and have it serve many, many use cases simultaneously.
Angela Jiang: For the rest of this talk, I’ll talk about, again, more detail about how these two properties affect the way that we deploy our products, and I’ll give some examples of each. Okay, starting with stochasticity. Right. Our models, again, are probabilistic, which means that every time you use it, you might get a different result. The result might not be what you want or maybe the result is what you want every third time you try the model.
Angela Jiang: It might be surprising that you can actually build real applications on top of this kind of behavior. What we actually find is that for a lot of tasks, especially very simple tasks, we can have very concrete and reproducible behavior for even tasks where the outputs are imperfect or are only correct some of the time, these results are actually still very useful for many applications in the right context.
Angela Jiang: We find that these models work really well, for instance, in creating productivity tools where there’s a human in the loop. A really good example of this is Codex Copilot. Some of you may have used it before, but this is a screenshot of it in action, where it’s a Visual Studio Code extension and it just gives you auto-complete code suggestions that the user can then tab to accept or reject.
Angela Jiang: This is great for a product like Codex because you have actually an expert there that’s telling you, is this completion good or not? And generating more. What we actually see is that most of the suggestions from Copilot are rejected by the user, but still, developers tell us that this is an integral part of their deployment pipeline or their development pipeline I should say. It does not need to be perfect every time we generate something. Those are cases where you don’t need a perfect result every time. What you’re looking for is just inspiration or getting over writer’s block. In some of the applications built on top of these APIs, you actually don’t have a correct answer to shoot for.
Angela Jiang: Applications that are doing art or creative tasks or entertainment, that’s a case where having variety really is helpful actually. Here I have a DALL-E prompt, which is a bright oil painting of a robot painting a flower. And here are different generations from DALL-E. And in this case, it’s really helpful actually to have probabilistic nature because, for one, I want to have different options. And it’s also really cool to have generations that you’ve never seen before or other people haven’t seen before.
Angela Jiang: I think it’s even counterintuitive that, in this case, we’ve actually learned that these AI systems seem to be doing really, really well and surprising us in creative tasks as opposed to rote tasks, which you might have expected the opposite earlier. Those are some examples where the properties of these generative models affect what applications are built on top, but it also affects how we deploy these models. Going into a little bit more detail, one property is that these products are really, really hard to evaluate.
Angela Jiang: This makes being a Product Manager, among other things, quite challenging because we really need to know how our products perform so we can position them in the correct way to the users, know what to deploy, and tell users how to use them correctly. For example, we have these text models, GPT-3, and they have different capabilities, right? One capability is they might be able to complete text, and another capability is they might be able to edit existing text. There’s some overlap there on which one you should use.
Angela Jiang: It’s really important that we understand exactly how this model performs on different tasks so that we can direct users to use the right tool at the right time. We try really hard to figure out creative ways to evaluate these at scale, given that we often need a human in the loop to be telling us if a generation is good or not.
Angela Jiang: We do things like large-scale application-specific A/B testing. Like we see if we use one model or the other, then which model gets more engagement for this writing assistant.
Angela Jiang: Next, I’ll just give a couple of examples of how generality also comes into the picture here. Like I mentioned, these models are capable of a lot of things simultaneously because the way that they’re trained is that they’re trained for a very long time, for months, on a lot of data. And the data spans the internet, books, code, so many different things.
Angela Jiang: By the time we get this model as the Applied team, we’re really still not sure what kind of capabilities it’s picked up at that time. But like I mentioned, it’s really important that we understand that quickly. This is also really exciting because, at any given time, we can always discover a new capability of the model, and it’s often discovered after training.
Angela Jiang: An example of this is that the original GPT-3 model was built with text for text. And it was really a discovery after the fact that it also happened to be kind of good at code as well. And these signs of life and discovery is what ultimately sparked this idea of a Codex series that specialized in code.
Angela Jiang: As a Product Manager, it’s really critical that we keep on discovering and probing these models to really understand what’s the frontier of what they can do. And again, we are still figuring out what is the best way to go about this. But something counterintuitive that I realized after I started is that traditional user research that we’re used to doesn’t actually work really well for this use case because you might expect that you would go to your users and ask, how is this model? What can you do with it?
Angela Jiang: What we find is that our users, by definition, are working on tasks that the model was already really capable at. Their focus is not on the new capabilities of the model and pushing that frontier oftentimes. What’s worked better for us is working with creatives or domain experts to really hack and probe at the model and see what the limits of it are.
Angela Jiang: That’s the exciting part of generality, that there’s always a new capability around the corner to discover. But there’s also some risk to it because there’s not only all these capabilities in the model, but the model can also generate things that are not useful for you, and it can also generate things that you really don’t want to generate.
Angela Jiang: For instance, GPT-3, when prompted in the right way, can generate things like hate speech, spam, violence, things you don’t want your users to be generating. This is also a big part of our goals as a research and applied organization is to figure out how to deploy these models in a way that doesn’t have toxicity in them.
Angela Jiang: There’s a lot of different approaches for doing this, spanning from policy research to product mitigations to research mitigations, but I’ll highlight two things here. One thing that’s worked really well for us is actually fine-tuning these models after the fact with a human in the loop, telling you exactly what kind of content is good content.
Angela Jiang: What we found by doing this is that we have a much better time at having the model follow the user’s intentions. And we have a much less toxic and more truthful model as a result of it. At this point, every time we deploy a model, we also fine-tune the language model in this way. We also do a lot of research and provide free tooling to help the users understand what the models are generating at scale so they can understand if there’s anything that they need to intervene in.
Angela Jiang: Okay. Finally, even though I think it’s very, very cool that these models can do a lot of different things at once, sometimes it really just can’t serve multiple use cases simultaneously well. Different use cases often will just require, for instance, different ways of completion.A chatbot for children is going to want a very different personality than a chatbot for support.
Angela Jiang: You also just have different product requirements in terms of accuracy or latency or even price. Copilot is a really good example of something that needs an interactive latency so that it can continue to be useful in real time. But contrast that with something like SQL query generation, which doesn’t need a fast latency, but actually just wants the most accurate response that you can give it. What we’ve found is that the combination of two things have allowed us to provide this flexibility to serve the use cases that we need to serve.
Angela Jiang: One is that we don’t just expose the best model that we have or the most accurate model we have I should say, we expose models with different capability and latency trade-offs for the users to choose from. And then we also offer fine-tuning as a first-class product so that you can take a model and then fine-tune it so that it has your personalized tone or your personalized data bank.
Angela Jiang: These are, hopefully, examples that give you a flavor of what it’s like to deploy generative models. This is also just the tip of the iceberg. If any of this stuff is interesting, please feel free to come and chat with me. I should also mention that we are hiring. We are. Thank you so much.
Julie Choi: Thank you so much, Angela. After DALL-E was launched, the productivity went a little down. We were so distracted by the DALL-E. I don’t know whether to thank or curse your team for that.
Julie Choi: Thank you, Banu. It is just a joy to introduce our next speaker. She’s a friend of mine and a former colleague, really an all-star. It’s so wonderful to have you here, Banu. Thank you.
Julie Choi: Banu Nagasundaram is a machine learning product leader at Amazon Web Services where she owns the go-to-market strategy and execution for AWS Panorama, an edge computer vision appliance and service. Prior to AWS, Banu has spent over a decade in technology, building AI and high-performance computing products for data centers and low-power processors for mobile computing. Banu holds a Master’s in EECS from the University of Florida and an MBA from UC Berkeley’s Haas School of Business. Let’s all welcome Banu.
Banu Nagasundaram: Thank you, Julie and team for having me here. I’m super excited to be here. One difference from the other speakers is this is not my area of expertise, the title of the talk. It is something I’m trying to do better at that I wanted to share with you.
AWS Product Manager Banu Nagasundaram speaks about seeking the bigger picture as a ML product leader at MosaicML Girl Geek Dinner. (Watch on YouTube)
Banu Nagasundaram: I’m trying to seek the bigger picture at work. I’m a Product Manager and I’m trying to see why companies do what they do and learn more in that process. What I want you to take away from this is how you can also seek the big picture in your roles that you do either as engineering or product leaders.
Banu Nagasundaram: With that, I wanted to share a little bit about the companies that I work with on a daily basis. These are concerns who use machine learning and AI and work with AWS to implement the services in production. This is different from the research that we spoke about.
Banu Nagasundaram: These companies are looking at getting value out of these systems that they put in place, of course, based on the research, but taking into production. What I implore you to think about is put on a hat of a CTO or a CIO in each of these companies and think about how and why you would make the investment decisions in machine learning and AI.
Banu Nagasundaram: For example, I work with healthcare and life sciences team. I learned a lot from the drug discovery talk earlier from Tiffany here, but I do work with healthcare and life sciences team to understand how they can take the vast amounts of health data that they have to translate into patient information that they can use to serve patients better.
Banu Nagasundaram: They use multiple services to personalize, to extract value from the text data, a lot of unstructured data that they have. That’s one category of customers.
Banu Nagasundaram: The second type of customers that I work with include industrial and manufacturing. The key component that they’re trying to improve is productivity and also optimizing their manufacturing throughput.
Banu Nagasundaram: The questions that they ask and they seek to improve include automating visual inspection. How can I improve the product quality across my manufacturing sites? I have thousands of sites in the US. I scale globally. How can I implement this process not only in one site but uniformly across those thousands of sites to achieve something like predictive maintenance on the tools, improve uptime of their equipment, etc?
Banu Nagasundaram: Third set of customers we work with include financial services. They are looking at data to improve or reduce the risk in the decisions that they make. They’re trying to target customer segments better so they can understand underserved populations but lower the risk in making those products and offerings that they want to do.
Banu Nagasundaram: They also look into fraud detection and many applications around financial services. Retail, this is one I work closely with because I work in the computer vision team right now. And retail is trying to use the insights from computer vision products to see how they can reduce stockouts, which is basically when you go to the store, is the product available? Can they sell it to you? How can they manage inventory? Can they keep track of the count or the number of people entering the store?
Banu Nagasundaram: You may have heard about Amazon Go, for example, a store with just a walkout experience. A lot of retail companies are working with us to understand how they can use computer vision to build experiences like that.
Banu Nagasundaram: And it doesn’t stop at a retail store. Think about the operations officer, a centralized person who’s sitting and trying to analyze which region should I invest more on? Which region should I improve security? Should it be in North Carolina? Should it be in California? Or should it be in a whole different country? They’re trying to collect and gather insights across their stores regionally, nationally, internationally to make those decisions.
Banu Nagasundaram: Then there’s media and entertainment too, which we touched upon a little bit around recommendation systems and personalization. Here we work with customers who are looking to improve monetization, who are looking to create differentiation in the marketplace, in the very highly competitive media and content marketplace, through those recommendation systems and personalization that they have.
Banu Nagasundaram: Across all of these customers, the core task as a product manager that I work on is understanding their requirements and then translating it into product features so that they are served better. But what I learned in the process is that it’s so much more in decision-making than just understanding about features or product requirements.
Banu Nagasundaram: It’s about what enables these CTOs to make those decisions is value creation. How can they use all of these AI ML systems to realize value from the systems that they put in place? One simple way to think about what value creation is, is that it’s an aggregation of data, analytics, and IT that brings the machine learning together. But there’s a second part to it, which includes people and processes.
Banu Nagasundaram: What I mean by that is all of this analytics and machine learning and data can help them understand something, but they still have to lean on people to analyze the data that they gather, make improvements in the process in order to recognize and create value. And that’s the workflow for decision-making across all of these companies.
Banu Nagasundaram: We can look at this as two buckets, one in data analytics and IT, and the second one as people and processes. For the computer vision product that I own, I wanted to talk to you about the value chain for the first part, which is the data analytics and IT portion.
Banu Nagasundaram: This might look a complex set of boxes. It is. Once I finish the pitch here, if this excites you, I’m hiring too, growing my team. Hopefully, I do a good job in explaining this value chain.
Banu Nagasundaram: I started my career in the bottom left, by the way, in silicon processor design. Pretty much in the bottom row, trying to understand how silicon design is then aggregated into components and how distributors sell those components to OEMs who are equipment manufacturers, and how from those equipment manufacturers, the equipments reach the consumers, which is through those equipment distributors.
Banu Nagasundaram: My product is currently both an appliance and a service. I do start from the silicon side, working with partners, I can give examples like Nvidia on the silicon side, Lenovo on the OEM side. And then once you have these equipment distributors selling these devices or equipment I should say to infrastructure providers, one of the examples of infrastructure providers is cloud service providers, but it can also be on-prem equipment providers.
Banu Nagasundaram: You then go to the infrastructure providers, which is the second row from the bottom, and these infrastructure providers then build the tools and frameworks either themselves or through the partner ecosystem. Those tools and frameworks essentially put in place to make efficient use of that underlying infrastructure. It’s a motivation for these infrastructure providers to offer the best tool and frameworks so that you can gain that value out of that underlying infrastructure.
Banu Nagasundaram: Then comes the ML services. You have the overall MLOps flow is what would fit in this bucket. Once you have the tools and the frameworks, how can all of these pieces get grouped together to build a robust system that can scale in production? This goes from data annotation, labeling, training to predictions, to model monitoring. “How can you maintain this when it is in production” is a key question for these decision-makers.
Banu Nagasundaram: Then comes the AI services that are built on top of these ML services. AI services can either be services offered by CSPs or it can be services offered by startups or companies who are trying to build the microservices or services on a specific use case. This is where the customer use case comes into play, where you have that specific use case. In case of computer vision, you can think of services like Rekognition or Lookout for Vision. Those are two examples that AWS offers for computer vision services.
Banu Nagasundaram: Then comes the final layer. This is the layer that the customers that I refer to, the CTOs, work closely with. The independent software vendors or ISVs are companies that build these software solutions, aggregating everything that I spoke to you about, but building the software components of it. But the software component by itself is not going to function in a customer’s premise. For the customer to realize value, the solution, the software solution for their use case has to integrate with their existing system. That’s where system integrators come into play.
Banu Nagasundaram: For example, Deloitte. You can think of Deloitte, Accenture, et cetera are system integrators who bring this whole puzzle together for customers, build that reference architecture for that solution. And they’re like, okay, so now we have this architecture in place. Then they bring in the value-added resellers, who are companies like Convergent or Stanley, for example, who take this entire system that’s put in place and roll it into individual sites.
Banu Nagasundaram: This is where it reaches the scale of thousands of sites. Once a solution is put together, proven in a pilot or a production pilot or a proof of concept, when you go to production, across globally, across cities, across countries, the value-added resellers roll the system into place in the customer’s site. But it doesn’t end there. There need to be managed service providers who can offer service contracts to maintain the system in place once it is on a customer’s premise.
Banu Nagasundaram: All of these building blocks in the value chain is what makes ML, machine learning, in production, intangible for a customer to realize value. It is a big journey. And this is the team I’m building who will work with individual partners across this value chain. And if this is something that excites you, we can talk after. In that value chain, we started with value creation. What is it that companies are looking for? We saw the value chain, but the value realization, let’s say the companies went through this process, put this whole system in place.
Banu Nagasundaram: What is it that actually helps them realize that value? When we think about data analytics in IT, many of the ML practitioners that I talk to, talk to me about the output, the visualization, the dashboards, the histograms for decision making, but it doesn’t stop there. That is not sufficient for these companies. There has to be people in those companies who take this output and actually work to achieve an outcome. The outcome is increase in productivity, increasing in throughput. It’s like, if I can know my demand better as a retail store, or if I can forecast the demand better, I can do so much better in my business.
Banu Nagasundaram: That is an outcome that I’m looking for from this data output that I’m getting from the machine learning models. But is that outcome sufficient? No, the outcome is like one place, one time, you are able to visualize that outcome, but you have to scale that outcome globally in order to achieve the impact. The impact that businesses look for is that you have to either increase revenue, reduce costs, reduce their risk, improve their sustainability, or create a competitive advantage. This is what the ML journey in production looks like. While I walked you through incrementally from the beginning on what it would take customers to get there, customers don’t work through individual steps, reach there and see what’s the next step.
Banu Nagasundaram: They actually have to make assumptions along the way and then understand what the impact might be and this is an AWS term that I’m going to throw and work backwards, which is understand what is it that you want and then build out what is it that you can, what you need to do towards achieving that end goal. That’s the big picture that I want to leave you with. In this whole ecosystem of having machine learning in production, can you think big on behalf of the customer, can you seek the big picture that the customer is looking for? If you’re working on a feature, what is the end state of that feature? What is the end state of that business? Who is actually your end customer? Your end customer may not be the team that you’re immediately working with and who is the decision maker for that overall flow?
Banu Nagasundaram: One of the simple frameworks, this might sound silly. It’s super simple, is to ask five whys, which is in a customer discovery or any feature that you’re building. Just ask, why is that outcome important? Why is that output important? How is it going to help? Why is it going to be something that helps the customer? Why is it needed tomorrow? All of these questions is just going to help drive a little bit more clarity into the bigger picture and motivations for the customer on how they make investment decisions and choices in your particular products and features that you’re building. And that’s it. I want to leave you with one fair warning. If you try that five why’s with your partner, that’s on your own.
Julie Choi: All right. Thank you. Banu, that was great. Thank you so much. It is my honor to introduce Lamya Alaoui, a dear friend of mine and I’m so thankful to you, Lamya, for agreeing to give the closing talk of the night. Lamya is currently Chief People Officer at Hala Systems, and she has been committed throughout her career to supporting organizations as they shift behaviors to align their talent strategies with their business objectives. Her corporate background includes over 15 years of experience in talent acquisition and management, where she has had the opportunity to build teams for companies such as Bertelsmann, Orange, Groupon, Google and Microsoft. Her work experiences span North America, Europe, Asia, Middle East, and North Africa. Let us welcome Lamya Alaoui.
Hala Systems Director of People Ops Lamya Alaoui talks about 10 lessons learned from building high performance diverse teams at MosaicML Girl Geek Dinner. (Watch on YouTube)
Lamya Alaoui: Thank you so much, Julie. Hi everyone. Thank you. I don’t see Sarah here, but I want to thank her personally because she has been so patient and Angie as well for inviting me here. How is everyone feeling today? This is the non-tech talk. This is the people talk. A quick background before we get into it. At Hala System we develop early warning systems in war zones. Not the type of things that you broadcast usually when you are in this type of settings, but we’re hiring as well, especially for our AI team. And Julie announced my promotion that no one else knows about, even in the company so, thank you. That stays here, please, in this room, until next all hands, on Wednesday. With that being said, a little bit of background about me. I’m Moroccan.
Lamya Alaoui: I moved to the US about 10 years ago. I will ask a lot of grace because my brain is wired in French. Sometimes there will be French words that will come out from my mouth so please be graceful about it if you can. With that being said, this is one of my favorite things because as a Moroccan who’s half Muslim, half Jewish, went to Catholic school, I thought I got it covered, but I was the lady the first time I landed in Germany because I was not aware that you’re not supposed to kiss people to greet them. But this is one of my favorite pictures to show and talks or even in workshops, because I’m pretty sure it happened to all of us one way or another. We show up and we think that we got it right. It’s what we’re familiar with and actually it’s not what the other person is expected.
Lamya Alaoui: Throughout my career, I have built teams in very, very different countries, different cultures. At Hala we have over 17 nationalities. Altogether we speak 25 languages. It makes it interesting for the meetings, believe me, where you have side conversational Slack in a whole different language, but you still need to deliver officially in English. The translation is super weird sometimes, but those are anecdotes for another time. That leads me to one of the lessons that I learned very early on. It seems obvious that when you’re building teams and you want them to perform, one might assume you want to know your mission and your values, but it doesn’t happen as often as one might think. The mission is basically your GPS. You want people to rally behind a common goal.
Lamya Alaoui: You want them to believe in that and it’s also what’s in it for them when they’re working for a mission. That means that it needs to be aligned with their own values or their belief system. SVlues start very early on. We all have seen or worked for companies where the values are stated. How many of you when you joined the company saw values listed on the website or have been talked to? How many of you were explained to what are the associated behaviors and expectations when it comes to those values? Oh, we have less people. This is one of the first lessons that I always recommend to people to kind of follow. I learned the hard way, by the way, is that know your mission, state your values. Values should come from the leadership of course, but also account for it in the hiring process, which we will get a little bit into it, but be very clear about what are your expectations are when it comes to values.
Lamya Alaoui: If I’m thinking about transparency, for me, it means having information that I need to do my job, but some other people, they think that they need transparency. They need to have access to everything. We all have someone like that in our companies. Don’t we? Setting those clear expectations and associated behaviors are very important. And mission is like literally your North Star. This is why it’s really important. The second one is seek first to understand. This is from Stephen Covey. There is a second part to it which is, then to be understood. When you are building diverse teams, we all come from different backgrounds, we have different understandings, often we speak different languages. We can also come from different cultures, high context or low context and if we don’t have mutual understanding we cannot succeed because then everyone is convinced that they’re right, that their approach is again the one that everyone needs to follow, but that kind of hinders teamwork, which is actually essential to high performing teams.
Lamya Alaoui: Listen. We all think that we listen pretty well, but we actually don’t because as human beings we have been trained, for the last few centuries, to listen to answer. We’re almost never listening for the sake of just listening and there is amazing research about that, that I will be more than glad to share. In working in very diverse teams or diverse teams in general, listen, help, create understanding, respect, which is a very important when it comes to having high performance teams as well. How many of you would say that they listen really, really, really well. Okay. We have four people in the whole room. That gives you something to think about. That’s another, and I’m a good listener, but still a lot of work to do with that.
Lamya Alaoui: Acknowledge that you will face cultural differences. When you join a new team, we all want to get along. Again, its human nature. We want to belong somewhere. We’re super excited. It’s a new job. We just went through the hiring process. We landed the job and then you show up in that meeting or you’re meeting your team and sometimes you see people that you have nothing in common with, and you still try to ignore that, which is again, human nature. Beautiful thing. Just as a manager, as a team member, be prepared that yes, we will have some challenges. We’re not seeing things the same way. One of the best examples that I always give here is like, I’m from what would qualify as a high context culture, which is there is no explicit direct messaging.
Lamya Alaoui: Basically I can be in my office and say, hey, I have a lot of work and I’m hungry. My expectation is that my coworker will just get it and go get me some food. Now, imagine if I’m having someone from a low context culture. In their head, they’re like, I mean, you’re going to go get your food when you’re done. That creates a fracture that it’s not even intentional because there will be resentment from my end. Why didn’t he get it? Having those conversations upfront is quite helpful because then people know again, what to expect. Assess the degree of interactions. When teams are being built, most of the time what happens is no one is thinking about how often the team members will have to work together, at what intensity.
Lamya Alaoui: Before starting to build the teams or adding members to your teams, just make sure that, hey, how often do they have to have, I don’t know, sprints. If it’s once a month, eh, you might want to have people that are pretty much aligned having somehow the same thought process and we’ll get to that in a little bit. If they’re not interacting a lot, you have a lot more room to assemble your team. Always, there are three degrees of interactions. We have low interdependency, which is people are doing their things on their own and sometimes it’s like going in a [inaudible]. Medium, which is a hybrid and then high interdependency, which the output from someone becomes the input for someone else.
Lamya Alaoui: Therefore, things need to be structured in a very certain way in making sure that people get along and understand, again, what’s expected from them. Communicate, communicate, and communicate and when you think you are done, communicate some more. One of the things about communication is that we all think that it happened. Again, the question, oh, disclosure, I ask a lot of questions. I mean, I used to be a recruiter. It comes with the territory basically. How many of you thought in a meeting that you were crystal clear and then two days later you discover that you were not.
Lamya Alaoui: When it comes to communication in teams, again, thank you, Slack, everyone in meetings, or usually where everyone is doing a lot of things at the same time. You’re talking, you think that you got your point across. Always ask a question at the end. What do you think? What are your takeaways? Just to make sure that people are on the same page and teams, high performing one’s, one of the best practices is having action items assigned, even if it’s an informal meeting or if a conversation happened on Slack, on, I don’t know, any other platforms. Even sometimes now, oh, we have also Asana now people add us to Asana and they think that, hey, we’re good.
Lamya Alaoui: No, make sure that the person actually went in, is understanding what you’re waiting for or expecting. Diversity and leadership. Clearly icons still have a long way to go in terms of diversity, but again, high performance teams when it comes to diversity need to see that diversity reflected in the leadership and in the management. Research shows that usually entry level management, there are sometimes good numbers, but as soon as we go to director, VP and above underrepresented groups tend to become less visible or they’re non-existent and C-suites and VP, EVP, SVP type of roles.
Lamya Alaoui: It’s also very important that the leadership reflects the diversity that the company is striving for. Again, an obvious one, hire the right people. This one is very dear to my heart because as I said, I’m a recruiter. Again, we’re only humans. We tend to hire people who are like us, who think like us, who share our values. Anyone wants to venture what happens when you hire people who are like you and your team and you’re building a team of clones, basically from a cognitive perspective. Anyone? Yes.
Shika: You wouldn’t get to learn something different from what you already know.
Lamya Alaoui: Yes. Anyone? Thank you so much. What’s your name?
Lamya Alaoui: Shika. Thank you, Shika for volunteering. When you build a team of clones and this is a tech talk after all, how do you think that you can innovate? How do you think you can perform if everyone is thinking the same way and not challenging the ideas that are being discussed? Chances are very, very, very slim.
Lamya Alaoui: When you’re hiring the right people for your teams you have to look at three things. The first one is the values, because that’s, again, what will be the foundation of the team that you’re building. The second one is what’s missing from your team. An ideal team has five type of people. You have a theorist, you have a strategist, you have someone who is an analyst, you have a manager and you have an implementer. When we hire our teams usually we hire in desperation because you know, you got to deliver on that project.
Lamya Alaoui: No one is thinking about the composition of the team in itself. In tech, my experience showed we have a lot of theorists and strategists and a lot of analysts, sometimes zero implementer and zero managers. Things don’t get done somehow. Always look at what’s missing and finally, the third part is people who are willing to learn, who have a curious mindset and are eager to grow, because this is how actually innovation happens. People who are not afraid basically of questioning the status quo.
Lamya Alaoui: The other lesson is constantly scan for misunderstandings and ways to clarify. Again, here I will go to the low context, high context type of thing so, let me give you an example. I think it might be easier to make my point that way and I will ask a question at the end. In high context cultures or in some cultures when you ask questions, no, is not an acceptable answer. You have to say yes and some others you have to say no three times before saying yes.
Lamya Alaoui: Imagine when you have people from different cultures in the same room and someone is asking a close question that requires a yes or a no. What are the chances, based on what I said, that you will get an accurate answer to your question? 50%, 60%, 20%, a hundred percent. For this always ask open ended questions to give people space to answer without having to break their own boundaries and to make sure that you are getting clarity in the answers. How many of you were in meetings where you felt that you were misunderstood when you were speaking? What did you do? Anyone wants to volunteer again? When you felt that you were misunderstood? Yes.
Angela: Have you summarized in a written notification? I gave action items to the people I wanted clarification from that helped with the misunderstandings.
Lamya Alaoui: Thank you, Angela. One of the things that is really helpful and again, best practice for high performing teams, always summarize and put things in writing and this setting English is the working language. We do have a lot of people, I know in my team, I have a lot of non-native speakers. I’m not a native speaker so to make sure that the message gets across and that it’s clear it’s always in writing, always, which we tend not to do because everyone is working long hours or you have back to back meetings. Then at the end of the day you just want to finally get to do your work and you never summarize in writing what happened in those meetings.
Lamya Alaoui: Finally, the structure. Build a structure that is around diverse teams. When we’re usually building teams we have, we’re looking at what now is referred to as visible diversity, which can be ethnicity, religion, gender and many other things, but we’re never looking at the cognitive one that also has a heavy part, especially in tech. Those are recommendation that I’ve always lived by every time that I join a new company. We do a reintroduction and relaunch of the core values. We redefine them. We create, and this is like a company work, it’s not just someone in their corner doing it, but it’s a collaborative work setting what are the core values, what are the expected behaviors and the ones associated with each one of the values, and then it’s communicated throughout the whole company. When it comes to performance review it needs to be designed with underrepresented communities in mind.
Lamya Alaoui: Performance reviews, if you look at the history of how it was designed, it was designed for a very specific community or population. It doesn’t fit other people. Just leave it at that. Work on creating a performance review management system that is more inclusive. Yeah. Targeted networking. Again, this ties to the diversity and leadership. One of the things, again, hiring in despair. When those roles are open, people tend to hire fast from networks that they know because they don’t have networks that are already established. This is something that is quite helpful. I don’t think there are any people in the HR realm tonight, but all of you can be ambassadors for something like this to be established in your companies. And again, clear expectations around the engagement and the roles. What are the pathways to promotions? Again, if you look at internal promotions or internal applications, woman tend for, this is just an example, women tend not to apply to internal openings when it comes to higher positions in the company.
Lamya Alaoui: Sometimes just because they don’t check all the boxes so the work starts before that, where we actually need to make sure that the job descriptions or the job openings are reflecting and are being thoughtful about underrepresented groups. Finally, this is one of my favorite things, Kaizen. There is always room for improvement. Performance and diversity are a very long journey. Each company is at a different stage or different phase. Just ask questions, be kind because not a lot of people think about it when it comes to those things and be as curious as possible because at the end of the day the only way to know other people is to ask questions. Thank you so much everyone.
Julie Choi: I want to thank all of our speakers tonight and for all of you, for being such an incredible audience. We have people who have been watching these talks outside in the overflow and I’m told that, I think it might be better if we just all go out and just mingle. Get some fresh air after all this time is for us to connect and network. Why don’t we continue the conversation outside? Thank you everybody.
This site reliability engineer discusses Ukranian borscht or machine learning, or both, at MosaicML Girl Geek Dinner.
MosaicML Girl Geek Dinner speakers after the event: Tiffany Williams, Banu Nagasundaram, Laura Florescu, Julie Choi, Lamya Alaoui, Shelby Heinecke, Angela Jiang, Angie Chang, and Amy Zhang.
Like what you see here? Our mission-aligned Girl Geek X partners are hiring!
When I was in high school, I didn’t have any industry role models. Most students didn’t go to a four-year college. School projects are valuable learning experiences and invaluable project-management and time-management exercises.Team work is a foundational building block for future success.
CCPA high school students experienced the journey of an entrepreneur building their minimum viable product (ideating, running surveys and customer interviews to validate assumptions, performing competitive analysis, building wireframes, managing features, and launching apps with native app builder Thunkable) as their Senior Capstone Projects.
As high school students, they likely do not know tech jargon. This is where industry mentors from companies across the San Francisco Bay Area come in to share their affirmative experiences and say hey, you’re not too far off from doing what the “professionals” and “techies” do at work.
Today’s presentations of students apps in Portable 2 focused on climate change (student education), shelter needs (items most in need at shelters), and healthcare (providing resources for free or low cost services when possible to low income and/or non-English speaking communities).
Volunteers attended student presentations and reinforced educator feedback by asking questions of the students presentations. Where did the data come from? Have you considered this additional use case? Why did this require building a native mobile app, versus just using Facebook to achieve your stated objectives?
After the presentations, Girl Geek X volunteers introduced themselves to the students. Marilyn works in user experience with a background in computer science, Katie is a Data Analyst at Playstation, James is a project manager, etc.
I talked about the thousands of early-stage startups that are great entry points (versus the big tech brands) for people new to tech. Together, we painted a picture of a diverse slate of roles in tech – filled by people of various education backgrounds.
We are excited to wrap up our first year with our “adopted” school CCPA in East Oakland with the Oakland Education Fund partnership, and excited for the next year!