“Designing AI for Designers”: Dr. Tonya Custis, Director of AI Research at Autodesk (Video + Transcript)

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!

Sukrutha Bhadouria: Hi everybody. Welcome to this session. As you know, AI is such a hot topic and has been for many years. It’s gotten into an even hotter, hotter topic now. So I’m very, very excited to have Dr. Tonya here for us today. Wanted to remind everybody to share on social media, Twitter, Instagram, all the amazing learnings that you will have today. Don’t forget to also post your questions in the chat. Yeah, so anyway, <laugh> Sorry about that, the first part of the confusion that I had with my tech technical difficulties. I’m gonna go jump right into the intro.

Sukrutha Bhadouria: Dr. Tonya Custis is the Director of AI Research at Autodesk. She leads the AI research team at Autodesk that conducts the fundamental and applied research in AI and machine learning with the goal of unlocking a new era of AI-powered design tools. We’re so excited to hear her speak today and to hear about how roller derby has played a part in both her life and career. Welcome, Dr. Tonya.

Dr. Tonya Custis: Thank you so much. Hi everybody. It’s such an honor to have been asked to speak to you today. I’m really excited to be here. First off, in the spirit of International Women’s Day, I would really like to start by thanking and giving a shout out to all the strong women in my life from friends that I’ve had since kindergarten, academic advisors, roller derby teammates, to all of the brilliant supportive women in tech that I have and continue to encounter every single day in my career.

Dr. Tonya Custis: How is leading a roller derby team similar to leading a team of AI research scientists? Today I’m gonna answer this question by talking about my current position and the work I do as the director of AI research at Autodesk and about how I ended up there. Although I am not sure I would’ve realize it until much later, I guess the first step in my journey to getting here was learning to program in Logo in second grade, and then teaching myself BASIC on my Atari 400 computer when I was in fourth grade. But fast forward a few years, and the next step was obviously getting my college degree in music clarinet performance, and music theory followed by graduate degrees in linguistics and computer science.

Dr. Tonya Custis: Now, you might ask yourself, what do music, linguistics, and computer science have in common? At the time, I’m not actually sure, I could have told you. I maybe would’ve told you that it was very unlikely I was gonna get a job as one of two clarinets in any major symphony orchestra, which basically only opened when someone dies. I decided to go to grad school for linguistics. I was really interested in how the brain processes language and specifically how children learn language. However, as I advanced my linguistics grad school career, it also became clear that like the clarinet gig, in order to get a job as the linguistics professor, I would probably have to wait for someone to retire, and when they did, have to be willing to move to wherever that opening was. What I haven’t mentioned that is that in grad school, I was also married with two small children. They were so cute. Just picking up and moving to wherever was not really an option, giving my family situation.

Dr. Tonya Custis: Re-enter computer programming, I love doing it. As a kid, I’d taken some programming classes in high school and in college, but I had never really seriously considered getting a degree in it or making a career of it. Until my interests in how children learn language expanded to how can I get a computer to learn language? And I guess that’s when it all came together. What do music linguistics and computer science all have in common? Recognizing, producing, and leveraging patterns. I was able to take the skills from each discipline and creatively apply them to the others. I finished grad school around the time when Google and other search engines were really starting to take off. Although there were no actual computational linguistics or natural language processing degrees at the time, the coursework I did that combined linguistics and computer science put me in the right place at the right time.

Dr. Tonya Custis: My first job as a research scientist gave me opportunities to do machine learning research on information retrieval and natural language processing. It was my first real job and there were a lot of rules to learn. I needed to figure out how my research and my work fit into the bigger picture of the team I was on, but one thing that remained constant between academia and being an industrial research lab is that research means failing a lot. It’s part of the job, and it’s also part of the job to then pick yourself up and learn from those failures going forward.

Dr. Tonya Custis: A couple of years into my career as a research scientist, I went to my very first roller derby bout. As soon as I saw it, I knew I had to play. I immediately bought skates and gear and found out when the next tryouts were. Here’s me at tryouts. I tried out and I became a member of the league. But much like getting my first job, I now had to figure out what the rules were and how I fit into the team, and so became began the period of my life where I was an AI researcher by day, and a roller derby player by night.

Dr. Tonya Custis: I played roller derby, competitive roller derby, for 10 years and during that time I had the privilege of playing with skaters of all ages, all body types, and from all walks of life. The Women’s Flat Track Roller Derby Association (WFTDA) serves as the international governing and sanctioning body for bouts, rankings and tournaments. WFTDA currently has about 450 member leagues spread across six continents. There’s even a roller derby World Cup. The rules of roller derby are complex. They involve a lot of safety measures, where you can and can’t hit people, and with what parts of your body you can and can’t hit them with. But here’s a real high level description of how roller derby works. There are five players on each team, and the players skate counterclockwise on the track. Each team has four blockers and one point scorer, the jammer. The jammer is the one with the star on her helmet, and the jammer earns points earns a point for each opposing blocker she laps on the track.

Dr. Tonya Custis: What this boils down to is you have to play offense and defense at the same time. You have to prevent the opposing jammer from passing you and your teammates, while also helping your jammer through the pack to pass the players from the opposing team.

Dr. Tonya Custis: The most effective way to slow down, or stop players from the other team, is to knock them over, or to push them off the track. Roller derby is a full contact, physical sport. In addition to countless bruises, I’ve had a concussion, a broken leg, and a broken collarbone. This picture actually shows my collarbone being broken. But <laugh>, I have really never had so much fun in my life. I learned so much from playing roller derby, but maybe the best lesson was sort of like research. It doesn’t matter how many times you fall down, as long as you get up again.

Dr. Tonya Custis: A year into my roller derby career, I became captain of my team. I really did not want to be captain of my team, but somehow I got elected anyway. Being captain meant making decisions about people’s playing time, art training program, and the overall strategy. It meant being a player and being a coach at the same time. I was a reluctant leader, but I approached it as a puzzle, as an exercise in recognizing, leveraging, and optimizing patterns, skills that I was able to transfer from the other areas of my life. I learned that none of us had the same strengths or weaknesses, but we could strengthen each other on the track, and mitigate each other’s weaknesses by playing to our strengths, and as long as we worked together and trusted each other. Our team, which had never won a game when I joined the league, ended up winning several season championships in a row.

Dr. Tonya Custis: A few years later, when I was offered the job of research director, I was once again a reluctant leader. I loved doing research, I loved being a player on the research team, and I didn’t know if I really wanted to be a coach. But oddly enough, I was able to creatively transfer my roller derby leadership experience to my new role, which I again approached as an exercise in recognizing, leveraging, and optimizing patterns – and also one in playing offense and defense at the same time! What I’ve found is that being a leader means protecting your team, so they can concentrate on and have the space to do their jobs, so this is playing defense, but it also means playing offense by proactively getting your people opportunities for career growth and development, and making sure they have the resources they need to do their jobs.

Dr. Tonya Custis: So to be honest, leading a team of research scientists isn’t that different than leading a roller derby team. Everyone’s different, we have different strengths, different weaknesses, and not everyone plays the same position, but we all do better when we learn from each other and trust each other, working as a team to solve difficult and interesting research problems.

Dr. Tonya Custis: Let’s fast forward, back to my current role as the director of AI research at Autodesk, I lead a global team of AI research scientists and we work on machine learning research primarily in the 2D and 3D geometry space. So you might be asking yourself why someone with a heavy background in NLP and linguistics is leading a geometry-focused AI research team, and honestly I asked myself the exact same question when I was offered this job. What drew me to the design space was the opportunity to join language and geometry, to use language to communicate the semantics of geometry, in order to make design more intuitive. I wanted to take my background in NLP and use it to creatively solve problems in a new domain. Increasingly, NLP techniques, like large language models are finding more and more success when being applied in new ways and to new modalities.

Dr. Tonya Custis: The recent explosion of generative AI models like ChatGPT, Codex, Dall-E and Stable Diffusion is especially interesting in the context of design software. Autodesk, if you don’t know, creates design and make software. We serve the architecture, engineering and construction manufacturing and media and entertainment industries. We help our customers design and make everything, from smart cars to skyscrapers, and from bridges to the visual effects in films and video games. So if you’ve ever been in a building, driven a car, used a computer, sat in a chair, or seen a movie, you’ve likely experienced the designs our customers have made with Autodesk software.

Dr. Tonya Custis: My team’s research has three main areas of focus, and a big chunk of the work we do is in publishing papers at top conferences, often with academic labs. I’m briefly gonna outline some of our recent papers to give you a flavor of what we’re working on. In JoinABLe, which is work we did with MIT, we used machine learning to learn a bottom-up approach to parts assembly. So given a set of parts, can a computer learn to assemble them? To do this, the computer needs to be able to reason about not only the shapes of the parts, but also how they might be joined together. And as it turns out, this is a task people can only do correctly 80% of the time, so the graph shows the experimental results for JoinABLe with blue at 78% accuracy, against other state-of-the-art machine learning methods in green, and also, human performance in pink.

Dr. Tonya Custis: In the UNIST paper work with Simon Fraser University, we investigate using machine learning for style transfer. So for example, you might want a chair that matches the style of a table. This algorithm, given the 3D representation of a table, can generate a chair that matches it in style. Although style is somewhat subjective, it can be learned by neural networks, and the ability to transfer style can save designers a lot of time.

Dr. Tonya Custis: Let’s pause here for a second and talk a little bit about generative AI. I’ve been doing research on language modeling and computational syntax for the past 15 years. ChatGPT and other large language models are an incredible achievement towards a lot of the things I have been working on for my whole career. And advances in multi-modal generative AI models like DALL·E 2, Stable Diffusion, and Midjourney point to the desire to translate our ideas between language and other modalities.

Dr. Tonya Custis: Linguistic generalization and the ability to generate novel human-like language is a huge milestone for AI. Large language models are capable of generating strings of words that no human has ever said before. This is the power of linguistic productivity, using a finite number of grammatical structures to produce an infinite number of utterances. However, we do need to remember in the hype of all this, that large language models are trained to optimize for linguistic structure. There is no intent, no information, no reasoning, nor knowledge behind the words being strung together.

Dr. Tonya Custis: Any meaning perceived in the string of words supplied by the computer is, the meaning is supplied by the user. Said differently, there’s no actual creativity behind how large language models generate language, just probabilities. Nonetheless, one of the criticisms of large language models like ChatGPT, is that they make up and hallucinate facts. These models again we’re trained to generate syntax, not meaning and not facts. Having trained on over billions of word tokens, the text they generate puts words together that are related enough and seem plausible enough that they might sound like facts, but often they aren’t.

Dr. Tonya Custis: Why and how is this relevant to the design space? My team has been working on research in which we apply large language model techniques to 2D and 3D data. What if we could learn a grammar for how sketches and 3D models are generated by humans, and then mimic that grammatical generation with an algorithm? In the design space, having the algorithm hallucinate new designs is actually an advantage, not a liability. In this paper, we proposed two transformer based generative models that generate engineering sketches. This paper shows the ability to tokenize engineering sketches and represent them in a transformer architecture showing that it was not only possible but a promising research direction. Other works we’ve published that use transformer models for sketch and 3D generation are SkexGen and SolidGen.

Dr. Tonya Custis: In both cases, we are trying to generate designs that will be helpful to the user, which means they have to walk the line between being realistic and manufacturable, but also novel, diverse, and inspiring. SkexGen uses sequences of CAD operations to train a generative transformer model. Here we can see how the CAD operations in their order are broken down into tokens, or words, so that the transformer model can learn the structure, or grammar, of how the operations go together to create an entire 3D model, or sentence, if you will. SolidGen is also a generative 3D model, but instead of learning from sequences of operations, it learns a grammar of the 3D structures directly, by predicting the vertices edges and faces that make up an object, using transformer based and pointer neural networks.

Dr. Tonya Custis: These three papers are all great examples of how we can transfer and apply the ideas and mechanics of large language models to other types of data. Language is how we describe the 3D world around us, and we’re also working on multimodal generative models that allow designers to use language to communicate their designs, design intent, and specifications to computers. There has been quite a bit of progress recently in text to images, but similar progress in text to 3D shapes has leg behind because there isn’t a lot of data that pairs text to shapes.

Dr. Tonya Custis: CLIP-Forge proposes a machine learn geometry sensitive method for generating 3D models from natural language, so a prompt for a round chair generates a 3D box model of a round chair as opposed to a square. One CLIP-Sculptor is the newest paper from our lab recently accepted at this year’s upcoming computer vision and pattern recognition conference. It improves on the results of CLIP-Forge and other previous state-of-the-art methods, both in terms of diversity and accuracy, but also in speed of generation. Language is a tool we all share, and right now we’re starting to be able to use language to help shape and build the world around us in an actual physical sense. It’s really inspiring to me to think about how these technologies will evolve to support designers who use Autodesk software.

Dr. Tonya Custis: I have come a long way from almost no one understanding why I would possibly be interested in combining linguistics in computer science, and from when no one understood what my job was or why I was doing it, and so it’s incredibly validating to have reached this place where computers can recognize, leverage, and complete patterns at such a large scale that they are useful to people in so many different ways. What I want you to take away from this talk today is that technology is always changing. Life is always changing, but your strengths and the skills you’ve learned in one place are often transferrable to new situations. In my experience, leadership requires you to recognize patterns and creatively transfer what you know from one place to new situations, so that you can play offense and defense at the same time, so that your team has the space they need to on the track and the trust, confidence ,and resources they need to do their job successfully.

Dr. Tonya Custis: The ability to take what we’ve learned from one place in life and creatively apply it to new situations is what makes us human. Be curious, learn and discover. Fall down and fail. But remember, in roller derby and life, it really doesn’t matter how many times you fail, as long as you get up again. Thanks!

Sukrutha Bhadouria: Thank you so much. This talk really resonated with a lot of our attendees. We see some amazing comments. We are at time now. Thank you all, everyone, and thank you so much Dr. Castes for making time for us today.

Dr. Tonya Custis: Thank you all.

Like what you see here? Our mission-aligned Girl Geek X partners are hiring!

• Check out open jobs at our trusted partner companies.
• Watch all ELEVATE 2023 conference video replays!
• Does your company want to sponsor a Girl Geek Dinner? Talk to us!