An interview with Josh Tobin, Research Scientist at OpenAI

This is a part of the interview series I have started on data science and machine learning. These interviews are from the people that have inspired me, that have taught me (read are teaching me) about these beautiful subjects. The purpose of doing this is to mainly get insights about the real-world project experiences, perspectives on learning new things, some fun facts and thereby enriching the communities in the process.

This is where you can find all the interviews done so far.

Today I have Josh Tobin with me. Josh is a Research Scientist at OpenAI working at the intersection of machine learning and robotics. His research focuses on applying deep reinforcement learning, generative models, and synthetic data to problems in robotic perception and control. Additionally, he co-organizes a machine learning training program for engineers to learn about production-ready deep learning called Full Stack Deep Learning. Josh made this amazing field guide on troubleshooting deep neural networks which is really one of a kind in its own course. He is also one of the instructors of the Applied Deep Learning Fellowship by Weights and Biases.

Josh did his Ph.D. in Computer Science at UC Berkeley advised by Pieter Abbeel. He has also been a management consultant at McKinsey and an Investment Partner at Dorm Room Fund.

I would like to wholeheartedly thank Josh for taking the time to do this interview. I hope this interview serves a purpose towards the betterment of data science and machine learning communities in general :)

An interview with Josh Tobin, Research Scientist at OpenAI

Sayak: Hi Josh! Thank you for doing this interview. It’s a pleasure to have you here today.

Josh: Thanks for having me!

Sayak: Maybe you could start by introducing yourself — enlightening us about your research.

Josh: My main research motivation is to build robots that are capable of helping people solve a wide variety of tasks in the real world. When I started in the field, I was surprised at how little real-world robots have changed in the last 20+ years. Then I came across deep reinforcement learning (Deep RL). Deep RL seemed to be one of the best candidates to make a big change in the field. The biggest bottleneck I saw to deep RL having a big impact in the real world is how data-hungry it is. It’s hard and expensive to collect data using robots in the real world. So early on in my Ph.D. I became interested in the question, “how can we reduce the need for real-world data collection in robotic learning”?

Since then, my work at OpenAI and Berkeley has mostly focused on training robots in a physics simulator, where data collection is cheap and safe. The hard part is transferring the learned behaviors to the real world. The best-known technique we explored is Domain Randomization, which involves massively randomizing the appearance and physics of the simulated world. The robot learns in such a wide variety of simulated environments that it’s able to generalize to the real world when it operates there, often with no additional training.

Sayak: That was enlightening, indeed! Your question on reducing the need for real-world data collection is highly impactful, I believe. I am very curious to know how did you become interested in deep learning?

Josh: My background is in pure math. When I went to graduate school in 2015, I knew I wanted to work on something more applied, but didn’t know what yet. I took a variety of classes in my first semester. I quickly became hooked on machine learning because it’s challenging, mathematically interesting and because I thought the impact could be massive. A lot of the credit for my interest goes to a machine learning and robotics class I took with Pieter Abbeel, who later became my Ph.D. advisor.

Sayak: Pieter Abbeel is so wonderful! I had started taking his Deep Unsupervised Learning course and it really got me hooked. When you were starting what kind of challenges did you face? How did you overcome them?

Josh: I started grad school in the applied math department at Berkeley. At some point in my first year, I decided I wanted to try to switch into CS and work with Professor Abbeel. I later learned his lab was one of the most competitive to get into at Berkeley. I felt like the other students were smarter and better prepared than I was.

When I was working with Abbeel on a trial basis, he asked me to run some experiments on the lab’s robot. He told me, “the only thing you need to be careful of is not to break the robot. We just have the one, and most of the lab relies on it.”

I had only taken one CS class in college, so I was teaching myself to program at the time. As I was trying to run the experiment, I ran into a TensorFlow installation issue. I spent hours trying to debug it. At 1am on Saturday morning, I did something stupid, and completely broke the robot. It wouldn’t even boot up. I was too embarrassed to ask anyone in the lab for help right away, so I came in early the next morning and thankfully managed to fix it before anyone noticed.

I think a few of the decisions I made early on helped me learn machine learning quickly, though I don’t necessarily recommend doing it the way I did.

• I tried to immerse myself in the subject completely. I didn’t think about much aside from machine learning and programming my first year.
• One thing I learned from studying math was being comfortable being stuck. Whenever I couldn’t solve something myself, I tried to figure it out on my own before asking for help. Sometimes it felt like banging my head against the wall, but I think I learned a ton from it. E.g., after my marathon robot debugging session, I felt much more comfortable with Linux and the software like ROS that runs on the robot.
• I made the decision not to start from the fundamentals, but to dive into the state of the art right away. This allowed me to figure out how to do research faster, but I’m not sure whether this hurt or helped in the long run. I later had to back-fill most of this, and it took me a long time to be confident in the basics.
• I constantly tried to push myself to work on things that felt really hard. Grad school is a great environment for this, and Pieter was also amazing at getting me out of my comfort zone.

Sayak: That was super comprehensive, Josh! I am really thankful for mentioning all these little details. Being comfortable being stuck is something that is staying with me. What were some of the capstone projects you did during your formative years?

Josh: I worked on a variety of class projects. The most memorable was a prototype of an approach to allow a fleet of robots to learn from their collective experience and a parallelized reinforcement learning system.

Sayak: That sounds pretty exciting! What was the motivation behind returning back to academia and start your Ph.D.?

Josh: I loved studying math as an undergraduate, but since I focused on more abstract fields like algebraic topology and knot theory, I didn’t manage to learn anything useful. After a couple of years in consulting, I realized that there is a limited window of time in your career when you can legitimately become an expert in something. I knew I wanted to get back to doing something more technical, and graduate school seemed to figure out what to become an expert in.

Sayak: That makes sense! Your field guide on debugging deep neural networks is so amazingly laid out. I have been benefitted by it in numerous instances in my own projects. In fact, it motivated me to write Training Neural Nets: a Hacker’s Perspective. Would you like to share any background story behind developing the field guide?

Josh: Thanks — I’m really glad you got something out of it. When I started mentoring more junior researchers, I realized that when they were learning the ropes we rarely talked about the fancy stuff like the nuances of the best model architecture or learning algorithm for a particular problem. Instead, we spent those early meetings just trying to get to the bottom of what was really happening with their model. How good is our performance really? Is there a bug? Are we overfitting or underfitting?

This knowledge is mostly trapped in researchers’ heads, but I felt like 95% of it could be boiled down to a simple decision tree, so I wrote up the first version of the guide as slides for a lecture in the first Full Stack Deep Learning Bootcamp.

Sayak: The field of deep learning is rapidly evolving. How do you manage to keep track of the latest relevant happenings?

Josh: Following interesting people on Twitter is the best way I’ve found so far.

Sayak: I too believe Twitter is a great source in this regard. Being a practitioner, one thing that I often find myself struggling with is learning a new concept. Would you like to share how do you approach that process?

Josh: I learned the following when I was studying math. First, read through the description. Find any terms that are unclear or you find confusing and brush up on them so the definitions are fresh in your mind. Try copying it down a few times and thinking about each word. Read through the proof if there is one, or try to justify it to yourself if not.

Once you feel like you have a handle on the concept, try to restate it in your own words. Once you can do that, try poking the edge cases — if you remove this assumption why does it fail? If the function is constant what happens? Etc. Once you’ve explored the edge cases, make sure you can restate it precisely.

I’d make a couple of updates to this approach for deep learning. Because the pace of the field is so fast, the original paper is not always the easiest to understand explanation of the concept. Try reading a few different ones.

One mistake I used to make was reading about an algorithm and feeling like I understood it, and then trying to implement it and realizing I really didn’t. Early on, it’s valuable to force yourself to implement things from scratch. Once you’ve done that enough times, you can start to just sketch how you would implement it and you will catch details that you would have missed otherwise.

Sayak: Again, thank you for sharing your views on this. I feel this is one of those points which many struggles with. Any advice for the beginners?

Josh: Stick with it! Deep learning feels mysterious at first, but you can get to the forefront if you’re persistent. Try to work on projects you’re excited about from the start. Don’t worry too much about what you don’t know, just focus on figuring out what you need to make progress on your project. Immerse yourself in the field. Start watching conference talks and reading papers, even if they seem confusing. Talk to other people working in the field. Often everyone is struggling with the same things you are.

Sayak: That was really a morale booster, Josh. Thank you so much for doing this interview and for sharing your valuable insights. I hope they will be immensely helpful for the community.

Josh: Thanks!

Summary

Josh’s experiences are so incredibly inspiring. The challenges he faced throughout his career and the way he approached them are truly very practical. I really could connect with his advice on being comfortable being stuck. His zeal for pursuing machine learning education is very admiring. Don’t forget to check the field guide on debugging neural networks mentioned in the interview.

I hope you enjoyed reading this interview. Watch out this space for the next one and I hope to see you soon.

If you want to know more about me, check out my website.