An interview with Arindam Pal, Senior Research Scientist at CSIRO
Let me welcome Arindam Pal for today’s interview. Arindam is a Senior Research Scientist at Data61 in CSIRO. He is also a Research Fellow at the Cyber Security Cooperative Research Centre. Prior to this, he was a Research Scientist at TCS Research and Innovation. He also has the experience of working at Microsoft, Yahoo Research Labs and IBM Research Labs. In the past, Arindam has held the position of Visiting Scientist at the University of Calabria.
Arindam’s research interests are in the areas of Algorithms, Cyber Security, Network Science, Machine Learning and Optimization. He has published high-quality papers in several top-tier conferences and journals. To know more about him you can check here.
I would like to wholeheartedly thank Arindam for taking the time to do this interview. I hope this interview serves a purpose towards the betterment of data science and machine learning communities in general :)
An interview with Arindam Pal, Senior Research Scientist at CSIRO
Sayak: Hi Arindam! Thank you for doing this interview. It’s a pleasure to have you here today.
Arindam: Thank you Sayak for inviting me. I am delighted to talk to you.
Sayak: Maybe you could start by introducing yourself — what is your current job and what are your responsibilities over there?
Arindam: I am a Senior Research Scientist at Data61 in Commonwealth Scientific and Industrial Research Organisation (CSIRO) and a Research Fellow at Cyber Security Cooperative Research Centre. For those of you who don’t know about CSIRO, it is Australia’s national science research agency. We solve the greatest challenges through innovative science and technology. I work on applying machine learning algorithms to solve cybersecurity problems. Some of the problems are fraud detection, anomaly detection, outlier detection, and user behavior modeling.
Sayak: My undergraduate dissertation was on using machine learning in cybersecurity (intrusion detection specifically). It’s such a lovely field. I am curious about how did you become interested in pursuing machine learning?
Arindam: From my childhood, I was highly interested in mathematics and science. When I studied computer science and engineering during my undergraduate program, I got interested in algorithms. Algorithms are methods to solve a mathematical problem in a precise, step-by-step manner. During my Ph.D. at IIT Delhi, I worked on finding approximate solutions to hard combinatorial optimization problems, such as scheduling jobs on a machine and routing vehicles in a road network. These problems are very interesting and challenging to solve. During my coursework, I took a course on machine learning. I was fascinated by the different types of problems that can be solved by machine learning algorithms, which can’t be solved by traditional algorithms. I did a course project in MATLAB, which gave me confidence in solving machine learning problems.
Sayak: That is nice to know. Thanks for sharing that, Arindam. When you were starting in the field what kind of challenges did you face? How did you overcome them?
Arindam: When I started to work on machine learning problems, there were not many books and courses available to learn the subject. Also, finding good datasets was a challenge. I tested most of my algorithms on the datasets given in the UCI Machine Learning Repository. I asked many questions on Stack Exchange. Then, many other resources became available. Coursera and edX came up. I took Andrew Ng’s machine learning course to learn more about the subject. Kaggle started offering datasets and conducted challenges and competitions. I also took part in some Topcoder and Google Code Jam challenges to hone my skills.
I took Andrew Ng’s machine learning course to learn more about the subject. […] I also took part in some Topcoder and Google Code Jam challenges to hone my skills.
Sayak: This is truly inspiring. On another note, Andrew Ng’s course has been the entry point to machine learning for a generation, I would say. What were some of the capstone projects you did during your formative years?
Arindam: Over the years, I did many interesting and challenging projects. Here are some that I would like to highlight.
- Evacuation Planning from Large Buildings
- Analysis of Scientific Publications and Patents with Machine Learning
- Legal Data Analytics and Mining
- Analysis of Interdependent Networks
- Warehouse Automation using Robotics
Sayak: I remember reading your work in the analysis of scientific publications and patents with machine learning during my stint at TCS Research. It was definitely something new and very interesting. Your research career has been so exciting so far. Would you like to pass along any thoughts for the budding researchers? You could also shed some light on your approach towards doing effective research.
Arindam: The most important thing about doing research is that you have to love the subject and be passionate about it. If you don’t like what you are doing, you will never be successful (isn’t that true about anything that we do in life?). You also have to work very hard. Nothing comes easily in research. You have to try many different things before you can come up with a new algorithm, which gives a better solution to your problem. You have to try it on many different datasets to be confident that it really works well, both in theory and in practice.
I would like to state the great physicist Richard Feynman’s algorithm to solve a problem. There are three steps.
- Write down the problem.
- Think very hard.
- Write down the solution.
Now the key part is “Think very hard”. How do you think about a new problem for which you don’t have much idea? Here is how you do it. Try to understand the structure of the problem. See, if you can think about a problem that you know, which is similar to the current problem that you are trying to solve. If you can find it, you are almost done. Most of us find solutions for a new problem, by reducing it to one or more known problems. It’s like the k-nearest neighbour algorithm in classification. You say, a picture is that of a cat, if it looks similar to a picture that you know is of a cat.
Most of us find solutions for a new problem, by reducing it to one or more known problems. It’s like the k-nearest neighbour algorithm in classification. You say, a picture is that of a cat, if it looks similar to a picture that you know is of a cat.
Sayak: What a tremendous analogy and a unique way of approaching problems! I am amazed! You have worked in so many different domains over the years. What’s the secret juice that helped you to be consistent and productive throughout this journey?
Arindam: Indeed, I have worked on many different domains over the last 20 years. I worked as a software engineer in Microsoft and other software development companies, where I used to write a lot of industrial-strength code. Now, I am more focused on design and research. I have also worked on many different types of problems: combinatorial optimization, software engineering, artificial intelligence, machine learning, and cybersecurity. Some of them are highly mathematical, whereas others are much more practical. I think it is my passion for computers, science, technology and mathematics that has motivated me to work in such diverse domains and remain consistent and productive throughout this long and eventful journey.
I think it is my passion for computers, science, technology and mathematics that has motivated me to work in such diverse domains and remain consistent and productive throughout this long and eventful journey.
Sayak: These fields like machine learning are rapidly evolving. How do you manage to keep track of the latest relevant happenings?
Arindam: I read a lot of books, papers, news articles and blogs to keep track of the latest research and development in machine learning. You have to read a lot of things to stay updated. I go to various conferences, workshops, and meetups. I attend talks given by professors and researchers. I give talks and discuss my ideas with my colleagues and fellow researchers.
Sayak: Being a practitioner, one thing that I often find myself struggling with is learning a new concept. Would you like to share how do you approach that process?
Arindam: Learning a new concept is tough for everyone, not just for you. The right way to approach this is to think about how you can relate this new concept to the existing concepts that you already know. Once you can make connections between them, it will become much easier to learn the new concept.
The right way to approach this (learning a new concept) is to think about how you can relate this new concept to the existing concepts that you already know. Once you can make connections between them, it will become much easier to learn the new concept.
Sayak: Connectionism almost always works in anything. Any advice for the beginners?
Arindam: In my opinion, the most important thing to remember is that you should clearly understand what you are doing. Don’t treat these machine learning and deep learning algorithms as black boxes. You should know how these algorithms are working, inside out. The mistake that many young people make is that they use some libraries like TensorFlow or PyTorch and think that these libraries will automatically solve all their problems. You have to understand the technical and business aspects of the problems very well to solve them.
… the most important thing to remember is that you should clearly understand what you are doing. Don’t treat these machine learning and deep learning algorithms as black boxes.
Sayak: Thank you so much, Arindam, for doing this interview and for sharing your valuable insights. I hope they will be immensely helpful for the community.
Arindam: It was a pleasure to talk with you Sayak. I hope that this interview will benefit young researchers, who are trying to build a career in machine learning.
Summary
Arindam’s take on reducing a new problem to an already existing problem is something I am definitely trying out in my own work (and you should too!). Arindam’s consistency and productivity are byproducts of his deep passion for computers, science, technology, and mathematics. It’s amazing and inspiring to see what this amalgamation has allowed him to pursue so far. Arindam’s rare experience of combining research and applied principles is also something to ponder on.
I hope you enjoyed reading this interview. Watch out this space for the next one and I hope to see you soon. This is where you can find all the interviews done so far.
If you want to know more about me, check out my website.
