Meet Kiran Bhat, who won the 2017 Oscar for technical achievement. The technology which he developed at IL&M has been used in movies like Avengers (for Hulk), Pirates of the Caribbean (for Davy Jones), Teenage Mutant Ninja Turtles, Warcraft, Star Wars: Episode VII, and Rogue One: A Star Wars Story. With his startup Loom.ai, Kiran is on a mission to build the best 3D representation for every face on the planet.
Kiran won the Oscar for technical achievement in 2017 (Sci-Tech Academy Awards) for developing the facial performance-capture system along with his team at IL&M. He holdsa degree in electrical and electronics, and mechanical engineering from BITS Pilani, and a PhD in Robotics from Carnegie Mellon University.
Kiran was born and brought up in Thiruvananthapuram. His father was a rocket scientist in ISRO, who later decided to move to Coimbatore to start his own company. Kiran recalls being an academically oriented kid who got exposed to programming early on. He was exposed to lateral thinking for the first time when he moved to a Coimbatore (after class seventh) school.
Kiran loved maths, physics, and chemistry. He could speak Malayalam, Thulu and Tamil. He even learnt French in class 11 and 12. Kiran came first in the district in class 12 board examinations.
He recalls his high school days, “I was blown away by the movies The Abyss and The Terminator. Looking at the computer-generated characters, I wondered what will it take to build such a thing.”
Kiran’s choice to go to BITS Pilani after class 12 was somewhat contentious in his family as he had secured admissions in many other nearby universities (Chennai, Trichy, and other southern cities) as well.
BITS Pilani days:
BITS Pilani had people from different geographies and it felt more than just an academic institute. Kiran says,
You could find your corner in the world and do what you wanted to do. There was no rat race of achieving just one goal. What made BITS special was that a lot of people came to the college with similar drive and aspirations. I’ve always believed that the best places to study are the ones where you can learn from others.
Since Kiran was interested in continuous math, calculus, and mechanics, he chose mechanical engineering (which had all three components). Because of his interest in robotics, which required him to learn control theory at the end of his first year, Kiran opted for electrical and electronics engineering (in addition to mechanical engineering). He adds, “It made a complete package as I could build the whole system and drive it (and give it a brain).”
He started tinkering with robots as a hobby but soon it became an obsession. In his second year, Kiran built PD controller. He says, “We had a robot arm in the lab. I was trying to get it to move in space. The control algorithm which allowed that, was written with fuzzy logic.”
Kiran was surrounded by some great minds in the robotics lab. He shares the story of Sartaj Singh, a hardware guru, who once hijacked a fully functional robot (bought for the institute) in the lab, removed its motors, and then put his own motors (and wrote his own controllers). He did all this just because the robot had a lousy operating system and a control box which was frustrating to program. Kiran recalls, “There were a bunch of guys like him who were not shy or afraid of anything. They convinced the professors to set up collaboration with Motorola so we had all these microcontroller chips, evaluations models coming in.”
Kiran and others in the robotics lab represented a diverse combination of people with interest in control theory, building mechanisms, DSPs, and fabrication. Every one fed off and challenged each other in a creative way. They spent a lot of time in the lab and almost started living there.
The robots and the mentors
Some of the robots Kiran worked on included the following:
Scara: It was the very first version of the robot arm designed fully in-house. The students mathematically modelled every single detail and later presented a paper which described how to build a robot. In the pre-Google era, students spent hours in library going through the IEEE magazines on robotics looking for solutions. Kiran says, “The experience of building Scara gave us an intuition about what not to design.”
Kiran with his first robot Scara
Hydra: It could do all-terrain locomotion. Kiran’s adviser, Prof IJ Nagrath, also pitched in when the team got stuck at a point. The mathematical model of how the wheel interacted with the ground was extremely non-linear and the team had to figure out a way to take the non-linear dynamics and implement in the computer. Hydra had a double-gear mechanism in its wheels and there was no way to cut the gear in that set-up. The team finally built it in aluminum and welded in a set-up in Coimbatore that allowed them to put things together without warping it.
Kiran and Krishnanand Kaipa with the Hydra robot.
Quadruped: This was Kiran’s undergrad thesis. He built the first prototype with an idea that most of the work was done fully by a mechanism and then introducing automatic controls in a few places.
Hexabot: Kiran went to the Centre for Artificial Intelligence and Robotics (CAIR) Bengaluru and built a hexabot. He had two papers from this project which came handy for his admission to higher studies.
Looking back, Kiran believes that the five years (for the dual degree) at Pilani gave him enough time to sink his teeth into something substantial.
Eventually, he graduated with 9 CGPA and enrolled for PhD at CMU in 1998. Choosing the college was a no-brainer given the fact that it was the best university in robotics then.
Robotics at CMU
Kiran felt that CMU was like BITS Pilani except that it was much smaller. His class had 16 students from different academic (AI, perceptive computing, hardware) and geographical background (Chile, Mexico, Europe, China, India, Canada, and the US). Since the robotics students tend to attend most of the computer science (CS) classes and a few dedicated robotics classes (like control theory), Kiran ended up interacting with a lot of CS students.
A lot of research work going then (1998-2004) formed the basis of deep learning and AI revolution of today. At that time, they were considered to be esoteric research topics.
In the first two years at CMU, Kiran was a part of two all-terrain vehicles (ATV) robotics team. The vehicles would roam around the grounds of CMU and communicate with each other. Kiran was working on the perception aspect of the ATVs, ie how to identify the sidewalks and roads. He says, “Pittsburgh has crazy weather and it snows a lot. Snow with slush becomes grey snow. The robot has to see the environment and make sense of the world.”
Baby steps to ‘Hulk’ steps
The Academy of Motion Picture Arts and Sciences’ Scientific and Technical Achievement Awards on February 11, 2017, in Beverly Hills, California.
In the beginning, Kiran and his team started with pilots like face replacement (automated using algorithms) for a few minutes in a movie. They then worked on games on Star Wars and Ninja Turtles among others. He recalls one of his biggest challenges,
Animators thought of characters as creative things and were not willing to believe that a computer could do that. We had to change this mindset. The technology is faithfully reproducing what the actors acted in an editable format.
Mark Ruffalo (who played the character of Hulk) was very happy with this technology. Duncan Jones was impressed by the work of IL&M with the Hulk and proposed to make his next movie specifically based on actors’ acting (and use technology to automate the entire animation).
But the biggest among all was when Disney decided to recreate a digital version of Rogue One. Kiran recalls, “It was a risky and a bold decision because it could have destroyed the credibility among the entire fan base (if it went wrong).”
But it paid off and the technology formed the front and centre of the movie. While working on the movie, IL&M team realised that Guy Henry’s mouth would never match Peter Cushing’s lips (very thin). It was difficult to mimic and animators had to do additionally patch up.
Finally, in 2017, Kiran and his team won the Academy Award for making CGI faces that look and feel like real human faces. He says, “The trick is same. You look at real people, analyse their faces with a program, and make the computer version do the same.”
Kiran gives an indication that Loom.ai avatars can be seen soon in few apps. The startup is working with some companies who’re to deploy this technology. Summing up his startup experience, Kiran says,
Starting a company is like having a child. I’ve twins so I know. It’s surprisingly satisfying to start something of your own. It’s challenging as hell most of the times but it’s unlike anything I would have done at Lucasfilm (IL&M). It tests you in a way that you’ve never been tested before. There’s no parachute. You’re in a free fall.
Kiran feels that the team at Loom.ai is very similar to the composition of teams he would have had at IL&M. He says, “If it works out, then more people would have used Loom.ai than any of the projects we’ve worked on ever.”
This article is an expert taken from Author: ALOK SONI. We thank him for bringing in this excellent story of Mr Kiran