Andrew Ng has worn many hats in his life. You may know him as the founder of the Google Brain team or the former chief scientist at Baidu. You may also know him as your own instructor. He has taught countless students, curious listeners, and business leaders about the principles of machine learning through his wildly popular online courses.
Now in his latest venture, Landing AI, which he started in 2017, he is exploring how businesses without giant data sets to draw on can still join in the AI revolution.
On March 23, Ng joined MIT Technology Review’s virtual EmTech Digital, our annual AI event, to share the lessons he’s learned.
This interview has been condensed and lightly edited for clarity.
MIT Technology Review: I’m sure people frequently ask you, “How do I build an AI-first business?” What do you usually say to that?
Andrew Ng: I usually say, “Don’t do that.” If I go to a team and say, “Hey, everyone, please be AI-first,” that tends to focus the team on technology, which might be great for a research lab. But in terms of how I execute the business, I tend to be customer-led or mission-led, almost never technology-led.
You now have this new venture called Landing AI. Can you tell us a bit about what it is, and why you chose to work on it?
After heading the AI teams at Google and Baidu, I realized that AI has transformed software consumer internet, like web search and online advertising. But I wanted to take AI to all of the other industries, which is an even bigger part of the economy. So after looking at a lot of different industries, I decided to focus on manufacturing. I think that multiple industries are AI-ready, but one of the patterns for an industry being more AI-ready is if it’s undergone some digital transformation so there’s some data. That creates an opportunity for AI teams to come in to use the data to create value.
So one of the projects that I’ve been excited about recently is manufacturing visual inspection. Can you look at a picture of a smartphone coming off the manufacturing line and see if there’s a defect in it? Or look at an auto component and see if there’s a dent in it? One huge difference is in consumer software internet, maybe you have a billion users and a huge amount of data. But in manufacturing, no factory has manufactured a billion or even a million scratched smartphones. Thank goodness for that. So the challenge is, can you get an AI to work with a hundred images? It turns out often you can. I’ve actually been surprised quite a lot of times with how much you can do with even modest amounts of data. And so even though all the hype and excitement and PR around AI is on the giant data sets, I feel like there’s a lot of room we need to grow as well to break open these other applications where the challenges are quite different.
How do you do that?
A very frequent mistake I see CEOs and CIOs make: they say to me something like “Hey, Andrew, we don’t have that much data—my data’s a mess. So give me two years to build a great IT infrastructure. Then we’ll have all this great data on which to build AI.” I always say, “That’s a mistake. Don’t do that.” First, I don’t think any company on the planet today—maybe not even the tech giants—thinks their data is completely clean and perfect. It’s a journey. Spending two or three years to build a beautiful data infrastructure means that you’re lacking feedback from the AI team to help prioritize what IT infrastructure to build.
For example, if you have a lot of users, should you prioritize asking them questions in a survey to get a little bit more data? Or in a factory, should you prioritize upgrading the sensor from something that records the vibrations 10 times a second to maybe 100 times a second? It is often starting to do an AI project with the data you already have that enables an AI team to give you the feedback to help prioritize what additional data to collect.
In industries where we just don’t have the scale of consumer software internet, I feel like we need to shift in mindset from big data to good data. If you have a million images, go ahead, use it—that’s great. But there are lots of problems that can use much smaller data sets that are cleanly labeled and carefully curated.
Could you give an example? What do you mean by good data?
Let me first give an example from speech recognition. When I was working with voice search, you would get audio clips where you would hear someone say, “Um today’s weather.” The question is, what is the right transcription for that audio clip? Is it “Um (comma) today’s weather,” or is it “Um (dot, dot, dot) today’s weather,” or is the “Um” something we just don’t transcribe? It turns out any one of these is fine, but what is not fine is if different transcribers use each of the three labeling conventions. Then your data is noisy, and it hurts the speech recognition system. Now, when you have millions or a billion users, you can have that noisy data and just average it—the learning algorithm will do fine. But if you are in a setting where you have a smaller data set—say, a hundred examples—then this type of noisy data has a huge impact on performance.
Another example from manufacturing: we did a lot of work on steel inspection. If you drive a car, the side of your car was once made of a sheet of steel. Sometimes there are little wrinkles in the steel, or little dents or specks on it. So you can use a camera and computer vision to see if there are defects or not. But different labelers will label the data differently. Some will put a giant bounding box around the whole region. Some will put little bounding boxes around the little particles. When you have a modest data set, making sure that the different quality inspectors label the data consistently—that turns out to be one of the most important things.
For a lot of AI projects, the open-source model you download off GitHub—the neural network that you can get from literature—is good enough. Not for all problems, but the main problems. So I’ve gone to many of my teams and said, “Hey, everyone, the neural network is good enough. Let’s not mess with the code anymore. The only thing you’re going to do now is build processes to improve the quality of the data.” And it turns out that often results in faster improvements to performance of the algorithm.
What is the data size you are thinking about when you say smaller data sets? Are you talking about a hundred examples? Ten examples?
Machine learning is so diverse that it’s become really hard to give one-size-fits-all answers. I’ve worked on problems where I had about 200 to 300 million images. I’ve also worked on problems where I had 10 images, and everything in between. When I look at manufacturing applications, I think something like tens or maybe a hundred images for a defect class is not unusual, but there’s very wide variance even within the factory.
I do find that the AI practices switch over when the training set sizes go under, let’s say, 10,000 examples, because that’s sort of the threshold where the engineer can basically look at every example and design it themselves and then make a decision.
Recently I was chatting with a very good engineer in one of the large tech companies. And I asked, “Hey, what do you do if the labels are inconsistent?” And he said, “Well, we have this team of several hundred people overseas that does the labeling. So I’ll write the labeling instructions, get three people to label every image, and then I’ll take an average.” And I said, “Yep, that’s the right thing to do when you have a giant data set.” But when I work with a smaller team and the labels are inconsistent, I just track down the two people that disagree with each other, get both of them on a Zoom call, and have them talk to each other to try to reach a resolution.
I want to turn our attention now to talk about your thoughts on the general AI industry. The Algorithm is our AI newsletter, and I gave our readers an opportunity to submit some questions to you in advance. One reader asks: AI development seems to have mostly bifurcated toward either academic research or large-scale, resource-intensive, big company programs like OpenAI and DeepMind. That doesn’t really leave a lot of space for small startups to contribute. What do you think are some practical problems that smaller companies can really focus on to help drive real commercial adoption of AI?
I think a lot of the media attention tends to be on the large corporations, and sometimes on the large academic institutions. But if you go to academic conferences, there’s plenty of work done by smaller research groups and research labs. And when I speak with different people in different companies and industries, I feel like there are so many business applications they could use AI to tackle. I usually go to business leaders and ask, “What are your biggest business problems? What are the things that worry you the most?” so I can better understand the goals of the business and then brainstorm whether or not there is an AI solution. And sometimes there isn’t, and that’s fine.
Maybe I’ll just mention a couple of gaps that I find exciting. I think that today building AI systems is still very manual. You have a few brilliant machine-learning engineers and data scientists do things in a computer and then push things to production. There’s a lot of manual steps in the process. So I’m excited about ML ops [machine learning operations] as an emerging discipline to help make the process of building and deploying AI systems more systematic.
Also, if you look at a lot of the typical business problems—all the functions from marketing to talent—there’s a lot of room for automation and efficiency improvement.
I also hope that the AI community can look at the biggest social problems—see what we can do for climate change or homelessness or poverty. In addition to the sometimes very valuable business problems, we should work on the biggest social problems too.
How do you actually go about the process of identifying whether there is an opportunity to pursue something with machine learning for your business?
I will try to learn a little bit about the business myself and try to help the business leaders learn a little bit about AI. Then we usually brainstorm a set of projects, and for each of the ideas, I will do both technical diligence and business diligence. We’ll look at: Do you have enough data? What’s the accuracy? Is there a long tail when you deploy into production? How do you fill the data back and close the loop for continuous learning? So—making sure the problem is technically feasible. And then business diligence: we make sure that this will achieve the ROI that we’re hoping for. After that process, you have the usual, like estimating the resources, milestones, and then hopefully going into execution.
One other suggestion: it’s more important to start quickly, and it’s okay to start small. My first meaningful business application at Google was speech recognition, not web search or advertising. But by helping the Google speech team make speech recognition more accurate, that gave the Brain team the credibility and the wherewithal to go after bigger and bigger partnerships. So Google Maps was the second big partnership where we used computer vision—to read house numbers to geolocate houses on Google maps. And only after those first two successful projects did I have a more serious conversation with the advertising team. So I think I see more companies fail by starting too big than fail by starting too small. It’s fine to do a smaller project to get started as an organization to learn what it feels like to use AI, and then go on to build bigger successes.
What is one thing that our audience should start doing tomorrow to implement AI in their companies?
Jump in. AI is causing a shift in the dynamics of many industries. So if your company isn’t already making pretty aggressive and smart investments, this is a good time.