Identifying Artificial Intelligence 'Blind Spots'
January 25, 2019 | MITEstimated reading time: 6 minutes

A novel model developed by MIT and Microsoft researchers identifies instances in which autonomous systems have “learned” from training examples that don’t match what’s actually happening in the real world. Engineers could use this model to improve the safety of artificial intelligence systems, such as driverless vehicles and autonomous robots.
Image Caption: A model by MIT and Microsoft researchers identifies instances where autonomous cars have “learned” from training examples that don’t match what’s actually happening on the road, which can be used to identify which learned actions could cause real-world errors.
The AI systems powering driverless cars, for example, are trained extensively in virtual simulations to prepare the vehicle for nearly every event on the road. But sometimes the car makes an unexpected error in the real world because an event occurs that should, but doesn’t, alter the car’s behavior.
Consider a driverless car that wasn’t trained, and more importantly doesn’t have the sensors necessary, to differentiate between distinctly different scenarios, such as large, white cars and ambulances with red, flashing lights on the road. If the car is cruising down the highway and an ambulance flicks on its sirens, the car may not know to slow down and pull over, because it does not perceive the ambulance as different from a big white car.
In a pair of papers — presented at last year’s Autonomous Agents and Multiagent Systems conference and the upcoming Association for the Advancement of Artificial Intelligence conference — the researchers describe a model that uses human input to uncover these training “blind spots.”
As with traditional approaches, the researchers put an AI system through simulation training. But then, a human closely monitors the system’s actions as it acts in the real world, providing feedback when the system made, or was about to make, any mistakes. The researchers then combine the training data with the human feedback data, and use machine-learning techniques to produce a model that pinpoints situations where the system most likely needs more information about how to act correctly.
The researchers validated their method using video games, with a simulated human correcting the learned path of an on-screen character. But the next step is to incorporate the model with traditional training and testing approaches for autonomous cars and robots with human feedback.
“The model helps autonomous systems better know what they don’t know,” says first author Ramya Ramakrishnan, a graduate student in the Computer Science and Artificial Intelligence Laboratory. “Many times, when these systems are deployed, their trained simulations don’t match the real-world setting [and] they could make mistakes, such as getting into accidents. The idea is to use humans to bridge that gap between simulation and the real world, in a safe way, so we can reduce some of those errors.”
Co-authors on both papers are: Julie Shah, an associate professor in the Department of Aeronautics and Astronautics and head of the CSAIL’s Interactive Robotics Group; and Ece Kamar, Debadeepta Dey, and Eric Horvitz, all from Microsoft Research. Besmira Nushi is an additional co-author on the upcoming paper.
Taking Feedback
Some traditional training methods do provide human feedback during real-world test runs, but only to update the system’s actions. These approaches don’t identify blind spots, which could be useful for safer execution in the real world.
The researchers’ approach first puts an AI system through simulation training, where it will produce a “policy” that essentially maps every situation to the best action it can take in the simulations. Then, the system will be deployed in the real-world, where humans provide error signals in regions where the system’s actions are unacceptable.
Humans can provide data in multiple ways, such as through “demonstrations” and “corrections.” In demonstrations, the human acts in the real world, while the system observes and compares the human’s actions to what it would have done in that situation. For driverless cars, for instance, a human would manually control the car while the system produces a signal if its planned behavior deviates from the human’s behavior. Matches and mismatches with the human’s actions provide noisy indications of where the system might be acting acceptably or unacceptably.
Page 1 of 2
Testimonial
"Our marketing partnership with I-Connect007 is already delivering. Just a day after our press release went live, we received a direct inquiry about our updated products!"
Rachael Temple - AlltematedSuggested Items
Light-curable Solutions for Reliable Electronics in Space Applications
10/15/2025 | Virginia Hogan, DymaxDesigning electronics for space environments, particularly those in low Earth orbit (LEO), requires careful consideration of materials that can withstand extreme conditions while supporting long-term reliability. Engineers designing satellite systems, aerospace instrumentation, and high-altitude platforms face a familiar set of challenges: contamination control, mechanical stress, thermal cycling, and manufacturability.
Analog Devices Launches ADI Power Studio™ and New Web-Based Tools
10/14/2025 | Analog Devices, Inc.Analog Devices, Inc., a global semiconductor leader, announced the launch of ADI Power Studio, a comprehensive family of products that offers advanced modeling, component recommendations and efficiency analysis with simulation. In addition, ADI is introducing early versions of two new web-based tools with a modernized user experience under the Power Studio umbrella:
Elementary, Mr. Watson: High Power: When Physics Becomes Real
10/15/2025 | John Watson -- Column: Elementary, Mr. WatsonHave you ever noticed how high-speed design and signal integrity classes are always packed to standing room only, but just down the hall, the session on power electronics has plenty of empty chairs? It's not just a coincidence; it's a trend I've observed over the years as both an attendee and instructor.
Beyond Thermal Conductivity: Exploring Polymer-based TIM Strategies for High-power-density Electronics
10/13/2025 | Padmanabha Shakthivelu and Nico Bruijnis, MacDermid Alpha Electronics SolutionsAs power density and thermal loads continue to increase, effective thermal management becomes increasingly important. Rapid and efficient heat transfer from power semiconductor chip packages is essential for achieving optimal performance and ensuring long-term reliability of temperature-sensitive components. This is particularly crucial in power systems that support advanced applications such as green energy generation, electric vehicles, aerospace, and defense, along with high-speed computing for data centers and artificial intelligence (AI).
Bluepath Robotics Optimizes AMR Fleets with Inductive Charging Solution from Wiferion
10/09/2025 | WiferionIn a dynamic and highly competitive industry such as logistics, efficient and uninterrupted material flows are of crucial importance. To ensure maximum uptime for its robots, Bluepath Robotics, which specializes in autonomous mobile robots (AMR), needed a reliable and powerful power supply.