Identifying Artificial Intelligence 'Blind Spots'
January 25, 2019 | MITEstimated reading time: 6 minutes
Once the feedback data from the human is compiled, the system essentially has a list of situations and, for each situation, multiple labels saying its actions were acceptable or unacceptable. A single situation can receive many different signals, because the system perceives many situations as identical. For example, an autonomous car may have cruised alongside a large car many times without slowing down and pulling over. But, in only one instance, an ambulance, which appears exactly the same to the system, cruises by. The autonomous car doesn’t pull over and receives a feedback signal that the system took an unacceptable action.
“At that point, the system has been given multiple contradictory signals from a human: some with a large car beside it, and it was doing fine, and one where there was an ambulance in the same exact location, but that wasn’t fine. The system makes a little note that it did something wrong, but it doesn’t know why,” Ramakrishnan says. “Because the agent is getting all these contradictory signals, the next step is compiling the information to ask, ‘How likely am I to make a mistake in this situation where I received these mixed signals?’”
Intelligent Aggregation
The end goal is to have these ambiguous situations labeled as blind spots. But that goes beyond simply tallying the acceptable and unacceptable actions for each situation. If the system performed correct actions nine times out of 10 in the ambulance situation, for instance, a simple majority vote would label that situation as safe.
“But because unacceptable actions are far rarer than acceptable actions, the system will eventually learn to predict all situations as safe, which can be extremely dangerous,” Ramakrishnan says.
To that end, the researchers used the Dawid-Skene algorithm, a machine-learning method used commonly for crowdsourcing to handle label noise. The algorithm takes as input a list of situations, each having a set of noisy “acceptable” and “unacceptable” labels. Then it aggregates all the data and uses some probability calculations to identify patterns in the labels of predicted blind spots and patterns for predicted safe situations. Using that information, it outputs a single aggregated “safe” or “blind spot” label for each situation along with a its confidence level in that label. Notably, the algorithm can learn in a situation where it may have, for instance, performed acceptably 90 percent of the time, the situation is still ambiguous enough to merit a “blind spot.”
In the end, the algorithm produces a type of “heat map,” where each situation from the system’s original training is assigned low-to-high probability of being a blind spot for the system.
“When the system is deployed into the real world, it can use this learned model to act more cautiously and intelligently. If the learned model predicts a state to be a blind spot with high probability, the system can query a human for the acceptable action, allowing for safer execution,” Ramakrishnan says.
Page 2 of 2Suggested Items
PCB Market Size to Grow by $29.06B from 2024-2028
05/17/2024 | PRNewswireThe global printed circuit board (PCB) market size is estimated to grow by USD 29.06 bn from 2024-2028, according to Technavio. The market is estimated to grow at a CAGR of over 6.6% during the forecast period.
Terran Orbital Joins the Mobile Satellite Services Association (MSSA)
05/16/2024 | BUSINESS WIRETerran Orbital Corporation, a global leader in satellite-based solutions primarily serving the aerospace and defense industries, announced its membership in the Mobile Satellite Services Association (MSSA).
Raytheon-Kongsberg System is Proving Itself – In Demos and In Combat
05/16/2024 | Raytheon TechnologiesOn a rocky island off the coast of Norway, Michael Sadlowski watched as a surface-to-air missile launcher pointed up into a dark nighttime sky.
Un-Jammable Quantum Tech Takes Flight to Boost UK’s Resilience Against Hostile Actors
05/13/2024 | BUSINESS WIREOSI Systems Receives Order for $9 Million to Provide Cargo and Vehicle Inspection Systems
05/13/2024 | BUSINESS WIREOSI Systems, Inc. announced that its Security division received an order from an international customer for approximately $9 million to provide the Company’s Eagle® M60 high energy mobile cargo and vehicle inspection systems including related service and support.