Measuring AI's Ability to Learn is Difficult
January 17, 2019 | University of WaterlooEstimated reading time: 1 minute

Organizations looking to benefit from the artificial intelligence (AI) revolution should be cautious about putting all their eggs in one basket, a study from the University of Waterloo has found.
In a study published in Nature Machine Intelligence, Waterloo researchers found that contrary to conventional wisdom, there can be no exact method for deciding whether a given problem may be successfully solved by machine learning tools.
“We have to proceed with caution,” said Shai Ben-David, lead author of the study and a professor in Waterloo’s School of Computer Science. “There is a big trend of tools that are very successful, but nobody understands why they are successful, and nobody can provide guarantees that they will continue to be successful.
“In situations where just a yes or no answer is required, we know exactly what can or cannot be done by machine learning algorithms. However, when it comes to more general setups, we can’t distinguish learnable from un-learnable tasks.”
In the study, Ben-David and his colleagues considered a learning model called estimating the maximum (EMX), which captures many common machine learning tasks. For example, tasks like identifying the best place to locate a set of distribution facilities to optimize their accessibility for future expected consumers. The research found that no mathematical method would ever be able to tell, given a task in that model, whether an AI-based tool could handle that task or not.
“This finding comes as a surprise to the research community since it has long been believed that once a precise description of a task is provided, it can then be determined whether machine learning algorithms will be able to learn and carry out that task,” said Ben-David.
The study, Learnability can be Undecidable, was co-authored by Ben-David, Pavel Hrubeš from the Institute of Mathematics of the Academy of Sciences in the Czech Republic, Shay Morgan from the Department of Computer Science, Princeton University, Amir Shpilka, Department of Computer Science, Tel Aviv University, and Amir Yehudayoff from the Department of Mathematics, Technion-IIT.
Suggested Items
Smarter Machines Use AOI to Transform PCB Inspections
06/30/2025 | Marcy LaRont, PCB007 MagazineAs automated optical inspection (AOI) evolves from traditional end-of-process inspections to proactive, in-line solutions, the integration of AI and machine learning is revolutionizing defect reduction and enhancing yields, marking a pivotal shift in how quality is managed in manufacturing.
Technica USA Announces New Strategic Partnership with I.T.C. Intercircuit Production GmbH
06/24/2025 | Technica USATechnica USA is pleased to announce a new distribution and representative agreement with I.T.C. Intercircuit Production GmbH, a globally recognized manufacturer of advanced equipment for the PCB manufacturing industry.
Smart Automation: The Power of Data Integration in Electronics Manufacturing
06/24/2025 | Josh Casper -- Column: Smart AutomationAs EMS companies adopt automation, machine data collection and integration are among the biggest challenges. It’s now commonplace for equipment to collect and output vast amounts of data, sometimes more than a manufacturer knows what to do with. While many OEM equipment vendors offer full-line solutions, most EMS companies still take a vendor-agnostic approach, selecting the equipment companies that best serve their needs rather than a single-vendor solution.
Sierra Circuits Boosts High Precision PCB Manufacturing with Schmoll Technology
06/16/2025 | Schmoll MaschinenSierra Circuits has seen increased success in production of multilayer HDI boards and high-speed signal architectures through the integration of a range of Schmoll Maschinen systems. The company’s current setup includes four MXY-6 drilling machines, two LM2 routing models, and a semi-automatic Optiflex II innerlayer punch.
MVTec, Siemens Expand Technological Cooperation
06/12/2025 | MVTecMVTec Software GmbH and Siemens are expanding their technological cooperation in the field of industrial automation. To reinforce their increasingly close collaboration, Siemens joined the MVTec Technology Partner Program in May 2025.