Cracking Open the Black Box of Automated Machine Learning

Estimated reading time: 5 minutes

ATMSeer’s interface consists of three parts. A control panel allows users to upload datasets and an AutoML system, and start or pause the search process. Below that is an overview panel that shows basic statistics — such as the number of algorithms and hyperparameters searched — and a “leaderboard” of top-performing models in descending order. “This might be the view you’re most interested in if you’re not an expert diving into the nitty gritty details,” Veeramachaneni says.

ATMSeer includes an “AutoML Profiler,” with panels containing in-depth information about the algorithms and hyperparameters, which can all be adjusted. One panel represents all algorithm classes as histograms — a bar chart that shows the distribution of the algorithm’s performance scores, on a scale of 0 to 10, depending on their hyperparameters. A separate panel displays scatter plots that visualize the tradeoffs in performance for different hyperparameters and algorithm classes.

Case studies with machine-learning experts, who had no AutoML experience, revealed that user control does help improve the performance and efficiency of AutoML selection. User studies with 13 graduate students in diverse scientific fields — such as biology and finance — were also revealing. Results indicate three major factors — number of algorithms searched, system runtime, and finding the top-performing model — determined how users customized their AutoML searches. That information can be used to tailor the systems to users, the researchers say.

“We are just starting to see the beginning of the different ways people use these systems and make selections,” Veeramachaneni says. “That’s because now that this information is all in one place, and people can see what’s going on behind the scenes and have the power to control it.”

• < Previous Page