Machines That Learn Language More Like Kids Do
November 1, 2018 | MITEstimated reading time: 6 minutes
The expression with the most closely matching representations for objects, humans, and actions becomes the most likely meaning of the caption. The expression, initially, may refer to many different objects and actions in the video, but the set of possible meanings serves as a training signal that helps the parser continuously winnow down possibilities. “By assuming that all of the sentences must follow the same rules, that they all come from the same language, and seeing many captioned videos, you can narrow down the meanings further,” Barbu says.
In short, the parser learns through passive observation: To determine if a caption is true of a video, the parser by necessity must identify the highest probability meaning of the caption. “The only way to figure out if the sentence is true of a video [is] to go through this intermediate step of, ‘What does the sentence mean?’ Otherwise, you have no idea how to connect the two,” Barbu explains. “We don’t give the system the meaning for the sentence. We say, ‘There’s a sentence and a video. The sentence has to be true of the video. Figure out some intermediate representation that makes it true of the video.’”
The training produces a syntactic and semantic grammar for the words it’s learned. Given a new sentence, the parser no longer requires videos, but leverages its grammar and lexicon to determine sentence structure and meaning.
Ultimately, this process is learning “as if you’re a kid,” Barbu says. “You see world around you and hear people speaking to learn meaning. One day, I can give you a sentence and ask what it means and, even without a visual, you know the meaning.”
“This research is exactly the right direction for natural language processing,” says Stefanie Tellex, a professor of computer science at Brown University who focuses on helping robots use natural language to communicate with humans. “To interpret grounded language, we need semantic representations, but it is not practicable to make it available at training time. Instead, this work captures representations of compositional structure using context from captioned videos. This is the paper I have been waiting for!”
In future work, the researchers are interested in modeling interactions, not just passive observations. “Children interact with the environment as they’re learning. Our idea is to have a model that would also use perception to learn,” Ross says.
This work was supported, in part, by the CBMM, the National Science Foundation, a Ford Foundation Graduate Research Fellowship, the Toyota Research Institute, and the MIT-IBM Brain-Inspired Multimedia Comprehension project.
Page 2 of 2Suggested Items
Un-Jammable Quantum Tech Takes Flight to Boost UK’s Resilience Against Hostile Actors
05/13/2024 | BUSINESS WIREIPC Focuses on Education and Onboarding
05/09/2024 | Andy Shaughnessy, Design007 MagazineI recently spoke with Carlos Plaza, senior director of education for IPC, about expanding educational efforts in the PCB design, fabrication, and assembly segments. As Carlos explains, PCB design is a hot topic, but onboarding may be the hottest one of all.
It’s Only Common Sense: Would You Join Your Own Company?
05/06/2024 | Dan Beaulieu -- Column: It's Only Common SenseIn the past few years, I have heard many company runners complaining about their workforce. They tell me that the government is paying people too much money not to work, too many young people are not interested in working every day, and there is just not the work ethic there once was when they were young.
Warm Windows and Streamlined Skin Patches – IDTechEx Explores Flexible and Printed Electronics
04/26/2024 | IDTechExFlexible and printed electronics can be integrated into cars and homes to create modern aesthetics that are beneficial and easy to use. From luminous car controls to food labels that communicate the quality of food, the uses of this technology are endless and can upgrade many areas of everyday life.
Scientists Propose a New Way to Search for Dark Matter
04/02/2024 | SLAC National Accelerator LaboratoryEver since its discovery, dark matter has remained invisible to scientists, despite the launch of multiple ultra-sensitive particle detector experiments around the world over several decades.