Defense Speak Interpreted: Update on the Continuing Resolution and Budget Process for Defense
American Made Advocacy: Reshoring—About Trust, Not Just Geography
Beyond the Board: The Future of Innovation—Why the Electronics Industry Needs You
Speaking Robot-nese: A New Multi-Modal, Multi-Step Robot Game
June 13, 2017 | New York UniversityEstimated reading time: 1 minute
As robots take over our world, they must not only learn how to communicate with us but also with each other. Recent scholarship has so far demonstrated that it’s possible for two robots to communicate in a shared language in the form of binary vectors. These conversations between the sender (Robot A) and receiver (Robot B) are typically mono-directional, and limited to a fixed number of yes/no answers.
But CDS Master’s student Katrina Evtimova, professor Kyunghyun Cho, Andrew Drozdov (NYU Computer Science), and Douwe Keila (Facebook) all believe that our robots can do better. In “Emergent Language in a Multi-Modal, Multi-Step Referential Game,” they have invented a new conversational game for robots that mimics human communication more closely.
First, the robots in their new game are prompted to have bi-directional conversations, meaning that they are allowed to send as many messages to each other as they would like. Second, they must use their shared language to exchange two different modes of information: text and image.
The game itself, then, unfolds something like this. The sender robot is shown photograph of a mammal, and then instructed to communicate what it saw to the receiving robot . The receiver is prompted to guess what animal the sender saw by reading the textual data, and then asking the sender additional questions about the photograph. This means that the conversation is not merely about shifting information from one robot to another, but exchanging different modes of information (e.g. textual and pictorial) between them. To help the robots, the researchers harnessed the power of neural networks and implemented techniques like visual and textual attention into the game.
The researchers found that the multi-modal, multi-step game not only improved the robots’ predictive accuracy, but that their conversations also became more human-like. Will these developments bring humans and machines closer together—or will we become too close for comfort?
Original by: Cherrie Kwok
Share on:
Suggested Items
RTX's Collins Aerospace Enhances Capabilities to Speed Marine Corps Decision-making in Battle
04/22/2025 | RTXCollins Aerospace, an RTX business, successfully demonstrated new technology that helps the military gather and use information from a wider range of sources at Project Convergence Capstone 5, a large-scale military exercise.
IPC APEX EXPO 2025 Learning Lounge: Education on the Show Floor
04/16/2025 | Andy Shaughnessy, Design007The conference portion of IPC APEX EXPO has been providing educational opportunities for attendees since the first show. But recently, show managers decided to expand education onto the show floor.
INEMI Sustainable Electronics Tech Topic Webinar: Enabling New Life in Storage Devices
04/07/2025 | iNEMIHard disk drives (HDDs) and solid state drives (SSDs) are ubiquitous in electronic products and a large number enter the end-of-life stream prior to their true end of life.
Gartner Identifies Top 12 Early-Stage Technology Disruptions that Will Define the Future of Business Systems
04/07/2025 | Gartner, Inc.Gartner, Inc. has identified 12 emerging technology disruptions that will define the future of business systems. Technology leaders must prioritize these over the next five years, as they present competitive opportunities in the near term and will eventually grow to become standard throughout businesses.
IPC APEX EXPO 2025: We’ve Got It Covered
04/01/2025 | I-Connect007 Editorial TeamIPC APEX EXPO is the largest electronics manufacturing trade show in North America, bringing together professionals from all sectors of the supply chain to educate, network, and share their products and services. We’ve put all our coverage of the show in one easy-to-find location. Just click on “Videos,” “Show Coverage” and “Photos” to find what you’re looking for. Check back regularly as more content is added. You won’t want to miss any of our unique coverage.