The Knowledge Base: A Conference for Cleaning and Coating of Mission-critical Electronics
Defense Speak Interpreted: Is DARPA Still Around After CHIPS?
Beyond the Board: Orbital High Ground—Why Space Superiority Is Slipping Away
Speaking Robot-nese: A New Multi-Modal, Multi-Step Robot Game
June 13, 2017 | New York UniversityEstimated reading time: 1 minute
As robots take over our world, they must not only learn how to communicate with us but also with each other. Recent scholarship has so far demonstrated that it’s possible for two robots to communicate in a shared language in the form of binary vectors. These conversations between the sender (Robot A) and receiver (Robot B) are typically mono-directional, and limited to a fixed number of yes/no answers.
But CDS Master’s student Katrina Evtimova, professor Kyunghyun Cho, Andrew Drozdov (NYU Computer Science), and Douwe Keila (Facebook) all believe that our robots can do better. In “Emergent Language in a Multi-Modal, Multi-Step Referential Game,” they have invented a new conversational game for robots that mimics human communication more closely.
First, the robots in their new game are prompted to have bi-directional conversations, meaning that they are allowed to send as many messages to each other as they would like. Second, they must use their shared language to exchange two different modes of information: text and image.
The game itself, then, unfolds something like this. The sender robot is shown photograph of a mammal, and then instructed to communicate what it saw to the receiving robot . The receiver is prompted to guess what animal the sender saw by reading the textual data, and then asking the sender additional questions about the photograph. This means that the conversation is not merely about shifting information from one robot to another, but exchanging different modes of information (e.g. textual and pictorial) between them. To help the robots, the researchers harnessed the power of neural networks and implemented techniques like visual and textual attention into the game.
The researchers found that the multi-modal, multi-step game not only improved the robots’ predictive accuracy, but that their conversations also became more human-like. Will these developments bring humans and machines closer together—or will we become too close for comfort?
Original by: Cherrie Kwok
Share on:
Suggested Items
2025 ASEAN IT Spending Growth Slows to 5.9% as AI-Powered IT Expansion Encounters Post-Boom Normalization
06/26/2025 | IDCAccording to the IDC Worldwide Black Book: Live Edition, IT spending across ASEAN is projected to grow by 5.9% in 2025 — down from a robust 15.0% in 2024.
Rethinking How Operators Interface With the Line
06/11/2025 | Nolan Johnson, SMT007 MagazineJurgen Schmerler, CEO of WaveOn, reveals how AI and large language models are revolutionizing electronics manufacturing. By integrating AI with machinery, operators can access real-time, multimodal information for troubleshooting and maintenance, significantly reducing training time and enhancing efficiency. He discusses the industry's challenges, the customizable knowledge bases, and the future of proactive maintenance and process control.
Standards: The Roadmap for Your Ideal Data Package
05/29/2025 | Andy Shaughnessy, Design007 MagazineIn this interview, IPC design instructor Kris Moyer explains how standards can help you ensure that your data package has all the information your fabricator and assembler need to build your board the way you designed it, allowing them to use their expertise. As Kris says, even with IPC standards, there’s still an art to conveying the right information in your documentation.
Future-proofing Electronics: ChemFORWARD Works Toward Collaboration for Safer Chemistry
05/19/2025 | Rachel Simon, ChemFORWARDThe electronics industry is facing a critical juncture. As consumer demand for sustainable products rises and regulatory pressures intensify, companies must prioritize the safety of their products and processes. This means not only complying with evolving chemical restrictions but also proactively seeking safer alternatives.
CACI’s Mission-Critical Technology will Accelerate the Delivery of Electronic Warfighting Capabilities to the U.S. Navy’s Existing Fleet
05/13/2025 | CACI International Inc.CACI International Inc announced today that it has been awarded additional work by the U.S. Navy to procure enhancements to the current fielded Shipboard Information Warfare Exploit system under its existing contract for Spectral, a next-generation shipboard signals intelligence (SIGINT), electronic warfare (EW), and information operations (IO) weapon system.