Developing Trustworthy AI to Inform Decisions When Every Moment Counts

Estimated reading time: 2 minutes

DARPA has selected performers for the In the Moment (ITM) program to create the foundation for trusted algorithmic decision-making in challenging domains, such as medical triage.

ITM has two phases that address decision-making at different degrees of complexity. The first phase will look at triage for small military units in austere environments, and the second phase will scale up the complexity of decision-making by looking at triage for mass casualty events. The program is separated into four technical areas that the following teams will support:

• Raytheon BBN Technologies and Soar Technology, Inc. will develop decision-maker characterization techniques that identify and quantify key human decision-maker attributes in difficult domains.
• Kitware, Inc. and Parallax Inc. will develop algorithmic decision-makers that demonstrate alignment with key attributes of trusted human decision-makers.
• CACI International Inc. will design and execute the program evaluation, focusing on how key human attributes may lead to the trusted delegation of decision-making.
• The University of Maryland Applied Research Laboratory for Intelligence and Security and the Institute for Defense Analyses will be responsible for policy/practice integration and outreach, including ethical, legal, and societal implications (ELSI) experts to advise throughout the research process.
• ITM will produce a framework for developing algorithms that can express key attributes that are aligned with trusted humans. If the foundational ITM technology is successful, the framework will inform future operational systems that rely on trusted decision-making algorithms.

“Key attributes might include how an algorithm evaluates a situation, how it relies on domain knowledge, how it responds to time pressures, and what principles or values it uses to prioritize care,” said Dr. Matt Turek, DARPA’s ITM program manager and deputy director of the Information Innovation Office. “From a technical standpoint, difficult decisions made in medical triage will likely require approaches that do not rely primarily on training data for their implementation, as those approaches can be notoriously brittle.”

Turek previously described difficult decisions as those where trusted decision-makers disagree; no correct answer exists; and uncertainty, time pressure, and conflicting values create significant decision-making challenges.

“The triage domain allows us to get at core issues around trust and delegating decision-making to go beyond the state of the art in AI,” he said. “The focus on triage will encourage research teams to work directly on some of the hardest decision-making challenges possible.”

Turek says he anticipates ITM advances will ultimately support fully automated and semi-automated decision-making where humans can choose to override the algorithm.