Building Trust key to pushing Autonomous Military Technology past "Tipping Point"

Autonomous Military Technology at a "Tipping Point"

goals are maturing rapidly, the Defense Science Board said in a newly published study.


“Autonomy, fueled by advances in artificial intelligence, has attained a ‘tipping point’ in value,” the DSB study said. “Autonomy will deliver substantial operational value–in multiple dimensions–across an increasingly broad spectrum of DoD missions, but the DoD must move more rapidly to realize this value. Allies and adversaries alike also have access to increasingly rapid technological advances occurring globally,” the study said.

The recommendations of the study are briefly outlined in Table 4 at the end of the report, on page 102. Details for each recommendation are included in the relevant chapter sections and provide additional information to assist with their implementation. The Board recommended that the Department of Defense undertake a series of pilot projects “intended to demonstrate the range of benefits of autonomy for the warfighter.”

Establishing Trust is key to Autonomy

The Pentagon’s science advisers want military robots to operate with far greater autonomy than they do today. Only one problem: There’s a cloud of distrust and misunderstanding hovering over the robots that the Pentagon already has.

That’s an unexpected conclusion in a July study from the Defense Science Board, recently acquired by Steve Aftergood of the Federation of American Scientists. The Board wondered what’s inhibiting the development of autonomous military vehicles and other systems. It found that the humans who have to interact with robots in high-stakes situations often labor under the misimpression that autonomy means the machine can do a human’s job, rather than help a human do her job more efficiently. And some simply don’t have faith that the robots work as directed.

The issue of trust is core to DoD’s success in broader adoption of autonomy. On the one
hand, an autonomous system must be designed to operate in a trustworthy fashion with respect to the missions for which it was designed. On the other hand, an autonomous system must be designed so that humans (and/or machines) can straightforwardly determine whether, once it has been deployed, it is operating reliably and within its envelope of competence — and, if not, that appropriate action can be taken. Establishing trustworthiness at design time and providing adequate capabilities so that inevitable variations in operational trustworthiness can be assessed and dealt with at run time is essential, not only for operators and commanders, but also for designers, testers,
policymakers, lawmakers, and the American public. The broad topic of trust shaped many of the recommendations that follow.
The first set of recommendations focuses on accelerating DoD’s adoption of autonomous
capabilities and includes:

• Tackling the engineering, design, and acquisition challenges
• Mitigating cyber issues introduced by increasingly autonomous and networked systems
• Creating new test and evaluation and modeling and simulation paradigms
• Integrating technology insertion, doctrine, and concepts of operations
• Developing an autonomy-literate workforce
• Improving technology discovery 
• Improving DoD governance for autonomous systems
• Countering adversary use of autonomy

These interdependent recommendations are intended to build trust in autonomous systems,
while at the same time accelerating DoD’s progress.

strengthen the operational pull for autonomy

The next set of recommendations focuses on strengthening the operational pull for autonomy, both by better understanding how others may use autonomy against the U.S., and by equipping our forces to counter such capabilities. These recommendations take the form of a series of demonstrations and experiments that demonstrate near-term military value while also building warfighter trust:

• Autonomous agents to improve cyber-attack indicators and warnings
• Onboard autonomy for sensing
• Time-critical intelligence from seized media
• Dynamic spectrum management for protection missions
• Unmanned undersea vehicles (UUVs) to autonomously conduct sea mine counter-measures missions
• Automated cyber response
• Cascaded UUVs for offensive maritime mining
• Organic tactical unmanned aircraft (UA) to support ground forces
• Predictive logistics and adaptive planning
• Adaptive logistics for rapid deployment

expand the envelope of technologies

The final set of recommendations is intended to expand the envelope of technologies available
for use on DoD missions. “Stretch problems” are proposed as a means to both strengthen the
operational pull and mature the underlying technologies, such that they would be trusted for
application on DoD missions:

• Early warning system for understanding global social movements
• Autonomous swarms that exploit large quantities of low-cost assets
• Intrusion detection on the Internet of things
• Autonomous cyber resilience for military vehicle systems
• Autonomous air operations planning

Previously, In the 2012 final study on Autonomy, DSB said:

Unmanned systems are proving to have a significant impact on warfare worldwide. The true value of these systems is not to provide a direct human replacement, but rather to extend and complement human capability in a number of ways. These systems extend human reach by providing potentially unlimited persistent capabilities without degradation due to fatigue or lack of attention. Unmanned systems offer the warfighter more options and flexibility to access hazardous environments, work at small scales, or react at speeds and scales beyond human capability. With proper design of bounded autonomous capabilities, unmanned systems can also reduce the high cognitive load currently placed on operators/supervisors. Moreover, increased autonomy can enable humans to delegate those tasks that are more effectively done by computer, including synchronizing activities between multiple unmanned systems, software agents and warfighters —thus freeing humans to focus on more complex decision making.

While the potential of autonomy is great, there have been many obstacles to general broad acceptance of unmanned systems, and, specifically, the autonomous capabilities needed to realize the benefits of  autonomy in military applications. Most Department of Defense (DoD) deployments of unmanned systems have been motivated by the pressing needs of conflict, particularly the threat of improvised explosive devices and the need for persistent intelligence, surveillance and reconnaissance (ISR) data collection. To date, most of the demonstrated benefits of autonomous systems have been in air or ground applications, but there exists no reason that they could not be effective in maritime and space missions as well.

The Task Force was charged to assist the DoD in understanding and preparing to take maximum practical advantage of advances in autonomy by reviewing relevant technologies, ongoing research and the current autonomy-relevant plans of the Military Services. The Department asked the Task Force to identify new opportunities to more aggressively use autonomy in military missions, to anticipate vulnerabilities and to make recommendations for overcoming operational difficulties and systemic barriers to realizing the full potential of autonomous systems.

Related/Background:

• Report of the Defense Science Board Summer Study on Autonomy June 2016 - DSBSS15.pdf
• The Role of Autonomy i n DoD Systems July 2012.pdf
• Defense Science Board Calls For Faster Adoption Of Autonomy | Defense Daily Network 
• Designing Unmanned Systems with Greater Autonomy: Using a Federated, Partially Open Systems Architecture Approach - RAND_RR626.pdf 
• Defense Science Board on Autonomous Systems - Lawfare  
• The Seven Deadly Myths of “Autonomous Systems” -IS-28-03-HCC_1.pdf