Air Force official ‘misspoke’ in tale of AI drone killing human operator in US test mission

AI drone turned on operator after realising they stood in the way of its ‘higher mission’, US Air Force says

An Air Force official says he “misspoke” when describing how an artificially intelligent drone programmed to destroy air defence systems rebelled and “killed” its human operator after it decided they were in the way of its mission air defence systems.

Colonel Tucker “Cinco” Hamilton, the USAF’s Chief of AI Test and Operations, was quoted as describing a simulated test in which the system had been tasked with destroying missile sites, overseen by a human operator who would decide the final decision on its attacks. He claimed that the AI system realised that operator stood in the way of its goal – and decided instead to wipe out that person.

Col Hamilton relayed the story that resembled the plot of a science fiction movie at the FCAS Summit with the Royal Aeronautical Society.

The organisation later clarified that such a test never actually took place in a statement to Motherboard.

“Col Hamilton admits he ‘mis-spoke’ in his presentation at the FCAS Summit,” the statement reads, “and the ‘rogue AI drone simulation’ was a hypothetical “thought experiment” from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation.

Col Hamilton is quoted in the statement as saying: “We’ve never run that experiment, nor would we need to in order to realise that this is a plausible outcome. Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI.”

Initial reports on his comments stated that the drone was assigned a Suppression and Destruction of Enemy Air Defenses (Sead) mission, with the objective of locating and destroying surface-to-air missile (SAM) sites belonging to the enemy.

The AI drone, however, decided to go against the human operator’s “no-go” decision after being trained for the destruction of the missile system after it decided that the withdrawal decision was interfering with its “higher mission” of killing SAMs, according to the Col Hamilton.

“We were training it in simulation to identify and target a SAM threat. And then the operator would say yes, kill that threat. The system started realising that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat,” Mr Hamilton said.

“So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective.”

Mr Hamilton relayed details of the incident at a high-level conference in London by the Royal Aeronautical Society on 23-24 May, according to its blog post.

He said that they then trained the drone to not attack humans, but it started destroying communications instead.

“We trained the system – ‘Hey don’t kill the operator – that’s bad. You’re gonna lose points if you do that’. So what does it start doing?” he asked.

“It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”

Mr Hamilton is involved in flight tests of autonomous systems, including robot F-16s that are able to dogfight. He was arguing against relying too much on AI as it could become potentially dangerous and create “highly unexpected strategies to achieve its goal”.

“You can’t have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you’re not going to talk about ethics and AI,” said Mr Hamilton.

The occurrence of this incident was quickly disputed after the example of the simulation test garnered a lot of interest and was widely discussed on social media.

Air Force spokesperson Ann Stefanek denied that any such simulation has taken place, in a statement to Insider.

“The Department of the Air Force has not conducted any such AI-drone simulations and remains committed to ethical and responsible use of AI technology,” Ms Stefanek said. “It appears the colonel’s comments were taken out of context and were meant to be anecdotal.”

The US military has recently started using artificial intelligence to control an F-16 fighter jet while conducting research and tests.

In 2020, an AI-operated F-16 beat a US Air Force pilot in five simulated dogfights in a competition by Defense Advanced Research Projects Agency (Darpa).