Ai's silent shift: autonomous weapons and the erosion of human control

Raúl Álvarez Prieto, director of Kallisto AI, isn't just concerned about the potential benefits of artificial intelligence in defense; he's sounding an alarm about the accelerating shift towards autonomous decision-making—a shift he believes will occur “in the near future” without human intervention. This isn’t about science fiction; it's a rapidly approaching reality with profound implications for global security.

The data deluge: ai’s quiet revolution

While the proliferation of drones often steals the spotlight, Prieto argues that AI’s most significant impact on defense is currently reshaping data management. Previously, vast quantities of data required laborious analysis by human experts. Now, algorithms and automated systems are absorbing and interpreting this information at speeds and scales previously unimaginable, effectively replacing hundreds, if not thousands, of personnel.

The applications are broad, spanning everything from logistics and maintenance to intelligence gathering and target recognition. But the most unsettling advances are in guided weaponry – missiles equipped with AI capable of identifying and neutralizing potential threats autonomously, and drones capable of striking targets without human guidance. We've already witnessed this in Ukraine, where drones navigate the final kilometer using computer vision and artificial intelligence, sidestepping the need for constant human control. This represents a tangible step toward entirely independent systems.

Colonel Gómez de Ágreda succinctly puts it: “It’s not a more intelligent AI that worries me, but a less intelligent human using it.” A stark warning that underscores the potential for misuse and unintended consequences.

Spain and the ai arms race: lagging behind

Spain, like much of Europe, finds itself trailing behind the United States and China in the race to develop and deploy advanced AI systems for military applications. While Spain utilizes AI for logistics, maintenance, and facial recognition, the foundational research and development are dominated by Washington and Beijing. The US and China have invested heavily not only in hardware but, crucially, in training algorithms and cultivating the expertise required to wield this Technology effectively. The advancements that European companies now leverage often originate across the Atlantic.

The real problem isn’t access; it’s agency. Today, the barrier to entry for autonomous AI weaponry has dissolved, making it readily available to smaller nations, terrorist groups, and other actors with potentially malicious intent. This democratization of advanced weaponry poses a significant and growing threat.

The human in the loop: a fading ideal?

The prevailing theory—and current practice—dictates that a human must remain “in the loop,” overseeing AI’s actions. However, Prieto suggests this may soon be a relic of the past. Consider missile defense systems designed to intercept supersonic threats in real-time; waiting for human authorization would be catastrophic. While regulation currently mandates human oversight, the reality is already blurring. Instances of autonomous decision-making are emerging in conflict zones like Russia and Ukraine, where drones are reportedly making independent targeting decisions.

“It’s a minimal leap,” Prieto observes. “It’s simply a matter of asking or not asking a human.” The temptation to bypass human control—to prioritize speed and efficiency—is a dangerous one. The ethical and strategic implications are profound.

The double-edged sword: efficiency vs. accountability

The benefits of AI in defense are undeniable: reduced costs, accelerated timelines, and even life-saving applications, such as autonomous search and rescue operations. But this technological progress carries a dark side. The potential for errors, inherent in any system, is amplified when those errors can result in loss of life. Military systems are rigorously tested and designed to minimize failures, but AI, like any Technology, is not foolproof.

Furthermore, the absence of human accountability in autonomous systems raises serious moral and legal questions. Who is responsible when an AI-powered weapon makes a fatal mistake? The programmer? The commander? The machine itself? As AI continues to permeate the defense sector, these questions demand urgent and comprehensive answers. The current patchwork of regulations is inadequate to address the escalating risks.

As AI reshapes the battlefield, one thing is clear: the future of warfare will be defined not by the sophistication of the Technology itself, but by the wisdom—or lack thereof—with which we choose to wield it.