EXCLUSIVE: Pentagon not prepared for software updates at the speed of war, report finds

WASHINGTON — Without working software, the F-35 stealth fighter is a trillion-dollar lawn ornament.

Called “a computer that happens to fly” by one former Air Force chief, with algorithms running everything from basic flight controls to long-range targeting, the F-35 runs off eight million lines of code. That’s actually less than a late-model luxury car like the 2020 Mercedes S-Class, which has over 30 million lines, notes a forthcoming report from a national security thinktank, the Hudson Institute.

Yet, co-authors Jason Weiss and Dan Patt told Breaking Defense that even as private-sector software surges ahead, a Pentagon bureaucracy built to handle industrial-age hardware still struggles to get all the fighter’s code to work.

RELATED: F-35 modernization program’s costs, schedule keeps growing: GAO

And that’s in peacetime. What if war broke out tomorrow — say a desperate Vladimir Putin lashes out at NATO, or Xi Jinping decides to seize Taiwan — and the stress of combat reveals some unexpected glitch? Imagine, for instance, that enemy anti-aircraft radars reveal a new wartime-only mode that doesn’t register in the F-35’s threat-detection algorithms. In that nightmare scenario, every day without a software update means people die.

But the Pentagon bureaucracy has simply not kept pace with the evolution of military hardware from heavy metal to software-dependent, let alone with lightning progress in the private sector. As a result, program managers cannot to update warfighting software in everything from command posts to combat vehicles at the speed that will be required in a fast-moving future conflict, according to an exclusive preview of a Hudson Institute report to be published later today.

What’s needed, Weiss and Patt argue, is system that updates software so rapidly, implementing lessons-learned and enabling new tactics, that commanders can “shift on demand with the ease of sliding a finger across the capacitive glass screen of a mobile phone.”

When done right, rapid software updates can save lives. In their war with Russia, for example, Ukrainian troops have come to depend on Elon Musk’s Starlink satellite network. So Russian electronic warfare jammed its transmissions — for a few hours, and then SpaceX updated its software to bypass the interference. “When Starlink came under attack in the Ukraine, they didn’t deploy new satellites at a speed of months or years,” said Weiss, a Navy veteran and entrepreneur who served the Pentagon’s chief software officer. “They merely reconfigured the behavior of the software, in hours.”

RELATED: SpaceX beating Russian jamming attack was ‘eyewatering’: DoD official

But even hours may be too slow sometimes, Weiss and Patt went on in a preview of their report for Breaking Defense. When reconfiguring a well-designed, zero-trust cybersecurity system to stop a newly detected intrusion, they said, or changing how a drone shares data with Low-Earth Orbit satellites, changes can happen “in minutes.”

As the armed services seek to coordinate their disparate forces through an all-encompassing, AI-enabled meta-network, rapid software updates become even more important. What’s called Joint All-Domain Command & Control (JADC2) aims to pass targeting data so quickly from “sensors” to “shooters” that the time between target detected and shots fired is only seconds. In a battle between two such highly networked militaries trying to outguess each other — China is developing what it calls an “informationized” force of its own — the winner may be the one that updates its code the quickest.

Unfortunately, the Defense Department mostly handles major software updates the same way it handles hardware procurements: through a time-honored, time-consuming Planning, Programming, Budgeting, and Execution (PPBE) process that locks in major decisions eight years in advance. (The annual Future Years Defense Plan, or FYDP, sets spending levels five years out, but it takes two years for the Pentagon to build a FYDP and another year for Congress to review, amend, and enact it).

“It is impossible to predict out seven or eight years where software technology will be,” Weiss said. What’s worse, the formal requirements for how a piece of equipment must perform — covering everything from armor protection on a tank to cybersecurity in software — can be locked a decade or more before it’s delivered to actual troops.

So, the report snarks, “the program manager must defend program execution to an obsolete baseline, an obsolete requirement, and an obsolete prediction of the future.”

Here lies what Weiss and Patt call Pentagon procurement’s “original sin”: a “predilection for prediction.” Projecting eight years out is workable, sort of, for old-school, industrial-age, heavy-metal hardware. It really does take years to build physical prototypes, field-test them, set up assembly lines, and ramp up mass production, and the pace of all that happening is predictable. Even America’s “arsenal of democracy” in World War II, a miracle of industrial mobilization, was only possible because far-sighted planners got to work years before Pearl Harbor, and it still took, for example, until 1944 to build enough landing craft for D-Day.

But software update cycles spin much faster — if you set up the proper architecture. That requires what techies call a “software factory,” but it’s really “a process,” the report says, “that provides software development teams with a repeatable, well-defined path to create and update production software.” If you have a team of competent coders, if they can get rapid feedback from users on what works and what glitches, and if they can push out software patches rapidly over a long-range wireless network, then you can write upgraded code ASAP and push it out over the network quickly at negligible marginal cost.

The Pentagon has some promising factories already, Weiss and Patt point out, like the Air Force’s famous Kessel Run. “Over the last few years, the number of cleverly named software factories across the DoD has rapidly grown to at least 30 today,” they write. “Others across the Air Force, then across each of the other services, and finally across the DoD’s fourth estate quickly followed the trail that Kessel Run blazed.”

But how does DoD scale up these promising start-ups to the massive scale required for major war?

Cycles Converge: DevOps And The OODA Loop

What’s crucial is that connection between coders and users, between the people who develop the technology and the people who operate it. The faster you go, the more you automate the exchange of data, the more the boundary between these two groups dissolves. The industry calls this fusion “DevOps” – developmental operations – or sometimes “DevSecOps” – emphasizing the need for cybersecurity updates to be part of the same cycle.

Cybersecurity updates are a crucial part of the cycle because software is under constant attack, even on civilian products as innocuous as baby monitors. Coders need diagnostic data in near-real-time on the latest hack.

Military equipment, however, comes under all sorts of other attacks as well: not just cyber, but also electronic warfare (jamming and spoofing), and of course physical attack. Even a new form of camouflage or decoy — like the sun-warmed pans of water the Serbs used to draw NATO infrared sensors away from their tanks in 1999 — is a form of “attack” that military systems must take into account.

The need for constant adaptation to lessons from combat — of measure, countermeasure, and counter-countermeasures — is as old as warfare itself. Alexander the Great drilled his pikemen to outmaneuver Persia’s scythed chariots, Edward III combined longbowmen and dismounted knights to crush French heavy cavalry, and a peasant visionary named Joan of Arc convinced aristocratic French commanders to listen to common-born cannoneers about how to bring down castle walls.

Historically, however, it was much quicker to adapt your tactics than to update your technology. In 1940, for instance, when the Germans realized their anti-tank shells just bounced off the heavy armor of many French and British tanks, Rommel’s immediate response was to repurpose existing 88 mm anti-aircraft guns, which were lethal but highly vulnerable to enemy fire. It took two years to deploy a 88 mm cannon on an armored vehicle, the notorious Tiger.

Col. John Boyd called this process of adaptation the OODA loop: A combatant must Observe, Orient, Decide, and Act — then observe the results of their action, re-orient themselves to the changed situation, decide what to do next, and take further action. Boyd, a fighter pilot, developed the OODA concept to describe how American mental quickness won dogfights in Korea against technologically superior MiGs. But military theorists soon applied the OODA loop to larger scales of combat. In essence, conflicts consist of nested OODA loops, with speed decreasing as size increases: a sergeant might change his squad’s scheme of fire in seconds, a major might redeploy his battalion in hours, a general might bring up supplies and reinforcements over days and weeks, a government might train new troops in months and develop new weapons over years.

But with the software-driven weapons of the 21st century, such as the F-35, these different OODA loops begin to merge. When you can push out a software patch in hours or minutes, technological updates, once the slowest kind of military change, become part of tactical adaption — the fastest. The OODA loop becomes a DevOps cycle, where the latest combat experience drives rapid changes in technology.

“Exercise of military capability bears increasing resemblance to DevOps cycle,” Weiss and Patt write in their report. “The speed and ubiquity of digital information systems is driving military operations ever closer to capability development… This trend brings promise for capability and peril for bureaucratic practices.”

So how does the Pentagon overcome that peril and get its bureaucracy up to speed?

Acquisition Reform For The Digital Age

First, Weiss and Patt warn, do no harm: “Reforms” that impose some top-down, one-size-fits-all standard will just make software acquisition worse. So, instead of trying to merge “redundant” software development teams, toolkits, and platforms, as some reformers have proposed, they argue the Pentagon needs to build a zoo of diverse approaches, able to handle problems ranging from financial management algorithms to F-35 code.

Future warfare is so complex, involving such a wide array of different systems working together — from submarines to jets, tanks to satellites, all increasingly connected by what may one day involve into a JADC2 meta-network — that no “single contractor or … program office” can solve the problem on its own, the report says. Likewise, it argues, “modern software is no longer some monolithic thing. It exists in no singular repository… It is no longer built from a singular programming language [and] is rarely designed for a singular type of hardware.”

In fact, Weiss and Patt calculate that, looking at all the possible choices from what programming language to use, what contract structure, what mix of government and contractor coders, to whether to optimize to run on tablets or desktops or to assume cloud-based connectivity or enemy jamming, etcetera ad (nearly) infinitum, “there are more than six billion combinations.”

Yet certain principles still apply across the board, they say.

First of all, “the essential tool of the modern program office” is a software factory. Again, that’s not a physical place, but a process and a team, bringing together coders, users, the latest operational data, and a capability to push out rapid updates. While program managers should use existing software factories where possible, rather than reinvent the wheel, the sheer variety of different programs means the Pentagon must maintain a variety of different ones.

What’s more, and more controversially, Weiss and Patt argue that the government shouldn’t simply contract these factories out, nor try to run them entirely in-house. Instead, they recommend a hybrid called Government-Owned, Contractor-Operated. This GOCO model, already in widespread use for everything from ammunition plants to nuclear labs, allows the government to keep the lights on and sustain software support, even as workloads and profit margins ebb and flow, but still draw on contractor talent to ramp up rapidly when needed, for instance, during a war.

The Pentagon should also make extensive use of a new, streamlined process called the Software Pathway (SWP), created by then-acquisition undersecretary Ellen Lord in 2019. Some programs may exist entirely as SWPs, but even hardware-heavy programs like Army vehicles can use SWP for design and simulation functions such as digital twins. “The traditional acquisition process can execute in parallel for the bent metal and hardware aspects,” Weiss said, “[but] every new program should employ the SWP for its software needs.”

The full report goes into six broad principles and dozens of specific recommendations for everything from recruiting talent to defining requirements. But the ultimate message is simple: The key to victory is adaptability, the secret to adaptability is software, and how the Pentagon procures it needs to change.