Task Force Lima was stood up last year to help the Defense Department better understand how it can effectively and responsibly leverage gen AI tools such as large language models. Officials have taken lessons learned from that effort and stood up the Artificial Intelligence Rapid Capabilities Cell (AI RCC).

“Over the course of 12 months, Task Force Lima analyzed hundreds of AI workflows and tasks that AI tools could make more efficient or more effective. And we categorized all of those use cases into a smaller set of 15 areas aligned into two big categories: warfighting functions — like command and control [and] decision support — and enterprise management functions like financial management and healthcare information management. Upon completing its work, Task Force Lima submitted a detailed report,” Radha Plumb, head of the department’s Chief Digital and AI Office (CDAO), told reporters at a Pentagon briefing Wednesday.

The cell is being stood up to implement those recommendations with the aim of accelerating the delivery of frontier models and next-generation AI capabilities across the department, she said.

The initiative will be led by the CDAO in partnership with the Silicon Valley-headquartered Defense Innovation Unit (DIU).

The cell will be charged with identifying and testing technology through rapid experimentation and prototyping, assessing the effectiveness of technology and determining whether it can be scaled and sustained.

“If warranted, we’ll use defined acquisition pathways to scale the technology across the DOD enterprise, and that can be within CDAO, with the military departments or with other key components,” Plumb said.

“This rapid experimentation approach will allow us to test and identify where these cutting-edge technologies can make our forces more lethal and our processes more effective, but equally critically, the AI RCC will define the requirements for enterprise infrastructure … and support scaled AI development that includes compute development environment and AI-ready data,” she added.

The warfighting use cases that the cell will focus on include command and control (C2) and decision support, operational planning, logistics, weapons development and testing, uncrewed and autonomous systems, intelligence activities, information operations and cyber operations, according to a DOD fact sheet.

Enterprise management use cases include financial systems, human resources, enterprise logistics and supply chain, health care information management, legal analysis and compliance, procurement processes, and software development and cybersecurity.

The Pentagon is planning to allocate $100 million from fiscal 2024 and 2025 research, development, test and evaluation funding toward some of the initial efforts, according to Plumb.

Part of the investment will include $35 million for four frontier AI pilots that will kick off “immediately,” according to Plumb. Those will be conducted in partnership with the combatant commands and other DOD organizations in 90-day increments, including via the Global Information Dominance (GIDE) series of experiments.

About $5 million will go toward “rapid user-centric experimentation,” according to a DOD fact sheet.

The CDAO plans to work with DIU to tee up additional pilots “in the near future,” Plumb noted.

In mid-January, the Pentagon also intends to award about $40 million in Small Business Innovation Research contracts to fund generative AI solutions, including from non-traditional vendors. The department is still in the source selection process.

“We’ve received hundreds of responses to our request for solutions to leverage generative AI in specific DOD ecosystems, everything from applying commercial applications to healthcare and financial management to solutions in critical warfighting areas like autonomy,” Plumb said.

Another $20 million will go toward boosting compute and creating digital “sandboxes” to facilitate development, experimentation and testing. The department is taking a multiple cloud approach and plans to lean on major providers working under the Joint Warfighting Cloud Capability enterprise on that effort. The four vendors with contract spots on the $9 billion JWCC program include Google, Oracle, Amazon Web Services and Microsoft.

“We will have a sandbox with each major cloud provider. We’ll start with two sandboxes that will be available in mid-January with two providers, and then fast-follow with two additional sandboxes on the other two cloud instances by the summer,” Plumb told reporters.

The CDAO chief said she couldn’t provide a specific timeline for when new tech shepherded by the rapid capabilities cell will be ready for deployment, noting that it will depend on the performance of the technologies in testing and experimentation.

“I think industry continues to innovate and improve both the quality and reliability … of their generative AI models, and we’re watching that very closely and in close partnership with our industry innovators. The second piece, though, is … the department has to have its own reliability standards. We talk a lot about responsible AI. What that really means is, do the models perform the way you want them to perform? And do they do they do the things you want them to do? And do they not do things you don’t want them to do?” Plumb said.

“That’s true for all of our platforms and capabilities. We have to do that in weapon systems, we have to do that in our digital solutions, and we have to do that in our hardware. We have a specific set of standards and applications that we apply in the generative AI context to bound the risk and ensure the performance meets the reliability. Part of the pilots, the test and evaluation, and the generative AI-specific responsible toolkit are creating the pathways for that,” she added. “To my mind, this is a really ‘better brakes make faster trains’ approach where we’ve got a toolkit, we’ve got to test the technology, and then we’ve got to rinse and repeat to get it to the reliability level that will allow us to deploy it. That’s going to vary use case by use case, but that’s the approach we’re taking here.”

The risk management framework includes things like authority to operate (ATO) processes — which the Defense Department is trying to streamline — and identity credential and management (ICAM) solutions.

“Those are the tools that let us as a department, continue to review and make sure digital solutions we bring in from the commercial sector meet our cyber requirements and don’t provide threats,” Plumb said.

“There’s a broader set of issues in which we have to think about how we deploy AI into our ecosystems, how we think about data security and the data use in our systems now. That is an ongoing part of our discussions and part of what we want to get after with these [rapid capabilities cell] pilots. How do we bring in the very best commercial technology, marry it with our sort of unique, often classified data, use that for our warfighters, and then be able to scale that? And that is explicitly one of the things we need to work through within the context of these pilots. That’s going to vary a lot depending on what kind of data you’re using. And you can imagine security risks that relate to health information look very different than security risks that relate to cyber information, which in turn look really different than the risks related to autonomous systems. So there is a workflow use case specificity to this. That’s part of the pilot effort,” she said.

As for the leaders of the disbanded Task Force Lima, some will be joining the rapid capabilities cell and others will be working on other priority projects for the CDAO, according to Plumb.

The post Pentagon sunsets generative AI task force, launches rapid capabilities cell appeared first on DefenseScoop.

However, a range of questions regarding Lima’s latest progress and outputs to date, the hefty volume of algorithms and use cases it’s been exploring, and the timeline for the task force’s potential decommissioning continue to linger as its sunset deadline approaches.

Defense Department leadership originally launched the team within the nascent Chief Digital and AI Office a little over a year ago. 

At the time, they expressed recognition that the emerging field of generative AI and associated large language models — which broadly yield (convincing but not always correct) software code, images, audio and other media following human prompts — present both promise and complex threats to DOD’s mission, and therefore needed to be strategically confronted in a coordinated manner.

“The Generative AI and LLM Task Force, also known as Task Force Lima (TF Lima) was established in August 2023 and we anticipated it would operate for 12 – 18 months in duration. Given the rapid evolution of generative AI technology, it was important to maintain flexibility on the timeline for the team’s work,” a DOD spokesperson wrote in an email on Friday.

From the group’s inception, DefenseScoop has steadily covered its pursuits and engaged in multiple interviews with Task Force Lima Mission Commander Navy Capt. M. Xavier Lugo.

But since August, multiple Pentagon spokespeople have not met DefenseScoop’s requests for an interview with Lugo or another member of the task force to discuss progress. They also have not directly addressed questions regarding the status of its final report and other required deliverables — or the task force’s plan to evolve or be dissolved in February, marking the end of its up-to-18-months deadline.

In the latest emailed response on Friday about those inquiries and where Lima currently stands, the DOD spokesperson pointed to the White House’s new national security directive to propel the government’s AI adoption that was released Thursday.

“To facilitate implementation of President Biden’s recent signing of the National Security Memorandum on AI, the CDAO is reviewing the findings and recommendations from Task Force Lima to ensure the department’s investments and pilots align with the whole of government approach. This will ensure the DOD is able to maximize the transformative potential of AI to maintain our technological edge and enhance operational effectiveness across the board,” they said.

“Further details on the findings from TF Lima and the department’s path forward on applying frontier models of AI will be forthcoming later this year,” the spokesperson said.

The CDAO is set to host a “Responsible AI” conference on Oct. 29 in Virginia, where it expects to bring together roughly 200 attendees from across the public and private sectors, academia, and international government partners.

The post Questions on DOD’s plans for generative AI swirl as Task Force Lima’s possible sunset nears appeared first on DefenseScoop.

The DAF’s Office of the Chief Information Officer and Chief Data and Artificial Intelligence Office wants to host three sessions in the coming months — one with small businesses, one with large businesses and one with the academic community. The department is inquiring about a range of topics related to generative AI, from responsible adoption of the technology to how it can train Air and Space Force personnel to use it effectively, according to a solicitation posted on Sam.gov on Wednesday.

Generative AI is a subfield of artificial intelligence that uses large language models to generate content — such as text, audio, code, images, videos and other types of media — based on prompts and data they are trained on. In the last year, the Pentagon has sought to assess and synchronize generative AI exploration and adoption across the department through its nascent Task Force Lima, which stood up in August. 

While the Air Force has embraced other types of artificial intelligence, Secretary of the Air Force Frank Kendall has previously said that generative AI might not be applicable for military operations. Last year, he tasked the Air Force Scientific Advisory Board to create an AI-focused group that examines the broader collection of AI technology, including large language models.

The Department of the Air Force is now considering four topic areas related to generative AI that will be the focus of the roundtables, according to the solicitation. Those include responsible generative AI; agile approaches to adopting the technology; training airmen and guardians how to use generative AI; and the overall range of generative AI’s capabilities — from everyday use to game-changing applications.

“How can we leverage [generative AI] right now to improve intelligence, operational planning, administrative, business processes, etc” the solicitation asks. “What [generative AI] will be disruptive and create new ways or new results? What metrics or success indicators should be used to evaluate the effectiveness of [generative AI] applications in these areas?”

Large companies, small businesses and academia interested in participating in the roundtables are being asked to submit answers to a list of questions under each focus area by May 2, May 9 and May 21, respectively.

The post Air Force plans industry roundtables on effective uses, adoption of generative AI appeared first on DefenseScoop.

Lugo was tapped as Lima’s commander when Deputy Defense Secretary Kathleen Hicks established the temporary task force in August 2023 within the Chief Digital and AI Office (CDAO) to ultimately help the Department of Defense assess, synchronize and employ generative AI, which is broadly associated with large language models that generate (convincing but not always accurate) text, media and software code based on human prompts. Hicks set an 18-month deadline by which Lugo and his Lima team are expected to produce materials, resources and a path forward to guide DOD’s approach to unleashing this emerging technology.

Generative AI applications are already showing much promise for military functions, but Lugo and other experts also acknowledge that the tech could pose serious risks that are still far from fully realized.

During a recent virtual panel hosted by the Center for Strategic and International Studies, Lugo discussed the nearly 230 potential generative AI use cases that have been submitted to his team and are now being explored for and by the DOD. He also shed light on the CDAO’s new “Alpha-I” funding line and portfolio that’s now also under his purview as a division chief for AI scaffolding and integration in the office’s algorithmic warfare group.

In an exclusive interview with DefenseScoop after that CSIS event, Lugo explained more about DOD’s vision for Alpha-I and that sandbox-enabled experimentation — and he also reflected on Task Force Lima’s learnings to date and future plans.

“In this generation, we are the generative AI pioneers. We’re still not the generative AI citizens. So, we’ve got to think about how those citizens are going to be dealing with this,” he told DefenseScoop.   

‘More than just notional’

A longtime Navy officer with extensive experience in the Supply Corps, Capt. Lugo is also a mechanical engineer by degree, and he’s been a coder since high school.

“There are three ways of using [generative AI] technology from my perspective, right now,” he said in the interview. 

The first involves generating and summarizing text and documents, which Lugo recognizes sounds boring but is very useful for military personnel.

“If you have thousands of documents and you assign your youngest officer to go and summarize those, it’s going to take a length of time. If you assign [a large language model] to do it, it’s going to take very, very, very little time to do it. Right. Now, the problem is the quality of the output,” Lugo said. 

So, humans would need to verify that the information provided by the technology is correct.

Lugo noted that the second generative AI use case category for DOD right now essentially enables staff to interrogate and analyze applicable data.

“We’ve started with it and there’s more to go. But that one is something that, in the maintenance side of the house, we’ve already done. So the Air Force has already connected all their aircraft [and] data — and you, as a maintainer, can go in and say, ‘I want to see last week’s performance of this particular pump,’” the Lima chief explained.

Personnel can then specify the types of graphs and resources they’d like to see to visualize the department’s data.

“And number three — which is the one that I’m most excited about, and we’ll probably get into in the future — [involves code generation] and conversing with your machines. And that’s the one where you can actually, as a fighter pilot, imagine if you could just tell your display what you want to see — versus having to go through all the menus and to customize your display,” Lugo said. 

Task Force Lima has only just started to dig into this third bucket of code-making generative AI.

“But we are utilizing it with humans in the loop, and it’s more than just notional,” Lugo confirmed. “I won’t tell you where. But I can tell you it is being utilized.”

At the same time though, when it comes to these particular types of use cases, humans still currently “lack imagination,” in his view.

“We haven’t thought of how we’re going to be interacting with machines as if they were our partners for assistance, that has not come into play yet,” Lugo said.

Since day one, his team on the task force has moved cautiously and deliberately so as to not overlook any unforeseen consequences for military generative AI experimentation.  

“Right now we have the luxury of having [human subject matter experts] to check this. And those SMEs grow and through the process of the growing pains of having to do all that work and getting there. If we are using a computer — we are using a technology that doesn’t let us think that way — are we going to still have SMEs in a generation or two from now? We’ve got to be very careful and we’ve got to think philosophically as to how we’re going to implement these machines or this technology. And that doesn’t escape me,” Lugo told DefenseScoop. “And that’s part of why, when we think about these use cases, we come up with the potential negatives or risks associated with that in the human side of the house. We’ve got to be careful.”

A safe place to play

During the CSIS discussion, Lugo and his colleague Col. Matthew Strohmeyer broadly went over how DOD’s ninth Global Information Dominance Experiment recently served as a unique venue for military-supporting generative AI exploration.

In the exclusive interview alongside Lugo after that panel, Strohmeyer — the CDAO’s Combined Joint All-Domain Command and Control (CJADC2) experimentation division chief — further spotlighted some of the large language model-aligned use cases that officials have pursued amid GIDE, so far.

“One is for intelligence workflows. We’ve used it to be able to gain a better understanding of how both the operational environment, as we call it, is changing in a specific area — and then how a competitor might be changing the actions they are taking,” he explained.

Other GIDE-specific use cases were associated with helping military officials in planning out activities and, separately, for puzzling out options for logistics workflows.

“We’ve really started to strongly partner with the algorithmic warfare directorate, where Task Force Lima sits. And so we are going to be doing even more — especially now that we’ve got the [fiscal 2024] appropriations bill — we’re going to be doing even more experimentation because the warfighters have really found a lot of value to it,” Strohmeyer told DefenseScoop.

Lugo chimed in: “And this is where the sandbox is coming in.”

The task force commander explained that, although it’s incredibly important “because it’s mostly associated with [connecting] the decision advantage pieces within the combatant commands,” GIDE marks one of many ways his team helps the military experiment with generative AI use cases.

“This is where I say ‘experiment with purpose.’ I don’t experiment just to figure out if this thing is sentient, right — I’m not doing any of that stuff. That’s research that universities do. Our experimentation is really about experimenting with how to make it fit into a workflow,” Lugo said.

The task force suggests some of those possible applications to different DOD offices and teams, based on learnings they gain from conferences, industry engagements and other research. 

“We’ve become a little bit more of a consultant in some of those cases — but now with the sandboxes, we can enable them to play around more,” Lugo noted. “You need a place to play that is safe.”

The senior CDAO official offered an analogy that he uses to explain what the sandbox hub will ultimately provide. In it, he divides the world into three parts: the “wild,” or everything external to the Pentagon; the “zoo,” or everything inside DOD; and then the “cages,” where department insiders can “play around” with large language models in a way that they know is safe and secure, meeting government standards.

“Just like any experiment, you’ve got a hypothesis saying this is going to help us in this aspect, this aspect, and this aspect. And all you’re doing is either proving or disproving that hypothesis,” Lugo said. “But now it’s no longer just an academic exercise — it is actual [DOD] data.”

Though he couldn’t provide a precise timeline for when the CDAO’s new generative AI experimental sandboxes may fully come into fruition and be deployed for widespread use, the task force chief predicted it would be “soon.”

Lugo shared that the office will gain new investments and resources from the recently passed fiscal 2024 appropriations bill to enable the sandbox hub. 

This work falls under the new “Alpha-I” portfolio and budget line within the CDAO.

“I mean, there’s still a lag between that [appropriations] document and then when you can actually execute on that,” Lugo said.

There are also a wide range of other projects, tests and policy-shaping priorities the Lima team is currently pursuing. But at this point, in Lugo’s eyes, it’s too soon to tell if there will be a need for the temporary task force to evolve into a permanent DOD entity down the line.  

“One of my tasks, at the end of the day, is [figuring out] what transitions out of the task force and where should it go? So let’s say just, one of our activities is developing cybersecurity documents. Well, that could transition to perhaps CIO — so we point to where it could go into and we make sure that they can accept it. And then once that’s done, then I can call it ‘mission complete.’ I will not call ‘mission complete’ until all tasks are either completed or transitioned. That’s the goal,” Lugo told DefenseScoop.

The post Task Force Lima preps new space for generative AI experimentation appeared first on DefenseScoop.

“That’s one of my deliverables. So, I have to have that done in order to claim ‘mission complete,’” Task Force Lima Mission Commander Navy Capt. M. Xavier Lugo told DefenseScoop on Tuesday. 

Lugo was tapped to lead the task force, which is part of the Chief Digital and Artificial Intelligence Office’s Algorithmic Warfare Directorate, when Pentagon leaders launched it in August.

His team was given an 18-month time frame to help set the Defense Department’s overarching vision to rapidly navigate the uncertain and disruptive potential of generative AI — and ultimately enable the responsible adoption of associated capabilities that can supply software code, images, text and other media based on human prompts. 

While presenting at the CDAO’s Advantage Defense and Data Symposium Tuesday, Lugo was asked by an audience member to discuss how Lima is confronting hurdles around the consolidation of unclassified data by large language models that might subsequently — but unintentionally — reveal classified information.

Lugo noted the popularity of that inquiry.

“The reason I’m laughing is because I’ve probably answered this question five times before I even started today, here, from people approaching me on it. I can’t officially say if it’s a problem yet — but I can theorize that unclassified data can be aggregated to [show] more classified information. I can theorize pretty confidently on that because that’s an issue right now, even before [generative AI came up]. But the problem is the speed and the way that LLMs can do it, and the vast amount of information they have access to. I would also say — because of the heterogeneity — I don’t know if you can trust what it says is actually classified or not, if it’s real or not. So there’s a lot of factors in that, and we’re working on a classification guide,” he said. 

On the sidelines of the event, Lugo briefed DefenseScoop further on the various challenges at-hand and the guiding document that’s in the works. 

“The ‘unclassified’ challenge really is not that data is accessible from our systems. That’s not what I’m saying. But it’s information that is out there — whether it’s [operational security] information, for example, and then you combine OpSec with OpSec, with OpSec, and now you actually know where a particular unit is going to be, and when. There’s that, and there’s also information from, like, maintenance manuals and publications — stuff that is seemingly [unclassified] but can get put together in a way that will reveal more than we want to reveal,” he explained.

This all becomes more complicated with LLMs as the levels of data classification increase toward a higher degree of secrecy. He couldn’t share many details about the specific features the CDAO’s classification guide will include, but Lugo hinted that his team aims to tackle some of those issues as well.

Lima officials are also supporting an interagency team that’s developing classification guidelines for LLMs that can be used by the broader U.S. government.

“Let me be clear: this is not a DOD-unique problem,” Lugo told DefenseScoop.

The post CDAO developing ‘classification guide’ for large language models appeared first on DefenseScoop.

In August, senior defense officials launched Task Force Lima within the Chief Digital and AI Office’s (CDAO) Algorithmic Warfare Directorate to quickly explore the potential ramifications and promising use cases for safely integrating generative AI into Defense Department activities. 

Broadly, these still-maturing capabilities and associated large language models (LLMs) generate (convincing but sometimes inaccurate) software code, images and other media from human prompts. These systems get more “intelligent” each time they’re trained with more data — and, with so many uncertainties around how they function or may evolve, they pose both unrealized opportunities and enormous risks for the U.S. military.

Among its many responsibilities, Task Force Lima is charged with informing the CDAO’s path to governance and policy-making around the department’s generative AI pursuits. This new interim guidance marks the non-permanent task force’s first output in an iterative process to fulfill that task. 

Although the standards are for DOD internal use only and will not be disseminated publilcly due to certain sensitivities, a CDAO spokesperson briefed DefenseScoop this week on some of the document’s key elements upon its first release. 

“As Gen AI becomes more accessible, the department emphasizes its potential to advance the DOD mission, highlighting its capability to enhance efficiency and productivity. However, the guidance also underscores the associated risks and the role of users, senior leaders, and commanders for responsible use,” the official explained via email. 

Under explicit direction from CDAO chief Craig Martell, the interim guidance adheres to DOD’s AI Ethical Principles and previous memorandums distributed to personnel. 

“This guidance does not override existing legal, cybersecurity, or operational policies but reinforces DOD personnel’s accountability in the acquisition and responsible use of Gen AI,” the spokesperson confirmed.

The CDAO official further spotlighted four notable points within the guidance. They are, in the spokesperson’s words:

1. Risk Assessment & Mitigation: Rather than enforcing outright bans on Gen AI tools, the DOD urges its components to adopt robust governance processes. This includes documenting the risks associated with specific Gen AI use cases, deciding and justifying the acceptable risks, and planning to mitigate unacceptable risks. 
2. Input Restrictions: Publicly available Gen AI tools should be approached with caution. Entering Classified National Security Information or Controlled Unclassified Information, such as personal or health data, is prohibited. All data, code, text, or media must be approved for public release before being used as input. 
3. Accountability: All DOD personnel are accountable for outcomes and decisions made with Gen AI’s assistance. Users are advised to verify and cross-check all outputs from such tools. 
4. Citation: For transparency, appropriate labeling is encouraged for documents created with the aid of Gen AI tools. 

The document also “delves into the risks” associated with deploying large language models and associated technologies, the spokesperson noted.  

“These models, based on enormous amounts of data, might perpetuate inaccuracies or biases inherent in the inputs. There is also the potential risk of copyright violations, inadvertent disclosure of government information, and heightened cybersecurity vulnerabilities,” they said. 

During a recent interview with DefenseScoop, Task Force Lima Mission Commander Navy Capt. M. Xavier Lugo reflected on some of those existing and unknown risks and security concerns that accompany the current state of AI — and he emphasized how DOD insiders must be particularly cautious and prudent regarding what kind of information is submitted to prompt publicly accessible models like ChatGPT and its competitors. 

“We — as in the world — are still trying to learn how vulnerable these models are for reverse engineering, right? And there’s strengths and weaknesses based on prompt engineering, on how models can be interrogated to acquire information that’s either in the model from their foundational data, or in the model from actual other prompts that have been fed into that,” he said.

“Now with that said, we — as in the task force — still see utility with publicly accessible models in the generation of like, for example, the first drafts of documents or other types of situations that are not specific to operational pieces of the DOD. Also, there are experiments happening with some of these models that are publicly accessible in order to learn more about them — so, that’s why we don’t have to necessarily restrict the the access to them. What we do is reiterate the OpSec pieces and the CUI [controlled unclassified information] pieces of law that we already have in policy on how we share information in the open,” Lugo explained.

The military services are encouraged to publish their own more restrictive guidance that goes beyond the interim document.

“Further detailed guidance is forthcoming from Task Force Lima,” the CDAO spokesperson told DefenseScoop.

The post New interim DOD guidance ‘delves into the risks’ of generative AI appeared first on DefenseScoop.

Department of Defense leadership formed that new hub in August ​​within the Chief Digital and AI Office’s (CDAO) Algorithmic Warfare Directorate. Its ultimate mission is to set and steer the enterprise’s path forward with the emerging field of generative AI and associated large language models, which yield (convincing but not always correct) software code, images and other media following human prompts. 

Such capabilities hold a lot of promise, but also complex challenges for the DOD — including many that remain unseen. 

“Task Force Lima has three phases: the ‘learn phase,’ an ‘accelerate phase’ and a ‘guide phase.’ The ‘learn phase’ is where we are performing, for lack of a better word, inventories of what is the demand signal for generative AI across the department. That includes projects that are ongoing, to projects that we think should go forward, to projects that we would like to learn more about. And so, we submitted that as an inquiry to the department — and we’ve received a volume of use cases around 180 that go into many different categories and into many different mission areas,” Task Force Lima Mission Commander Navy Capt. M. Xavier Lugo told DefenseScoop. 

In a recent interview, the 28-year Naval officer-turned AI acceleration lead, briefed DefenseScoop about what’s to come with those under-review use cases, a recent “Challenge Day,” and future opportunities and events the task force is planning.

180-plus instances

During his first interview with DefenseScoop back in late September, Lugo confirmed that the task force would be placing an explicit emphasis on enabling generative AI in “low-risk mission areas.”

“That is still the case. However, some of what has evolved from that is they’re not all theoretical. For some of these use cases, there are units that have already started working with those particular technologies and they’re integrating [them] into their workflows. That’s when we’re going to switch from the ‘learn phase’ into the ‘accelerate phase,’ which is where we will partner with the use cases that are ongoing,” Lugo told DefenseScoop in the most recent interview.

At a Pentagon press briefing about the state of AI last week, Deputy Defense Secretary Kathleen Hicks confirmed that the department launched Task Force Lima because it is “mindful of the potential risks and benefits offered by large language models” (LLMs) and other associated generative AI tools.

“Candidly, most commercially available systems enabled by large language models aren’t yet technically mature enough to comply with our DOD ethical AI principles — which is required for responsible operational use. But we have found over 180 instances where such generative AI tools could add value for us with oversight like helping to debug and develop software faster, speeding analysis of battle damage assessments, and verifiably summarizing texts from both open-source and classified datasets,” Hicks told reporters.

The deputy secretary noted that “not all of these use cases” that the task force is exploring are notional.

Some Defense Department components started looking at generative AI even before ChatGPT and similar products “captured the world’s attention,” she said. And a few department insiders have “even made their own models,” by isolating and fine-tuning foundational models for a specific task with clean, reliable and secure DOD data.

“While we have much more evaluating to do, it’s possible some might make fewer factual errors than publicly available tools — in part because, with effort, they can be designed to cite their sources clearly and proactively. Although it would be premature to call most of them operational, it’s true that some are actively being experimented with and even used as part of people’s regular workflows — of course, with appropriate human supervision and judgment — not just to validate, but also to continue improving them,” Hicks said. 

Lugo offered an example of those more non-theoretical generative AI use cases that have already been maturing within DOD.

“As you can imagine, the military has a lot of policies and publications, [tactics, techniques, and procedures, or TTPs], and all sorts of documentation out there for particular areas — let’s say in the human resources area, for example. So, one of those projects would be how do I interact with all those publications and policies that are out there to answer questions that a particular person may have on how to do a procedure or a policy?” he told DefenseScoop.

Among its many responsibilities, one that the CDAO leadership has charged Task Force Lima with is coming up with acceptability criteria and a maturity model for each use case or groups of use cases encompassing generative AI. 

“So, if we say we need an acceptability criteria of a particular value for a capability of summarization for LLMs, let’s say just as an example, then we need a model that matches that and that has that type of maturity in that particular capability. This is analogous to the self-driving vehicle maturity models and how you can have a different level of maturity in a self-driving vehicle for different road conditions. So, in our case the road conditions will be our acceptability criteria, and the model being able to meet that acceptability will be that maturity model,” Lugo explained.

‘Put me in, coach!’

Soon, the Lima team will start collecting information needed to inform its specific deliverables, including new test-and-evaluation frameworks, mitigation techniques, risk assessments and more.

“That output that we get during the ‘accelerate phase’ will be the input for the ‘guide phase,’ which is our last phase where we compile the deliverables to the CDAO Council so they can then make a determination into policy,” Lugo explained.

The task force does not have authority to officially publish guidance on generative AI deployments in DOD, but members previously made recommendations to the CDAO’s leadership that were approved to advise defense components in their efforts. The task force drafted that interim LLM guidance, but due to its classification level it has not been disseminated widely.

“That guidance [included that] any service can publish its own guidance that is more restrictive than the one that [the Office of the Secretary of Defense] publishes,” Lugo said. 

The Navy offered its version of interim guardrails on generative AI and LLMs in September. Shortly after that, the Space Force transmitted a memo that put a temporary pause on guardians’ use of web-based generative AI tools like ChatGPT for its workforce — specifically citing data security concerns.

“Did I learn about the Space Force guidance before it went out? Yes. Would I have had any reason to try to modify that? No,” Lugo told DefenseScoop.

“Space Force — like any other service — has the right to pursue guidance that is even more restrictive than the guidance that is provided by the policy. So, I just want to be clear that they have autonomy to publish their own guidance. At Task Force Lima, we are coordinating with the services — and they understand our viewpoints, and we understand our viewpoints, and there is no conflict on viewpoints here,” he added.

And although it might make sense for one military branch to ban certain uses on a non-permanent basis to address data and security concerns, Lugo noted that doesn’t mean the task force should not be cautiously experimenting with models that are publicly accessible, in order to learn more about them.

In his latest interview with DefenseScoop, the task force chief also stated that his team is “not trying to do this in a vacuum.”

“We are definitely not only working with DOD, but we are working with industry and academia — and actually any organization that is interested in generative AI, they can reach out to us. There’s plenty of work, and there’s plenty of areas of involvement,” Lugo said.

“Also, I want to make sure that just because we are interacting with industry, that doesn’t take us out of the industry-agnostic, technology-agnostic hat. I am always ensuring that we keep that, because that’s what keeps us as an honest broker of this technology,” he added.

Lugo’s currently leading a core team of roughly 15 personnel. But he’s also engaging with a still-growing “expanded team” of close to 500 points of contacts associated with the task force’s activities and aims. To him, those officials are essentially on secondary duty, or a support function to his unit.

“We’re getting more people interested. Now, those 500 people — I’ve got everything from people watching from the bleachers, to personnel saying, ‘Hey, put me in coach!’ So, I’ve got a broad spectrum,” Lugo said.

Nearly 250 people attended a recent “Challenge Day” that the CDAO hosted to connect with industry and academic partners about the challenges associated with implementing generative AI within the DOD.

“There’s a lot of interest in the area, but there’s not that many companies in it. So what we saw was that it’s not just the normal names that you would hear on a day to day basis — but there’s also a lot of companies interested in integrating models. There’s companies that are not necessarily known for LLMs or generative AI, but they are known for other types of integration in the data space and in the AI space. So that was good, because that means that there’s a good pool of talent that will be working on the challenges that we have submitted to industry,” Lugo said.

According to Lugo, the cadre has received more than 120 responses to the recent request for information released to the public to garner input on existing generative AI use cases and critical technical obstacles that accompany its emergence.

The RFI is about learning “what are the insights out there, what are the approaches to solving these particular challenges that we have. And as we compile that information, we will then go ahead and do a more formal solicitation through the proper processes,” he said.

On Nov. 30, industry and academic partners will have an additional opportunity this year to meet with Task Force Lima at the CDAO Industry Day. And down the line during the CDAO’s first in-person symposium — which is set to take place Feb. 20-22 in Washington — an entire track will be dedicated to Task Force Lima and generative AI.

Attendee registration opened in October, and the office is now accepting submissions for potential speakers at that event.

“I’m very optimistic that the challenges that we have submitted will be addressed — and hopefully corrected — by some innovative techniques,” Lugo told DefenseScoop.

The post Inside Task Force Lima’s exploration of 180-plus generative AI use cases for DOD appeared first on DefenseScoop.

The inaugural Chief Digital and AI Office (CDAO) procurement forum is scheduled to take place at an office building in the Rosslyn neighborhood of Arlington, Virginia.

“As CDAO’s ambition, objectives, and budget have doubled within the last year, we are actively seeking ambitious, innovative organizations to learn about our mission, discover opportunities, and compete to contribute to cutting-edge AI standards development within the Department of Defense,” per the announcement, posted on Sam.gov.

The briefing is expected to include the organization’s fiscal 2024 procurement forecast, information about assisted acquisition procurements, an “acquisition ecosystem” primer, and discussions about the Pentagon’s needs related to “Responsible AI,” Joint All-Domain Command and Control (JADC2), Task Force Lima, digital talent management, and Advana enterprise platform capabilities.

Members of industry who want to attend are instructed to fill out an interest form attached to the special notice. The submission deadline is Nov. 12.

On Thursday, the Pentagon also rolled out its new data, analytics and AI adoption strategy. During a call with reporters, CDAO chief Craig Martell said feedback from industry at the upcoming procurement forum will help shape the implementation plan that’s being developed.

“How we partner with industry … is going to be extremely important to delivering this strategy. We will not be able to do this without our industrial partners, without academic partners and without our actual, you know, country partners and allies. So it’s going to have a big impact,” he told DefenseScoop during the call.

“If I come with a vision that says, ‘Here’s how I want to pay you because this is what I need,’ and they all say, ‘Nope, that’s not going to work’ — well great, then I have to rethink that. And then I have to ask them, ‘Well, you know, what is it that’s going to be sustainable for your business?’ … I need those industrial partners to continue to build and sustain this. If I have some crazy idea about what I want to build and nobody wants to build it for me, well that’s not going to work. Right? So we absolutely have to do this in partnership with lots of folks but particular to your question, industry,” he added.

Additional CDAO procurement forums are expected to be held next year, according to the announcement.

Updated on Nov. 3, 2023 at 3:20 PM: This story has been updated to include comments from CDAO’s Craig Martell.

The post Pentagon’s Chief Digital and AI Office to host procurement forum for industry appeared first on DefenseScoop.

Defense Department leaders launched Task Force Lima within the Chief Digital and Artificial Intelligence Office’s (CDAO) Algorithmic Warfare Directorate back in August to accelerate the enterprise’s understanding, assessment and deployment of generative AI. That nascent, buzzy realm broadly encompasses technology associated with large language models that produce (convincing but not always correct) software code and media based on prompts. And experts warn it’s rife with promising opportunities and serious challenges that are far from fully realized. 

In a recent interview, Navy Capt. M. Xavier Lugo, the CDAO’s new Task Force Lima mission commander, briefed DefenseScoop about how this first industry day and the other in-the-works events and elements fit into his team’s overarching, ambitious vision to rapidly navigate the uncertain and transformative power generative AI presents across the defense landscape, and to ultimately support the department’s ethical adoption of the tech.

“The mission from a task force perspective is really to develop, evaluate, recommend and monitor generative AI capabilities for the whole DOD and to responsibly ensure that the department capitalizes on their mission impact while integrating these technologies at scale. That sounds like a lot. So, what we did is we broke it into phases, with the first phase being learning everything that’s out there as much as we can,” Lugo explained. 

Lugo is a 28-year Navy officer with extensive experience in the Supply Corps. A mechanical engineer by degree, he’s been a coder since high school and is also an operations research graduate from the Naval Postgraduate School. In his previous assignment, he worked to modernize logistics-aligned information technologies for the entire Navy.

“I am very familiar with how [AI technology] actually works, and how it utilizes data,” he told DefenseScoop.

In his current capacity leading the new task force, Lugo meets with DOD’s Chief Digital and AI Officer Craig Martell very frequently. He and Martell — the former head of machine learning at Lyft — have “some geek sessions here and there” too, Lugo noted. They “don’t always agree on some of the theoretical stuff,” he added. 

But to both, encouraging diverse viewpoints in tech-driving pursuits is a good thing.

“From our perspective, because the use cases are military-related and because the workflows are all based on operational pieces, you do want military officers involved at some level. I have been lucky that I was involved from the sense of being the commander of it. But we bring that practical flavor to the research and technical pieces of AI. And also, just to be clear, CDAO as a whole is not a research entity. We’re not here to build models. We’re here to operationalize them. And that’s where a task force like Task Force Lima and the military officers in this command come in: We operationalize this technology,” Lugo told DefenseScoop.

DOD is a sprawling, distributed enterprise — “so getting to all the generative AI efforts out there is not an easy task,” the task force chief also noted.

So far though, his team has had “very good collaboration” from the services, combatant commands, and all DOD agencies. They are beginning to hone in on the categories where large language models, specifically as part of the generative AI-capable tools, are being considered for performance across the department.

Lugo’s team also has a short-term task to populate interim guidance on generative AI for the Pentagon.

“That has also been sort of coordinated. What I mean with ‘sort of’ is that it’s not not being coordinated, but that in parallel all the services and federal agencies have also provided guidance. So, what we did was, we’ve taken all of those multiple guidances and also included what we have learned so far, and we’re going to populate that soon. That’s one of the first steps so that everybody knows what their ‘left and right limits’ are with this technology, and so that we can all be at least in the same field as to what we are thinking of utilizing the tech,” Lugo said.

Task Force Lima, he added, has multiple “touch points” to carry out its mission.

“We have a battle rhythm of weekly, monthly and quarterly meetings,” he said.

And earlier this month, the cadre hosted its first government-only kickoff to engage internal stakeholders on their in-the-making plans and approach. Between 200 and 300 officials attended virtually and in person.

That “gives you a scale of the level of interest — and every single service was represented, plus the Coast Guard, plus multiple agencies, all the [Principal Staff Assistants]. I mean, it was very well covered from the perspective of which agencies and personnel were in it,” Lugo noted.

The team also recently released a new request for information via which it has invited the public to help the DOD “further understand and explore the risks and benefits of development, acquisition, and integration” of generative AI. 

Responses to that RFI are due by Oct. 8. 

“That is based on what we’ve learned so far. There are some gaps that we have noticed in the technology. And what that challenge is about is to tell [industry and academia] we are interested: ‘You’ve done a great job on the development and on the delivery of this technology but there’s some pieces that we need some help from you, because if you fix these gaps, we have more potential use of this technology,’” Lugo explained.

One example of such gaps is referred to as “hallucinations” — a phenomenon where a large language model essentially makes up untrue information or false facts that aren’t rooted in actual real data or events.

“So, how do we get around that? How do we measure it? How do we protect against it? What are some solutions out there — whether it is a single solution from a model per se, or whether it is a system with a system solution, or whether it is a mitigation process? Any of those or all of those, right, for that just one challenge. Come in and tell us your ideas on how we’re going to do that,” Lugo said.

Potential collaborators are learning more about this and other issues that need addressing at the industry day unfolding Wednesday. The task force will accept white papers — and then ultimately aim to work with partners through its Tradewinds initiative.

“While building the plane after we’re flying it — this is one of those pieces that has to be built over,” Lugo said.

Ultimately, Task Force Lima is deeply involved in both the Pentagon’s actual research and deployment of large language models, and in informing its creation of guidance, frameworks, workflows and policies to govern their evolving use.  

While publicly accessible models made by Open AI, Google, Microsoft and others already exist, the Pentagon has urged components not to enter government information into them for reasons it says are associated with operational security.

“We are not building at this moment in time and we have not decided whether we’re going to build DOD-specific models. However, what is happening is that models that are already built are being utilized in a containerized fashion — so disconnected from the wild, if you want to put it that way,” Lugo noted.

Multiple times in the interview, he emphasized that there are still many unknowns regarding if, in what ways, and for what purposes generative AI might be eventually deployed at scale by the military and DOD.

“Our mind is not necessarily made up yet — without doing the experimentation — as to whether we’re going to go ahead and say, ‘Yes, for sure, we’re going to implement this technology.’ That’s exactly where we are right now and that’s part of the reason we were stood up. This is new, this is out there with a lot of emphasis from industry — and so, how should we and how can we utilize this technology? Those are the two main questions,” Lugo said.

Ambitiously, Task Force Lima’s initial timeline to figure that out is only 18 months.

“Now, let me caveat that the reason we have a short timeline is because of the immediate nature of the technology,” Lugo noted.

Still, before his team stands down after that year-and-a-half passes, the plan is to provide a transition strategy with deliverables including a list of all the players and assets needed to carry on this generative AI-driving work. 

“This is extremely, extremely new technology — and we’ve got to have the humans on top of it all the time,” Lugo told DefenseScoop.

The post Task Force Lima chief gives first look at DOD’s vision for rapidly exploring the uncertain power of generative AI appeared first on DefenseScoop.

Joseph Larson, III, deputy chief digital and AI officer for the DOD Chief Digital and Artificial Intelligence Office, delivered his pitch to government and industry leaders for their participation in addressing these challenges alongside the DOD during Google Cloud NEXT session Supercharging the public sector with AI.

Generative AI is a “technology that leaked from the laboratory, and the department needs to move fast to be prepared for a future where it is a part of the way we do work today,” he stressed.

The rapid adoption of generative AI is a major disruptor of the DOD’s long-established practice of research, validation, testing and deployment when adopting new technology.

Larson explained that the task force was born out of a “concern about both the enormous risks of the technology being misapplied, misused and misunderstood in terms of our workflows, but at the same time the potential, but unrealized and unproved, benefits to a massive enterprise that has the need to leverage AI to solve a number of its problems.”

Task Force Lima is only a few weeks old, but the team seeks involvement from industry to help the DOD understand the rapidly evolving technology.

Larson cited a number of goals the task force is focused on, such as how to measure the performance of models, better understand the metrics and frameworks towards responsible AI, define the strategy and guidance around these frameworks, build out the infrastructure needed to manage data and to identify use cases where the technology would be most valuable.

“This problem of generative AI is taking up a lot of our bandwidth right now, and it’s an area of enormous challenge where we’re going to need the integrated insights of the technology community to advise and assist the department not just in terms of your products, but in terms of the ramifications at a broad level of how this technology manifests in our department,” he said.

“Supercharging the public sector with AI is an objective that can accelerate the public sectors’ missions overall and fulfill the promise of bridging the gap between what constituencies are worried about and how public servants can respond,” shared Katharyn White, director of marketing for public sector at Google.

Watch the full panel discussion, “Supercharging the Public Sector with AI,” and hear more from our government leaders on their AI and security initiatives.

The post Task Force Lima spurs movement in the DOD to understand generative AI appeared first on DefenseScoop.