Task Force Lima's Mission and Challenges with Captain Manuel Xavier Lugo

Bonnie Evangelista [00:00:03]:

All right.

Captain Manuel Xavier Lugo [00:00:03]:

Good afternoon. This is Bonnie Evangelista with the chief Digital and Artificial intelligence office. We're here at DoD Advantage 2024, and I have my esteemed colleague, Captain Lugo, or as most of us call you, X.

Bonnie Evangelista [00:00:15]:

Right, yes. Hello, Bonnie.

Captain Manuel Xavier Lugo [00:00:17]:

How you doing?

Bonnie Evangelista [00:00:18]:

I'm doing good. It's been a very exciting time over here at advantage.

Captain Manuel Xavier Lugo [00:00:23]:

How's the conference been for you?

Bonnie Evangelista [00:00:24]:

It's been very, very insightful. That's probably the best word I can use.

Captain Manuel Xavier Lugo [00:00:31]:

In what way?

Bonnie Evangelista [00:00:32]:

So we weren't sure what the state of AI, specifically on generative AI, which is what I'm focused on. We weren't sure what the state of it was and state of the art. And to be honest, I'm very happily surprised with what I'm seeing from industry. Not just industry, industry, academia, and also the DoD at large. The problems are not gone, but at least everybody has identified pretty much the same challenges across.

Captain Manuel Xavier Lugo [00:01:12]:

Did you assume or expect people would be a little bit further behind in terms of working towards the space?

Bonnie Evangelista [00:01:20]:

I wasn't 100% sure what, what, what to expect, but I was expecting that we, as a task force, had missed or had some blind spots out there. And what I. And what actually what I found is that a lot has been validated, the thought processes that we've had, but also that the challenges, although, although large, they are quantifiable, so it's not as mystical in certain areas. Now, don't get me wrong, that doesn't mean that we don't have challenges. It doesn't mean that trust and confidence is where it needs to be on this technology. But at least we know we have some ways ahead of how to get to that trust and confidence.

Captain Manuel Xavier Lugo [00:02:11]:

So let's level set up a little bit. You mentioned the task force. Let's start with you. Who are you in CDAO? Like, what's your area of responsibility? What are you tackling?

Bonnie Evangelista [00:02:21]:

So I am the initiatives and capability delivery branch chief under algorithmic warfare in the CDAO. That is my day job, but I'm also the commander for Task Force Lima. And inside the initiatives and capability delivery, I'm leading the AI scaffolding, or also known as Alpha one.

Captain Manuel Xavier Lugo [00:02:50]:

Let's start with task force Lima, because I feel like that is a little bit of a buzzword right now. What is that? What was the genesis and what are you doing?

Bonnie Evangelista [00:02:58]:

So task force Lima was chartered by the deputy Sec Def on August of last year. She basically tasked the CDAO to tackle the regenerative AI problem space in the way that the charter is basically started. As large language models. Back in the summer, right before the official sharder was signed, we identified as a department that large range models could have potential benefit and also a lot of risk. But we also understood that there was some urgency into getting that technology applied to the DoD, or at least if it's not applied, safeguarded guardrail, and come up with some sort of mechanism to understand it better. So the task force purpose is exactly that is to learn about the technology, accelerate it where it properly fits, and then provide guidance to the department on how to best evaluate the technology, how to best make it fit in current workflows. And we have deliverables associated with how to have informed policies around this new technology. But also, if we are going to decide to implement it in the DoD, how would we do it at scale, and should we do it at scale? So there's a lot of questions that the task force is tackling and answering, and it's doing it in those three lines of effort of learn, accelerate and guide also is term limited to 18 months.

Captain Manuel Xavier Lugo [00:04:58]:

The task force?

Bonnie Evangelista [00:04:59]:

The task force, yes.

Captain Manuel Xavier Lugo [00:05:01]:

Did the depth Sec DeF give an outcome or an end state she's looking for at the end of this 18 months?

Bonnie Evangelista [00:05:07]:

Correct. So there's a set of deliverables, everything from, and we are tackling those time wise, but everything from interim guidance that had to be put out within 30 days of the task force established, which we did and followed with what is our RMF or risk management frameworks for this technology, also understanding the workflows and use cases that apply to the, specifically to a DoD, and again, all the other aspects of the technology or from any type of new software or new technology out there? Are there any cybersecurity risk? Are there any other risks out there that we are not addressing with current instructions and policies, as well as what will be the implementation plan for this technology to include, potentially, how we would resource it and sustain it for the department.

Captain Manuel Xavier Lugo [00:06:13]:

So the learning part of that is very interesting. How are you going after learning about what's happening and how to get to the implementation plan?

Bonnie Evangelista [00:06:20]:

So the first thing we did was we, we started collecting use cases. We're at 218 use cases right now from across the whole DoD. I can tell you, obviously, that's just a sample. However, when we analyze that sample, they're all converging to about three to five capabilities or, or categorizations of the use cases. For example, summarization, use cases can be utilized in multiple areas of the DoD, just as an example or code generation. It also has a lot of applicability. So even though we have 218 use cases, we are aggregating them to just a small number of capabilities that are desired for this technology and areas of implementation of the technology. That's the learn side from the demand side.

Bonnie Evangelista [00:07:28]:

The other piece of the learning, which has been extremely interesting, is learn what is actually out there in terms of capabilities. Correct? In terms of capabilities, most of the capabilities are coming from industry, but there is academia capabilities also developed from academia and also a lot of open source. So we're working on learning what all those are. And for the first time ever, I believe that from a government, first time ever, for me, from a government perspective, I'm only a couple weeks behind, was actually out there. And now that's not because the task force is great at seeking this. It's also mostly because the technology is getting pushed to us with such frequency that we are not getting surprised by things that are coming out because we either get a heads up or we're only a couple weeks behind on it actually coming out.

Captain Manuel Xavier Lugo [00:08:31]:

I see.

Bonnie Evangelista [00:08:32]:

Yeah.

Captain Manuel Xavier Lugo [00:08:32]:

How are you trying to either merge or connect? What I'm hearing is lots of understanding problem sets in which we could leverage this technology. And then I'm learning, you know, you're learning what's out there capability wise. So are the two marrying up or what's. Is there a plan there? You're smiling, so I'm hoping you're smiling.

Bonnie Evangelista [00:08:55]:

Because that's exactly what we're trying to do here today, this week. Right. So we've designed a plan that will involve analyzing the use cases to come up with what we're calling acceptability criteria for a workflow. So I think it's safe to say that we're abandoning the black box mentality or the LLM as a model, as a single solution for any particular workflow. I mean, there's some workflows where all you need is a chatbot, but for the most part, the workflows are systems of systems and to include human machine teaming. So we're coming up with acceptability criteria from the use case perspective. So we need this technology or this actual LLM system, not just the LLM and LLM system, to perform these particular functions at this level of maturity or at this level of automation.

Captain Manuel Xavier Lugo [00:09:58]:

When you say at this level, my mind went to technology readiness level. What do you think?

Bonnie Evangelista [00:10:05]:

What are you. No, we are doing a parallel of autonomous vehicles. So right now, for autonomous vehicles, there's a zero to five scale on autonomy for them. Now, that's a. As hard as that is is a singular use case, right? So what you have everything from the 1973 Chevy pickup truck that has no autonomy in it, right. And you have to do all the manual driving and then you go and progress. Well, now we add a cruise control. So that's another level.

Bonnie Evangelista [00:10:36]:

Right. Then you got adaptive cruise control. Adaptive cruise control with lane navigation. Finally to a point where you just sit on the vehicle and say, I need to go here, and it'll take you there. Right. So those are the, that's the scale. And that's what we're talking about when talking about of level of readiness in the sense of or maturity level for that particular functionality. For that particular functionality of the LLMs.

Bonnie Evangelista [00:11:02]:

We haven't figured it out yet. So it's. Even the word. Even the words themselves are not totally codified. What are we going to call it? However, that scale, whatever the scale end up being, will signify impact to the workflow if that capability is not at that particular scale value.

Captain Manuel Xavier Lugo [00:11:25]:

And what workflow are you.

Bonnie Evangelista [00:11:28]:

So the use case workflows. Okay, so the particular use case that a. That an emission owner would come with. Right. So that's one side of the equation. Then the other side of the equation is, okay, where's. So we're challenging industry to now meet us at that acceptability criteria. And they got to come up with not only matches on what those functions are, but also methods of measuring and testing whether they can actually meet those levels.

Captain Manuel Xavier Lugo [00:12:05]:

I'm going to put this on the table. And the task force is not meant to buy or produce anything, right?

Bonnie Evangelista [00:12:13]:

That is correct.

Captain Manuel Xavier Lugo [00:12:14]:

So even though you're. I'm hearing you say industry is going to meet us where we're at. And so when I meet.

Bonnie Evangelista [00:12:19]:

So us in this contact is Dod. Got it. Yeah. So, and to that point, the use cases, all of them have owners. We own a couple. We as in cDal, we as in Lima own a couple, but it's very minimal number. The use cases are owned at all echelons, everything from OSD level all the way down to units. So you're.

Captain Manuel Xavier Lugo [00:12:45]:

I'm hearing you're in an 18 month sprint.

Bonnie Evangelista [00:12:47]:

Yes.

Captain Manuel Xavier Lugo [00:12:48]:

Or maybe that could be dissected into mini sprints. You're gonna collect and analyze things and information in order to produce some guidance. And what other kinds of deliverables can those use case owners take with them? Right. To go buy the thing.

Bonnie Evangelista [00:13:10]:

So three things. Although each one of these are lines of effort, so means they're continuous the whole 18 months. We were focusing on learn first. Right. If we had to, it makes sense to learn what your AoR is and what's. We have a good feeling right now that even though we're still accepting use cases, and by the way, most of these use cases were llms, we are generative AI, so we are accepting also other modalities. It doesn't all have to be large language models. Right.

Bonnie Evangelista [00:13:45]:

But with that said, we have a pretty good descriptive analytics on the use cases. We slice and dice them into many, many ways of looking at the use cases, whether it is, what capability is needed, where it is, in what area they're working. What we did was we set up four tiers. Tier one are use cases that have a path to completeness right now.

Captain Manuel Xavier Lugo [00:14:14]:

A path. What is a path to completeness?

Bonnie Evangelista [00:14:17]:

There we go. A path to completeness means they've clearly identified where they're going to integrate this technology.

Captain Manuel Xavier Lugo [00:14:25]:

I see. I see.

Bonnie Evangelista [00:14:26]:

They have the resources to actually purchase the technology that they need and they have the platforms that they're going to run this in, all identified. So what we did with those, and there's only eight of those. What we've done with those, out of 200.

Captain Manuel Xavier Lugo [00:14:41]:

And how many?

Bonnie Evangelista [00:14:42]:

218. Eight out of two.

Captain Manuel Xavier Lugo [00:14:44]:

Okay.

Bonnie Evangelista [00:14:45]:

Yes, those eight we're calling the exemplar use cases. And what we're doing is we're partnering with the use case owners in order to learn from their implementation. Right. We're not an obstacle to them, if anything. What we're doing is we're also providing them with any frameworks, any idea, any guidance that we have from our experts or from learning from other areas on the DoD, on this technology. So that's the tier one. Tier two is about 21, is the. It's more of the not just do it, but they're, and that's probably the wrong term is more the easy to acquire, easy to perform.

Bonnie Evangelista [00:15:32]:

But they're missing some of those elements that I talked about for completeness. They may be missing the resourcing, or they may be missing the platform, or they may not know exactly what model to use in their work, but their workflow is specific enough that we know that we can help them out. If we can feed their gap. Then there's tier three use cases, which are either ill defined, don't fall into those characteristics that we talked about, or they're what I call Sci-FI it's like we're not even, the technology is not even close to do those, but we're still collecting them because we want to know what the demand is out there.

Captain Manuel Xavier Lugo [00:16:19]:

So you mean the use cases or there's no technology to fill the gap.

Bonnie Evangelista [00:16:28]:

For the use case at this time?

Captain Manuel Xavier Lugo [00:16:30]:

At this time.

Bonnie Evangelista [00:16:31]:

Okay, that's correct. But that's not the only characteristic of the tier three. I'm just saying in tier three. And then there's the tier four use cases, which is either the risk is too high, we're not going to implement that period, or they're not LLM cases. They can be solved with something else. Techniques, all with NLP or other techniques.

Captain Manuel Xavier Lugo [00:16:55]:

Okay, so where, and I know you're still very much in your learning, it's still kind of early in the task force. Where are you finding interdependencies, like in terms of, in order, especially with the exemplar use cases? I'm thinking of that path to completion. What, what other relationships or connections or bridges do we need to be making in this process?

Bonnie Evangelista [00:17:26]:

Okay.

Captain Manuel Xavier Lugo [00:17:28]:

To make that successful? So riff with me and just see if that makes sense. But I think we'll get it.

Bonnie Evangelista [00:17:33]:

So without going into details of what those eight are, one of them had been already implementing this, this or similar technology for two years.

Captain Manuel Xavier Lugo [00:17:48]:

Okay.

Bonnie Evangelista [00:17:49]:

So it's already part of their workflow.

Captain Manuel Xavier Lugo [00:17:51]:

Okay.

Bonnie Evangelista [00:17:52]:

The difference is that the tech was not at the level that is now. So now all they're doing is pretty much a plug and play. It's putting a new engine in their, in their Formula one car. Right.

Captain Manuel Xavier Lugo [00:18:03]:

Okay.

Bonnie Evangelista [00:18:05]:

However, they needed, they needed some resourcing to continue through that path. And we're able to, we, as in the algorithmic warfare division, were able to help them out for that.

Captain Manuel Xavier Lugo [00:18:23]:

That's a good example, though, just as a thought experiment. Like, yes, you were able to help them out now, but that's a dependency. Right. So they need to, like who, who does own that if that, if they're what they, if they found something good.

Bonnie Evangelista [00:18:35]:

Right.

Captain Manuel Xavier Lugo [00:18:36]:

And you want to build on that success or you want to make it, you want to either scale it or make it a thing.

Bonnie Evangelista [00:18:42]:

Exactly. So what do we do for sustainment, right, for that particular use case?

Captain Manuel Xavier Lugo [00:18:46]:

Yeah.

Bonnie Evangelista [00:18:47]:

So what we do for sustainment for that particular use case is if we go and decide, and I'm being very careful by saying that, right, because we haven't made a decision yet whether this should be an enterprise level, this total thought experiment.

Captain Manuel Xavier Lugo [00:19:01]:

In my head, I'm just trying to demonstrate the work you're doing, right.

Bonnie Evangelista [00:19:05]:

But if we were to decide that, that these models have to be, should be centrally or enterprise level, then what we benefit is that we get the information that we need in order to, in terms of architecture, in terms of implementation of the workflow, in terms of risk, in terms of all these different dimensions that we have that we're exploring, and then they get the benefit of, they'll get the, the enterprise level tool to do it. So the sustainment in theory should be absorbed by the DOD at large for that in theory. And I say that I'm being, again, each use case is different. So right now they're paying it from command funds. So we need to figure out should they even continue to do that? Should they pump for it? You know, there's so many questions in that, in that particular area.

Captain Manuel Xavier Lugo [00:20:06]:

How about are you looking at or have you thought about the people part of what we just talked about? So scaling and doing enterprise is a people change. Any, any thoughts on that?

Bonnie Evangelista [00:20:27]:

Yes, although we're not there yet. And the reason I say that is because we're working it the way that we're working the exploration of this particular technology. We're going to get to what are the actual requirements? And the requirements is going to be also people. What kind of people, what kind of talent? What is it that we need? Do we contract out the personnel? Do we rely on industry?

Captain Manuel Xavier Lugo [00:20:55]:

I'm thinking more. Sorry, adoption, no, adoption. Like if there's a new tool, yes, it could be for acquisition, right. It could be that's a potentially, might be cultural barriers or organizational change, things you got to worry about. Right. So I'm just curious if that's been part of, if you've just heavily focused on tech or if you have had any conversations about the people.

Bonnie Evangelista [00:21:19]:

So when we talk about the workflow, the people are definitely integrating into that conversation. So each of the use cases is we're definitely Personas, we're defining who are the actual users in here and where do they fit for the outcome of that workflow. So it is all maybe with the exception of the Sci-Fi problems, they're all about human machine teaming. So it's how do they make decision makers either do better informed decisions or faster in their decision space? There's a lot there. Then there's also just on the workflow per se, of automating pieces that are very mundane or very just time consuming that don't require much thought process in the human perspective.

Captain Manuel Xavier Lugo [00:22:14]:

Right.

Bonnie Evangelista [00:22:16]:

And or tasks that are about summarization of a lot. Thousands of pubs or tens of thousands of pubs or documents or lessons learned or information. We, there's no lack of data in the department, it's just the way the data is stored. Right. We got everything from, from microfiche to cloud. Right.

Captain Manuel Xavier Lugo [00:22:41]:

So, all right, so based on where we are today, this. Nothing's absolute right now in this conversation, but what are you most fearful of?

Bonnie Evangelista [00:22:53]:

So, as I tell people, don't be scared of the technology, but be skeptical. My problem is that I think we can be a little reckless in the adoption of it and start performing decisions, start making decisions without really knowing how this technology is working in the sense of, and I know everybody talks about hallucinations, but that's not a trivial problem. But that doesn't eliminate the technology's usefulness. You just need to know it, understand it, and see where it fits in your workflow. And what are your mitigation techniques to it. The problem I have is that I don't know what percentage of the. Of the users really understand what that means. And look, users are, users are smart, and they might learn by pain as well.

Bonnie Evangelista [00:23:57]:

So the one time that you submit something that had 30% hallucination rate on it and was errors, and your boss comes back to, like, why you give me this? That might be a lesson point, and that might be fine in some use cases, but we do have some no fail missions that we can't afford at. So. So am I scared that we're going to do that? No. I trust our commanders. We all should look at this with skepticism, and I think there's healthy skepticism on the technology out there, but I also think that there's a little bit of this is magical, and this is going to solve all my problems out there as well. And I don't know what the ratio of those two is.

Captain Manuel Xavier Lugo [00:24:45]:

Okay. On the other side of the coin, what are you most optimistic about now that you have seen what you've seen and we're here.

Bonnie Evangelista [00:24:53]:

Yes.

Captain Manuel Xavier Lugo [00:24:54]:

At this event.

Bonnie Evangelista [00:24:55]:

So I think it's almost a flip of what I just said. Right. So the fact that we're all converging to the same characterizations of this technology gives me a lot of hope because means that we have identified the potential, but we have also, at least from the practitioners and the experts, we understand what are the limitations of the technology and what. What we need to do to either understand it or mitigate it. We don't have the answers. We don't have the science figured out yet, but we have identified that we need to do that and that this is not a magic pill or a black box that will solve all your problems. I think that is starting to become understood. I'm also optimistic by looking at the use cases, how creative people are with this technology, and that it's just going to be the beginning of more and more creative ways of thinking that I feel that, yes.

Captain Manuel Xavier Lugo [00:26:05]:

I think the more we can push down not just the technology, but the ability to play at that level like you're describing, we'll see lots of goodness from that.

Bonnie Evangelista [00:26:20]:

There's no point of us pushing down to the use cases. Right. That's why we're doing a pool, because that's where we decree. So don't underestimate a sailor's laziness to come up with creativity. Right. So. And that's exactly what this technology is showing. Right? Yeah.

Captain Manuel Xavier Lugo [00:26:41]:

Yeah. Are you looking for an end state or are you kind of navigating everything you just talked about, not knowing where this is going to land?

Bonnie Evangelista [00:26:50]:

So. Very good. I forgot to mention that part. So we have the liberals, right.

Captain Manuel Xavier Lugo [00:26:57]:

Right.

Bonnie Evangelista [00:26:58]:

And the deliberables will do one of two things. They'll either be completed prior to the 18 months or they'll be out. Some of them might be outstanding at the 18 month mark. Right. That's just, that's a binary, binary piece that's going to happen there. However, part of the deliverables is also to find what is the right transition point for that particular deliverable. So right now we are, as a task force, we're tackling pretty much a vertical stack of this problem. Right.

Bonnie Evangelista [00:27:30]:

But reality is that there's a lot of pieces here that are going to have to be divvied out across the department. So we'll have transition plans for anything that is either an open question that may remain out there or hopefully not for outstanding deliberables. But if they're outstanding for a reason, we should be able to go ahead.

Captain Manuel Xavier Lugo [00:27:52]:

And a decision point, right? Yeah. Okay.

Bonnie Evangelista [00:27:55]:

Yeah.

Captain Manuel Xavier Lugo [00:27:55]:

Okay. So let's talk to both industry and government. What do you want to leave industry with, like biggest takeaways for task force Lima? Either how can they help you or what do you want them to know?

Bonnie Evangelista [00:28:08]:

Biggest takeaway is we have a set of challenges out. We need you. We need your help in getting those challenges resolved. Also is do not skip the basics. There is a confidence factor in this technology that has to be achieved if we want even higher adoption. So don't skip that. Right now. We hear a lot about the potential, and I agree, but the potential comes with assumptions that hasn't been, that haven't been validated.

Bonnie Evangelista [00:28:43]:

That would be my, let me double click that.

Captain Manuel Xavier Lugo [00:28:46]:

You said challenges are out. So like, is there, how do they get plugged in? How do they respond?

Bonnie Evangelista [00:28:51]:

So we got challenges in the trade winds portal, we got some of them out. There were challenges to industry with white paper. So there's a list of that that is still there.

Captain Manuel Xavier Lugo [00:29:04]:

They're active.

Bonnie Evangelista [00:29:05]:

Yes. The list of the challenges is we're not receiving any more white paper. So challenges are out there. And now we're talking about with maturity model, there's challenges are coming out here too.

Captain Manuel Xavier Lugo [00:29:16]:

Okay. Uh, for government, what do you want? Any takeaways for government with regard to task force lima either?

Bonnie Evangelista [00:29:23]:

Yes. So keep pumping the use cases right as they come along. We, I didn't speak too much about the accelerate piece, which is us building sandboxes and us providing tools for government to actually experiment and explore. That's coming up. We just had a senior working group meeting last week and we talked about what are the, in the next 30 days, what we're going to come out with in the sense of sandboxes, access to tools, foundational models and building these models or building something as simple as a chatbot. We proved it in a hackathon that it's doable by government employees. We can do this. So we're going to.

Bonnie Evangelista [00:30:12]:

Probably not probably. Our plan is to actually start exposing those out there and make sure that, and see how they fit into your work, into your use cases. So reach out to us if you have a use case, if you have a need for actual, an actual model or you just want to learn about generative AI and, and it's.

Captain Manuel Xavier Lugo [00:30:33]:

Can anyone be, can anyone be part of the task force or your working group? Whatever.

Bonnie Evangelista [00:30:38]:

Yeah. So, so we have right now about 800 people associated with the task force. Everything from people in the bleachers to put me in coach. Right. So if you're interested in this technology, we'll probably make available the link. There's a confluence page that anybody with a CAC can access and ways to join the task force or in any type of the areas that we're looking for. Yes, we need help. We need help because we need to.

Bonnie Evangelista [00:31:12]:

This has to be a distributed way of thinking. It can't not be just us pushing down information.

Captain Manuel Xavier Lugo [00:31:18]:

Is there any other way people can reach out to you? Group inbox or something like that or what?

Bonnie Evangelista [00:31:23]:

Yes, there is a group inbox.

Captain Manuel Xavier Lugo [00:31:25]:

So we'll, we'll put it in the show notes. How about that?

Bonnie Evangelista [00:31:28]:

Yes, please.

Captain Manuel Xavier Lugo [00:31:28]:

Yeah.

Bonnie Evangelista [00:31:29]:

Okay.

Captain Manuel Xavier Lugo [00:31:29]:

Okay. Thank you so much. That was a lot. I think we had planned on talking about more, but we'll have to do a follow up.

Bonnie Evangelista [00:31:38]:

Thank you, Bonnie.

Captain Manuel Xavier Lugo [00:31:38]:

Thank you.