Streamlining Government Contracting (Bonnie’s Interview on Closing the Loop Podcast)

Bonnie Evangelista [00:00:18]:
All right. Hey, team. Bonnie here from the Chief Digital and Artificial Intelligence office. This week, we're sharing an interview I did on the Closing the Loop podcast with Andrew Campbell, the VP of OpenView. Andrew is doing a series of interviews with executives from large global businesses, and he wanted to talk to the likes of me to figure out how the Department of Defense is handling the same issues that we're seeing on the private sector side. So we talked about the inefficiencies in government contracting and how we can reduce our workflows from three months to 30 days. And, yes, that's a real thing.

Andrew Camel [00:00:52]:
I hope you enjoy it, Bonnie. So excited to have you on.

Bonnie Evangelista [00:00:55]:
Absolutely. My pleasure.

Andrew Camel [00:00:57]:
I feel like it's very appropriate that you have the American flag in the background. It's not often that I get to talk to somebody in such an interesting role as yours. So let's just jump right in. I think it'd be great if we could just get a bit of context on where you sit in the DoD and what your roles and responsibilities are and yet where you fit into the picture.

Bonnie Evangelista [00:01:17]:
I work for an organization called the Chief Digital and Artificial Intelligence Office for the Department of Defense. So this is the best way to explain it, because there are so many layers and tiers within the department, is that I sit above where the services are. So we have the Army, Navy, Air Force, and I sit a layer above that, and our office reports directly to the Deputy Secretary of Defense. So even there's a ton of bureaucracy, even in the office of the Secretary of Defense, and we don't even sit within that. We're kind of off to the side, and we have our own special mission doing digital and AI and data analytics for the department, trying to figure out how to do that in this modern day of digital transformation, maybe there's a group of us just trying to bear down, figure it out, and get after it. Like, how do we get capability into the hands of soldiers, as I like to say, my role within this organization is? It's funny you want to talk to me because I'm a self proclaimed contracting nerd, and I have been doing federal contracting literally my entire government career. But I've kind of landed at the intersection point of technology, especially emerging technology, as you can tell by the office that I work in. So although I am very much in what we like to call execution of contracts and basically procurement and buying things, I have to be in this world just as much as all the smart people do.

Bonnie Evangelista [00:02:49]:
So I'm surrounded by lots of smart people, but I like to call myself.

Andrew Camel [00:02:53]:
The contracts nerd of the yeah, yeah, okay, great context. I think before we jump into all the interesting things you're working on, it might be helpful for the uninitiated on government contracting within the DoD, how that typically works. Like what is the workflow for somebody wants to go buy a tank? How does that actually happen?

Bonnie Evangelista [00:03:13]:
Okay, I'm going to have to really be succinct in this, Earl. An entirety of that process can overwhelm people. Essentially, though, before you can buy something there has to be a requirement. So somebody in the department has to raise their hand and say, I need a tank. And that's usually at the I call that the tactical level. And then someone else in the department has to help the user or the consumer essentially develop the requirements around. Okay, well what does the tank have to do? Like how fast does it have to go? How much are there weight restrictions? Does it have to be ruggedized? Does it have to do X, Y or Z? Does it have to be the Cadillac or the Corolla type style? Someone has to define what are we buying and the requirements. Process in itself is very much an art as much as it is a science, I would say.

Bonnie Evangelista [00:04:09]:
And this is an area admittedly, where the department gets criticized a lot because it's very prescriptive and it's a very lengthy process and there's good reasons for that too. But in this day and age, I think part of jumping ahead a little bit, we have to figure out how to do this faster. So once you have a requirement, then someone in the acquisitions or the contracting group has to help figure out how to acquire the capability and how an acquisition I'm not going to get into what is that we can go 50 layers deep on that. That's different than procurement though. So essentially in order to buy, there's a ton of regulation and rules and laws that dictate how we spend taxpayer dollars, essentially. And a lot of those rules involve the contracting principles of fair competition, transparency. And I'm going to stick with fairness to all. And fairness to all has essentially equated to everybody gets access to all the same things.

Bonnie Evangelista [00:05:15]:
It's what we call full and open competition. Everybody gets a bite at the apple kind of thing and there's exceptions to that. But I'm just going to speak in generalities for now and with that becomes a very lengthy process. So there is a government we call it the government point of entry, like the place where everybody can see all the opportunities within the government where that requirement is publicized. And then there's lots of terms and conditions and there's lots of instructions for how to respond to the solicitation for the government. And we're talking lengthy technical proposals potentially. And then there is something called source selection. So once the proposals are received.

Bonnie Evangelista [00:06:00]:
There is a panel of subject matter experts or other stakeholders who might be involved in terms of delivering that capability, who are going to determine what right looks like or what is the best solution to addressing the requirement. Source selection can take months and I'm not going to get in unless you want to deep dive into that as well. And usually then you can roll right into a contract award once source selection is done. But source selection is not just there's different tactics you can use. Most people use what's called best value or trade off. So just because you have the lowest price or the highest price doesn't necessarily mean maybe there's other benefits in your proposal that the government's willing to pay for. So there's lots of trade off discussions and there's lots of attorneys and there's lots of leadership depending on the dollar value. Because if you're getting into the millions, tens of millions, potentially hundreds of millions of dollars, the higher the scope of the award, the more people that are involved.

Bonnie Evangelista [00:07:04]:
Right. Because there has to be more due diligence on the government side.

Andrew Camel [00:07:08]:
Understandably?

Bonnie Evangelista [00:07:09]:
Yeah. And everything I just described can take anywhere between twelve to 18 months just for a single contract. That's kind of the elephant in the room when we talk about emerging technology.

Andrew Camel [00:07:23]:
Yeah. Okay. Very helpful framing. And so maybe it's a good transition point to say, where do you see those opportunities to accelerate that process? Because it feels so multidimensional and sort of loaded in bureaucracy and slowness. You have really interesting insight into how to improve that. So I'm curious where you see that opportunity.

Bonnie Evangelista [00:07:44]:
I love that question. Thank you. When it comes to emerging technology, I'm not trying to solve world hunger overnight. So these comments are for anyone out there who's trying to get after speed to contract because of how quickly technology is moving and how volatile this space is right now and the necessity to get your hands on something, just to try it and learn from it. That's the audience that I'm particularly talking to as I'm sharing this with you. But we got to stop defining requirements and we got to start defining problems. So I think we're way past the government trying to predict what's going to happen in industry where I don't think we have the expertise or the capacity you just mentioned. We're bureaucratic, we're slow.

Bonnie Evangelista [00:08:31]:
We're never going to go, or I don't think we can go the way we're currently the construct of our department is currently designed and built and working under. We can't go that fast. So can we define the problem that the requirement is meant to address? So rather than us telling industry, I need a tank, and we can tell industry, hey, I need the ability to, I don't know, shoot this thing.

Andrew Camel [00:08:58]:
I think we talked about the situation before. If you need to develop a targeting system, you don't want to say specifically, I want this type of computer and this type of lens and this type of you want to basically state the problem you have, the thing that you want to go solve. And then industry should come to you and say, here's what we have, here's what we can offer, what we can possibly develop.

Bonnie Evangelista [00:09:17]:
Absolutely. And I like to or kind of the place I've landed or my team has landed when we try to do this, because this is also our approach to how we're issuing challenges and things of that nature. We really focus on outcomes are end states. So the army doctrine way is current state, end state, and then there's a gap between your current state and your end state. So, I mean, that's a good place to start. Identify the mission gap or identify the place that the pain points that you have to getting out of your current state, man, maybe you don't even know what your end state is, but what's your objective? Like, I just want to go faster, or I want it to be easier, or I want my current process to be less expensive. My hypothesis is there are way more solutions in industry out there than we realize and we're just not allowing ourselves to be open to that. So I think that's like opportunity number one for sure.

Andrew Camel [00:10:09]:
And when you think about that process of defining requirements, that feels pretty baked into law. Maybe I'm wrong about that, but how do you envision that actually changing? If it can in the short term?

Bonnie Evangelista [00:10:20]:
Yeah, I would offer it's more baked into policy than it is to law. I'm sure someone out there can check me on this, but based in my 15 years of being in the government, it's more policy driven. It's more our defined way of doing business. To meet due diligence. There are things required by law, such as market research. Sometimes we have to conduct analysis of alternatives, type of document if we're initiating a new venture from an acquisition perspective. But other than that, I think everything else has really been it served the department at a point in time to do business that way. And I'm not sure it's serving us now.

Andrew Camel [00:10:58]:
Yeah. And it's actually maybe a good point to transition into some of the things you're working on on a technology side to sort of better enable your internal customers, those people that need whatever the tank example to more quickly do that. And then also on the vendor side, how you're helping those people better access those opportunities and satisfy the contractor requirements to ultimately provide those much needed services and accelerate the process.

Bonnie Evangelista [00:11:22]:
Yeah. Anything specific?

Andrew Camel [00:11:24]:
Yeah, I mean, why don't we talk about the solutions marketplace? I know that's been a big focus of yours for a while.

Bonnie Evangelista [00:11:28]:
Okay. I would say one of our premier offerings right now in which we're trying to do business differently. So it's very much just like we were talking about our challenge to industry or our ask of industry that is published right now is not asking for a response to a specific requirement like we were just talking about. It's more about, hey, we have some strategic focus areas where we think there could be a lot of improvement. Industry, tell us what solutions you have and what problem you're solving, and you can enter what we call the solutions marketplace. And by the way, you're not submitting a paper, you're submitting a five minute video. So this is framed very much like a venture capitalist pitch. And all we want to know in this video is what is your solution? What problem are you solving? What's the impact if you solve my problem? What's the magnitude? Why should I care if you solve that problem? And why are you different in your market? So how are you differentiate yourself from your competitors? How are you innovative or creative? That in a five minute pitch.

Bonnie Evangelista [00:12:34]:
All videos are assessed by a peer panel. And if you're essentially assessed favorably, or we call it if you have technical merit to enter a DoD marketplace, you're in. And when I say you're in, that doesn't mean you get a contract. All that means is you're determined to be, quote unquote, awardable. And since you're awardable, that gives a government buyer like me a mechanism to purchase directly from you if you have a solution that meets admission. So I've already satisfied that competition thing I talked about where everyone that can be a very lengthy process and I have to compete things. There's been outs of it, but the authorities we're using allow us to do business this way, where we can use a peer panel and we can satisfy competition in this way. So I can do business directly with a vendor who has a solution to my problem, so it can accelerate that contracting process tremendously.

Andrew Camel [00:13:31]:
So, just to be clear, this all happens out of band with the traditional requirements process where you sort of say, look like we're developing a effectively library of options that address certain problems we've stated. And then when somebody says, hey, I have this set of requirements, then they can go to that marketplace and say, or maybe you can even help them with that. Say, look, your requirements actually indicate that the problem you're trying to solve is X. And we have these three or four forefront of innovation vendors we've already identified. Are these of interest? Is that sort of the right way to think about it?

Bonnie Evangelista [00:14:01]:
I would offer a slightly different way of thinking about it before we even get to the requirements process. Somebody could be doing market research in the marketplace and being like, oh man, we were building a requirement for X, but did you know Y is out there that wasn't even on the landscape right of the art of the possible. That's a possible use case or scenario. Another would be is outside of the requirements process. I like to think about the marketplace as a mechanism to enable piloting and experimentation. So even before you get to developing the requirement, maybe you need to run an experiment to help inform what the requirement looks like. And it could be making the requirement better or it allows you to learn before you even get to that part of the process.

Andrew Camel [00:14:48]:
Totally, yeah. Makes a lot of sense. Okay, so basically you're envisioning it as the front door before you even get to the requirement stage.

Bonnie Evangelista [00:14:55]:
Yeah. So a term I hear people latching onto in the department is entry point. It's supposed to be an easy entry point. So for the government buyer, it allows you to engage directly with a vendor on the basis of competition, which is important for government business in a very collaborative and streamlined manner. And then on the industry side, you don't have to have I don't know if this is a term that resonates with you or your audience, but you don't have to have a government wide contract like a GSA schedule or some massive government contract to do business with the government. The mechanism is there. You're in the marketplace, you just have to find that demand signal on the other side.

Andrew Camel [00:15:36]:
Right, but ultimately, are you then required to go through that contracting process once you sort of establish who your partners are going to be?

Bonnie Evangelista [00:15:42]:
You do. Yes.

Andrew Camel [00:15:44]:
Okay, got it.

Bonnie Evangelista [00:15:44]:
It's designed so that the services can implement their own contracts. We didn't want to dictate at the OSD level, the office of Secretary of Tents, what the contract would look like. So we made it flexible enough that, again, you could be doing business directly with that solution provider. They have the thing I need to buy the thing. So we've enabled that. But the services or the components still have to execute their own contracts the way they would normally do it. But we've made it flexible enough that they can execute what we call more traditional procurement contracts, which is leveraging the Federal Acquisition Regulation. Or there's other authorities available to us, something called other transaction authority, where you kind of get to use a different set of rules and it's easier.

Bonnie Evangelista [00:16:27]:
And we've accommodated for both. So we've tried to make it super flexible. So that the optionality on the government buyer side trying to make it easy to say yes.

Andrew Camel [00:16:36]:
And is your vision for this that is the front door, as you said it for all contracting? Or is it really just like, look, there's going to be things like pencils and chairs and air conditioners that you're going to need to buy through some standard flow, but if you need to do something that's slightly more forward leaning, that's when you would come to tradewind. Or this mechanism.

Bonnie Evangelista [00:16:55]:
Yeah, definitely the latter. This is not meant to replace office supplies, janitorial services, like some of these recurring needs or even maybe I don't know if I'll stand by this statement in the future, but jury is still out. How about that? Maybe even, like, major weapons. This isn't really solving out. This is like, I see a commercial solution. This also accommodates early R and D type of zones. I see an idea, and I want to buy it right now because it might be obsolete three months from now. That's the kind of scenario we are trying to address.

Andrew Camel [00:17:30]:
Yeah, okay, awesome. Love it. And now I think it'd be great to transition into how you guys are integrating LLMs to your credit, which is super interesting across variety of areas in the process we've discussed to make this process easier for all parties involved. So do you mind just spending a minute on that? Or probably more than a minute?

Bonnie Evangelista [00:17:47]:
Absolutely. Yeah. I don't know if we're the first. Maybe I'll find out soon if that's true or not, but we're the first that I'm aware of. They're actually applying Generative AI, using LLMs to the contract writing process. This was born out of my former organization. So before I joined the CDAO Chief Digital and Artificial Intelligence Office I was called the Jake. The Joint Artificial Intelligence Center.

Bonnie Evangelista [00:18:11]:
And the Jake was reorganized under CDAO. But when we were just Jake, we were thinking about this problem before OpenAI opened the floodgate with Chat GPT. So somewhere between like nine and ten months before that happened, we were thinking about, like, how do we do this? What is the art of the possible? We wanted to experiment in this space just like we were talking about. So I issued an Open challenge because I don't know the answer, just like we were describing. There's no way I could have written a requirement for this. I have no idea what this would look like. I don't even know if it would work. And so I issued an Open challenge to industry, and I was like, I put kind of a low cost risk bounty maybe to industry.

Bonnie Evangelista [00:18:52]:
And I was like, I'm willing to pay a little money to see what's good out there. Most of the responses I got were people telling me they would build me something. There was one response, though, where somebody said they had also been thinking about this problem on the industry side. So, submitting proposals to the government is also a very lengthy, tedious, very manual, hands on type of process. And so they had already kind of started developing things, and they were willing to kind of extend that to the government. And I thought that was worthwhile, I'm willing to see, to throw a low cost type of bounty over to them to see what they could do. And the first MVP, I think it was around 45 days, that's when I first saw what Generative AI was. And again, this was like nine or ten months before Chat GPT, and they were able to show me how to write a problem statement.

Bonnie Evangelista [00:19:42]:
We were talking earlier about how to define our problems. I was like, can you help me get customers to define their problems? That's a real big that's hard when you've never done that before or can be hard. And I want to make that process easier because that's kind of also our intake process. That's kind of where we start the conversation with our customers. We're like, what do you need to buy? What is your problem? So if we had a collaborative way to do that using a tool like this, I thought that'd be pretty cool. So the MVP was to show me how to do that. And they wrote a descriptive title, and I don't remember what it was. And they were able to generate like, two paragraphs of text.

Bonnie Evangelista [00:20:15]:
And it blew my mind. I was like, this is insane. I'm just the contracts nerd. I didn't even know that was a thing and this was out there. So then it was like, what else can we do? And so we started iterating from that foundation, and we now have this tool, and we call it Acbot, and it helps us craft those problem statements with our customers. And I kid you not, I've done this, I've repeated this multiple times is I can now do what would usually take weeks or months. I can do it in 30 minutes with the right people in the room. It's not like Chat GPT where you query it and it gives you an answer back.

Bonnie Evangelista [00:20:55]:
It's more like I can start in plain language with the customer and say, what do you need? What do you want? Even if you're not sure, just tell me something in plain language. And it helps me start to translate that into something more specific to, again, going back to that framework, current state end state gap. And we're not expecting the bot to give us the answer from the plain language. So we use human reinforcement, like two or three times in the workflow. So at first, the bot will generate like a couple of sentences, and we try to make the need statement really concise. And then the humans have to verify. And if you're doing this right, you've got the primary stakeholders on the call all looking at the language that's generated. You're all verifying and editing the text in real time.

Bonnie Evangelista [00:21:41]:
And then you move to the next part and it starts to generate like three paragraphs. And then you're editing and validating that text. And that's done. It generates like five paragraphs. And now you have that 85, 90% draft of a problem that I could give to industry, and they can tell me if they have solutions to that. And then we build some RPA type of workflows on top of that to get to the point where we're generating what we call a call to industry or a solicitation, and we can publish that and everything. It's been a very fun journey to see how far that's going. And we're continuing to build on top of that.

Bonnie Evangelista [00:22:19]:
We've got a ton of interest from Air Force and Navy right now to build similar workflows for their more traditional procurement workflows. So we're going to see where this goes. But right now we're using it very cool on my team to really get after the stuff we've been talking about when we're not using those traditional workflows and we're going after, like, prototyping or things that are a little bit more nontraditional, we define those problems and we get it out to industry as fast as possible using the tool.

Andrew Camel [00:22:46]:
Amazing. So cool.

Bonnie Evangelista [00:22:47]:
Yeah.

Andrew Camel [00:22:47]:
How did you guys get it to sort of understand the if you have an idea for how the sort of back end works, like understand the contract structure that it's trying to build toward, how do you guide it into the format that's required, I guess is my.

Bonnie Evangelista [00:23:00]:
Question for the contract.

Andrew Camel [00:23:01]:
Yeah, because ultimately you're trying to get to this problem statement or request to industry. I forget the wording you use. I assume there's some sort of set standard for that. Is that the model understands how to take the statement and then map it into the different sections of that document or how does that actually work for.

Bonnie Evangelista [00:23:17]:
The generative AI part? The generative AI currently is only building the problem, and you're right, it already knows where in the document. And this is more on the Solicitation side, not the contract side, where in the Solicitation we want to put that section. And then we have, I'll say templates. So in addition to when you're really telling industry, like, I have a problem and I want you to submit responses to my problem, there's really three basic things you got to do. You got to tell them what is the problem? And then two, you got to provide instructions on how to respond to the Solicitation, and three, maybe some criteria on how you're going to evaluate those responses. We figured out, like, our team has a way of doing business in terms of the instructions and the criteria, all of that. That's where we use robotic process automation. On top of once we've generated the problem with the customer, we have templates and standard language for this is how you're going to submit.

Bonnie Evangelista [00:24:16]:
This is like, some criteria we're going to use. I'm actually hoping the tool will help us maybe generate some criteria, some new criteria that's more applicable depending on what the problem is. That's a future feature, potentially. The generative AI part, again, is really only in that beginning part currently. But we've got some experiments going on where the generative AI part could expand to other workflows or other administrative tasks.

Andrew Camel [00:24:43]:
I could imagine. Yeah. I mean, also, have you done any work yet to figure out if it can actually process instead of expanding on ideas, sort of compressing long documents into the core ideas for when you're getting proposals from industry and you want to.

Bonnie Evangelista [00:24:57]:
Understand what they do, that's actually an interesting idea where we were going with it, at least currently on the roadmap. Like, one of the things probably in the next, I don't know, 60, 90 days we're working on is actually integrating the marketplace into that intake process. So as we're writing, here's my problem. Can the AI start recommending videos for me to watch? So maybe I don't even need to solicit and there's already a solution out there, and so can we integrate the two? I like your idea, though. That's a good one.

Andrew Camel [00:25:30]:
It's funny because if you take it to the end, you sort of imagine a world where you have LLMs writing the contracts, reading the contracts, and it's like, wait, why do we have a little bit like an anti pattern?

Bonnie Evangelista [00:25:44]:
You sound like some of my friends on the Air Force on the intel side. That's exactly what they said. I don't know. I mean, I could be wrong, right? This stuff is moving so fast that I don't think any of us need answers and we just have to do our best to continue to make sure. I do agree with some of those out there who are I'm not doomsay about it, but I do think there is an element of wisdom we have to be exercising before deploying these type of capabilities. I 100% agree with that. Tristan Harris at the center for Humane Technology is really big on this, and some people might take his criticisms of what's happening in AI to be very like, oh, he's don't I don't get that at all. I think he's just asking for people to not be where the sort of.

Andrew Camel [00:26:33]:
Edges are of capabilities. Yeah, he's really been built around that.

Bonnie Evangelista [00:26:37]:
Yeah, he's really big on harnessing technology and not just deploying it blindly. So I think there's some good wisdom there. So it's the same thing for right now. Unless we're proven wrong in the next, who knows, 612 months. And wherever technology takes us, then there is always going to be a human validating, at least currently from a contracting perspective, especially in the government, there's always going to be that human validating what's happening. The AI is never going to make the decision, make the contract. They can make recommendations just like a human would make recommendations. We pay people all the time to support the government, help advise, or manage or write our contracts.

Bonnie Evangelista [00:27:26]:
To me, this is no different to asking AI to do the same thing. But you're always going to have that subject matter expert, someone like me, a practitioner like me, on the other side, to make sure yes, we didn't, I don't know, break the law. Or like you just said, going from proposal to contract, we didn't miss the intent. Or we got our mutual meeting of the minds, all the things.

Andrew Camel [00:27:51]:
Right.

Bonnie Evangelista [00:27:52]:
Kind of how I feel about it.

Andrew Camel [00:27:54]:
Yeah. No, it makes sense. I was going to say one of the things that you're making me think a lot about is this problem in self driving cars that I love to bring up, which is self driving cars are objectively better drivers than humans. But there's a lot of the problem with adoption is that they fail in ways that are different than humans and if they failed in ways that were similar to humans, we'd be actually much more comfortable with it. But the issues are really not coming from the fact that they're worse, they're objectively way better drivers than humans. But it's just we are not comfortable with dealing with how they're failing, which is creating a whole new set of problems. And I wonder if it might end up being similar here where it's like, look, I do imagine a world where as great as you are, the all seeing AI model is probably going to be better at writing contracts at some point.

Bonnie Evangelista [00:28:38]:
It's already better at writing problems, you know what I mean? Like problem statements.

Andrew Camel [00:28:43]:
Right. So the issue is that when are we comfortable with saying, hey, maybe we should be handing this off because actually that AI might be better than me, the human, at writing these contracts or doing whatever. The job is interesting to think about.

Bonnie Evangelista [00:28:56]:
Yeah. I will say if we're making predictions on the government side, that is going to be a way longer adoption than.

Andrew Camel [00:29:04]:
The industry side because of the policies.

Bonnie Evangelista [00:29:07]:
Regulations, et cetera, law. I also say liability. Who does the liability go to? And I think these are questions in the public conversation right now about even self driving cars or yeah, totally. I can't even imagine like in the medical field, it was the AI who said you were going to get this disease, not me. I can't even imagine.

Andrew Camel [00:29:27]:
It's funny I was going to say because if you take this in some ways, it will be solved at the front lines of combat first. Because if the AI is in fact better at shooting a gun, sending a know whatever the task is, there's much more incentive to say look, we are willing to because we are forced to win against our adversaries on that level. We can't compromise on capabilities. But as you walk back further and further and further from that front line where you can really see the impact of having worse capabilities, that's when it becomes slower to change. Because it's a less clear argument that we should be depending on the AI because it's subjectively better when we get all the way back to contracting, even though ultimately it does flow through to winning. At the end of the day, as a country, it's harder to see the ultimate impact and so you're less willing to say, look, objectively the AI is better, we should just go with it, right?

Bonnie Evangelista [00:30:22]:
Yeah. All of this, everything we're talking about, it's funny. We started with contracting but you could really talk about any aspect of our lives, right? All of this is making us uncomfortable in ways maybe we didn't even know we were going to be uncomfortable. And I think I'm not saying get comfortable with the uncomfortable, but the more I think we can be honest about where we have these concerns because that's where we learn, that's where growth is. If we're able to go into that space rather than avoid it or just say I'm not doing like the more we can learn and understand it. And I think this is like my CTO. Dr. Bill Streline, he talks about this, too, about the ability to talk.

Bonnie Evangelista [00:31:03]:
I'mixing up my words because he talked about, you know, the more we can assure the technology and maybe get that trust we're looking for, but just the more I think we can be vulnerable about this is just because it's moving at a pace maybe we're not comfortable. I think there's some legit fear there and the more we can just say that and own it, there's no judgment there. But we're never going to get past what you're talking about unless that comes to the forefront. And that's just Bonnie talking. I feel like we're getting a little bit exactly. Yeah.

Andrew Camel [00:31:41]:
So what maybe last area of questioning, there are going to be a lot of entrepreneurs, vendors listening to this that are they I DoD is very large bit complicated to understand, a little bit slow moving and bureaucratic. But I do have this really interesting thing that I'd love to either help with on this contracting LLM process or something that ultimately will be in the hands of the DoD. Broadly, what advice do you give them about how to best engage with you? What is the first step that they should take to try to add value to this effort and ultimately drive revenue for themselves?

Bonnie Evangelista [00:32:17]:
I will say before you engage with me though, my first piece of advice is get clear on your value proposition. And when I say value proposition, get clear on how you, whatever it is, a software tool or maybe a model like you were talking about LLMs, you have something in this space that is worthwhile and you think will help. Get clear on how that helps the end user, the soldier. And end users aren't always soldiers in the department, but they're the bread and butter of the department. If you're not helping them, I can't help you. Don't get me wrong, I'm an end user, right? If you've got a contracting tool, I want to know about it. But whoever your end user is, get clear on who that is. It's a soldier.

Bonnie Evangelista [00:32:58]:
It's your business analyst. It's anybody who has to do reporting. Figure out who that is. You don't necessarily have to know every functional lane in the department, but figure out who benefits and how your thing makes us win and why it's better than what we have or what we don't have and whatever. So get clear on that first shameless plug, but really get on the marketplace. And if you have that thing and you get on the marketplace, I can now buy from you. And there is still a little bit of a hustle that I haven't solved that problem with the marketplace. You still have to hustle a little bit.

Bonnie Evangelista [00:33:37]:
You have to make your contacts and go to your industry engagements, because, again, once you're clear on who that end user is, that's how you start to figure out maybe, possibly a path into the government, into that revenue path into the government. The marketplace is the easiest way right now for anybody to point at your solution and buy it, I promise you. So that's what I would say.

Andrew Camel [00:33:59]:
Yeah. Well, look, I think that I might add third thing to take away, which is that the fact that you are able to put out this proposal say, I want to go try this LLM thing, or just even say, the problem statement and then be surprised by the LLM, adopt it and make it a real solution in such a short period of time. I think indicates that there are non bureaucratic channels or thoughtful people that really just want to innovate in the organization, in the department. And if you have a solution that, to your point, adds value to the right people, there's a way for that to move forward in a not massive headache way. So I think it's a great message. I think it's awesome.

Bonnie Evangelista [00:34:35]:
Well, thank you. I don't know. Often I wonder sometimes I don't know why I care so much about this, but I think it's worthy. I think it's worthwhile. There are some of us within the department who are trying to move the needle in the right direction, and you just have to find them. And there's not an easy place to do that, but hopefully things like this will get the message out and the right people who should be finding us will find us. And we're going to keep moving the needle as much as we can.

Andrew Camel [00:35:04]:
I love it. Awesome. Bonnie, this was so much fun. Thank you so much for doing it.

Bonnie Evangelista [00:35:08]:
Yeah, thank you. Had a blast.