Bloomberg Audio Studios, Podcasts, Radio News.
Hello and welcome to another episode of the Odd Lots Podcast.
I'm Joeisenthal and I'm Tracy Alloway.
So Tracy recording this March twenty fourth, And of course almost all of our episodes.
Lately have been about the war in Iran.
But what's interesting, or what's a little weird is that just prior to the war, literally days or maybe hours, the biggest story in the world was actually about defense and you know the DoD that's right.
So you are referring to anthrop a. Yeah, and it's disagreement to put it mildly in the Department of War.
Yeah, exactly.
This was the biggest story going right up on the eve of the war in Iran. And of course obviously there was this contract topics technology was used by the Defense Department. So it was not a disagreement about the use of AI per se in war, but the question of the degree to which AI could be used for autonomous weapons systems on their own without a human in the.
Loop and surveillance and surveillance this was another key element.
But you're right, so we've heard this expression autonomous weapons, yeah, pop up more and more especially in recent days. And I've a lot of questions over what exactly that means, because my impression is the US military certainly has been using AI for some time.
Yes, and so we're really talking about.
Degrees here of autonomy, right, And so if you think about an autonomous weapon, I think your mind could go fully terminator and you know there's like a murder robot out there that's making its own decisions on which.
People or places to target.
And then you get levels below that, right where AI is kind of helping humans to come up with strategic decisions.
Right.
So, if there is a missile coming and you have a missile defense system, I don't think you want a human in the loop. Is like, Okay, here are the coordinates X, y Z that we think it's going to hit. At this point in time, we think the missile will be here, Are you cool with firing it?
I think everyone's.
Probably okay with that level of autonomy. But I have a feeling that to your point exactly a lot of this discussion, and maybe it's core to what Anthropic and the Department of Defense who are disagreeing on, I have a feeling a lot of this is going to come down to definitions. My guess is that there is not one shared agreement of this is an autonomous weapon system and this one is.
Not absolutely And of course there are also questions over exactly how places like the Department of Defense, not only how they define it, but once they have those definitions, whether or not certain companies trust them. Yes, so stick to those policies, because the US will say, well, our policy is not to survey are citizens at the moment, So if you're anthropic, you don't need to worry about that.
Clearly anthropic feels otherwiseer say they do. So there are all these really interesting thematic questions that pop up from all.
Of this totally.
And then there's the question of Okay, here's a technology and the government says, we believe that we can use this to make the country safe. For what, You're not going to let us do it like private corporation. There's some very interesting questions about the role of corporate power visa vi the government and so forth.
Anyway, this is something that has.
Become even more timely in the reports, even in the early days of the Iran War, about these AI systems having been used perhaps in target selection, but we don't really know. None of the reporting is like that.
Clear.
I don't think that they're going out and advertising this strike.
This is exactly how we're using AI.
This is exactly how we're using AI, and so forth. But this is obviously a huge debate and war aside. It's totally going to grow, and just as an AI is going to grow, it seems in so many different areas. Anyway, I'm really excited to say, we really do have the perfect guest, someone who's been writing and thinking about this
stuff for a long time. When we talked to an AI expert, I marked the dividing line AND's like, where you talking about AI prior to when Chad Gipt was released, as like, I take a little bit more seriously the people who are in this space prior to November twenty twenty two. Anyway, I'm very excited to say we're gonna
be speaking with Paul Shari. He's the executive vice president at the Center for a New American Security and he's the author of two books related to this, one as the most recent for Battlegrounds Power in the Age of Artificial Intelligence, and then prior to that, he is the author of Army of non autonomous weapons and the future of war. He was previously in the Office of the Secretary of Defense, also a previous Army ranger, so truly
the perfect guest. So, Paul, thank you so much for coming on odlots.
Oh thank you for having me. Very excited to be here.
What do you we start?
I mentioned I had a feeling that maybe the definition of an autonomous weapon is a contested one. But if I say to you, what's an autonomous weapon, what's an autonom this weapon?
So I think you're right from the beginning that there is not a unified definition that everyone agrees on. The Defense Department has their definition that's written in their policy. I think conceptually, I think the distinction really is a weapon that is choosing its own targets on the battlefield, and it's not where we are today. Right now, today people are choosing those targets. But it is kind of a spectrum because we do have examples of weapons that
have some measure of autonomy. A good analogy might be self driving cars, where conceptually, like, okay, a self driving car would be where the AI is driving the car, but then you get into an actual car today and a lot of them have intelligence cruise control, automatic breaking, automated self parking, They have all these like automated features that are kind of creeping you in this direction where the AI is taking over more and more control for
what the vehicle can do. And it's actually pretty similar thing in the military space as well.
So Joe mentioned that had we been having this conversation even a month ago now, it probably would have had fewer concrete examples of AI enabled weaponry, let's say, or strategy. When the Pentagon talks about its advanced AI tools that it's deploying for the Iran conflict, what are some examples that you're seeing right now that are different to say, maybe just a year ago when we had another Iran conflict.
Right, So, there's a couple ways in which the Pentagon is using AI right now. One is narrow AI systems that have around for over a decade now that do image classification. For example, So this was the military's original project Maven almost a decade ago, where they took machine learning image classifiers to sift through drone video feeds and saddle images to identify objects. Okay, here's a building, here's
a person, here's a vehicle, that's pretty mature technology. Now, what's come out in just the last couple of weeks that's really quite interesting is that in the midst of this huge, messy public breakup between Anthropic and the Pentagon, we found out that in fact, Anthropics AI tools are being used by the US military to help plan the
war against Iran. That's obviously a different kind of AI tool, AI Large language models, AI being used to write code AI agents, and that's being used in a different ways, really helping intel analysts sift through just the massive amounts of data that the US military has. And so you can imagine the problem that the military is facing right now when they're looking at targets in Iran. US military has flown over six thousand sorties against Iran. The Iranian
military architecture is degraded in a lot of ways. US militaries are bombed a lot of targets. There are mobile targets, senior Iranian commanders, mobile missile launchers and air defense systems and drone launchers. US militter has got to bring off it in f together and find out where are these targets right now and where is there an aircraft that is the right bombs on it to take these targets out. And that's how AI is being used to help basically process and understand all that information.
When I think about the description that you gave for that, I sometimes think like, could it be that?
Now?
I don't think that using anthropics technology mean so that they go into Claude dot AI and say, give us a list of suitable targets for sorties. But could it be something like that? But I'm sure there's a different interface and so forth. But is that a completely ridiculous way of essentially framing the service that AI is providing right now?
So the way that these AI tools are being integrated are through a existing system called the Maven Smart System okay, which is built by Palenteer, and that fuses all this data together. So you basically have an existing architecture for data management for Intel analysts that the Milita already has that brings together all these different forms of data. You might have satellite imagery, geolocation, data, signals, intelligence, other forms
of information. That's pretty great for Intel analysts, but that's also really unwieldy because how does a human understand all that data and process it? And that's where the large language model tools, whether it's Clawed or other companies can be valuable. Is that could be a way for a human to interact with that data, to basically task a large language model to say, Okay, here's a bunch of data I'm giving you. I want you to look for
intersections and things, right. I want you to look for a place where we have satellite imagery and some other forms of intelligence that can help identify the location of you know, some missile launcher for example, and then humans can look at that and help one just like find where are all these targets, and then it helpfully planning too. A human could say, okay, here's this list of potential targets that I have. Now they're scattered all over run
around is a really big country. I want to map these two locations for US aircraft at different bases across the region. What are available aircraft and what are available munitions on those aircraft? The commused to take out those targets to help build a strike package. So like, the AI is definitely being used to help understand the battlespace and to plan operations. But in I would say ways that are pretty narrowly directed by people It's not quite as simple as like, dump all this data into a
context winder for LM and say, oh, AI figure it out. Okay, people are asking the AI some really specific questions, So.
I'm thinking how to phrase this question diplomatically, But I get that the difference between fully autonomous weapons is, you know, the human as a decision maker in the current setup, how meaningful is the human actually? Like, what's your sense of it? Because I'm imagining if you're an intelligence officer and you're getting realms and reams data from Iran and you ask the AI to pick out certain patterns or identify potential strategic targets, how much due diligence are you
actually doing on what that model spits out. Because of course, the tendency when a lot of people use LLM certainly is just you accept what it shows you on the screen.
And just to add on to Tracy's question, because there's on to go, which is then in the early.
Days of the war, we hit that school.
Yeah, and I'm reading a New York Times report and that was the result of quote outdated data provided by the Defense Intelligence Agency. Now we don't know exactly what that means, but Okay, various outputs come out, then what happens, like how much is the human layer currently in terms of Okay, here are targets, here are ships that are
on a battleship. This could be plausible. What do you think or what do you know about the level of human decision making that happens between some output and then the ultimate call for a strike on whatever it is.
Yeah, I mean, I think, first of all, I think it's a really important question because it is one of the possible failure modes, if you will, of AI and how we use it, because you can end up in a place where humans are nominally in the loop and you could say, well, it's not an autonous weapon humans making these decisions. But if the human is not meaningfully engaged and they're just kind of rubber stamping some kind of decision, and that's not really what we're looking for.
So I think that's it's been a longstanding concern for many years about people worried about autonous weapons. I think that's a very real risk with how AI is used now. Based on my understanding of how the AI technology is used in Maven today and based on what I've seen of demonstrations of it, because I have seen some demonstrations
of this in action. I think humans are pretty involved right now in terms of actually looking at the output from AI, giving pretty specific guidance to the AI systems.
I do think there is an underlying challenge that the strike on the school highlights, which is, when you're talking about thousands and thousands of targets, what's the degree of vetting that's gone into all of that information, both in the run up to the war, which in this case, that school was a fixed object, and so that's likely something that should have clearly been much more vetted prior
to the we're kicking off. That someone could have identified that that building that was struck had actually been at one point in time part of an Iranian military compound. But you could see based on public available satellite imagery that it had been moved out of that compound some time ago and it had been converted to a school, and it would appear, based on what's been reported in the Times, that that information had never been updated in
this DEIA targeting database. Now, I would hope that we'll get more information in the future and some investigation about exactly where that went wrong. But I think that does speak to this underlying challenge of how good is the data going into this AI system and how thoroughly are people vetting it? And again, in principle, AI might be able to help you with those things, but you got to use it the right way and people still have to be meaningfully engaging THECC.
Why don't we back up for a second, tell us about the work that you've done in this area, really several years ahead of the curve, and talk about this stuff, planning for this stuff. Give us a little bit of sort of your background and what got you on this train again several years before Chad Shept.
Yeah, so really over a decade ago now, around say twenty eleven, I let an effort inside the Pentagon when I worked at the Office the Sectary of Defense on developing the Pentagon's policy on the role of a ton I meet in weapons, the one that's still in effect today in fact, and that was really part of at the time. We weren't at all where the military is
now in terms of integrating AI tools. I mean, these types of large language models just did exist at the time, but the military had kind of woken up to what I would call this accidental robotics revolution during the wars in the Rock and Afghanistan, where the military had deployed thousands of air and ground robots, drones in the air and ground robots for diffusing bombs, and the military was starting to think through where is this going in the future,
And one of the things that everyone could see would be valuable would be having more autonomy in these systems, the ability to not be tortally reliant on a human remotely controlling them, which was really the case at the time, but that raised all these obviously the any questions about like well, how much autonomy should they have and what are the legal and ethical implications of that, And that was actually the topic of a lot of discussion among people in the military at the time and in the
Pentagon for people working on these issues, and so that ultimately led to that policy directive that's still in place on the role of autonomy and weapons. And then when I left the government, I continued to work on this topic. Is we've seen discussions internationally through the United Nations as we've seen the technology evolve in really amazing ways but also ones out of risks with artificial intelligence.
So when you were doing that job.
I get that you're on the policy side, but did you ever see anything on the contractor side similar to what we're seeing with Anthropic right now. Was there ever a contractor who said, actually, no, I'm really uncomfortable with the way that the department wants to use this particular tech.
Not at that time. Now, a few years later, after the US military launched Project mavin there was a big dust up when it came out publicly that Google had been a part of Product Maven, and a number of Google employees signed and open letter protesting and Google eventually
discontinue their work on Project Mavane. And you know, it's not exact replica here of what's going on, but there's certainly some similarities in terms of a disconnect between how some people in the AI community are thinking about how their technology ought to be used in war and how the military is thinking about it. And I think part of that's like there's this underlying challenge of AI is really different than a lot of traditional military technologies because
it's coming out of the commercial sector. So in a way, it's kind of like the opposite of stealth technology that was invented in secret defense labs and doesn't have a lot of commercial applications. AI is all of these different applications. It's not being invented by the military. The military is having to be import it in and there are a lot of debates about how AI should be used, you know, in the military and more broadly in society.
Actually, on that note, I think this is really interesting and definitely a pivotal point in I guess, the history of the military industrial complex. But why can't the US government, with all its resources, actually develop AI in house and just avoid the seeming complication of having to deal with a commercial enterprise.
Part ofly, it doesn't have the technical skills the AI scientists and engineers are. Really there's a fierce competition for talent in the AI space, and so the military just like can't buy that talent they don't have it. And the government spends a lot of money, hundreds of billions of dollars annually on defense. But we've seen actually in the last few years that private enterprise is able to mobile massive amounts of capital towards building data centers, to
training AI models. And partly because the commercial applications for this technology are much bigger than the defense applications, and so for a lot of these tech companies there's some at least maybe not in this particular instance, but in the past there could offer be some prestige associated with saying, oh, you know, the Air Force uses our AI system or the Navy users our technology, but the defense sector is
actually kind of small for them as a customer. I mean, the dollar amount that's been talked about publicly for the Anthropic contract, it's two hundred million dollars. That's not a lot of money for these AI companies. Yeah, and so I think that actually we've seen the defense sectors struggle to just keep pace with the amount of investment that's needed in the space.
Cherioucy, I think it's a good question. And then you remember, well, the government couldn't build a good healthcare website to sign up.
For health insurance.
And I hate to bring that up because it's old, but it's true. Right, So it's like, are they gonna build a world class LLM or can a government build a good employment insurance website. We've done multiple episodes the answer continues to be not the case. I do find it fascinating. However, your point about there is this novelty. It is impossible to imagine and say Lockheed Martin inventing get technology and they's like, no, you can't use it.
Because Lockheed Martin's entire raise on death Right is building.
Technolog for the government.
It is inconceivable what that would be. But it is sort of novel when you're getting these defend technologies. And you know the Google was also an example of Google obviously had technology that did not originally serve a purpose of defense. We saw the we remember the employee revolt. Let's talk more about that disagreement though, between Anthropic and the Department of Defense. In your mind, where does Peter
heg Seth want to go with this technology? And is that deviate from some of the policies and the directives that you were working on when you were when you were working on this stuff.
So it's kind of crazy about this whole dispute is particularly an issue of autonomous weapons. Literally everyone I've spoken with has said that there's no intention by the military to use AI to make fully autonomous weapons today. Okay, I think anybody that's actually worked with a large language model with any kind of chat bought, whether it's Claude or Gemini or chat Chie knows that if you use
these to write an email, you need to double check it. Like, in no way, shape or form are they reliable enough to make life and death decisions. I don't think the
military actually wants to do that. What's at dispute here is a more fundamental disagreement about well, who sets the rules and so the origins of this really was that when the Pentagon came out with a new strategy for AI in January, one of the things in their strategy was that going forward, they wanted their contracts with AI companies to allow the military to use their AD tools for any lawful use. Basically, look, anything that's legal, we
want the ability to do it. And that has conflicted with how a lot of these tech companies have been thinking about their AI tools. They're very nervous, many of these companies about harms from AI. They're conscious of these risks, and so a lot of them have various use policies in place. You can't use AI, you know, to launch offensive cyber attacks for example. That's kind of thing that
actually like the government might want to do. And so this is a This was really the rub with the government was like who sets the rules rather than necessarily like a near term question of fully autonomous weapons.
So what we've already seen is Anthropic has this disagreement with the government, and then OpenAI steps in and raises its hand and says, okay, Anthropic doesn't want to do it, We'll do it happily. Does this just leave us in a situation where it's sort of a race to the bottom, Right, It's like the lab with maybe the least amount of safety concern or the least amount of reputational concern is able to do this, and so we still wind up in a situation where the government is using AI.
Well, I think what's unfortunate here is that when you think about would be optimal for the government one, I think it would be ideal for the government to have access to this technology and to have access to all of the best in class models available, because they are good at slightly different things sometimes, and it's much healthier for the government to have access to a number of different providers so that there is healthy competition in the market.
You don't get locked in with one vendor. But also, if the AI scientists are saying, hey, it's not reliable for this, you to listen like that seems like a thing you want to hear them like out about right, And so I think like in order to use AI and ways that actually are effective for the US military, we've got to have a healthy dialogue between the AI community and people in the military profession. But what the
technology can and cannot do. And I think it's unfortunate that we've seen that dialogue break down in such a dramatic way over the speed.
Just going back to the idea, who actually makes the rules you mentioned earlier that you know you can't use claude to hack into to illegally hack into a system. Supposedly it is unable to do that. It has like a kill switch within itself that prevents it from doing that. If you're anthropic, could you not just hardcode some of these systems and say you're not going to be able to be used for domestic surveillance of Americans or for war crimes.
So yeah, So this is where it gets a little more technical. It has to do with some of the ways in which the companies may be providing their technology to the government. So there's a couple of different ways in which an AI company can put safeguards in place to make sure that their models not being abused. One is training the model itself to refuse certain requests. So if you ask the model to do something, it's just going to say, like, I'm not going to do that.
That's not consistent with the guidance that I've been given, and the model has been trained to do that response. Another way is that the company can put classifiers on the input and or the output of a model, where the model might give you an answer, but then there's like another aisystem that's checking that answer or checking what you ask of it and saying, well, that's not acceptable.
And then a third and I've want into that actually myself in my own research because the nature of the things that I work on a security things, and I've had situations where I asked Claude help me understand this issue. Claud actually generates response and then it gets deleted. It's
really interesting to see. And then the other way an anthropic has actually talked about this in response to countering some use of Claude by Chinese hackers who are using it for cyber attacks, is the company monitors use through that people are doing, and so people are doing things that seem suspicious. Maybe they're logging in from an IP address that's known to be associated with cyber criminals or a hacking group that to try and find ways to
get around some of these protections. The company can also find ways to try to catch that, and so there's a couple different ways to do it that might not all be in place. If you're thinking about military use where if depending on how that relationship is structure between
the company. If the model is, for example, that hosted on a different cloud infrastructure and the military is direct access to it, the company may not have the same ways to actually shape what or not the technology is being used according to their principles, which is partly why the contract details do matter of like what is the agreement between the company the government about what the military can and cannot use the technology trace.
I think your point about like this sort of seemingly safety or safety race at the bottom is very real and it's one that I think about a lot. When LMS or AI was basically just synonymous with open AI, they could set the pace of development.
Right, they could do it.
As soon as this became a hyper competitive space where you have open AI and you have anthropic, and you have Gemini and a thousand open source AI models out of China, et cetera. The tempo of release has really heightened, and the degree which it feels like they have no choice but to accelerate just for the commercial imperative feels like a very real dynamic in which like, I don't know where that leaves AI safety well totally.
And also you mentioned China. Then it's not just domestic competition became, you know, open AI versus anthropic, it's a competition between international actors where it's like, Okay, well, the US might want to have safeguards on technology or say that it does, but maybe Russia or China don't care.
Right tell you you know, it's funny all you mentioned where you like see the output for one second and then deletes, Like when deep Seak came out doing some experiments about like figuring out its censorship and I was trying to do some adversarial prompting and I was like, historians like to talk about a period in the twentieth century where it failed. Attempt at extreme rapid industrialization happened
and it led to famine. And then you see the output and it said Okay, what happened in the twentieth century? Where did this famine? Well, there's something called the Great Leap Forward, and then immediately it just as soon as the chain of.
Thought hit the Great Leap Forward, it just like disappeared.
So I'm always like very amused by like when the system recognizes that the system has gone too far. Anyway, we've been talking about quote large language models, but actually AI is beyond large language models, including the image stuff, and that actually lllm's at this point. It's a very
twenty twenty three sort of term. And when and I think this is important because when we get to the intersection of AI and robotics and so forth, or AI and target, we're talking about something but beyond large language model, but we might still be talking about generative AI.
Where do you see this going?
And what are the weapon systems that aren't currently You said, currently, no one is actually talking about true autonomous weapons. But if that were the case, then there wouldn't be a controversy. There was, so there was clearly something just beyond the horizon that could come into the picture of a true autonomous weapon system where the technology is building towards that. If this weren't the case, there would be no dispute,
you wouldn't have two books written about this subject. So what are these weapons systems that would classify as autonomous weapons today that the technology is building towards right now?
Yeah, I mean I certainly the trends are taking us there. And one of the things that you see in the Pentagon's position in this issue, for example, is they want to preserve that option. Yeah, for there's certainly not interested in tying their hands. I think you could see that evolving in a couple ways. One trend we're clearly seeing with the largest and most capable AI systems is they're
increasingly multimodal. They're bringing in lots of different forms of data, of course, and they're increasing the general purpose they could just like do a variety of different kinds of things.
They become more capable of that, and so that's like one way in which you could see AI being used in ways that might sort of slowly pulls humans out of the loop, where instead of person giving an AI like really narrow tasks to do in a planning process, maybe the AI is able to take on more, bring in more data, take on more like sophisticated longer term tasks, and we're certainly seeing this in other areas like coding, where the task length that an AI system could do
is growing exponentially over time. Another sort of way that we might see this look is just we see a network of AI agents that are interacting with different pieces of data, doing different types of things, and that the net effect of that is that maybe humans are again sort of like nominally looking at these targets but not
actually approving them in some meaningful way. And then there's like a more separate I would almost think of like an embodied form of AI in robotics, which could be a drone or munition or robotic system that has some kind of onboard autonomy that might be partly a distilled model so that it can be operating at the edge on lower computing on this actual munition or drone, or it might be some hybrid system that has partly machine learning but also just a lot of hand coded code.
It's more like expert level system that's going out into the battle space and hunting targets directly and attacking them. So something kind of like the low cost drones that we're seeing Iran launch, but once the can loiter and identify targets that attack them.
Whether that doesn't exist today. We don't have drones loiter that are just hanging out there. And then when something flat, then there's a system is like this looks like a target attacks That actually doesn't exist currently as far as you.
Know, well, I mean, they're not in widespread use. So there have been some narrow examples, I would say historically dating back to the eighties, in fact of lording munitions that could search over wider and would queue off of radars, and so radars are what the military would call a cooperative target that when they're emitting in the electromaenetic spectrum, if you know the signature of the radar you're looking for, you could see it. You can just home in on
that radar. Now, if they turn off, that's different than kidden and they're harder to find. But there have been some examples. A system that the US Navy had in the eighties called the Tomahawk anti shit missile, not actually the same Tomahawk cruise missile that the military is using now, a different one that was designed to fly a search pattern and hunt Soviet chips. There was an Israeli system called the Harpy drone that was designed to go after
radars that would loorder for a period of time. But these lording musicians have never really been in widespread use by militaries.
We got to invent one of those, like high pitched alarms to deter the loitering drones from hanging out outside targets. I guess I mean we have electrical jammings.
Yeah, that does exist.
Okay, So when I think about as we move towards more autonomous weaponry, I think about bots basically interacting with bots at that point, and then I think back to previous examples of bots interacting with bots, and there are numerous ones where things tend to go off the rails.
They just start debating the meaning of life, right or.
They start talking in like a language that no one understands accept them.
Stuff like that.
Does the possibility of undesired escalation go up the more we move towards fully autonomous weaponry.
I think that is a very serious risk. And so the mental model that I have for this are things like flash crashes that we've seen in financial markets due to the interactions of different algorithms that are executing trades where you get these emergent properties of how the algorithms might interact in the market, and it's a competitive environment. Companies aren't going to share the details of what their
algorithms are doing, and you can do strange behaviors. Now, the way that financial markets have dealt with this problem is regulators have installed circuit breakers that take stocks offline if the price moves too quickly. There's no referee to call time out in war, and so I think that's like, particularly in cyberspace, one can envision a future where that is a risk, where things are happening at machine speed and you have autonomous offensive cyber operations. You need to
defend against that. You need some measure of autonomy on the defensive side to defend it. Machines space need and you could get situations where you get weird interactions that might escalate the conflict, or it could also happen between
drones interacting in some kind of crisis situation. Now, a situation where like there's a big shooting war in the way people are already attacking, there might be less of a concern, although you still could worry about escalation geographically against bringing new countries into a conflict, or maybe attacking really sensitive sites that are tied to nuclear command and
control that you'd rather not go after. So I think that's a very real risk when we think about how this technology might be employed going forward.
What about AI in really difficult ethical questions strikes where we know that civilians, for example, are going to be killed, which that happens all the time in war and presumably tries to be minimized, but war planners will find some level of acceptable they called collateral damage. Is AI playing a role or do you expect it to play a role in some of these strikes?
That may be great areas.
I think you could envision in ways that a I would be used that would make warfare more precise and more human and ethical, and ways that it could be
used that would not and what can be opposite. So, for example, if you had an AI system that could look over all this targeting data and then identify if a strike is within a certain distance using you know, musis of a certain size of protected targets, whether it's schools or hospitals or critical civilian infrastructure, and say hey, well, like warning here, you should not carry out the strike or it's a higher level of approval, or maybe you
should use small or more precise munitions. That would be a really beneficial use of AI, particularly when you're talking about a military campaign that hits a lot of targets in a short period of time, That could be really
valuable and may reduce civilian casualties. You know, the risk of all of this is you could end up in a world where humans are just less engaged in this process, right, and so there's both mistakes that humans miss or humans just don't feel as morally responsible, which I think is like a really tricky thing to think about morally because on the one hand, as a democratic society we make a decision as a nation to go to war. It's a very small number of people that have to carry
that burden. And if someone if you could say, well, look, what's the benefit to someone having like PTSD years after a conflict that they're haunted mess having happened, that doesn't seem great. Maybe we could reduce that. On the other hand, if we thought of war and nobody felt morally responsible for the killing that occurred, that doesn't seem good either, and that could lead to more suffering and civilian casualties
and war. So I think that's certainly a concern we think about how to use the technology.
Yeah, this is very enders game coded, right, where you have someone who's basically playing like a video game and wiping out entire civilizations and they think it's just a video game, just an exercise, but it turns out it's actual warfare. And we're seeing some degree of that in the way that the Department of War is portraying this conflict so far. It's very video game, yes.
Especially in public presentation.
Yeah, literal animated gifts of video games exactly.
So, Paul, you mentioned something you mentioned the word circuit breaker, and circuit breakers are nice things to have in markets. I think they'd be even nicer things to have in armed conflict and war. Is there any possibility that you could design something like that for a major conflict.
I think it's possible, like at a technical level, to figure out how you would do that and where you put protections on your side in the military, and what you would do with even maybe cooperatively with an enemy. The challenge is how do you avoid what we were talking about earlier, a race to the bottom on safety. Yeah, and right, we're seeing this in the private sector between the AA companies as the Russian to get products out
the market. I think it's especially hard in the military space, where countries are investing in the military because they're worried about what some other adversary might do and then want to get a leg up on them. And so it's not that cooperation in the midst of conflict never happens. It does, and countries have agreed to take certain weapons off the table chemical and biological weapons, for example. It doesn't mean that they're never used, but most civilized countries
that were not going to use them. But those examples are pretty rare and it's pretty hard to do, and some think that dynamic is the really challenging one. It's like, how do you find ways to cooperate with your enemies to avoid some of the biggest dangers here.
So I think there's the last question for me. You know, you mentioned that drones are kind of robot and there are other robots that have been existence in either national security or police work for a while. I think there are robots on the subway sometimes that seem to be really yeah, but I don't think they really.
See the robots at the grocery store and they end up like chasing me while I'm trying to buy like rare or something. Yeah, the ones that sweep the floors and stuff.
Yeah, there's the robots.
Yeah, I think there is a Eric Adams did a contract with some company that was doing like subway for cards or something like that. But these are really different ai as we talk about it in robots are two different technological trees, but they are going to merge, and there's the possibility of their ultimate merger. Do you foresee a world in which essentially we don't have human soldiers and wars are fought with who has the most advanced autonomous robots? We know China is investing a lot in
humanoid robotics. Do you foresee a world in which that is the nature of a ground invasion as you and it happens with robots or various other sorts, talk to us about how far that could go.
Yeah, So, I mean, look, I think where we see robots whose more and were for absolutely the long arc of technology in war, from the first time someone picked up a rocket through it at somebody else has been towards great distance between adversaries moving up through bows and arrows and rifles and intercontinental ballistic missiles. And I think robotics will be the next evolution of this trend of finding ways to find the enemy strike the enemy without
putting yourself at risk. And there's certainly a role for robotics out on the battlefield. I think a vision of like future wars of just robots fighting robots, there's no humans in wealth. That's not realistic for a couple of reasons. One is I think militaries are going to need people relatively forward deployed to execute command and control for robotic systems.
The US military right now can fly drones remotely from the United States in a relatively uncontested environment against more sophisticated adversaries who could jam your communications like and we see, for example, like a lot of jamming on the front lines in Ukraine. That's one of the ways you should go after these drones. Then you need people close by because it is easier to have shorter range protected communications. When you go to a longer distances, that's just hard
to do. So I think given be people relatively close for that reason. I think if you want to control territory, you have to put people there eventually to get out of a vehicle and walk around and control it. But I think the other reason is maybe a little dark, which I think realistically, in order for wars to end, there will have to be some human price that's paid.
I think that's an unfortunate reality that if it's just machines that are being destroyed, that we may not get to the place where one side or the other is willing to suit for peace. And I think, unfortunately, war is likely to involve people and human costs for a very long time.
I have one more question as well, and I guess it's a thought experiment, but if we think back to sort of pivotal moments in military history and their intersection with technology, one of them that comes up is the Russian officer who decided not to press the button in response to the US and thereby, you know, supposedly save the world from nuclear disaster. Happen in a fully autonomous military environment nowadays, I.
Mean today, it would still happen because there's people involved. Right, So this incident standis la Petrov, He's sitting at a terminal and it gets this warning that there's a ballistic missile launched from the United states against the Soviet Union, and then another missile, another five missiles coming in. And the thing that's interesting about this is when Petrov talked about it afterwards, and we could hear what he said because we all lived because he made the right decision.
Here is he talked about how he said he had a funny feeling in his gut and that he knew that the Russian system, the Russians had just deployed, or said the Soviets rather just deployed a new satellite based early warning system to detect us ICBM launches. That it was new, and he knew that a lot of the Soviet technology doesn't work that great at first, so he was skeptical of it. Turns out it was in fact faulty.
He was detecting the reflection of sunlight off the top of clouds and the system was identifying that as a missile launch, and that's what it was. And he went and then called the early warning radar stations and staid, are you seeing his missiles come over the horizon and said, no, there's not missiles. So he reported up the chain that the system was malfunctioning. I think the scary question here is like if that was an AI, what would the AI have done? And it's kind of like whatever it
was programmed to wherever it was trained to do. And obviously we're seeing more general purpose AI systems like large language models, have the ability to bring together more information, to understand better context, to have just like a more contextual understanding of the questions that you're asking of it. But it still doesn't know the stakes of a conference. It still doesn't know, like in some visceral level, what
the consequences are. And so I think that's a strong compelling reason why we need to have humans involved these decisions. Even as the AI becomes more capable, there's still gonna be things we want humans to do because humans understand why it matters.
I started the conversation by mentioning that you know, it's not very common traversial to say, have an anti missile system fire a missile when there's one coming in, but that could be wrong, and you want to make sure that it is in fact a missile and not a
civilian air jet or something like that. Be So even there where it seems like a canonical example, if you just want to have the missile system go off, you would want to have human safeguards and human oversight and human understanding of the system such that it is in fact shooting down missile anyway, Paul Sharie, fascinating conversation. Really appreciate you coming out on the up lots and talking about your work.
Thank you. Really joyed the discussion.
Thanks so much, Paul. That was depressing and fascinating.
Yeah at the time, Yeah, no, it was great. It was Thank you so much.
Yeah, Facebook cover.
I kind of get.
Choked up at the end thinking about that decision that saved humanity at the end, And.
It's actually it's a crazy story.
It's a crazy story. It's one of those stories that like, why doesn't everybody know.
That that person's name?
I mean, when you think about how many people, I couldn't remember it either.
But shout out to another podcast if you want to learn more about this. Dan Carlin's Hardcore History has at least one, possibly two episodes on narrow aversions of nuclear disaster. So very good to listen to, if not terrifying.
You know, there's another point in that exact story that I think is really interesting and it's something I've been thinking about a lot across AI because There's something similar about humans and AI, which is that there is definitely a gap between what we know and what we can articulate. And this is certainly true with AI. Right, so the bot makes some decision, or it determines something, it does not mean it's going to be able to spit out in words, how it arrived at that decision.
But that's true for humans as well.
And so the idea that like, okay, maybe we do get funny feelings about or take you know, like, again, we're still pretty good at determining the difference between AI generated texts and human generated text can we but often I mean we're still like often could get it right, but could we write down exactly what we saw that we understood? There is that gap, and when we're talking about life or death decisions being made, it is scary to think about that role of instinct that we can't
articulate having been taken out of the decision. Well.
I think also technology is very good at pattern recognition, right and responding to patterns and preset paths. It's been programmed to do certain things, and I think in a war environment, that's one of the most uncertain environments that you can possibly imagine, and so you have to think that there should be some element of flexibility in your response, But I don't know how you actually encode that into a thing that like runs on rigid numbers and lines
lines of code. The other thing I was thinking was the anthropic situation and just how new that is from a sort of military history perspective, in the sense that here we have this really important pivotal piece of technology that hasn't come out of like actual military demand. Right
to Paul's point, it's a commercial product. Its commercial uses are arguably a lot more profitable than its military ones, and so seeing that now interact with the Pentagon and the Department of War really interesting.
It's been flipped right.
Yeah.
The closest example actually that comes to mind, there is one example that's in fairly recent history, and it's starlink oh yeah co And of course that was developed for commercial internet purposes, but it played a role in Ukraine and so forth, and at one point, if I recall, there was a tension point about the degree to which the Ukrainians could use and so I do think that is sort of an interesting parallel here.
The other thing that we didn't.
Get to this and I this is gonna be a little cynical, but I think it's right, which is that there's another element I believe to the Anthropic situation, which is, like Anthropic is the last.
Big lib tech company.
We're perceived as such, right, and we know that there's been this fairly sort of right word turn in Silicon Valley over the years. And I don't think like Anthropic is like totally part of that. I think there's still sort of lib coded. I also think it's incidentally why a bunch of people who probably like work in media are like end up using Claude even though they're all kind of the same. Like, I do think there's something there and that they have this thing they say, we're not going to ever.
Have ads, and we know that.
Like Andreas and Horowitz, Mark Andresen just talked about ads are good. Ads democratize the internet. Ads enabled the Internet to be spread to everyone. There are some other politics at play. Because again, like from my understanding, and it would take a lawyer, it's like, I don't think that the agreement that open as Ey sign was that different than what the agreement that Anthropic had. There is probably a little bit of difference. I just think there's some
other politics and play here. Per to Paul's point, maybe nobody is talking about currently autonomous weapons right now fully autonomous, but it can't be long, and I think this is going to be a real tension.
Sooner resan later.
Can I say one thing and I'm going to be slightly facetious, but also not Can you be slightly facetious.
I'm going to be.
Facetious, which is I have a solution to modern warfare.
Don't do it?
Okay, but no for real?
Yeah, if we're just gonna have bots fighting points and it's going to cost a lot of money and result in people's deaths, every country should have to build its biggest, best, most technologically advanced robot and just have them fight it
out gladiatorial. And my twist is to Paul's point about war always having to be painful in some way, Everyone in that particular society has to be engaged and dedicate some amount of time or money to building that particular robot, and you just have to iterate on the robot forever until you feel comfortable to have them fight, and that way everyone shares the pain, but without the loss of human life. Am I high.
I don't think so well.
I think you should write a book. No, I don't think you should write a sci fi book.
All right, shall we leave it there.
Let's leave it there.
This has been another episode of the Authoughts podcast.
I'm Tracy Alloway. You can follow me at Tracy Alloway.
And I'm Joe Wisenthal. You can follow me at the Stalwart. Follow our guest Paul Shari. He's at Paul Underscore Shari. Follow our producers Carmen Rodriguez at Carmen armat dash, Ol Bennett at Dashbot.
And kill Brooks at Kelbrooks.
For more Odd Lots content, go to Bloomberg dot com slash odd Lots, where the daily newsletter and all of our episodes, and you can chaut about all of these topics twenty four to seven in our discord Discord dot gg slash onlines.
And if you enjoy all thoughts, if you like it when we talk about the future of autonomous weapons, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad free.
All you need to do is find the Bloomberg.
Channel on Apple Podcasts and follow the instructions there.
Thanks for listening.