So Crean and I will tell you about some of the work going on at the Centre for the Governance CBI, which is mentioned, is that the Future of Humanity Institute and the philosophy of the University of Oxford, we can begin by just asking What is this term governance? So that sort of two key terms ethics and governance are often used and it's valuable to try and reflect on one of these.
And one definition notion of governance is a very descriptive definition, says governance is really the processes by which decisions are made and things like shape. This notion of governance are norms, policies, institutions, laws, but also things like technology code, infrastructure, right?
All of this shapes how decisions are made. Now, to motivate the importance of governance, I think it's useful to quote what we might think of as a theory of how A.I. governance could work from Vladimir Putin. So the quote because whoever leads in the eye will rule the world. This is a theory of governance, if you will, because it tells you how I could change the way in which decisions are made, in particular, it's a stark theory of governance.
It says I will concentrate power in whomever listening in this quote, I received a lot of media attention throughout the world, but also in national security conversations and hallways, because I think it reflects a fear many have that I could be displacing in power in world order. And also, of course, it's resonating here with our concerns about the role that Russia plays in the world. Now it's worth remarking that when Putin made the statement, he was not staring ominously into the camera.
In fact, he was hosting a middle school science fair televised where he was broadly just encouraging. These are children, these these young budding scientists and their various projects. And this one was a remark in relation to a robotics project. He had a lot of cooperate in the nice things to say about AI, but this was the quote that was really pulled out of context and echoed around the world.
And I think this probably illustrates the challenge we have having a mature, calm global conversation about AI governance because of the fears, because of the ways in which claims can be amplified and taken out of context. So our job will not be easy. Now this motivates a normative notion of governance, which is that we don't just care about how decisions are made. We also want those decisions to be made in a good way where good means something like effective, legitimate, inclusive, adaptive.
OK, so at the Centre for the Governance of AI, we are both interested in how decisions are made and how technology and institutions and so forth shape those decisions, but also in how we can get to a good set of governance structures when we think about the governance of the AI. There's a narrow definition which focuses on specific systems algorithms deployed in a particular domain, be a criminal justice, making loans and so forth, or things like autonomous vehicles, robotics.
There's also a narrow interpretation of ethics, the ethics of a particular system being deployed. There's also a broad interpretation of the governance and ethics of AI, which says I could have a wide range of impacts on things that matter to us, and we need to think about how we can manage those impacts. So impacts such as labour, displacement, challenges to democracy and social epistemic, how as a community, we make decisions and strategic instability, which means nuclear instability.
So the risks to the risks of nuclear war, in short. Now, a question I had for myself is how does our remit what we're working on relates to the ethics and aims to inform conversations with Peter. I've come to understand that ethics in AI is very much understood in this broad sense to the full range of potential impacts. So I would say that the the problems we're studying are the same.
The scope of the problems are the same, but perhaps the emphasis and the toolset are somewhat different and complement each other. We tend to bring the tools of social scientists, policy, science and in particular, an emphasis on the geopolitical character of the challenge. OK, I'm. OK, so here we are. And in summary, our mission is to help humanity achieve the benefits and avoid the risks associated with advanced artificial intelligence.
Now it may seem like the kind of mission that this group could complete on our own, or maybe with a bit of help, but that's meant to be sarcastic. Actually, it's it's not going to be easy. The governance of A.I. will be a very difficult challenge, and I'll briefly explain why. If you think about AI's a general purpose technology, this is a concept that's been used to think about AI technologies like electricity, the combustion engine.
Then you can see that I will not just have these impacts in various narrow applications, but really in a deep, transformative sense, transforming the economy, society, politics melting. And we could go through the properties of AI as we understand it and see why it poses such challenges for governance so that the harms and the benefits tend to be diffuse.
The technology is so fast moving and require such technical sophistication to understand that so many of the developments in IoT, dual use and the sense that they have scientific, commercial humanitarian benefits. But those same technologies with slight modification can be misused, can cause harm, can be used for the military. So if we try to enumerate the governance challenges, we'll get a long list. Here's one such list. I've group them into several categories.
Maybe I'll point to one subcategory that's perhaps neglect in this conversation, which is air safety. This is really a set of work that needs to be done in collaboration with computer scientists and researchers. The Future of Humanity Institute has an air safety group, and we collaborate with researchers at other labs and elsewhere.
The work that we do is mostly on the right hand side. So thinking about domestic political challenges, international political economic challenges and then especially challenges for international security. So I'm now going to start moving quickly to just give you a sampling of some of our work.
So to begin, if you want to have a more, more lengthy overview of how we think about some of these problems and how they relate, you can look at this research agenda document that I wrote, which breaks up the problem into four categories slightly different. So the first is the technical landscape. This is work that needs to be done by email researchers and economists, and some others that tries to understand what's the current state of email and not just about machine learning.
And also what it would look like in the future. So how will it change economic structures? How will it change the demand for different kinds of labour and so forth? The next category is politics. This is where most of our work to date has been. This is thinking about the various ways that political institutions can help manage and develop A.I., but also the disruptions and opportunities that A.I. poses for political institutions.
Ideal governance refers to the more normative challenge of how can we envision what it is as a society? We want what our institutions that might do a good job of managing the risks and eliciting the benefits. And then finally, policy refers to near-term policy recommendations that we can make to universities, to government, international bodies, labs and firms.
OK, so now I'm going to go quickly. So one of our first are projects that got a lot of attention was what's called the malicious use report. This was led by Miles Brundage, who's now at OpenAI, and this was our attempt to just sort of think through and catalogued the various risks that I could pose. For example, one of the risks that we talked about in here was deepfakes, which at the time was relatively unknown.
It was sort of a technical curiosity and speculation, even though it wasn't so long ago. Now, of course, we're seeing this in the news. All kinds of challenges to privacy, decency and also political conversations. One of our collaborators is a DPhil at Oxford Objecting. And he's done a lot of work on Chinese A.I. policy, innovation policy, global technology policy and how that relates to China.
And if you're interested in this, he has a weekly newsletter where he sometimes translates works coming out of China and also just reflects on the many interesting issues in this space. Another collaborator, Bao Bao Zheng, is the lead on our survey works. We've done surveys of publics and experts. This depicted here is a survey we did of American citizens, and we also have a survey in the works on Chinese citizens.
And then we're going to survey European citizens. And I don't know if you can read the left hand side, but this is one of our interesting results. We asked these Americans which institutions, which organisations they most trusted to develop, manage and develop AI in the public interest. And we found such things as Americans trust university researchers. So that's good for us, I guess.
And the U.S. military is another institution that has a lot of trust in maybe second place where tech companies so so Google, for example, scored well and Microsoft and some others, with one big exception now is Facebook. And interestingly about Facebook, even though we surveyed this after the Cambridge Analytica results scandal occurred, we did a pilot beforehand and the results were basically the same. So this concern about Facebook governance has a long history.
And then Americans also don't put up especially large amount of trust in the government. So federal government, state government and international governmental bodies like the UN do not do not score well there. Another interesting result here is we asked these respondents which A.I. governance challenges they thought were most likely to cause to have a large impact in the world in the next 10 years and which were most important.
One takeaway is that all of these issues scored very high on importance on the y axis. You can see they're all somewhere around 2.5, which somewhere between important and very important. But another interesting result is that there was meaningful variation in these different issues. So, for example, the issues that were regarded as most likely to impact a large number of people were data privacy, cyber cyber-attacks, surveillance and digital manipulation.
Now there's, of course, always ways in which you might want to ask these questions differently to better elicit their beliefs. In this case, I think because we asked about the challenges in the next 10 years that gave a certain kind of response, whereas if you're thinking longer term on other issues like technological unemployment, I expect would be judged as more impactful. OK, so we've also done surveys of A.I. researchers and experts on this.
These figures refer to an older paper of ours from 2017. We also have a recent survey we've done. The results will be coming out soon, and I won't say much about this for some time, but we had some interesting results, for example, on the left figure asking these experts to just forecast when various capabilities would be achieved. And for example, in the bottom, you can see Starcraft that refers to superhuman Starcraft performance.
And what's nice about some of these questions is that they are they have to do with tasks that should be achievable in the near term. When your term is like five to 15 years. And so what that means is after about 10 years, we can start evaluating how well these experts did at forecasting technological developments. We can also analyse by demographics are certain kinds of experts better able, more calibrated and their technological forecasting.
And so the reference to Starcraft, of course, is that DeepMind has sense results when this task would be achieved. We have some work this is with a computer scientist who's now a police colonel looking at the social implications of data efficiency. And maybe I'll just remark that I think work in the space and ethics and governance of the I really benefits from a deep conversation between the technical experts and the social,
ethical, philosophical experts. So in this case, data efficiency is when you can sort of do more with less data. And there's a first conceptual perspective that's useful, which is to think of this data efficiency in two ways. One is to think about it in terms of the access effect. If you need any given level of performance, say it's an autonomous vehicle that is sufficiently safe, that the regulators will approve it, you will be able to achieve that with less data.
That's the access effect. The second perspective we call the performance effect is that for any given level of data that you have, you'll be able to achieve a higher level of performance. OK, now, given those two perspectives, there's often some intuitions that emerge from it. For example, some people think, well, data efficiency means that the market, the technological, the market and I will become more competitive.
Instead of having a few big tech companies, you'll have more entrants. Right, because you can do more with less data. What this paper did is really reflect on some of these intuitions and show that they're not as straightforward as you might think. And I for want of time, I won't explain, but I'm happy to do it. You and I are references to the paper. OK, I'm going to skip. So another set of projects we have this is with a historian of technology of this particular period.
Let's look at moments in history when humanity has confronted powerful technologies here, depicted as the aeroplane. And then on the right hand side is the United Nations Atomic Energy Commission. Our deliberations were taking place under the control of nuclear weapons. So here are circumstances where humanity has confronted an emerging, powerful technology that many regarded as posing a joint danger.
A danger to sort of all the great powers and many individual scientists, members of the public, but also political elites, tried to find ways to build global institutions to minimise those risks. And of course, I should qualify. I'm not saying that I it's the same as nuclear weapons or the bomber. These are very different kinds of technologies, different circumstances. But there are lessons, I think, to be drawn from history when we have confronting technologies.
And in both these cases that were perceived to be highly valuable commercially, but also posing a disruptive impact for military stability. And I think there's a lot of really interesting lessons to be learnt here. Many of us didn't even know this took place, that there was a movement to control the bomber and even to have an international air force, so bombers would only be possessed by the League of Nations. No one country would be allowed to build bombers.
Fighters were allowed because those are defensive technologies of thought. And then many of us have forgotten this historical moment when the US and the Soviet Union had a multi-month conversation in the UN about moving all nuclear materials and nuclear weapons over to the U.N. for control. Right. In the current global climate, we think this is just impossible. But this conversation did happen because it's provoked by the fear of a nuclear arms race, which of course, ended up happening.
So in conclusion, there's a lot of governance challenges and ethical challenges that we're going to face. And I do think many of these become especially difficult in the presence of geopolitical competition. And this is something that resonates with many of you. So even something like privacy becomes much harder in a world that's competing economically between these economic blocs. So Europe, for example, might want to have more stringent privacy policies.
But there's a concern that if Europe adopts that, then Europe has no chance of cultivating any champion the way Silicon Valley and China have. And similarly, in the US, there's debates about regulation of Facebook and others, and they often use retort is if you regulate us, if you break us up, then China will win.
Then you know, the Chinese air champions will win. And so you can see how even what seemed like domestic political ethical issues cannot be understood except without understanding the charter of global competition. So with that, I'm going to turn it over to my colleague a. So I would be talking about a bunch of different projects that we've been doing it. I bet it would start with a project of mine on human autonomy in particular.
It's a question of whether and how AI systems may affect human autonomy. Now, autonomy is one of these concepts that has been popping up in guidelines in ethics principles recently. Quite a lot. But surprisingly, there is not a lot of academic work that has been done on the topic. And I think it's a bit like with fairness, where philosophers have been thinking about these concepts for a long, long time.
It's the same with autonomy we've been there's a lot of philosophical work on the philosophical thought on autonomy. And the question is like what can philosophy contribute to it to a topic important as this? Now what do we mean by autonomy? Of course, philosophers never agree on anything. But broadly speaking, we can say that autonomy refers to the ability to self-governance.
So it's the ability of human beings to be their own persons, to have actions guided by beliefs, preferences, values that are in some sense in some important sense, genuinely their own, as opposed to externally imposed by a manipulation or correction. Now, when we look at the principles and the guidelines in ethics of AI, then we see that autonomy actually is used in a in a different a variety of different ways.
So of course there is. The autonomy is personal autonomy, which was the concept that I just introduced and that it's incredibly rich. So I won't have time to go into a lot of detail of what philosophers have said about personal autonomy. But the High-Level Expert Group on Artificial Intelligence, for example, has the principle of autonomy and rights that A.I. systems should not unjustifiably subordinate, coerce deceive, manipulate condition or hurt humans.
But that's about it. In terms of personal autonomy, the AI ethics guidelines usually warrant when these guidelines speak about autonomy, what they mean is more control or autonomy of control. So I mean, I would call this the principle of control, but they use the term autonomy. Usually it's it's about maintaining and exerting control about over AI systems, over the AI system question.
Sometimes people also speak about maintaining the power to decide which tasks are being outsourced to AI systems. But it's very much about controlling the system. Now, these two are clearly not the same. So, for example, I may not have control over which advertisement pops up on my screen or which recommended recommendation system recommends the next song to me. But this doesn't mean that my that my personal autonomy in some sense, is affected by this.
So, yeah, so I mean, both of these are very rich topic, and I think there is some there are some serious concerns underlying the principle of control as well. But it's up to interpretation whether they might be more referring to consent or more about questions about meaningful, meaningful human control. So instead, I'd like to talk a bit more about personal autonomy.
And like the the concept of that, our beliefs and values are genuinely our own and just give a subset of possible ways that I may interfere with personal autonomy. So there is a paradigm case of manipulation. So Cambridge Analytica, for example, attempted large scale money from large scale voter manipulation. It's quite it's very unclear whether they succeeded.
Succeeded at that. Probably not. But what's important here is the scale and the potential for abuse and the potential for manipulation by AI systems. Now then there's also adaptive adaptive preference and the deformation by system. So what I have in mind here makes the most recommendation algorithms that give them like those are algorithms, as the name suggests, that first predict the user's preferences on the basis of data they they have access to,
and then present the user with options that best fit those preferences. So we know this, they're being used across the bench. We know this from Spotify, Netflix, YouTube and so on. Now they may like it turns out that they may actually alter preferences as opposed to merely adapting to our preferences.
And now their first, their first studies and one of them, I linked the paper at the bottom by adding my features and who shows how preferences change on the basis of a of a random of a fake recommender system, giving giving ratings to a certain to certain songs and and videos. I think they did that. Now finally, we may. There is also the concern for the loss of competence so that we that we lose the competence of making authentic decision in some sense.
Again, this is a very complex issue that I. You touched on now, maybe some more advertisement of how great autonomy is. The thing is like it overlaps with a lot of other areas that we're already looking at in in ethics like privacy and surveillance, the need for transparency of AI systems and the use of AI systems, but also like questions of responsibility. Now the main takeaway here is that the way personal autonomy might currently not be, it might be not at stake.
Like the more sophisticated AI systems get, the more problematic. This could be this could be a problem, and they are getting very sophisticated. So here is an example of an of Typekit you, which is a language model developed by OpenAI. And what it does is you enter a sentence by hand or a title or not or a paragraph, and it completes the article. So here I enter just for fun the like, the title of the ethics of autonomy and artificial intelligence, and I was looking at what I was producing.
And fortunately, it didn't write an abstract about my work, but instead it came up with something about the moral responsibility responsibilities of robots. There was another case where it was talking about autonomous vehicles. So this very free hearing is very surprising and very advanced A.I. system. Now, the reason why it took a detour became became famous was not only because it was one of the breakthroughs in AI development and in language models.
It was also because OpenAI made the choice of only partially releasing the model. So OpenAI, as a company, decided, well, maybe the model could be used for malicious purposes or so we're going to we're going to have a stage for these.
We're not going to release a fully trained model. And this sparked a lot of controversy in the machine learning community, which is usually known for being quite open for publishing a lot of open source for publishing their algorithm so that other people can access it. Now, the question is whether this is from this norm of openness is problematic in some cases.
And here I want to I want to briefly outline a paper that was written by Alan and Toby Shevlin on openness and on this culture of openness in in machine learning. And they were particularly they were looking at arguments that were made by the machine learning community against against open AI's decision to partially release in favour of openness. And a lot of their arguments were based on examples from cybersecurity.
So in cybersecurity, they have this model of responsible disclosure, which means that they publish all the secure, like all the vulnerabilities in software publicly, but only after a few a few weeks. So people have time to patch the patch, the leaks. But if the reason to publish these these vulnerabilities publicly is so that other people can learn from the mistakes and hopefully it builds more secure software now in their paper.
Toby and Allen show that the comparison between these arguments and cyber security and I don't really don't don't really apply in all cases. And that, in fact, well, you can. When you patch a security leak in the software, you you patch a piece of code, whereas when you have you publish your AI system, then in many cases it's irreversible and it's much harder to patch to patch the new vulnerabilities that pop up.
And yeah, I think, for example, of voice deepfakes that fake the the voice of a loved one, and they get quite LaGuardia available with a snippet of voice. You can you can create these deep fakes and that are being used for fraudulent cases. Now, once these models are out there, it's really hard to take them back. So the comparison the comparison doesn't, really doesn't really work that well in all cases. Now how much time? How much more time do I have?
Five minutes. Seven minutes. OK. OK. So we also we also working on trying to implement these insights and more practically. So I'm just going to briefly outline here we have developed it or we are in the process of developing a guideline for machine learning researchers about how to conduct responsible research.
It's going to be based on work that has been done in the academic community, and it's going to be a hands on guide that takes me such a step further step by step for the research progress. Now here we've been focussing on the machine learning research community and how the machine learning research community, whether machine learning research community is heading and how that might affect the safe development of artificial intelligence.
Now, another interest of ours, though, is to figure out like, what does our own community do? And one thing that has been emerging in the last few and the last few years is that there seem to be two parallel communities working in the AI ethics and society domain.
And these communities can be roughly described as those working on near-term issues like algorithmic bias, talking as vehicles, fairness and those who are working on more long term oriented projects on what is usually called the long term community, such as superintelligence or advanced A.I. Now it's a bit all there, and there seems to be an almost a rivalry about which is more important. So here we see Kate Crawford, who is one of the founders of AI now and has an excellent work on on fairness.
And Ryan Cato, who say, Oh, you know, I think this whole talk about superintelligence is overblown and we should focus on the real issues. And then we have others from the larger community. You say, Well, you know, I think it's really important that we talk about this now and not in the future because the effects are drastic. So with my colleague Jess Whittles down in Cambridge, we were wondering, OK, what's going on? Is this actually a useful distinction to and that is being made here?
And so we tried to figure out whether the the division of the community is into these two blocks is not a bit too crude and in fact, not very useful when we think about the ethics and society space. And what we find is that when people talk about near-term and long term, they mean very different things by it, in fact.
So people defining near-term and long term, sometimes they refer to the capacity of an AI system or are about they think about the impacts or their research priorities are set because they believe they want to focus on more extreme risks or they want to focus on certain risks. So we've identified these four dimensions, and we think that all of these are not binary, by the way. Like these are very much gradients and a lot of these a lot of research projects are going to be somewhere,
are going to fall somewhere in the middle here. And we've just focussed on capacities and impacts and just some of the research developments. Some of the research priorities actually onto this onto this graph, and you can see that it is far from clear that they actually these two clusters that emerged like the long term, the near term cluster. In fact, there is a lot of overlap and the communities could benefit much more from engaging with each other,
exchanging methodologies, exchanging knowledge. OK, so here is a paper that far for future reference. OK, some of our work has also been done on privacy of more with a focus on how can we use email to actually enhance privacy. This is by Ben Garfinkel, but sadly, I won't have time. So I would just briefly skip to the teaching card because Peter asked me to talk a bit about this course that I gave last year with the Oxford Artificial Intelligence Society.
It was a six week course and each one each last was two hours. Now, the benefit of not doing it as part of the university but with a student society is that you can actually have the first. You can actually also invite people who who give expert tasks. So what we did is we really had the first hour of the course as a as an introduction, the topic on the various topics that are listed here.
And the second course we had people from the Oxford and my friends was there from the exit to the interest rate institute, but also people from AI. I talking on the on the take on this topic, specifically now for half a minute, I'm just going to talk about some of the policy engagement that we've been doing it if it's AI. So there are a lot of but we've published a lot of work in obits, in newspapers and engaged with the public.
And one of those more remarkable engagements is that we actually working together with the partnership on the AI, on the so-called windfall across now. The windfall clause was originally suggested by Nick and his book Superintelligence, and it's like it's a legally commitment. Is there a legally binding commitment by private firms that in the case of a firm's profit in the case of for profits, skyrocket because it has developed some transformative artificial intelligence?
Then every part of the profits about a certain threshold go back to the community or to humanity as a whole. Now there's a lot of substantial work that needs to be that me to go into determining how exactly this will apply. But the good news is that our analysis finds that the windfall class actually is legally permissible. So, yeah, so this is a project that I think I've been more interested in. Think contact Jade or Elon. And please get in touch with us. OK, so thank you very much.