Does AI threaten Human Autonomy?

Welcome to a discussion of the relationships, positive or negative, between artificial intelligence and human autonomy. Like our event four weeks ago on ethics and in education. This is a seminar associated with the new Institute for Ethics in A.I. at the University of Oxford.

If you're interested in finding links to other related events both past and forthcoming, together with recordings and podcasts categorised by topic, then go to the Philosophy Faculty Home page and click on the ethics in a I think. I'm Peter Milliken. Gilbert Ryul fellow and professor of philosophy at Hartford College, Oxford.

And I'll be chairing tonight's event. Autonomy is a core value in contemporary Western societies, a value that's often invoked in debates about practical ethics and politics and indeed lies at the heart of liberal democracy. Understandably, therefore, advocates of artificial intelligence are keen to emphasise how A.I. systems can enhance role autonomy, giving us easy access to information on which to base our autonomous decisions, enabling us to achieve our personal goals more efficiently.

And so. But on the other hand, there are growing worries that A.I. systems, in fact, pose a number of serious threats to our the. One prominent example in recent years, not least in various elections and referendums, has been the use of manipulative techniques that take advantage of our weaknesses and biases, leading us to make decisions or to be persuaded of views that might be quite different from those that we would otherwise have adopted.

Such tricks have been part of the advertisers repertoire for generations, of course, but machine learning on our social media sites and so forth makes it possible for us to be individually targeted and exploited both by sales people and political groups in ways that were previously unimaginable. All this raises the question of whether it's possible to harness the considerable power of A.I. to improve our lives in a way that's fully compatible with respect for Tom.

And whether we need to reconceptualize both the nature and value of autonomy in the digital age. That's the focus of our seminar tonight. I'm delighted to be joined by three Oxford academics who are particularly interested in these two. First, Carina Pringle. Korina is a research fellow at the Institute for Ethics in Azi University of Oxford. She's also research affiliates at the Centre for Governance of A.I. in the Future of Humanity Institute, again here at Oxford.

Guerena works on the ethics and governance of A.I. with a particular focus on autonomy, and there's both publicly advocated and published on the importance of accountability mechanisms for A.I. Welcome, Katrina. It's good to be here. Second, Jonathan PUE. Jonathan is senior research fellow at the Oxford You ahero Centre for Practical Ethics, researching on how far A.I. ethics should incorporate traditional conceptions of autonomy and moral status.

He recently led a three year project on the ethics of experimental deep brain stimulation and neuro hacking, and in 2020 published a book, Autonomy, Rationality and Contemporary Bioethics with Oyuki. And he's written on a wide range of ethical topics, particularly on issues concerning personal autonomy and informed consent. Welcome, Jonathan. Thanks for having me. And third, Jess Mallie.

Is policy lead at Oxfords Data Lab, leading its engagement work to encourage use of modern computational analytics in the NHS and ensuring public trust in healthcare records. Notably those developed in response to the coded pandemic, Jess is also pursuing a related doctorate at the Oxford Internet Institute Digital Ethics Lab. As technical adviser for the Department of Health and Social Care, she co-authored the NHS Code of Conduct, the data driven technologies.

Welcome, Jess. Hi, Peter. Thanks for having me. Each of our speakers will give a short talk, Paul, about the discussion and then the next talk and so forth, and the event as a whole will last for 90 minutes or about that. So we'll have plenty of time for discussion. And you're very welcome to offer your own questions to the speakers. Indeed, you are encouraged to do so. So please feel free at any time to pose questions by typing them into the comments box on YouTube.

I'll be noting these as we go and posing them to the speakers at various points. So the sooner you get your questions in, the more opportunities there will be. There will be for having them addressed. OK, I hope that's all clear, and now we'll move to our first speaker. Karina? So Carina's, as I mentioned, this research fellow at the Institute for Ethics and I research affiliates at the Centre for Governance of VII.

And Korina, you're one of our new appointments under the Ethics in A.I. Initiative at Oxford. So it's it's great to have you here. I noted that you're a strong advocate for accountability in A.I. development and you've addressed the Mexican Senate on this topic. So that sounds interesting. And you were featured as one of the 35 under 35 future leaders by the Barcelona Centre for International Affairs and the Banco Santander. So over to you. Yeah.

Thank you. Thank you very much, Peter, for having me. So my task today is to provide an overview over human autonomy and the various ways that you might affect affected. Autonomy, radio and radio. As Peter already mentioned, is one of our core values in our society and a very broadly refers to our effective capacity to make decisions, to live our lives according to our own choosing. And according to our own standards. So basically, me points to our capacity of being self-governing agents.

And the question we're exploring today in the seminar is whether any assistance might actually threaten this ability or capacity and if so, in what sense they might do. Now, the first point I'd like to make is that when we try to assess the impacts of a on human autonomy, I think it's really important to be clear on what aspect or or what dimension of autonomy one is interested in.

So, for example, one possible dimension concerns the effects of A.I. systems on our ability to make decisions and especially our ability to make decisions that are in some important sense our own and that are not subject to a distorting external forces. And here's some of the main issues that arise in the context of a AI, as Peter already pointed out. Our online manipulation and deception. But clearly our case is assuming the economy is undermined by the use of AI systems.

But also, more generally, there's the question of how the AI systems might affect and shape our preferences. So, for example, by now, there are quite a few studies that show how people change their preferences to match the recommendations they're presented with and by recommendation. In the case certainly has been made that this to some extent, to some extent, fundamentally undermines our autonomy.

So, I mean, this is one of these cases where it's probably a bit more complicated than this, because in order to show that these systems actually undermine autonomy, one would also need to show how is it different from ordinary cases of preference adaptation that we experience on a day to day basis when we ask our friends, you know, which movies they recommend, for example. Now, I'd just mention the potential of A.I. systems to influence our beliefs and our decisions.

But there is another aspect of Upon Tomie which has to do rather with the execution of those decisions instead of the formation of the decisions. I remember reading an article several years ago, which was during the very early stages of social robot development, and there was the case that some Japanese engineers tried to build a robot which was supposed to display affection and to give lots of hugs to people.

And for some reason, the robot became very, very eager and I mean very, very eager up to the point that it would not let you leave the room. I assume simply constantly hugging her and not letting go. Now, this is a case where the engineer can you have made up her mind about leaving the room. So she's clearly made the decision, but their boat was actively preventing her from executing the decision.

And note of the ethics guidelines on. I actually bring up this dimension of autonomy and emphasise that human beings need to remain in control of our systems and especially that they need to be in control over but over whether when and how tasks are outsourced Joy. Now, the question of control is especially a concern that has been raised in the context of opponents I systems, but also in the context of decision making and guarantees, too.

The worry is very much that decisions of important decisions of importance are outsourced. Try AI systems, then humans are users might not be able to reclaim decision making power. Finally, another concern that is sometimes raised in the context of a AI and human autonomy is that through this process of outsourcing, an increasing amount of tasks to assist us, we actually make ourselves dependent on the technology.

And as a result of this dependence, we might lose our ability to act autonomous with these kinds of arguments. I think it's important that we're a bit careful to not equate human autonomy with Merrilee. Independence and feminist philosophers, for example, quite rightly pointed out that no human is fully self sufficient, and independence and self sufficiency particularly seem ill chosen as values that is worth striving for. And those as part of our conception of autonomy.

And I also think it's important to note that dependence on technology either is a bad thing. Nor does it imply a lessening of our autonomy. So you cherry user, for example, could be very dependent on their wheelchair. But it seems it seems rather misguided to say that as a result of this dependence, they would be less autonomous. In fact, if anything, I think that future probably increases their autonomy because it provides them with.

Now, this brings me to my second and last point, which is that the answer to the question of whether or not a Swinton's human autonomy, Righi, very highly depends on the context and the system in question. So there is no there exists unfortunately, no one fits all approach and neither are there. One fits all policy responses. So, for example, it makes a big difference whether manipulation by assistance is intentional or whether it's unintentional.

So Cambridge Analytica, the Cambridge analytical scandal is an example where the attempt to manipulate voters through targeted political advertising then was very much intentional. So it was intended by those employing the. Manipulation through, let's say, biased search rankings or through YouTube video recommendations. If they take place in many cases, can be unintentional and can be a side effect of their recommendation algorithm question.

So that's more technical problem or a problem of mis specification of of the goal of the goal of the algorithm. But the kind of governance responses that we'd need to implement in order to address these two cases of manipulation that say both of them are be manipulation. And there's, of course, also a lot of debate about what counts as manipulation.

Let's assume these are true cases of manipulation. Then the two the government responses need to be it would need to look very different for the case of Cambridge Analytica and for the case of that same bias. Google search rankings. And similarly, just to just to stay with the example of manipulation for the moment. From a moral point of view, not all cases of manipulation are equally morally reprehensible.

So, for example, it makes a big difference whether I am nudged into buying a more expensive car insurance or whether I am nudged into paying my taxes at the state. So and similarly, in the case of it, it just makes it more different than my health care. Not just to undergo treatment, but in my heyday. Not just me to spend even more time playing the games on my iPhone.

And so, again, we really need to really need to assess these applications on a case by case basis and B, be especially sensitive to the context they are deployed in. I'm very glad that the remainder of the seminar is dedicated to explore a very specific case of failure and autonomy. I would also become very clear that even then, there remains plenty of room for disagreement. OK. So to wrap up, I'd just like to briefly reiterate the two main points I just paint.

The first was that me as a concept, it's terribly ambiguous and that we need to care. But one aspect of autonomy we are talking about before we can even begin to ask the question of whether it poses a threat to it. The second one was that even then, we have to be sensitive to content and just have to accept that there is no silver bullet that allows us to assess all possible threats to once. Thank you very much. Mute. Thank you, Carina. There we are.

Very interesting. Could I just press you on a couple of questions arising from what you've charged? So you gave the example of human versus machine autonomy. You gave the example of a wheelchair user. Now, I think it's it is quite tempting to think that in any situation, there's a certain certain total amount of autonomy which is divided up between the various agents involved. So if you have more autonomy for the all the automated system, then that's less autonomy for the user.

And somebody might press that a bit further with the wheelchair user and they might say, look. The wheelchair increases the autonomy of the user, but only to the extent that it's under their control. As soon as you start putting A.I. systems into the wheelchair, that means that it's making decisions, then that's reducing the autonomy of the user. So then they might try to suggest that actually it kind of is a limited amount of autonomy to go around.

And therefore, more autonomy for the automatic system means less for us. How would you respond? Yeah. So first of all, I'd say that there is a fundamental difference between what we consider to be human autonomy and what we consider to be system autonomy. Our economy is used in the context of autonomous systems. I think human case autonomy is an intrinsic value and it plays an important role for moral agency responsibility and so on is something to strive for.

In the system case, there is at least no such. There is at least no such moron agency. And so there is no autonomy in this sense. And autonomy more refers to something more like independence. So independence. Human operators or also in the case of of learning algorithms or machine learning first to their ability to learn on the basis of experience.

So basically, this means that autonomy in the case of machines is not an intrinsic value, but it's like at most an instrumental one by suggesting that there is some kind of a shared amount of autonomy and that there might be even a Trade-Off between our autonomy and the autonomy of the system, we're basically putting human autonomy and system autonomy one on one and the same level.

And this I mean, as I as I just pointed out, cannot be right because the two refer to very different things and fulfil a very different functional roles as well. So I do think that this idea of a Trade-Off is somewhat misguided. And I think you brought up the wheelchair at example. I agree with the amount of control. But then again, we get this question of control is often very underspecified. It's very unclear what amount of control and we actually would actually want in this case.

So, for example, you might want to be able if for a car to be able to break it, to be able to stop or to choose a particular to particular route. Otherwise, you could say, well, I'm not having the economy to steer the car. But in other cases, these kind of issues would never come up. If you enter a roller coaster, you don't have the you don't want to control the rollercoaster with a stop or two. It took two to leave the rollercoaster, so. Yeah.

So maybe it maybe just this question of control is very, very contentious in itself because it's supposed to have control over the system. So, yeah, things get very complicated. But coming back to the Trade-Off, I do think that there is that there is no such Trade-Off between the autonomy. Right. Thank you very much. Can I just follow up just a bit on the concept of human autonomy and you say that SINNEN shouldn't take value? I think most of us would sign up to that.

But is that a value that geographically and historically universal or some might think it's a relatively local thing for modern Western cultures? Yes. So this is a very, very a criticism one hears very often. The idea that autonomy is a has this Western overly individualistic flavour to the individual at the centre of action. And I think here philosophy and in particular Western philosophy, actually has moved on quite a lot and has moved away from this individualistic notion.

So on the one hand, there are now a lot of concepts of relational autonomy that are emerging from the coming from the feminist literature on autonomy that put very much the relation, the relational nature of autonomy at the heart, saying, you know, we are all people, we are all embedded in our society. We are all reliant on our social interaction to solve the problem.

And then furthermore, there are also some procedural accounts of upon me that identify and person as appointments, virtue of the procedure by which the person has made the decision or hold certain beliefs. And these accounts are also value neutral. And so basically there are no values baked into these accounts and they can't be.

And I think our successfully employed across cultures and it seems that also when you're when you ask people to think countries like China are known to have more collective mentality and a very different way of thinking about individual action and social action even there, which kind of these notions of autonomy and do find resonance. And the people would actually ask. Thank you very much indeed. We will now move on to our second speaker. See you later, Karina.

We'll move on to our second speaker. Jonathan PUE. Welcome back. Jonathan mentioned when I introduced Jonathan, he's senior research fellow at the Oxford Heroes Centre of Practical Ethics. One thing I said then was that you had a project on the ethics of experimental deep brain stimulation, neuro hacking, hacking, the software incorporated into brain computer interfaces. Can you just a little bit more about that? Sounds intriguing. Yeah.

It was a very interesting project. So over the course of that grand project, I was collaborating with experts in neurosurgery and cybersecurity. And there's some emerging evidence now to suggest that it would be possible to hack the wireless elements that are incorporated into brain stimulation devices. So historically, it's been established that it's possible to hack pacemakers, cardiac pacemakers, and now that wireless systems are being incorporated into things like the brain stimulation.

This raises some interesting questions philosophically as well as just generally for society. Do you mean do you mean hack for evil purposes? Some third party interfering with it? It technically could be possible. So there have been studies showing that it would be possible to hack a cardiac pacemaker to potentially stop it from functioning.

That hasn't yet been fully established of neuro hacking, but the technical possibility is certainly something that researchers are are interested in and certainly worried about. So there's a lot of interest in how we can develop the security of these devices far more robustly. Mm hmm. Very interesting. And today you're going to talk to us about a particular example of autonomy or lack of or threats to in health care.

But you're you're more on the positive side. I think so. But I want to say how we develop over the course of the evening. Good overview. Okay. Thank you, Peter. Let me just share my screen. Good luck. Hopefully that's that's come up. So Kareen has given us a fantastic introduction to the various ways in which we can think about autonomy and how that's implicated in AI systems. We decided to look at a particular case study to focus our discussion.

We're going to use the study of inhealth tools. And I think that's a really interesting case to think about, because these M health tools, which I'll introduce a little later, represent something of a middle ground, if you like, between questions in the burgeoning field of A.I. ethics and the field of bioethics, which is perhaps a little more well established and which is my home discipline.

So in the first part of talk, I'm going to say a little bit about how I think about autonomy in bioethics. And fortunately, Kareena and I agree on most things, so I can move for that fairly quickly. But I'm going to present a visualisation, if you like, of how you can view the concept of autonomy and draw on that to make some arguments in favour of the idea that MLV tools can be in a system which might be used to enhance autonomy.

So let's start with that. So, as Karina mentioned, the fundamental concept, the rearms cultural autonomy, is this property of self governance? One thing I would like to add to that is this point that we can think about autonomy in two different ways. We can first talk about it at a local level. So we might be interested in whether an individual is autonomous. With respect to a particular decision or a particular action.

But we can also be interested in whether an individual is autonomous over extended periods of time, perhaps with respect to a long term goal or project. Now, one reason it's important to notice this distinction is that sometimes our global autonomy can demand quite different things of us than our individual autonomy.

Sometimes when we left to our own devices about making decisions about how to achieve our long term goals, we might in fact be less successful than if we perhaps outsource some of those decisions. So that's going to be a really important point, I think, for thinking about the overall implications of a I for autonomy. Now, as Korina suggests in her presentation, there are two at least two dimensions we can talk about when we think about autonomy.

Now, in bioethics, there's a fair tradition which really emphasises the importance of freedom and ability to self-government, to autonomy. So the key question here is gonna be, am I able to act on the basis of my own decisions? And that's, again, going to be really important when we think about the implications of a I for the primary reason that A.I. systems promise to hugely enhance our abilities to do the things that we want to do. Luciano, Floridian colleagues capture this idea really nicely.

I think in a recent paper I wrote that put at the service of human intelligence A.I. can enhance human agency. We can do more, better and faster thanks to A.I. And that's really an important point, I think, coming into this conversation. But of course, again, during on screen his earlier presentation, that's not that or that's not all that autonomy amounts to, as well as being able to do the things we want.

It's also crucially important that we're autonomous with respect to our decisions about what it is that we want to do in the first place. And there are some different elements that can feed into that. Of course, in order to make autonomous decisions, I need to have sufficient understanding of what my options are like. That's a point that's familiar from the medical context. It provides the foundation for the doctrine of informed consent.

And there's also essentially this reflective element, we might think that autonomous decision making must be grounded by motivational states that somehow reflect my values. Now, once they have that quiet general and somewhat basic framework in mind, we think we can begin to see how different Freda's to autonomy feed into the different elements involved.

So deception and information manipulation, it seems to me, a ways in which we can undermine the level of understanding that autonomy might require. And as Kareen mentioned, in a way that's going to be a particularly important source of threats to autonomy.

We've seen the prevalence of fake news. The development of filter bubbles and echo chambers really pose threats to the kind of understanding that autonomous decision making requires other kinds of interference, perhaps target the reflective element of autonomy, perhaps nudging us into making decisions we wouldn't be effectively endorse or inducing addictive forms of behaviour.

So I'm just going to reiterate Carina's conclusion. Absolutely. The answer we can give to this general question has to be it depends. The A.I. systems can both enhance and diminish these different aspects of autonomy. Now, what I hope visualising that framework will help us to do is to bring out this point that the very same application of I can enhance some elements of autonomy whilst at the same time threatening others.

So I think with respect to these applications, there can be trade-offs about the kind of effects they're having. So that bramblett mind. Let's now turn to the case study of the health tools. So very broadly speaking, M health tools are mobile and wireless technologies that support the achievement of health objectives. These can take a wide array of forms. I'm going to use just two examples for the purposes of the presentation.

First, Fitbit switch. I'm sure many of you are probably familiar with. After all, there are roughly 28 million active global users of these things. So Fitbit pictured that the small bracelet essentially that you can use that can monitor your levels of physical activity, counting the number of steps you take, for instance, and also certain how parameters such as your heart rate, other health tools are targeted. More specifically, are individuals living with particular medical conditions.

So the my sugar out there is targeted at individuals living with diabetes. It allows users to monitor their blood glucose levels and also to log their eating activity. Now, the earliest iterations of these MLV tools were essentially monitoring devices, but there they are increasingly becoming more and more sophisticated and they're beginning to integrate AI systems so that they can tailor personalised recommendations to their users.

And the more sophisticated they become, the more likely it is they're going to raise some of the issues about autonomy. But we've already seen coming up in our discussion so far. Now, what I want to do is drawn the framework I talked about a few minutes ago to give some reasons for thinking why am I have tools? Might be the kind of AI system which could potentially be used to enhance our autonomy. Later on, when just give her presentation. I think she's going to raise some points.

Why am I a little misguided about this? But hopefully I can give a positive case in favour first. So the first argument draws in this point that inhealth tools, it seems, could be used to enhance the user's understanding. As I mentioned in the framework earlier, an autonomous decision maker is typically going to be an informed decision maker in order to make autonomous decisions. We have to have some understanding of what our options are like.

We need to know how we can go and apply our values in the world. And when they're functioning properly and health tools can give us a range of information which might be really quite useful for how we make decisions concerning our own health. So Fitbit, of course, can give you real time information about your physical activity and your your heart rate. And that can really be useful when you're trying to plan an efficient exercise regime.

Of course, we are now fully aware that physical activity and exercise is hugely important for our overall health. Now, of course, that argument does have to be caveated in a number of ways. And one crucial way is that it assumes that the information that we are receiving from mental health tools is always going to be accurate.

And now that's perhaps a slightly problematic assumption at the moment, because there are a number of reasons why consumer grade monitoring devices won't necessarily give you highly accurate information. So that is absolutely a caveat. However, the hope might be that this is a technological bug, which over time could be could be overcome. And the general point remains that if we can enhance understanding using these tools, so much the better for individual autonomy.

The second argument moves away from the autonomy of our decisions over to the more practical dimension of autonomy. I mentioned earlier. So one of the big obstacles we face when trying to achieve long term health objectives is maintaining adherence to healthy behaviours. Again, let me just focus on on the case of exercise. We all know that regular exercise is important for health, but many of us don't particularly enjoy it. Vigorous exercise can be a very unpleasant experience.

So one of the key goals of trying to get people to engage in more physical activity is trying to ensure that we can maintain adherence. And so what we see in these more sophisticated health tools is that they have started to use motivational strategies to help ensure that users are achieving the long term goals that made them sign up for the tool in the first place. So Fitbit, for example, will now offer badges once you've passed certain milestones, almost literally.

In this case, my sugar app uses an even more kind of complex motivation strategy, if you like, a game ify defeating diabetes. So the app displays diabetes as a monster that you have to try and tame by achieving certain healthy behaviours. So there is a sense in which these strategies can help users achieve their global health goals. Remember, I made that distinction between local and global autonomy. But again, there's a caveat here. And this, again, draws in some of the remarks Katrina made.

We might worry that the motivational strategies that these tools are using are perhaps manipulative at the local level. They're getting users to engage in individual decisions and actions which they perhaps wouldn't reflectively endorse. Now, I think there are two really important questions we have to ask here. The first is, are the motivational strategies themselves manipulative?

Are they nudges that circumvent our effective capacities? That's perhaps a question that we can come back to and discussion. But the second question, which I think is equally important, but perhaps overlooked sometimes, is we have to ask, how much does this matter from the perspective of autonomy? And this is where the distinction between local autonomy and global autonomy becomes really quite important.

I mentioned from the perspective of our global health goals, these motivational strategies might be highly effective in getting us or helping us to achieve the goals that we want to achieve. And so we might feel that if there are problems at the local level, these perhaps from the perspective of autonomy, could be outweighed by the manner in which they provide a boost to our global autonomy.

The third and final argument I want to highlight is what relates to this narrative of empowerment that has surrounded MLV tools. Now just is going to go into a lot more detail about this. But the general idea I take is that these tools have been marked in a way to which is suggested they're going to empower patients to give them responsibility for their own health care.

Now, one thing I think is interesting about that is generally when we think about autonomy, we tend to think that giving people more responsibility to make their own choices is one way of fostering their autonomy. That point will be familiar to anyone who's been involved in raising young children. We try and foster their capacity for autonomy by giving them more and more decisions as they grow older.

How applicable is that to the healthcare context? Well, I think this is something we're going to talk about. I think one of the interesting things about this development is in medical ethics. Traditionally, the patient's autonomy has to a large extent been the responsibility of the physician. Part of their duty of care is to ensure the patient is able to make autonomous decisions about their own healthcare.

Perhaps these MLV tools are one way in which we're finding a shift away from that physician based responsibility for patient autonomy. That's an interesting development and we have to weigh it against the preferences of those who perhaps don't wish to have that kind of responsibility. With responsibility for health care choices comes accountability. And there are, of course, problems of that. So just going to raise some very interesting points about this.

Now, as a parliament, I would suggest there is one way of viewing that narrative through lens in which it can be understood as enhancing autonomy in one quite general sense so that there the owners wanted to present. And now I'm back to Peter. Thank you very much, Jonathan. Wait a minute. Sorry about that technical glitch. Thank you very much indeed, Jonathan, that was very interesting. Could I ask you to.

So just a couple of questions that might help to shed light on on on some of what you're saying. Joe, do you do you think in general that providing users with more accurate information, for example, generated by a systems? Will that pretty much always enhance their autonomy? Yeah, that's a really interesting question, Peter. I mean, traditionally in medical ethics, there has been this view that as soon as we can get information to patients, then the more the better.

But we've seen that it's actually, of course, hugely problematic. And that is going to be exacerbated in the context of by the first problem is the sheer amount of content that we can give to users with whether health systems. You know, that these systems can generate huge amounts of data and providing all of that to users would simply overwhelm them. So the first problem we're going to face is that we need some way of filtering the information.

And as soon as we introduce filtering, we introduce the possibility also for some kind of influence or decision about how to filter the information we give to users is always going to involve some kind of value judgement. So that's one form about the content of the information we provide. The second is the manner in which information is presented. And again, this is a problem that has become more and more apparent in medical ethics over recent years.

It is extremely difficult to present seemingly even very scientific medical information in an entirely value neutral way. So the framing effects are a good example of this. You can influence the weight that information is afforded in an individual's decision making by the manner in which you present it. So providing information is not always going to enhance autonomy for the simple reason that simply providing information doesn't entail enhancing understanding.

Right. Thank you very much. That's raising some really interesting issues. I'm going to postpone a follow up question there until everybody's onboard, because I think that I'll be interested to see what you will have to say about it. Different question. I obviously is a new thing in the world. It's making a lot of noise, but it's not the first technology that's raised the spectre of manipulation. Does it raise fundamentally new questions then?

Yeah, I it's it's a question that I think of, to be honest, in some ways, I think the answer is no. And you touched on this, Peter, in your introductory remarks, because you're a systems when they do influence or manipulate us, are taking advantage of the same flaws in our decision making about many other technologies we've already taken advantage of. And you alluded to the fact that advertisers have known about these rules and our decision making for many, many years now.

So in that way, the forms of influence aren't necessarily new. But the threat posed by ISIS items is, I think, undeniably greater. The forms of manipulative interference are more pervasive, and they're also likely to be much more effective because of their highly targeted and personalised nature. And again, that's that's something you alluded to. Now, for me, I think that raises interesting questions about manipulation.

But perhaps the more theoretical question it raises is that it draws this really quite tight links between the concepts of privacy and autonomy to this point. In practical ethics generally and certainly in medical ethics, these two moral concepts of autonomy and privacy have really been separated as quite discrete ethical concepts. Now, in theory, that might still be true. But I think what we're learning is that in the practise of A.I., these two concepts are going to be much more symbiotic.

So any investigation into one really has to pay close attention to the other. Very interesting. You may be aware we have an event recently which was a launch event for my colleague Charissa Valises. Privacy is power. You can absolutely. So Caressa Crystal is my office mate. So we had some great conversations about that. I see you will have discussed lots about this. Yes. And I suppose that's the book. Yes, indeed. These are interesting issues. And they're so interlocked, aren't they?

Which is one of the strengths of the ethic today. I initiative here that all these things are coming together in a really beautiful way. Well, thank you very much, Jonathan. And now we move on to Jess. Jess Smally. Hello. Yes. Welcome back. As I mentioned, when introducing you to start with your policy, lead the data lab at the University of Oxford. And you're also doing a doctorate in the digital ethics lab.

But previously with technology adviser at the Department of Health and Social Care and the subject matter experts at NHS X x. This tongue twister and you co-authored the NHS Code of Conduct for data driven health care technologies. I love to you. Just a word about the contrast between working in the NHS and then coming and doing research at Oxford. Yeah, it's an interesting one. One one sort of led to the other, I guess.

I had been in Oxford previously to do my undergrad and then I went and worked for the NHS straightaway. I was supposed to be there for six weeks and then I ended up staying for about six years.

But I was it was when we were developing all of these policies with regards to how do you use data and in particular how to use machine learning and AI in healthcare contexts responsibly, that I sort of say I'm not really sure we have enough knowledge in policymaking settings in order to be able to make these decisions. And so my solution to that was to come back to Oxford and sort of try and generate some of the knowledge and understanding myself.

I suppose the contexts are slightly different when when you're in an academic context, you have. A lot of freedom of thought. So you can almost think in a constraint, less matter manner, and you can't really do that when you're in a policymaking context, because there are so many constraints that you have to think about. You have to think about stakeholders engagement. You have to think about interlocking policies.

And how does this interact is that you have to think about relationships between other departments. And you also have to balance quite complicated trade-offs sake, as we've seen throughout the coronavirus pandemic, you know, trade-offs between promoting public health versus promoting the economy. And sometimes these are presented as dichotomous and they're not necessarily.

But you don't have to think about those kinds of things so detailed in in an academic context, it's still it's OK to just think about one concept and think about it in a quite narrow way. So I suppose that's the main difference. If it's really interesting, I mean, you're getting both perspectives to her to a rather extreme degree.

I mean, philosophy is one of the subjects perhaps, which promotes thinking all sorts of weird and wonderful things in a strange thought experiment, perhaps more than others. And then, on the other hand, healthcare must be one of the areas where you're most constrained by considerations of cost and politics and public acceptability and sort of other things. Anyway, over to you, Jess. Thank you. I will send my screen. But hopefully you can all see that.

So I'm going to basically pick up on the point that Jonathan made right at the end of that talk in response to a question from Peter. That information does not automatically enhance a person's understanding and explore why this has implications for autonomy, in particular with regards to the empowerment narrative, with how M house tools or health apps are positioned in the world of policy making as part of an overarching health strategy.

Basically, I'm going to be the killjoy for the next ten or fifteen minutes or so, and then we can have a discussion about whether or not I'm right to be the killjoy or whether I should in fact be more positive. This is all largely based on paper that I wrote with the John F. Kennedy, who Jonathan also mentioned, called The Limits of Empowerment.

So. To start with, I think it's about it's a burden to know that the empowerment narrative is very heavily entrenched in policymaking documents throughout the NHS and has been probably for about the last 10 or 15 years. But it's become considerably more. Announced in the last five years, as we have seen the rise in the use of wearables and apps, etc., and this is a quote from the previous secretary of state said this. Jeremy Hunt, not Robin and Matt Handcock making a speech in September in 2017.

And as you can see, that latter half of that sentence is an accompanying. All of this is likely to be a big shift in power from doctors to patients as patients use technology to put themselves in the driving seat of their own healthcare destiny. And this really sums up the argument that's underpinning the empower the person narrative.

This idea that if we give people empowerment, empowering tools and we give them access to more information and we give them access to algorithms that can make their healthcare personalised and predictive and preventative, then they will be better, enhance their entire autonomy, will be enhanced, and they will be better equipped to take care of their of their own health.

This idea isn't new. It's kind of been around this idea of empowering people to take hold of their own health care has existed since about the 1970s. You can see in that graph it starts to pick up around the 1970s and it was present in sort of neo liberal House policy. But it's really boomed in the late 1990s and and the early 2000s, in part in reaction to the sustainability crisis faced by the NHS under the NHS in the 1990s, 2000.

Especially now, the population is vastly different to what it was dealing with in sort of 70 or so years ago. We have people who are much older. We have people with much more complicated health needs. People tend to be dealing with chronic conditions, rabel and acute conditions and things that require Long-Term Care rather than acute short term, very serious illnesses or injuries. And that's started to put pressure on the resources of the healthcare system.

And so this started to prompt the government and policymakers to pivot towards a proactive paradigm of care. So rather than being reracked reactive, want something? Somebody was sick. Encourage the shift towards health care that is sometimes known as P4 medicine. So personalised, preventative, putit, predictive and participatory. And it's often in that context that we hear arguments about the use of A.I.

So I will lo know everything about you and it will be able to predict when you're going to get sick and it will predict based on your digital phenotype, when all of the information around you as well as your genetics. And we will basically be able to intervene and stop everyone from getting sick. This argument, you can follow it through and becomes quite extreme in some cases, but it's particularly relevant for the use of health apps, this idea that the data is generated only on you.

It's your personal information and it's giving you personalised recommendations. And this shift and this idea between reactive to preventative medicine sort of coincided with the Internet becoming commercially available. And this led to info, liberal arguments. So this idea, which is partly the idea that Jonathan was just picking up on that the idea of giving patients more access to information and a sort of date keeper less way.

So democratising access to to information via the Internet would lead to the emergence of what's known as a generation of expert patients. And these patients would be have a greater level of self esteem. They would have they would be empowered to be active participants in their own healthcare, and it would ultimately result in them having a better quality of life.

And this boom of the arrival of M health and the Internet, the later adoption's of the Internet and the mobile web and particularly wearables, has resulted in this empowerment narrative taking off even more than it ever did before. But the problem is, in all of these policy documents, ranging from sort of early, early narratives, even from the sort of late 90s of the National Programme for I.T. right through until last year, 2000 and nineteens.

Sort of their next step of the NHS strategy is that it never actually says. How does the data empower people? We take them word empowerment to literally mean a transference of power in order to be able to, you know, be the person who is responsible for taking care of your own health. How does this actually work? And that's when we start to see that there are limitations to this argument here. You can see to the icon reflections of me as the self.

And there's my data that will. And this is why I find the context of healthcare. And I so interesting because healthcare is very physical. It's very kinaesthetic. It's like I understand my body. I can touch my body. I can feel it. I know if I have taken some medicine, I know if I've had an operation, something that you have quite a defined and a group, you can quite easily conceptualise it in your mind.

The problem with A.I. and with health tools is they're not necessarily operating on your physical self, operating on your digital self. And the idea is with this empowerment narrative is you use words. I turn the digital medical gaze. This idea of, oh, I look at my phone, my phone has told me that I took 10000 steps today, but I could probably take 15000 tomorrow. And that will make me better. Or oh, I ate too much chocolate today and maybe I should eat some more broccoli tomorrow.

It is this idea that you look down at your health, your phone, you're looking at a version of your data self, your data self highlight to you the limits of your health, and then it automatically triggers a response in you to make better health care. But there are many potential barriers to that actually working. For one thing, we don't know who set the baseline. So you can only be unhealthy or healthy depending on what the definition of healthy is.

And that is always going to be a value laden decision. There is no such thing as a neutral definition of what weight is. What about what the perfect weight is, for example, or what is the perfect diet? These things all have value laden like implications, but it's not often clear to people that those decisions have been made in a value based way. And it's also not clear to people what those baselines are. The Ten Thousand Steps thing, for example, is largely based on the logic.

It's better for people to move more. There is no real scientific evidence suggests that 10000 steps is the ultimate outcome and that the peak number of steps that people should be taking. If you take in 10000 steps a day every day of your life, taking 10000 steps tomorrow is not necessarily going to make you fitter or make you lose weight. And we have to expose the fact that these things can be politically driven.

There might be political reasons. There might be nefarious reasons why one baseline is set for another. And there also should be aware of the fact that sometimes these these baselines are biased. Often we talk about the fact that healthcare data or A.I. algorithms can be biased. We don't always talk about the fact that health care is often biased. Now we know more about people who use health care systems more. We also know more about people who have been prioritised in health care system more.

And we often know more about people who look like me rather than people who might have different different colours of skin, who might come from different ethnicities. And all of those baselines based on me get translated into em house tools. And I used to police people's behaviour who come from different contexts, who have different bodies, who have different health needs, and who have different cultural contexts.

And yet at the same time, we are still arguing that they are being empowered to improve their own health. Based on the baseline over which they had no control and you never see this happening. It's all happening on your data self. It's never happening on your physical self. And this undermines what can be called your integrity of self. So do I understand what is happening to me? What decisions are being made about me and why I am being nudged to move in a particular way?

And so ultimately, these health tools and apps are effectively policing people's behaviour, and these values become internalised constantly and reinforced by push notifications and by the Buzz's and other gamification features, all of which are supposed to be motivating but are actually making people complicit in their own self surveillance.

And you were basically enslaving people to this idea that I must improve myself constantly rather than empowering them because healthy has become institutionalised. The idea of your behavioural norms obligates people to constantly improve their house. So you are never healthy. You're always somewhat sick. You're always somewhat incomplete. And you were always supposed to be striving for continuous improvement. Oh, you can do 10 birdies. Can you do 12? Can you do fifteen? Can you run a marathon?

Now, can you run an ultra marathon? Can you eat only green vegetables and kale rather than just green vegetables, kale and maybe a chocolate biscuit? We're always trying to improve people. And so rather than enhancing people's autonomy, what we can actually see is, um, have to start being seen as hyper nudging.

They are reducing people's autonomy and altering and replacing this idea of self-improvement by libertarian, paternalistic algorithms that alter the presentation of the digital self to nudge people into taking predetermined actions which they have not themselves decided to take. And they have been made on about policing them against baselines, which individual people were not involved in taking or deciding or knowing whether they were the right ones for them.

So this comes to the point that Korina was making right at the beginning. What is the difference between procedural and relational autonomy and health apps? And this idea of empower the person are largely based on the procedural definition of autonomy, which is a very narrow definition of autonomy. And it puts all of the attention on the decision. So I have empowered a person to know that potentially that behaviour is not very healthy and that they could make a difference.

They could make a decision to take a different type of behaviour and that would make them healthier. But this has got nothing to do with whether or not that person actually has the ability and therefore the power to make that change. Let's give an example. If we take someone, you should be breaking more as you should be doing more exercise. We don't know if that person has access to a gym. We don't know if they have small children at home that they cannot leave if they need to go for a run.

We don't know similarly about people's foods where people have easy access to fruits and vegetables and fresh food, or is the only food that is accessible and affordable to them. A McDonald's and a Mars bar. We don't know all of these things. What sometimes known as the social determinants of health over which people have limited control and are really the responsibilities of public health programmes.

And you cannot be saying and suggesting that people's autonomy has been enhanced if we have told them and policed them and made them feel guilty and responsible for their own ill health through these nudging, hyper nudging apps if they actually have no power and ability to change it.

Instead, we are making that person feel is an overwhelming level of guilt and responsibility, which can in fact alter the autonomy, particularly if they don't really understand why they have been told to take this action in the first place, because it has been done on their data self, not on their physical self. And so ultimately, these things could, in fact, being really limiting on the person's autonomy because it is acting on procedural autonomy.

And really what we should be talking about is relational autonomy. So people's ability to make autonomous decisions in their wider context and in-house apps and apps have no real implications for this wider relational autonomy. But we make them feel like they do. And that undermines people's autonomy instead. And so finally, to talk to some of the points that Jonathan was making about informed consent, informed consent is at the heart of ethical care and current medical practise.

So his idea that, you know, I am an attorney, I make an autonomous decision about my health care because I fully understand the risks and I fully understand the benefits and I understand those trade-offs. And the doctor can explain them to me. And I know that the doctor has explained correctly because there is evidence proving that this works has been tested in a randomised controlled trial and that has been reported on the particular harms that I might face.

So I can make an a genuine autonomous decision that I am happy to accept the risks because I believe that they are outweighed by the benefits in the world of health apps. This doesn't happen. Health apps are poorly regulated. They're poorly governed. Very few of them have actually got evidence search showing that they worked.

This is because there is currently no internationally agreed upon standard or specific regulatory or accreditation system that is designed to protect individual consumers from the risks. You go on the App Store, of which there are 300 more than 300000 thousand apps, you will see them all making these wild claims about how they can improve your health care. Very few of them mention risks if they do mention risks.

It tends to be a disclaimer, which is a legal requirement in order to not be a medical device. And that disclaimer will say this is not a medical device. Do not take this advice to be medical advice. If you need medical advice, seek a doctor, but is often buried very small and very, very small. Small print doesn't take into account other types of harms, in particular harms and people's autonomy, such as their feelings of self access, efficacy, their self esteem.

If this app has promised me that I would lose weight and I didn't, I must be a failure. If this app is, promise me that this will reduce my pain levels. Maybe I'm just really sensitive to pain and I'm being a wuss. We don't we don't have the evidence that these things work. There are exceptions. There are some exceptions. There are some apps that are really well tested. They are reported in public. But that is actually the exception rather than the rule.

And so if we don't have the efficacy that they are actually working on, we don't know whether the descriptions are accurate. We don't know whether the risks are balanced with the potential benefits. How can somebody really be giving informed consent to use an M health to. Therefore, how can they really be making an autonomous decision? That's what they want to do. And so all of these things are concerning. But we can be positive.

You know, Jonathan has shown us that the positive, not everything is terrible. And I don't want to be a killjoy all the time. I love data and health and apps. And that's why I got into this space in the first place, because I think the potential for transmission is huge. But we need to be questioning and we need to reframe how we position these things. They shouldn't be the centre. They shouldn't be the be all and everything. And house apps are not going to be the solution to all else.

And there are a couple of things that we should be making our policy in a strategic level sort of decisions and reframing issues in order to help us think a little bit more critically. First of all, we should think about the informational environment or your infosphere as being social determinants of health. Not everyone is at is given equal access to equal information. Not everyone is given equal accurate access to accurate information.

Different people have different levels of ability to judge what is accurate and what is trustworthy. And this includes in health apps, some health apps come up with wildly inaccurate information. There a really great study. You can find it online, which showed, for example, that there are a number of apps which we're recommending people who are suffering from very severe psychological issues that they should, in fact, treat this by drinking hard liquor.

This is not a medically recommended treatment, but people don't necessarily know how to judge those types of information. That's a, quote, extreme example, but there are more nuanced examples. And potentially we should think about whether, in fact, the role of data and the role of health apps in the collecting and analytics and algorithms and all of this in the health care system is,

in fact, P4 medicine. So that personalised, predictive, participatory type medicine or whether it is in fact that better targeted at improving population has and so should we in fact shift the narrative and the level of abstraction or the way we analyse this problem from empower the person to enable the group? And finally, we should be making a buyer beware market. We should make people who are using these health apps better aware of the fact that there are limitations to their use.

We should make them aware of what evidence there is for them to work. We should make them aware of the fact of whether this app has been tested on one group of people, but not on another group of people. And we should just be more transparent and we should be holding people who make these health apps accountable to the same standards and holding them accountable to the same standards that we hold other people who make medical devices accountable.

And that is it. Thank you very much. I'm looking forward to the discussion. Thank you very much. I'm just trying to restart my video. There we go. Thank you, Jeff. That was really interesting and powerful stuff in there. Can I ask you to. From one point of clarification, so a natural answer to quite a lot of what you've said. Or start, at any rate, in responding to it would be to say, look, there's a distinction between devices that give information and devices that give advice.

And when I go and stand on my bathroom scales. It just gives me information. It doesn't tell me, you know, Peter, you ought to have your. You shouldn't have that extra drink or something. Now in it, do you think that's that's OK, that's harmless. It's when these systems advise us what to do. They're, as it were, taking things for granted about us perhaps usurping our autonomy and so forth.

Would you be happy if the devices we wore on our wrists, etc., just gave information about our heart rate and all the rest? And when combined with advice and I think that's an interesting question and it's a really important thing to to investigate. I think it's partly it's the way that that can that information is information only. What's the why were you doing it? Why were you looking at that particular piece of information?

And also comes to Jonathan's point about the fact that information can be presented in different ways that alters whether or not it has a harm on a person's autonomy. And so if I look at the scales. And I'm not particularly bothered how much I weigh. I just need to know because I'm I might be entering a competition that requires that I know how much how much I weigh. Then then that's that's one thing. If I'm looking at their information because I want to.

I do, in fact, want to lose weight. And then how that information is presented does make it quite big difference. And it's also depends on the information or context in it. Why has that person looked at it in the first place and what's their ability to understand what it means? And so one of the things we've looked at recently is misinformation online and different people's abilities to interpret it.

And we've known for a while that pro Einasleigh pro anorexia communities exist on the Internet and there's people who promote anorexia as a lifestyle rather than as a psychological illness. And on those types of forums, you see people promoting tips around how do you lose weight and how do you how do you minimise your hunger? And they will have on there things like optimum weights.

If people go and still look at information or they're using a Fitbit to tell them how many steps they have taken, even though it's not necessarily telling them what to do, they still go on seeking that information for different for different reasons. So informational, informational, behavioural seeking context makes a difference. Can I just ask you to clarify that? I mean, you you were you've talked quite a lot about how there's a lot of misinformation out there.

I think you were suggesting towards the end that there should be some sort of regulation. Now, what you might get from a regulator is here is the government's advice on optimal BMI or whatever it is. Would you be happy if the advice were all were well authenticated or as well authenticated be? Or how much is your objection to the fact that we're receiving this bombardment of advice on how we should be, how we should be and how much?

Is it an objection to the fact that a lot of that advice is bogus? And quite a large part, a reaction to the fact that I think that the information and the advice that people is giving is bogus. It's not just that. I think it's also this. It's more also to do with this idea that we're assuming that just because we've given people information, they know what to do with it and that they, in fact, can do something about it.

And it's particularly this idea that by giving people information, we have shifted responsibility for their health care from the health care system to that particular individual. And then the issue of that is that if people don't act on it and the right way and their health care doesn't, their health care doesn't improve even if the advice has been verified.

But because their context, it makes it impossible. We've framed that person as a bad user and a person who is become sick on purpose because they didn't do what they wanted or what we wanted them to do. And if you're constantly reminding people of that nudging, then internalising that value, that's really what worries me.

And just finally, I mean, a question I asked them earlier to be how much these issues knew, how much of a distinctive they are systems, because there have always been books on health nudging people to behave in a particular way. Some of the million informed social is the difference. Simply be the fact that it's on your wrist. They're nudging you all the time. Yeah, it's it's it's there is an it's an exacerbation of the problem rather than something that is completely new.

And it is because it's exactly the answer that was given earlier. It's all pervasive. You can't really escape it all the time. And in particular, it's it's not even just this that these types of behaviours and nudging and towards people's health behaviours isn't even just happening on people who choose it. So, like, I can choose not to Fitbit because actually I have before and I really didn't like the experience because it made me too aware.

But I will still go online and just be scrolling Instagram and I will see a thing that tells me I'll actually, hey, have you drunk this new skinny tea. But it is it's it's impossible to avoid. And that's partly happening because Instagram knows. Oh, right. Well, is young and female probably. She does care about how much she weighs because society is telling her that she should care about that. So I'm going to show her all of these things all the time, too. It's it's all pervasive.

And that's really what makes a difference. All right. Thank you very much. So I'm going to invite our other two speakers back now. They will. Think magically appear. Jonathan and Carina. Hello, welcome back. Carina. So to start with, we've got quite a few questions from the audience that I'm going to get my head around in the next couple of minutes and start throwing at you. But first of all, Jonathan, I'd like to hear your responses to what YESSes said.

Sure. Well, I think there are maybe some things we disagree about, but there certainly is a lot of agreement. I certainly agree. We have the need to regulate these apps if they're going to have to take such a prominent role. And Jesse's hugely well informed about the many, many problems with the apps as they currently stand. So I learnt a lot and I find it really very interesting.

I suppose one kind of general thought I have about this is concerned the worry for moving responsibility from healthcare professionals to the user. I do take Jess's concern, but this comes back to something she said at the beginning about this idea of sustainability and health care and how much the current status quo, if you like, is actually able to meet patient demands that we might want it to me.

So let's take physical activity as an example. The ten thousand step guideline, I agree that there are lots of problems with that, but there are some pretty well validated baselines for what a good baseline of physical activity police should be is a hundred and fifty minutes in the NHS. That's been very well researched and very well validated.

Now, one of the problems we have trying to achieve a good level of physical activity is that when people go to their GP, as the GP is, are having to rely on self reports of how much exercise people are doing. So let's take them health tool out of it. How honest are people giving information to their GP, particularly about valuable things like the amount they drink, the amount of excise they do things better, so they probably overestimate it a little bit.

That's, of course, going to influence the kind of advice the GP is able to give them. Well, I said a couple of years ago, coming back to the physical activity example, a group, a GP wrote a letter to the General Medical Council basically saying that I felt very uncomfortable with the fact that they didn't feel able to prescribe exercise as a medicine because prescribing excises as a medicine is very different from simply saying don't follow the guidelines.

So I wrote a paper about that a couple of years ago. Now, one way in which these M health tools could fill that gap is a you've got potentially and I understand the problems with accuracy here, but you could have a technology which reliably monitors levels of physical activity and potentially fills a gap by providing recommendations based on well validated baselines of physical activity, where there is at present a bit of a lacuna in primary healthcare provision.

So I want to present that as one way in which I think maybe the shift, the responsibility from healthcare professional to the user or the patient doesn't necessarily always have to be bad because the way that healthcare is currently set up, there are gaps and giving people when I said responsibility in the sense of holding them wholly accountable by giving them a bit of responsibility to say,

look, we need your help, guys. We need you to monitor some things that we as primary care physicians might not catch on our own because we simply can't maintain the healthcare system sustainably if we have to pay that much attention to everyone. So that's a counter to some of Jessi's very important and excellent points. Okay. Right. I think I'm I'm going to move on at this point to questions from the audience.

You've got some some very interesting ones here. But I'm going to also give you an opportunity at the end to respond to each other. Sure. So a few of these have been directed towards particular individuals. One for you, Korina. This is from Oliver. If the machine if the human machine interaction cooperation is construed as a single system in and of itself. Can you think of that? I mean, what would you say about the Highbridge autonomy that that system might have?

So I think I think the thought there is that you are distinguishing between human and artificial autonomy. But suppose you've got a. If you think of it as a single system, that's quite an interesting question about how to conceptualise the economy. Yeah, and I think it's an excellent, excellent conceptualizer conceptualisation of of the topic and also just coming back to the wheelchair example.

I think this is a very, very good example where, in fact, we have a technological artefact that is not that isn't there we don't consider separate from the individual who uses the artefact. So I would say that a lot of a lot of people were relying on wheelchairs, would not consider that wheelchair as a as I mean, obviously something something separate. But like in they would in their conception of autonomy, it really becomes part of their.

And how they move around, it becomes part of their life. So I think it's a I think it's a very, very interesting way of conceptualising it. I think there's one thing maybe that we should be wary about. And then maybe this comes back to the idea of control. It is about how much control you have about this knowledge about the system itself and how much are we dependent on, let's say, manufactures and how make the incentive structures of private firms and manufacturers.

When we when we make use of these systems, and I think it's it's an excellent idea. So just coming back to this distinction between the human autonomy and the system autonomy there. I would say this system contributes to. I mean, I would say it's a we should understand this because there is a human being involved. We should stop and say the system is contributing to the intrinsic intrinsic value of human autonomy.

And so that becomes the main. So when you've got the two together, the main focus becomes the human autonomy. And how far the machine feeds into that. Exactly. So then the machine basically becomes a becomes a tool of the human rather than a separate agent in itself. Right. Okay. Thank you. Question for you, Jess. Is there a difference between the value laden health and the values of the health care professionals the patients can't be interacting with? That's from well, really.

I mean, I think at the point there is that we're we might be nudged by health apps, but we might be nudged by health professionals. Did you see a significant difference that. Thank you for the question as a great question. Off of him, when I talk about A.I., an in-house guy and sort of pre face, what I want to say by the fact that I do actually think there is a lot of opportunity for these things to make things better.

And I I and that often when you are the the ethicist in the room, you have to be the negative person saying, oh, this is this is bad, this bad. And in doing that, you automatically set it up to make it so that analogue or the existing system is already perfect. I don't think the existing system is like the values that doctors use right now are necessarily always perfect. And therefore the apps are always bad. A bit a bit like, you know, Jonathan was saying, I do.

I do think that there, in fact, can be used in ways that are good and helpful to people. My concern is that they are you don't have to be a clinician in order to develop a healthcare app. You can just be anyone who knows how to write a line of code. And you can just be tech bro in Silicon Valley who thinks, hey, I know how to do this and will make me some money if I make this afternoon, will nudge people in this way or nudge people in that way. And we don't have any oversight over that process.

We don't know where the idea originated from or who decided that this was how I was going to design it. Right. Sorry. From the way baselines were also through to the way that it manipulates a person's behaviour through the way that it presents information. You know, it's really hard to go and see on the App Store and see who who even made it and therefore follow that that chain of accountability back.

And it's really that that concerns me. It's not the idea that healthcare systems or currently in doctors automatically good at automatically bad. But we are shifting in the way these things are and we are losing sight of the accountability change and who to go and say, why did you make that decision and have you thought about the implications and who do I as the person who's used this app and I maybe had a negative outcome.

Who do I tell that I had a negative outcome as a result of using your app so that I know it doesn't happen to another person? And that's what worries me more than anything. Right. Thank you very much. Jonathan, a question for you. Would you say more about the merging of autonomy and privacy and how this promotes an idea of A.I. exceptionalism? OK. It's a very interesting question. So the merging of autonomy and privacy, again, I'm gonna be drawing on a little bit on the book.

That's fine. You hate it. There's an awful lot in Chris's wonderful book. So I urge the person who asked the question to go and cheque that out. But something I learnt from Chris's book was that many of the invasions of privacy that are apparent in A.I. systems, and I think that's also relevant from justice spoken about these health tools use a huge amount of personal data.

Now, the way in which these A.I. systems tend to influence our decision making makes a great deal of use of that personal data. So the reason that I think the threat that AI systems potentially posed to autonomy is so pervasive and effective is that these systems use the data they have obtained by invading our privacy to target very specific forms of manipulative influence. And so already there's a very close relationship. Does it go the other way?

I mean, well, in medical ethics generally, I mentioned earlier that autonomy and privacy have been treated as slightly separate concepts. That's not wholly true at a practical level, of course, because we talk about informed consent to handing over parts of our privacy so that there's always been a bit of a link between the two.

But for the reasons I've just said, I think that becomes much more, much tighter in the context of AI with regards to exceptionalism, I suppose hopefully I've maybe answered a bit of that question in the sense that the fundamental point is going to be that the way in which manipulate manipulative interference operates by our systems is by using the data that has been obtained by invasions into the privacy. And I think that's something of a new feature of AI today.

Thank you. Back to you. Karina from from Maia, how can the difference between human autonomy and system autonomy be communicated well to the public? When we're talking about A.I. and she was particularly interested in what Carina said about how feminism much inform our definitions. Yeah. So I think a good way to communicating is first by understanding what is what is behind human autonomy and namely that has something to do with moral agency and with responsibility.

And in contrast to what we actually talking about when we talk about autonomous systems in the context of why we're talking about autonomous vehicles, for example, or lethal autonomous weapons systems. Now, arguably, these are infant or autonomous systems. So you could say maybe when they ask us to reach them, if they reach the point that we can consider the moral agents.

Then this discussion would need to be changed. But feminism comes in because feminism questions this this idea of independence. I said it was a very independence and self-sufficiency to see is a very it was it used to be a very or still is a very male centric idea. And in fact, women long knew that there was a social relationships and emotional relationships are are important. I mean, I think so. Men Vectron. But it was certainly not in the forefront of political philosophy more generally.

So feminism rejects this idea of autonomy as independence. And in the case of autonomous systems, when we talk about autonomy, that is independence. It is independence from human beings. So I think that here is a really nice connexion to be drawn. Tried to if you want to communicate it to the public. Talk about independence and autonomy and then show the concept of autonomy in the human context is much, much cheaper than just.

Thank you very much. It's really interesting. Moving on to jets now, I think this is one for you. How can an individual who fit this trim roof? How could an individual who feels that they are being subjected to guilt inducing nudges based on their data self rather than their physical self advocate for themselves? Especially if this is via NHS services. I realise you might be walking on thin ice in your sleep. I mean, that's it. That's a fascinating that's a fascinating question.

I mean, I guess my my first question is my first point is that if you have become aware of that, you are already in a position to advocate for yourself, then. Then if you you want to wear the back that is actually happening. And, you know, you can you can question it and you can say, look, I mean, this is often one of the things that we have these discussions about and explain ability and I and II and why explain ability is important because you can say actually,

I don't think this is right. So if you can say I don't think this is right because of X, Y and Z, you know, like the OP is telling me that I've got pain care and actually I have yellow hat like this. You can point out the obvious differences just by the very fact that you're aware and you can sort of champion it that way.

But the and I think this is really interesting argument to bring out in a really good thing to be having a discussion about in public, because I think often what happens is you get the reverse of this question and you get this idea that doctors are being presented by patients who say, hey, Dr. Google told me this was what was wrong. And you disagree with Dr. Google and I believe Google over year, which probably does happen.

But I also think increasingly that the situation that you're describing happens as well. They're being told to do one thing, whether it be a decision making, an automatic triaging tool. You go do this or you go do that or something that is actively acting on what you should do in your healthcare and no patient not feeling as though they have any put any means of questioning it. So find the specific differences point. Point them out.

If that person or the NHS services are still not listening to you, then find somebody else who can. And if that's not possible, then you should be making a fuss about it. And and I say, may you do any kind of public advocacy if you really want to do it, try and do it in public. But it's to interest rates. A bit of a worry here I have about the very concept of enco the very concept of autonomy. And I I was I was thinking when Korina was speaking that this this might be an issue.

And also, you know, what you were saying about the Health Care Act. Now, suppose I go to a doctor and I ask the doctor how I should be treated and the doctor gives some advice. And I want to know why. Can you explain it? Now, it might be to explain it in any terms that I can understand. We'll just completely distort, distort the facts of the maths. You know, so much research going into this, so much understanding of the physiology and the biochemistry and all sorts of different things.

And it's not clear always that getting more information increases your autonomy. Right. You're in it. For most of us, being getting all the medical information that's relevant. Would just completely confuses. And it may be that maybe that the concept of autonomy is a rather problematic one. You know, we tend naively to think that we're autonomous when we fully understand the reasons for doing what we do, something like that, and we do it with a knowledge of those.

But actually, in any really complex scenario, like in medicine, we're always relying on the authority of others, only a vast majority of it in. They. Is that a serious problem with the concept of autonomy? Should we just be making do with. Partial autonomy. I'd just be interested in your views on that. All of you. I I can maybe speak to that first. This is something I talk a fair bit about in the book that you mentioned at the beginning, Peter.

I think the medical ethics. There's always been this view that, look, of course, we can't expect autonomy to require full understanding of the sort that you're talking about because you would simply overwhelmed patients with information. So the concept we tend to invoke in these discussions is that of material information. So the kind of autonomy that's required for informed consent requires that the

patient understands information that is material to the treatment decision. Then very controversial question is, well, how do we define information? That's material from information that's not. And that conversation that's been largely carried out in the legal domain and in the past five years or so, we've got quite big changes in the law in England, at least regarding how we define information that is material to treatment decisions.

So I don't think it's an insurmountable problem. I don't think the concept of autonomy for use in medical ethics or for that matter in a I. Has to be wedded to this idea of full understanding. Of course, that's not obtainable. But that doesn't get us around all the very difficult questions. And we're certainly seeing this play out in medical law in England today. And I don't think the discussions about that are over. Right.

Just do you want to add anything on that? I mean, our largely largely agree. I think I think it's complicated. I think one of the things that we mentioned in the paper, which I haven't mentioned, is also as sort of a more metter version of autonomy and the like. Making an autonomous decision to give over the decision making ability to someone else is still an autonomous decision, as is the autonomous decision to not want to know.

So I don't want to know that you've predicted that this has happened and this has happened, and I still think that's worthwhile. So like Jonathan just said, I don't think it's an insurmountable problem. I think it becomes problematic when we don't look at the way it's potentially being harmed critically. And there I take it there would be a crucial difference between deferring to an authority that's well authenticated.

And rightly respected. And so basically going to the doctor, as opposed to taking your advice from some app that's been written by someone in Silicon Valley who may or may not be reliable. Yes. At the moment. Yes. I don't say that that it's not impossible that we could get to a point where we have enough trust and reliability in the other definition of like autonomous, autonomous decision making that we would feel comfortable doing that.

And also, we should be aware, like the point I made earlier. Not all doctors are fantastic all the time. So we should we shouldn't make make this this distinction between the always perfect human and that always imperfect algorithm. Those those two extremes don't exist. But we should be critically analysing both. I mean, presumably it's not impossible. We get to a stage where technology.

Is in general more reliable than the doctor, precisely because the very best decision making algorithms can be put into it and reproduced all over the place in lots of handheld devices. Completely, completely possible from from. Especially from a very cold technical definition either. There are many other which we don't have time to talk about today. But there are many other aspects of housecat like empathy and my ability to understand you as a person, which in an algorithm cannot advocate.

But from a purely technical perspective, yes, that's true. Yes, actually, that Anna asks a couple of questions. Well, I mean, one of which is rather resonantly. How does this metric cultural obsession of the empowered, quantified self. Converse with the almost mystical discourse around wellness? Interesting. And can the researchers speak to the gender specific gap in algorithm design for health care?

The well-known fact that most algorithms apparently based on male data rather than female data storage historically. Do you have anything to. Oatmeal. Does any. All Korina. One is Duncan sorry, my room connexion is somewhat shocking. So I understood half of the question, so maybe I'll pass it over to two jets and then make a comment after.

So the fact that the first question about this sort of forever quantified, quantified self and the mythical example of wireless, I think those two things go hand-in-hand because the wellness in this industry is booming on this idea that you can how to fly and data fly every single aspect of your life. And this was the point I made about this fact that you're made to feel as though you're almost constantly in somewhat sick, in the sense that you're always somewhat suboptimal.

And there is an out there somewhere an optimal version of you that you can achieve if you continue to listen to this app. And then this happened and this apple, you know, I'm gonna follow this influencer and they're going to tell me that I should only drink kale and then that I do yoga five times a day. But then at the same time, you'll have that a different person on a different app telling you that you should go running 100 miles a day and only eat red meat.

It is constantly over bombarding people and that is what's driving the profit making. And I think we should be very conscious of that. And we should accept the fact that maybe we don't want to like how to quantify every single aspects of our lives. Now, how do you how do you quantify the very beneficial from a health mental health perspective and therefore a physical health perspective? Things like having a good conversation with a friend or laughing.

These are all things that are aspects of the human existence that make you feel better. But we shouldn't necessarily be quantifying them. And I think we should be therefore very critical of this idea that you can quantify every aspect of wellness. No doubt. Right. Well, thank you very much indeed. That's been a really interesting and enlightening discussion.

What's the food for thought there? Just a shame we have to stop because I think I feel we've raised a lot more questions than we've answered, but it gives us all an appreciation of what a complex and multifaceted topic this is. The session's been recorded. It will be added to the very rich collection of A.I. ethics resources that we're building up at Oxford.

As I said at the beginning, you can find links to past and forthcoming events, the growing set of recordings and podcasts categorised by topic. If you go to the philosophy faculty page and click on the ethics, you know I link. Before saying goodbye, I'd like to thank Wes Williams, Vicki McGuinness and the whole team. Torch for helping us with the organisation and technical arrangements for this seminar. They've made everything much easier for the four of us. And this is hugely appreciated.

Thank you. Thank you very much to the speakers. Of course. Carina, Jonathan and Jess being really, really interesting. Interesting what you've said, and I'm sure our listeners will have enjoyed that. Thank you, listeners, for watching. This is for watching. I'm not just watching our viewers, especially those who ask questions. The really interesting questions, it's a shame we didn't have longer to spend on them to look out for our future events.

From the link I mentioned. Thank you again and goodbye.