Your phone can use tiny skin-colour changes to measure your heart rate - podcast episode cover

Your phone can use tiny skin-colour changes to measure your heart rate

Jun 03, 202618 min
--:--
--:--
Download Metacast podcast app
Listen to this episode in Metacast mobile app
Don't just listen to podcasts. Learn from them with transcripts, summaries, and chapters for every episode. Skim, search, and bookmark insights. Learn more

Summary

Discover how your smartphone's front-facing camera can unobtrusively monitor your resting heart rate using machine learning to detect tiny skin color variations. The discussion covers the technology, its accuracy across skin tones, privacy safeguards, and the broader potential for accessible health monitoring. Additionally, the episode revisits Richard Feynman's half-century-old mathematical solution to the classic 'restaurant dilemma' problem, exploring its modern testing and broader applications in decision theory.

Episode description

In this episode:



00:57 How your smartphone’s camera could measure your heart rate

Research article: Liao et al.



08:55 Research Highlights

Nature: A star gone rogue tears through the Galaxy

Nature: Gold keeps glittering courtesy of surface chemistry



11:04 Should you try something new in a restaurant? Maths has the answer

Nature: Feynman solved the ‘restaurant dilemma’ 50 years ago — now a study confirms his mathematics


Subscribe to Nature Briefing, an unmissable daily round-up of science news, opinion and analysis free in your inbox every weekday.

Hosted on Acast. See acast.com/privacy for more information.

Transcript

Intro / Opening

F

Meet Shark Cryoglow, the new medspa inspired mask with under-eye cooling IQ LED technology using high-powered LED and deep infrared for visible results in just eight weeks. Better Aging Treatment boosts collagen and is clinically tested to reduce fine lines and signs of aging. Skin clearing provides chemical free acne treatment on and beneath the surface, and even visibly tightens under eyes in one use. Developed with dermatologists for radiant results at home. Learn more at sharkninja.ca

🎵 Music

I

When you picture the First World War you probably think of barbed wire, artillery fire and mud.

H

What you probably don't think of? Drag shows. Love letters pass between those in uniform. Queer lives hiding in plain sight.

I

In our new series, we're looking at the stories that didn't make it into the official record.

H

Because they were too tender.

C

Dangerous to preserve.

I

Canadian soldiers performing in drag at the

H

To forbidden love in the trenches.

I

But we don't.

H

Carbon.

I

And I'm...

H

And this is Lavender Ranks. Listen wherever you get your podcasts.

🎵 Music

D

Creators launch, grow, and monetize their podcasts everywhere. Acast.com

🎵 Music

C

In an experiment.

H

Why?

E

Like so far

I

No idea.

C

But now the data's peak.

🎵 Music

A

Nature.

🎵 Music

Welcome and Heart Rate Monitoring Intro

E

Welcome back to the Nature Podcast. This week, how your smartphone's camera could measure your heart rate, and testing Richard Feynman's solution to the restaurant dilemma problem. I'm Benjamin Thompson.

🎵 Music

E

First up on the show this week, reporter Julie Gould has been finding out about a way that a person's mobile phone could be used to quickly and unobtrusively measure their heart rate.

Smartphone Camera Heart Rate Technology

B

Resting heart rate, literally the number of times someone's heart beats in a minute while at rest, is a pretty simple number to calculate, but it carries a lot of information about a person's cardiovascular health. A declining resting heart rate over time can signal improving fitness, but a consistently elevated resting heart rate can be associated with cardiovascular disease, diabetes, and even early mortality. So, knowing if this rate is increasing can be useful.

D

Having the ability to to be notified if you see trends are different. Your heart rate resting heart rate is different. It has been increasing over the last couple of months. That could trigger some further investigations as to why. Is it stress buildup or is it something else going on in your health?

B

This is Ming Zo Po, a staff research scientist at Google. His goal is to simplify taking these measurements so that these warning signals can be picked up faster and, if need be, treated sooner. But clinical measurements of these numbers can be time consuming. It requires dedicated time and effort to go to the doctor or to take a few minutes out of your day to rest, measure and make a note of it.

There are devices already out there that make it easier, like smart watches which track resting heart rate without any effort from the user. But these devices cost money, require charging and need to be worn consistently. A large portion of the world's population, particularly in lower income countries, simply doesn't have access to them.

But what if something that is ubiquitous could do the job? Well, this week Ming is part of a team who have a paper out in Nature reporting a system that estimates a person's resting heart rate simply from brief videos captured while they're using their phone. It's called PHRM or passive heart rate monitoring and it uses machine learning to measure resting heart rate during everyday smartphone interactions without the need for using a specific app.

D

PHRM is designed to run entirely on your smartphone device. And every time you unlock your phone to use it, to check your email or to browse the web, it activates the front facing camera to capture a very short video clip, about eight seconds. We train a deep neural network to analyze the video frames captured in this short video clip to infer your heart rate.

at that moment. And at the end of the day, we can aggregate all these intermittent readings throughout the day to estimate your resting heart rate.

B

The technology being used by the front facing camera is called remote photopletismography.

D

Every time your heart beats. It generates this pulse wave that propagates through the blood vessels and changes the blood volume, including the vessels that are underneath the skin of your face. So blood absorbs specific wavelengths of light, primarily around the green, yellow spectrum. So these are the subtle color changes that the front-facing camera or the smartphone camera is analyzing to estimate your heart rate. Human eyes can't really pick up on these subtle color shifts.

Because the changes are on the order of something like one percent. of change of the baseline. But a a digital sensor is able to by measuring the the underlying raw pixel intensity and tracking the changes over time.

B

Ming and his team then trained a neural network to estimate the heart rate from the videos using these minute colour changes brought about by pumping blood. But real world smartphone use introduces huge challenges. People move, they walk or they sit in cars and lighting is constantly changing.

So to address these challenges, the team built in automatic image stabilization and cropping to ensure a person's face was the focus, and they also allowed the system to assess how good it thought each video was.

D

The way we train them the deep neural network is to instead of just allowing the network to output a single heart rate number, we also allow the network to output a confidence estimate. So in other words, we allowed the network to express uncertainty. So if conditions were really bad and inconducive for a reliable heart rate measurement, the output confidence would be low and the algorithm could then discard that unreliable measurement. We conducted a real world

study with participants using their personal phones in unconstrained conditions. And these two aspects are important, I'd say, both the personal phone use and and the unconstrained used.

B

In this part of the research, these two factors were important as they didn't want the participants to feel like they were being watched by the researchers. So in this experiment, the team collected more than one hundred sixty thousand videos from one hundred and seven participants using twenty six different smartphone models, and this gave them a wide variety of conditions to test their system on.

Ming and his team then compared the passive heart rate measurements from the study to clinical standards, which it was able to meet when measuring daily resting heart rates passively. Right.

D

We set the target of achieving a mean absolute error of under five beats per minute. And the system was able to produce those accuracy targets across three skin tone groups in our study.

Accuracy, Privacy, and Future Goals

B

But while the team was able to measure resting heart rates across all skin tones, the system still encountered issues.

D

We d deliberately try to recruit and over represent darker skin tone groups in this in this study. Because one thing we also observed was that the measurement success rate, which we defined as the number of video measurements that return a sufficiently high confidence level from the PHRM model to be deemed valid was lower.

in the darker skin tone group. This meant that there's a lower number of valid high-rain measurements across the day for the darker skin tone group. So that's an area I think for further investigation and improvement.

G

Yeah.

B

And there are other things to be considered too. The idea of a smartphone camera silently monitoring physiological signals obviously raises privacy concerns, so Ming and his team have recommended strong safeguards for the future use of the PHRM technology. The most important is making sure that informed consent is given, as well as PHRM only being run locally on the device and kept separate from the rest of the phone.

D

You run it in an environment that the video frames are not accessible by other apps. So it's a safe, secure environment to execute the inference on. And we discussed tying it to face authentication.

A

So that

D

Yeah, the system would be smart enough to make sure that if you pass your phone to somebody else, it's not going to measure their hard rate and attribute it to you instead.

B

While the team's results show that the system was able to measure a person's resting heart rate to a high accuracy, From a medical device perspective, they're careful not to overstate what these measurements can reveal. For example, the simple snapshot heart rate measurements cannot infer someone's psychological state.

A spike in heart rate could mean many things, you're feeling a bit anxious, you've had some coffee, or you've gone for a short run. So the system is not ready for prime time yet, and there's still a lot of testing to do before it can be rolled out. But heart rate can be a useful indicator of potential health conditions, and for Ming, the broader goal is making health monitoring almost invisible and available to everyone.

D

This was an example of the potential of what we could do with devices that we already own and how we can make health monitoring as as seamless and invisible as possible. So I think if we if we do it in the right way, I mean the responsible way, I hope that Yeah, it can really make help tracking easy for everybody and make it accessible.

G

Very well.

E

That was Ming Zerpo from Google in the US there. To learn more about the heart monitoring system, head over to the show notes for a link to the paper.

Galactic Wanderer and Gold's Luster

Coming up, should you stick to your normal order in a restaurant or should you try something new? Mathematics has the answer. Right now though, it's time for the research highlights with Dan Fox.

G

A rogue star seen streaking across the cosmos was probably hurled by the black hole at the center of the Milky Way. Astronomers analysed light emitted from the star using the dark energy spectroscopic instrument, and plotted its trajectory using position and velocity data from the Geyer Space Telescope. They found that this sun-like star is unusually rich in heavy elements, suggesting that it comes from an intense star-forming region such as the galactic center.

Given its trajectory and current velocity of 314 kilometers per second, the team suggests that it's likely to have been catapulted at high speed when its partner star was captured by the gravitational pull of the black hole. The authors say that this hypervelocity star could provide a way to study the chemistry of the inner galaxy, which is usually shrouded with dust. Get drawn into the gravitational pull of that research published in Astronomy and Astrophysics.

Atoms on the sparkling surface of solid gold spontaneously arrange themselves in ways that prevent oxygen from tarnishing the metal. Gold is not completely inert. Its atoms can form compounds of oxygen and other chemicals, but in its bulk form, the metal tends to avoid chemical reactions, and in particular, the oxidization that helps to make copper green and iron rusty.

To understand why gold never loses its luster, a team used supercomputers to assess the surface chemistry of gold crystals. These crystals are known to be composed of atoms arranged into cubes. But in some cases the atoms on the surface rearrange to form hexagonal patterns instead. The author's calculations showed that while square arrangements help oxygen molecules to split on contact, the first step to oxidization, hexagonal patterns make that less likely, helping to keep the gold shiny.

Findings, the authors say, that could help researchers to design gold based catalysts. You can find that gold standard research in physical review letters.

Richard Feynman's Restaurant Solution

E

When I go and see my mum, we often get food from a terrific restaurant that's around the corner. I always order the same thing, it has to be said, vegetable bolti and sag paneer, since you're asking. And I love it, but I do wonder if maybe I should be less of a creature of habit. There's loads of good stuff on the menu, so maybe I should switch it up and get something else. It might be better, but then I might not like it as much.

And it turns out there is a mathematical solution to this mouth watering conundrum, and one that has quite a story together. History. It's the subject of a new research paper this week. And joining me to talk about it is Nature's Davide Castel Vecchi, who joins me now. Davide, hi.

C

Hello, Ben. Thanks for having me.

E

So I'm not alone, Daviday, in this restaurant situation. Now, the story in this case starts in a Thai restaurant in nineteen seventies California.

C

Correct. In Glendale, California, which is a community right next to Pasadena, where the California Institute of Technology is located and where Richard Feynman, the legendary, some would say notorious Nobel Prize winning physicists was teaching. He and a friend, they were regulars at this restaurant. And Feynman's friend Ralph Leaton wondered, you know, should I keep getting ginger chicken?

Because I like it, or should they be, you know, adventurous and explore the rest of the menu? And Feynman being Feynman, pulled out a notebook and started calculating. And soon enough he had a mathematical solution. He had like formalized the problem into what we would call a decision theory problem.

E

Right, and other people call this a stopping problem as well, right?

C

Correct. And picking your best dish at the restaurant may not be the most consequential decision. of your life, but it's a very similar problem that we encounter in many other situations in life, like for example, when you shop around for a new home or new place to rent When do you stop? Like at what point do you think you're satisfied enough as opposed to keep searching?

E

And so Feynman then jotted down an answer to kind of figuring this out. And the story continues then, right? I mean tell me about this solution that he wrote down.

C

So the solution for many years was only known to Feynman because his handwriting was really difficult to read. But Ralph Leaton kept the notes. And many years later, in twenty thirteen, Tom Griffiths, who is a cognitive scientist at Princeton, heard about this and became interested in he actually went through the painstaking work of transcribing the whole set of notes.

E

Wow. I mean longtime listeners to the podcast will know that my handwriting is absolutely atrocious as well, and my colleagues can't read it. And trying to figure out what someone means can be absolutely maddening. And was this pages and pages and pages of notes, or was this a very short equation what were we talking about?

Testing and Broad Applications

C

It was only a few pages. In fact the solution is quite simple. But what Tom Griffiths subsequently did with two collaborators is analyze the mathematics and also generalize it. And they confirmed that Feynman's solution was kind of the optimal solution, at least within certain assumptions. And not only that, they also recruited more than two thousand volunteers to play basically an online game and test whether their choices would be consistent with this solution.

E

And this is what this new paper then that came out this week is all about, right? It came out in PNAS. How did they go about testing this? Stick or twist problem

C

So they rephrased the problem a little bit. Instead of you visiting the same restaurant every evening, let's say you're visiting a new town, you spend a week there. And you have the option of visiting different restaurants every evening. You have your own rating for each restaurant every evening and the goal of the game is at the end of the week to accumulate the highest possible score. Now if your first evening you already hit a restaurant that is a ninety nine out of one hundred

Then obviously your best solution is to keep going to that same restaurant every evening. But if you start with, say, a fifty on the first evening, then you know you're probably better off trying a different one the second evening. The odds that being adventurous pays off, go down, however. So for example, if you haven't had better than a sixty,

on your fifth evening, then chances are that the last two evenings you are going to go worse. So you better stick to your sixty for the last two evenings.

E

And could our listeners go to a restaurant with a calculator and try and work out their best options, David? What's this equation look like that these results show is the kind of the correct one for this situation?

C

Yeah, in principle you could imagine visiting a new town and having, say, seven evenings, seven choices to make. And it's a simple calculation. You know, you enter the number of remaining nights, you take square roots and you plug it into this very simple formula that tells you what score you have to get that evening. to make you want to change.

E

But of course I guess there is the X factor because we have this idealised situation where you can work out your square roots. But sometimes you just get bored.

C

Yeah, so this particular model doesn't take the boredom into account. I mean, chances are if you visit a town for a week that you probably wouldn't get too bored if you order the same dish two nights in a row.

E

And more broadly then, David, of course we have this unusual situation, this thought experiment, which did have a mathematical solution. Does it have broader applications, do we think?

C

Well the field of decision theory and more broadly game theory is one that economists have been using in applications for a long time and it's been the subject of multiple Nobel prizes for economic And results like this you could imagine marketing experts applying them to say, for example, you have a restaurant chain and you want to figure out how often or for how long

to have a special dish on the menu, you know, for a limited time only, because maybe you have modeled people's behavior and figured maybe after a month they would get bored of it.

E

When you've been reporting on this story for Nature, David A, do you think it will change your ordering behavior when you go to your favourite restaurant, Little Beast Road, somewhere in London?

C

That's a good question. I also tend to o order the same dish at my favorite Indian restaurant here near our nature offices in London. Um I don't know, if nothing else I will probably keep thinking of Feynman and getting amused whenever I encounter this dilemma in front of a menu.

E

Wonderful Davide, we'll leave it there and we'll put a link to your news story in the show notes. Davide Castelvercke, thank you so much for being here.

C

Thank you very much, man.

E

And that's all for this week's. If you've enjoyed the show, do let us know. You can leave a review on your podcast app of choice, or you can reach out to us on social media. We're at Nature Podcast. And we're on email too. Podcast at naturepod. I'm Benjamin Thompson.

🎵 Music

F

Meet Shark Cryo Glow, the new medspa inspired mask with under-eye cooling IQ LED technology using high-powered LED and deep infrared for visible results in just eight weeks. Better Aging Treatment boosts collagen and is clinically tested to reduce fine lines and signs of aging. The skin clearing provides chemical-free acne treatment on and beneath the surface, and even visibly tightens under eyes in one use, developed with dermatologists for radiant results at home. Learn more at sharkninja.com.

J

Do you keep hearing podcast ads like this one for example, but always wonder how you actually get involved with them for your own brand or organization? We're Acast, and we give you the platform to do it all yourself. Browse thousands of popular podcasts, choose the shows that match your perfect audience, set your budget, and launch. And if you want a hand, Our podcast specialists are there to help you launch with confidence. This is podcast advertising without barriers. Get started.

Acast dot com slash advertise

This transcript was generated by Metacast using AI and may contain inaccuracies. Learn more about transcripts.
For the best experience, listen in Metacast app for iOS or Android