All episodes of the Breaking Math Podcast are available commercial-free on Patreon at patreon.com-forodslashbreakingmath, starting at the $5 a month tier. This week we are featuring special guest Levi McClain who joins us to talk about the mathematics behind music, audio, engineering, sound, neuroscience, and the human sense of fear. Without further ado, here's this week's episode. What makes the sound scary? Well, I was curious.
I built an instrument specifically designed for horror music, and then I proceeded to rescore the entirety of the 1922 horror classic, Mous Faratou, in order to figure that out. Without further ado, I want to introduce our guest Levi McClain. How are you, sir? I'm doing fantastic. Thank you so much for having me on.
Levi, I am sorry. I'm ecstatic to have you on. I can't believe you're here because I literally just saw your content and said, hey, have you ever thought about artificial intelligence applications to this? And you responded. Thank you. Yeah, of course. It's always great to talk about things just outside of my own discipline. Thanks for reaching out.
Absolutely. Absolutely. Yeah, so you have a whole lot of content. I watched almost all of it, and I tried my best to reduce it into something that we could talk about in this one episode. It's a long outline. So, so who knows, we may have to do this in a two-parter, but that's okay. I think that Levi and I have been on the phone preparing for this episode. I want to say for about three and a half hours, would you say that that's probably correct?
Yeah, that's probably about it. We've had some good times on the... Oh, yeah. Yeah. So, that sounds weird. Before we... That's all good, man. We've had some good... No, we have added... Don't be ashamed of that. There is no shame. In our bromance, Levi, I think it is a beautiful thing. I'm just kidding. I'm just kidding, dude. Yeah. So... It's a beautiful union. Yes. Thank you, sir. Thank you, sir. That's how... You know, a sharing Brotherhood of Science and Knowledge.
And for our listeners, I already talked about this book. I actually got Levi a copy of this book. I think you ended up buying it yourself. I owe you for that, but next time, right? I'll buy you a few drinks. Yeah, this is the book that... We kind of base the discussion on, which is a brief history of intelligence by Max Bennett, where in this case, we will zero in on audio... audio engineering.
Before we dive into the content, I have a quick, awesome story that illustrates the power of audio engineering and knowledge in that area. As I mentioned earlier, I had many conversations with Levi about the science of audio engineering and machine intelligence. And what I loved is that we kept asking each other questions back and forth, and had an opportunity to share our knowledge.
I was on the phone with Levi when I was driving to this studio, and I was walking in. I said, hey, Levi, I'm in studio now. And Levi said, I know. And I said, how... What? I thought, you know, are you following me?
Or something? Or are you tracking me or something? It left me really weirded out. And basically, Levi explained, and I'll let him take over this point in a second, he basically explained that the changes in my own voice from when I was in the car to when I was walking on the street, when I walked into the studio, he could tell absolutely just based on the echoes and what he heard in the sound itself, that freaked me out.
Can you tell us a little bit about that? What was it like hearing me? Yeah, yeah, sure. So I guess this is probably something that it seems like a super power of concert. It's really not. I think most audio engineers end up developing something like this, because we're sitting around and we're mixing sounds all day, and we're in studio.
So we get really accustomed to listening to the room of a sound, because that changes so much of the sound of the instruments that we play, you know, a singing in a cathedral is going to sound vastly different than singing in a little bedroom closet. So we've kind of developed this keen ear for being able to hear things about the room, certain qualities like I can tell generally speaking about how large a room is, just with my eyes closed, you know, if I walk into it.
Also with that, we can kind of be able to discern certain sounds of the material of the walls, you know, what is the structure that we're in made out of, and it's this really interesting and kind of cool things that humans can do that we, I don't think we utilize very often, but in talking to you, I immediately thought of the applications of how can we implement something like this into artificial intelligence.
So like would it be possible to take like a microphone, put it in the middle of the room, train it on a bunch of different rooms, and then from the ambient sound of that room, could a computer, could an artificial intelligence machine learning system be able to identify qualities about that room. I think that's a really, really neat thing that I don't think has been explored yet.
That is absolutely incredible, and I think you're right, and that's, it's funny because these conversations about how you can garner or how you can gather information about your surroundings forces us to realize there's so much that we don't know, and that we don't know that we don't know part of that a bit of an awkward phrase here, but that's absolutely true.
And you spoke of audio engineers who learned that skill over time, and suddenly it just became aware that your brain could pick up patterns in these things, and you learn to associate them with that. You suddenly have this ability to know information about your surroundings. It sends a bit of chills down my spine. I'm not going to lie, but at the same time, it's absolutely amazing.
You talked about an AI application with this, and basically, so an AI hypothetically, we could trade an AI with the technology we have in microphones to pick up a wide variety of sounds that are outside of what humans can hear, and based on your own experience on what you can decipher just from the frequencies you're able to hear, as you said earlier, forgive me for just restating what you said.
I like to do that. It makes me feel smart. You could learn the size of environments. You could learn properties of the materials. It's incredible. And I think the example we use is, you know, like if I choose a nice, you know, sunny day scenario, I'm not talking about a terminator scenario here, but a nice scenario where you've got a robot that is responsible for being something like a firefighter robot, and the visual spectrum is blocked from smoke or from debris.
And makes you wonder what information it could get just from audio signals itself. And yeah, you know, you brought up the possibility that there's just a lot that you can get. I don't want to go to down this rabbit hole, but I brought this up with some friends of mine who talked about this is a very developed science in areas such as so darn what, like with submarines and things like that.
So yeah, there's a lot of research in various areas on sound. So it's amazing. Now, without further ado, I'd like to actually deep dive into your talk on, on, on fear. That was the video that we opened up with. And can you tell us a little bit about, tell us the story of why you decided to focus on fear. What, what was the process like when you put that script together, both in terms of your research and just your inspiration. And we saw you completely destroy a violin.
Do you feel sorry about that at all? Not at all. The violin had it coming. So I originally started this idea when I was watching the 1920s horror film, news for Ato, which is it's like a knockoff Dracula. Film from the 20s. It's a silent film. And I noticed that something interesting about it, which is that I didn't find it scary whatsoever. And all the people I was watching with also was like, this is just kind of boring, honestly.
And, and then I was kind of doing some research into it and then reading a little bit about horror film history. And I came to understand that the original soundtrack for the news for Ato film was actually lost because the, I guess the widow of Bram Stoker, who was the original author of the Dracula films, sued the director for news for Ato. And there was a court order to like destroy all the movies. And so basically all the copies got burned and the original soundtrack was lost.
I thought that was a really interesting opportunity to explore this idea of scary sounds in a way where I can, I can see if I can make news for Ato scary just through sound. And, you know, also kind of have a chance to, you know, rescore a really iconic movie. My gosh, that's, that's, that's incredible. That's, that's cool. So, quick question here. Have you submitted that YouTube that you made to any artistic competitions or any, you know, doctuary?
Okay, there's gotta be, and if there's not one, I will make one. I will reach out to my colleagues in the Department of Education, or I will find colleagues there, and I will insist that we met at some conference. And I'll say, you have to have, you know, an award show for educational content. Let's make it a thing. Okay, note to self. I'll do that. Okay, so that video has a lot of neuroscience that you discuss. And actually, I watched the entire video, and I filled up, I want to say six times.
I want to say six pages of a notebook with every term and every concept that you put in it. As you mentioned, some specific mathematical concepts. I'd like to bring those up here right now. If you don't mind, I'm gonna scroll on my fun outline here, and we'll go right down to it. Okay, so you talk a lot about what do humans find that is pleasing and calm, and what do they find that is frightening or disturbing, or things that would cause our, our, our, our,
our, our, our, our nervous system to react. Is there a mathematical description of that kind of thing? Yeah, so, um, the way I like to approach it is, you know, most topics in psychoacoustics are quite complex because sound is so multi-dimensional. So there, there's different ways to describe a sound and, and kind of all of it comes together in its entirety to describe a sound as a whole.
So with that, with that video and that, and this project, I was trying to figure out more so like, what is there something specific? Is there any one dimension of those, of, of this quality, the sound that contributes more so to something being frightening than, than anything else? And what I, what I came to understand is specifically in certain types of fear, mainly, quick fear. So there's quick fear and there's slow fear.
We'll discuss quick fear. Maybe we can, we can jump on the slow fear a little bit later. One of the qualities in, in the anatomy of a sound, if you will, is called roughness. And now roughness is a, basically a description of how quickly a sound rises in volume and amplitude.
And if it has a high degree of roughness, typically humans will assign that kind of scare, that, that kind of sound to being scary. So in conversation, roughness will fluctuate, you know, like three to five to maybe ten times per second. And then in, in something like a human scream, it's much more. So like you're talking 30, 150 times per second or something like that. And so we can kind of start to classify certain sounds mathematically through that one dimension in as scary or not scary.
And so that's, that's one part of the puzzle. So that's fascinating. One of the things that I wondered about, and I think you and I either talked a little bit about it where I wrote in the side notes on my notebook is, okay, so our ears working a way rather, our ears plus our brains and our essential universal system, differentiate the frightening sound based on the, the hurts or, you know, the amount of amplitude changes.
And I think from the video you said, the scream is literally between 30 and 150 hurts in a scream. It made me think about all the sounds that we hear in nature or that any, any living thing would hear in nature and how unique the scream is. And possibly how it evolved to be a unique signaling thing that there's danger, you know what I mean? Like it's just a very unique thing. And I ask this because with machine learning machine learning learning learns certain things based on a reward function.
And well, first of all, machines don't have a self preservation instinct machines aren't worried about predators chasing it. So, so it's reward functions are a little different. So, what that said, whether it's machine learning or biological things, you learn things for a reason. And in the case of say Darwinian evolution, you learn things like the reward function is you get to keep living and produce offspring.
And, you know, the punishment function, if you will, is obviously you get into the, danger or die. So, I don't mean to be morbid here, but that's, that sort of explains the, the, the form and function of, of, of fear.
And I just found it interesting that it's that we specifically evolved to, to hear, you know, this characteristic of sound. And it makes me think like, would we have identified something else as scary if the circumstances were different? Like what is scary is a learned behavior, at least I assume more so than an absolute, you know, this is scary, you know.
Yeah, yeah. So, I mean, you bring up a couple interesting points there, which is that one, I think a lot of this understanding what makes something sounds scary and maybe just fear in general, maybe that you could apply this out to, to encompass a wider, wider range here, but specifically in talking about like fearful sounds.
I think it comes down to anything that breaks with expectations. So, in bioacoustics, this would be called a nonlinearity. So, if you are in sticking with the, with the example of a, of a human scream versus human talking, if you're in a room with a bunch of people and you're talking, and then someone screams in the back room, that breaks from expectation.
So, you don't, you don't expect to hear that in a, in a crowded room. And so, everyone's attention is drawn towards it, right? So, that, that's an example of a nonlinearity. It's essentially something that breaks from expectation. That's right. Yeah, and actually, I am, yeah, I'm sorry, please continue. I just cut you off like a, like a T-bone car act.
Oh, you're good. You're good. Okay. Okay. Yeah, that would definitely, I tried to scare you there just to show it, you know, yeah, I'm just kidding. Okay. Now, in the video, I didn't even think of this. I didn't bring up a diagram. There's beautiful diagrams in your video. There is phenomenal pictures. And for that reason, maybe our listeners should just go watch your videos so they can see your great diagrams.
That said, there's a whole section on a fear short circuit that is based on when your brain, or your, your ears in your nervous system, here a sound that it has identified as the trigger for a fear response. Can you walk us through the short circuit, sorry, the fear short circuit process in our brains? Yeah, sure. So, it's essentially a five neuronic acoustic startle circuit is what it's referred to.
And again, this goes into the kind of fear that's immediate, the scream at the corner of the room, which articulates your attention towards it. So, what's basically how it works is, you know, pressure wave, sound wave will hit your ear.
And they'll go to from your ventral, sorry, I have a, we'll note here, a ventral cochlear nucleus, which is encodes intensity and temporal information. So, where the sound is coming from, it'll then go through your lateral amnesus to your particular pontine nucleus. At this point, that sound has articulated your eye movement and has brought your attention to the direction of the scary sound.
And the sound wave then travels through spinal internarons to your motor neurons. And at this point, you are fully aware of the threat and you have completely reacted. And what I think is interesting about this is, is that it's such a fast process, right? And it only takes about 10 milliseconds to articulate your attention towards something that's fearful. And what's interesting, I think, is when you juxtapose that with visual information.
If you look at something like a, you know, maybe a scary bear or, you know, someone coming at you with a knife or something more of it like that, it's a much slower process, even though, you know, light travels faster than sound. It's very interesting.
You'll see the threat and then your brain has to go through a decoding process to be able to recognize that it's a threat. Remember that, like, oh yeah, bears are scary. We should have, avoid them. And then, you know, your brain has to decode all this information to make you aware and present of the threat. It's because of the visual process is a chemical one. There's a chemical conversion that has to happen with light, whereas the five neuron acoustic startle circuit is a very quick process.
It happens immediately because it's a mechanical process. And it's only has to go through five synapses. So that, yeah. What I really enjoyed about this video is exactly you, man, you do a great job of breaking it down from you basically said earlier, the slow fear response and the five of the fast fear response.
And you broke it down systematically as an engineer. I love this because as an engineer, I think, oh, okay, as I'm seeing this, I see the process. I could literally code this myself if I wanted to. I could make my own robot from this if I wanted to. So all my engineers who are enjoying this episode, check out the video and think about how would you encode a fear response?
You know, if you have to for an assignment, another bonus assignment. Here I am assigning homework, right? Is how would you train machine learning to do it? You'd have to identify a reward function that then identifies things to, shall we say, fear or avoid or to follow and create criteria. I'm going to blaze through this part real quick. So I just love all the detail here.
So basically says the there's a fear response short circuit that is described in the video includes the ventricle cochlear nucleus, which encodes intensity and location to the origin of the sound. Look at that. That's the first point. I'll stay it again. Ventricle cochlear nucleus and the bits of information that it includes. I shouldn't say bits.
The information that include that it encodes are intensity and location. So you have to have those two things first. But then the signal travels down and it goes to a piece called the nucleus of the lateral limesquist that I pronounced that correctly. Yeah. Okay. Okay. Okay. I told you guys I took crazy notes while I was watching this video.
And then it goes to a part called the reticular pontine and sorry, the particular pontine nucleus where the eye movement and the attention is coordinated. This is amazing. It makes me think about how much science was done on this process to where we literally boil it down system by system.
But further, it literally talks about how information from one sense is integrated with with other senses and decisions are made in this short cut function. Short cuts are interesting because when you have a short cut of information, you can make a rapid response. But you also don't have the full processing and the full decision making potential of the entire brain. That's another conversation altogether. But we're talking about fear here. So it makes sense.
And then it goes on to say that the processing comes later. And to that point, that's one of the core things that I like to do whenever I'm explaining something or understanding is like breaking it down so it's replicatable.
And I think that's so important because for me, you can tell me some information. But I'm the kind of guy who's always like, but why? But I want to be able to build this like it's Legos. And I think it's really useful to explain processes in the brain whether they're audio or otherwise in that way where it's like, oh, well, we have the ability to, you know, we have this idea of like, oh, this is what we think happens.
It's five neurons that does this. Well, then you can actually replicate it and you can actually test it and then have a have a very fine green understanding of certain processes and brain. I think you could not have explained it better. And I think that's what a lot of engineers are thinking about when you have a, should we say, a satisfaction of your own understanding. I don't remember in my own life, there was a point where I realized that I think I was inspired by Richard Feynman actually.
I don't know how familiar you are with Feynman. He has a story in his book, sure, you must be joking, Mr. Feynman, where somebody just off the cuff asked him, he said, hey, when you take a strand of spaghetti and you bend these spaghetti, why does it always break in two places and not one place. And Feynman thought about it and he could not let go of it. And just for his own satisfaction of understanding, he obsessed with that question for a long time until he had an answer.
I can give you the spoiler if you want or I can save that for your homework. What do you guys think? No, I want to hear it. Okay, okay. Okay. So, so you know, you can look up all kinds of stuff. You talk about when you add tension to a piece of spaghetti, where is the pressure going and where is the is the instability in the spaghetti strand.
I need to let you know real quick, for disclosure, I had this conversation with an engineering consultant of mine, so I need to confirm this. Originally, I was thinking that it had something to do with the mathematics of nodes. And I'm sure as a sound engineer, you know what nodes are.
So, for example, if you're a child and you have a jump rope and you're spinning it, you can make one giant node or if you spin it at a certain frequency, it'll split into two nodes, you know what I mean? Or however many nodes. My assumption originally is it had something to do with the energy going through it as you attempt to break it.
So, I mean, if you want to do a node causing instability, that turned out to be wrong. I was just making it up as I went. So, you got to watch out for smart sounding stuff that just made up lots of folks do it. According to my boss, it was simply that if you put one end in a vice and you break the other end or you bend the other end, it'll break at one point.
So, as you are applying pressure from two hands, that's two points where you are applying pressure. The energy is kind of similar to the nodes in the sense that there's two fracture points. Sorry, long conversation. I wonder, how long talking about that? And we were just obsessed with this problem, that's engineering for you.
Also, what I think is a human advantage is that our striving for engineering isn't simply one explanation for a phenomenon. Human beings in our, I'll just go and say what we think of as our exceptionality, maybe other creatures do this too.
But our level of understanding that we strive for is that can we rebuild this phenomenon and can we describe it in with that amount of fidelity, that we can recreate it? That's why I made this show in part because I want to do that with a lot of the unknowns and artificial intelligence. So, yeah, we'll get onto that later. Now, once you have a systematic and kind of a scientific understanding for the fear of spots, you really know it very, very well.
You can then apply your knowledge of a fear of spots to something really cool, like making a scary movie or a soundtrack, which is exactly what you did. I wanted to ask you about some of the things that you went for specifically, like what sounds you chose, what you peppered in there, and then like, what was your strategy in making a scary sound or scary music for this clip?
Yeah, so there's a lot of ways you can go about it. And having this knowledge of inputting non-linearity is basically gets you scary sounds. There's a lot of things you can do with that as a composer. So that involves some distorted sounds. So if you distort sound, you're pushing up that quickness to get to a high amplitude. So that's the roughness quality that you're really taking advantage of.
So I definitely use that, but also some composition techniques, because a non-linearity, if we take a step back and just think of it as breaking from expectation, we can do certain things with that. So that's where the idea with the breaking the violin and turning it into this Frankenstein instrument kind of came into play.
Part of that is to play on the movie a little bit, because it's essentially the early horror movies. You have Dracula, you have Frankenstein, the mummy, and those kind of things. So it's a little bit of a hark and back to that, but it's also, I think plays into the soul of the music a little bit, because what is Dracula, what is news for out to at its core?
Well, it's a perversion of beauty, right? So it's something that's deeply human that's been twisted into something into some villainous monster. So to do that to a physical instrument, I think plays on that pretty well. And what you can do then is break the instrument, rebuild it into something a little bit perverse.
And just from that, your ears are very used and accustomed to hearing a violin as it is, which is this highly engineered structure, this instrument that we've been using in Western classical music for years and years and years and years and years. To hear it in this new context, your ear will automatically pick up on, oh, this sounds different. It sounds like a violin, but it's not. Nusferatu looks like a human, but he's not.
And so that plays on the fear a little bit. And then, you know, we can also do certain things that play with dissonances, so that's taking chords that and, you know, adding these strange notes that probably shouldn't be in there in these interesting ways.
And then also like putting certain things like the melody, if you put it way up high in an instrument, you're pushing the tolerance of that given instrument and you're really leaning into the fact that, oh, this is not where this instrument is usually played.
So if I'm playing at the very top range of a violin, it's discomforting to play as a player, but it's also discomforting to experience as the listener, because listeners, you know, we have a really adept ear at being able to intuit discomfort, you know, whether that's in music or in life. And so just playing on all of those things kind of kind of goes into the mix when it came to this project.
Awesome, awesome. I have a lot of fun. I just have a lot of fun with using elements that, that can, you know, cause fear or comfort when I'm doing my own art. It's kind of like, you know, whenever I'm designing a Halloween costume or I'm writing a script or trying to create a scary story, I know it's just really fun. And I don't think that I think it's pretty common knowledge is you specifically said that this part of making something scary is the subversion of expectations.
But also that is often a moving target and what's scary in one time will then, you know, populate that knowledge will become a populated more in a, I'm sorry, I'm not saying that right. Once people become familiar with something, it has a diminishing effect on how scary it is. So I think you said in the video that when this movie first released, I think it was banned somewhere, right? What was it banned?
I forget which country it was banned in, but it was, it was, it was banned in the country. There's a bunch of reviews, you know, with these things, it's like, oh, it's a horror movie so we assume that it's going to be scary. And so like a lot of what I do is, I don't, I usually don't trust my, any of my intuitions at first glance. So, so one of the first things I did is, I went back to see about reviews at the time to see like, wait, was this, it's not scary to me now.
Maybe it was never scary. So I need to double check that. So I go back and, you know, check reviews and what people were saying at the time. And people thought it was very, very scary, which, you know, plays into what you're saying, because now I would say that most people would not find it as scary as maybe a modern horror movie.
So you're right, the target does move. And that kind of plays into the idea that like fear is more than just the immediate quick fear of a jump scare sound. It, it, and that plays into into the idea of this slow fear concept.
So a lot of what we find scary changes over over time, over generations, as we get used to certain conventions, like the idea of a Dracula has been played out so many times in so many movies that you really need to up the game in the modern era in order to kind of reclass this idea of, of scaringness.
Oh, and I just realized earlier you talked about the entire process of destroying that violin and making a Frankenstein instrument. We have that video as well. I want to put that video in before we go any further in the outline. Can we, can we pull up the making of the Frankenstein violin video? I built an entire horror instrument in order to figure out what makes something sound scary.
Now, if I asked a composer, they might say it has something to do with the timbre of the instrument in concert with certain compositional techniques, appropriately placed dissonances, stinger chords, and things to do with tension and release. An acoustician by contrast might examine the anatomy of a scary sound, and observe a high degree of roughness in their waveform. This is an acoustic property which refers to the rate at which the amplitude of a given sound changes.
A property high and not just scary sounds, but also human screams. The neuroscientists might note that some types of fearful sounds are purely mechanical process, oriented and actuated by a five neuron acoustic startle circuit embedded in our brains. And the psychologists? Well, they might discuss how our relationship with fear changes as we understand ourselves and the world around us better and better through the decades, implying that some fears are of our own making.
I say it's a really complex question. Perhaps a full 30 minute deep dive into the complex realm of psycho acoustics is in order. So, hey, and look at that. That's exactly what I did. So, if that's what you're interested in, go check out my video, what makes the sound scary over on my YouTube channel in the link in my bio, hoping to get past a thousand views on this one that would be nice. So, any support helps.
And that is all for part one of my interview with Levi McClain. We had so many notes that went into this episode that we realized we were forced to separate it into multiple episodes. Please join us next week at the same time on the same channel for part two of the interview with Levi McClain. We will finish talking about music theory, about neuroscience, about artificial intelligence, and the attempts to use machine learning to attempt to decipher whale language.
And again, it will be the same time, same channel next week, and that will be part two.