Abbey Road Studios Spatial Audio Forum

Hello, and welcome to the Sound on Sound podcast on our People and Music Industry channel. I'm Hugh Robjohns, SOS Technical Editor, and today I'm talking with Mirek Stiles, who is the Head of Audio Products at Abbey Road, but he also heads up the Spatial Audio Forum, which is what we're going to be chatting about today. The Spatial Audio Forum was started back in 2017 to bring together the expertise of artists, producers, engineers, manufacturers, and academics from the worlds of music, film, broadcast, gaming, and virtual reality. The idea was to exchange research and the results of practical experimentation, finding practical, creative approaches and solutions for recording and mixing in three dimensional sound. Forum members currently include academics from the Universities of York, Imperial College London, and Huddersfield amongst others, boffins from the BBC, from Waves Audio, Magic Leap, Qualcomm, and creatives like composers Stephen Barton and Andrew Dubman amongst many others. So, hello Merrick, thank you so much for joining us today. I guess the first question I want to ask is, how did you get involved in this forum in the first place? Um, I got involved off the back of Abbey Road Red, which is our tech incubation programme. Um, it's actually Europe's first music specific tech incubator. Um, so we work with entrepreneurs, startups, and help them in any way we can with their development. And off the back of that, I was introduced to the world of VR, um, and AR, augmented reality. And something just sort of struck with me. Um, we were Working with a company called Ossic at the time, who had a pair of headphones that had head tracking enabled and had an array of small speakers that went around the outside of the ear to give very realistic 3D sound over a pair of headphones. And I was just fascinated by this. And I started to explore a little bit more about sound for VR and AR. Um, and that, that led me into the world of, of, uh, real time engines like Unity and Unreal Engine. And it just kind of, I just went down the rabbit hole from there, really. And what I quickly worked out, though, was that there was, uh, lots of different players involved, um, in terms of personalities, um, lots of, different software available, different formats, people were sort of saying the same thing but using different, uh, language and terminologies, and it just felt like really, really wild to me. It's like, you know, I described it as the Wild West, really. It, it, and I was finding it very hard to digest it all. So, just through going to various, Um, kind of trade shows and meetups. It just made sense to me to try and get some sort of forum together to sort of try and demystify this all a bit. Um, and kind of sort of work our way through it together. And it's been really successful and I think we've all learned a lot from each other. I like that Wild West description. That seems very appropriate from what I've seen of this so far. But, um, I'm sure we'll remember if you go back You know, 20 years actually to the, to the 2000s, there were all these music surround sound formats. We had DVDA, we had SACD, we had a lot of, um, DVD videos coming out with, with surround tracks, uh, for music performances, concerts and things. But over the sort of 10 years or so, it pretty much died a death and DVDA has completely gone now. So what is it that's changed? Why are we now looking at 3D immersive audio again? I mean, I was, uh, I was an assistant engineer on some of those remixes that were done back in the early 2000s. Yeah, there was a bit of a, um, bit of a wave, wasn't there, going on with, with a lot of concerts being remixed and re and released on DVD with, uh, 5. 1 soundtracks. And the labels really got behind it and really invested in it. And, uh, I mean, I think the biggest one I was involved in was the, they remixed The Beatles Anthology in 5. 1. And it was, like, really good fun. Um, I, I, you know, some of those mixes sound absolutely phenomenal. Uh, the problem is history always repeats itself. So, uh, the quad was introduced in the early seventies, for example. And so you're kind of expecting people to add these extra speakers into the room. And there's usually like a. A door in the way or bookshelf in the way or something. I mean, people's front rooms really aren't cut out to have multiple speakers around the room, really. It was just inconvenient for the consumer. So, yeah, as you say, it kind of went away. But where we're at now is a couple of things. One, we've got the concept of sound bars. Providing, um, things like 5. 1 or 7. 1 or even Dolby Atmos using side firing and up firing speakers, using the walls and the ceilings to sort of reflect the, the surround information, uh, which, which does actually work remarkably well. There's even, at the moment, there's early versions of binaural soundbars as well, which actually track the user's heads and beam direct binaural audio to each ear, which is very effective. Um, it's, it's certainly not mainstream at the moment, but I think it will get there. So anyway, my point being is that it's convenient. You no longer have to have these speakers all around the room. So there's that. And then on the other side, you've got like binaural rendering over headphones has really come into fruition over the last uh, few years, people are really starting to pay a lot of attention to it now. And again, it's convenient now. It's over headphones, it's convenient, and there's not this need to have all this extra equipment. So, I think where we're at now is very different to where we were in the early 2000s. I certainly think the binaural thing is a strong one because as you say, everybody's going around with headphones on these days. And if you can make accurate binaural That's going to be a very popular thing. But the problem there surely is in, everybody's hearing is slightly different, and you have these head related transfer functions. Everybody needs their own kind of mapping to make it really accurate and effective, don't they? So everyone, because everyone's anatomy is slightly different, what the HRTF that's provided in whatever format you're listening to surround sound on over headphones, it's, it's from a fixed, generic, uh, ideal. Uh, having said that, it does work remarkably well for a large proportion of people, but there will always be exceptions to that. And, and as you quite rightly say, everyone's head shape is different, ear shape is different. Nose shape, torso, everything's different. And, and the head related transfer function, that's what it's based on. It's, it's, it's kind of the size of your ears, the, the delays you hear between your ears, the shadowing, the masking that your face provides between the left ear and the right ear. But there is the concept of, of um, providing your own HRTFs, um, and I think that's the way it will go. There's early examples of it now, for example the Sony 360 Reality format, um, does enable you to take photographs of your ears and upload those photographs into the app and then it will give you a personalized HRTF off of those photographs. Wow. does make a difference and again it has to be convenient. I mean, I've had my HRTF measured in an anechoic chamber and I was sat on a bicycle seat for like the best part of two hours being turned a degree at a time and it was just horrible. So obviously people aren't going to do that. Um, but from my experience, having the personal HRTF can make a big difference to the sense of spatial immersion. Um, but I think that's the way it's going to go. I mean, just reading between the lines on, uh, things like the Sony, uh, PlayStation 5 and their Tempest audio engine, um, it sounds like that they will eventually go down the route of providing HRTF because that their motto is, um, 3D audio for all. So I think they're pushing the kind of whole 3D audio over headphones thing because everyone's got access to headphones. Uh, I'm sure Dolby Atmos will have their own version of HRTFs eventually. I think. Eventually, everyone will probably go down that route because it does provide a benefit. But having said that, what I'm hearing at the moment is still really, really good. Do you think the idea here is just to give a sense of immersion within an acoustic environment, or is it about the accurate location of sounds in space around you? Well, it's a mixture of the both, really. Um, you kind of want to get that, that sense of direct, um, uh, sound from all around you, but as with anything, really, that always sounds nicer when it's in a, in a beautiful sounding environment or beautiful sounding room, so of course you want to emulate that also. Um, so it's a mixture of the two. Hmm. Okay. So some of the early surround sound formats were based on on individual tracks, like the 5. 1 and the 7. 1, you had dedicated channels of sound. And then they extended that and we have height channels. And then we got Dolby Atmos, where you have more height channels. Where do we go beyond that in terms of creating this sound? Because I keep reading about things like higher order ambisonics, where you've got dozens and dozens of channels. I mean, how practical is this? And how far do we actually need to go? Yeah, so ambisonics and the beauty of ambisonics is that you can convert your ambisonics to any channel array you want. So you can convert ambisonics to Dolby Atmos, which works really, really well. Or you can convert it to 5. 1 even, or even down to stereo if you want. But yeah, the concept of ambisonics is having a microphone with, um, many capsules. So first order ambisonics has four capsules. So you're capturing sound direction from, um, top to bottom, um, left to right and front to back. And then the high order ambisonics, um, is, is more capsules and, and hence more virtual speakers in, in the playback. So I mean, third order ambisonics is probably, um, the point where it's practical and sounds really good. Quite a few of the doors Um, allow you to work in high order ambisonic workflows, so Pro Tools, for example, Reaper. But I mean, it, it, what it effectively means is you've got 16 channels of audio, for example, for, for third order ambisonics, which I guess when you think about it, it could get impractical, but I mean. Pro Tools and Reaper seem to handle it pretty well. It's just all your plugins are 16 wide plugins instead of like a stereo plugin, for example. Um, and there are plugins out there that are working in high order ambisonics. So I don't think it's impractical. It's could be considered a steep learning curve, maybe. And a bit out of people's comfort zones. I think that's more of the issue. So how do you go about, uh, if you've got an orchestra in Studio One at Abbey Road or whatever, how do you go about recording them in three dimensional sound? It depends on what your output is. I mean, so for Dolby Atmos, What's quite common at the moment is, is to have height, um, microphones placed in the room. So, for example, in Studio One, there's, there's four, um, microphones hanging off the ceiling above the orchestra, which is obviously great for putting in your height channels in Dolby Atmos. Mm hmm. And, I mean, beyond that, what I've been experimenting a lot with recently is this concept of six degrees of freedom in, in real time engines, game engines, if you like, like Unity or Unreal Engine. Um, so, the six degrees of freedom Effectively, you know, your head can turn left to right, it can tilt up and down, it can move left to right, your body can move up and down, your body can move left to right, your body can move forwards or backwards, and the idea being is that the audio in your headphones will change depending on what your body's doing, which direction you're facing, and where the sound is coming from. So, um, how we record for that, or the experiments I've been doing, um, the last recording I did was in Studio 2 with, uh, composer Stephen Barton, who you mentioned earlier on, um, and Dr. Hewn Cooke Lee from Huddersfield University. We recorded a string octet in Studio 2, which is, for those who don't know, it's the famous Beatles Room, um, and they played, um, Ellen Rigby. We've set them up in the round as it were. So, uh, the violin's in front of you and then the viola and Shelly behind you. And we still have spot microphones on each of the desks. 'cause, 'cause you need that, that clarity. Mm-Hmm. if you like, as you, you do in any recording. But we also had an, a high old Ramas, so microphone, an igon microphone in the middle of the room. And then around the outside we had clusters of spatial arrays to pick up the, the, the room ambience, um, and kind of the spatial environment. So the idea being is when you mix all that. Um, create your stems and import those into a game engine. Um, you have a graphical representation of Studio Two and a graphical representation of the instruments. You attach the close microphones to the instrument graphics. You have the ambisonic microphone in the middle of the room and then towards the outside of the room. You kind of have these spatial arrays, so effectively what you can do is you can walk through the performance as it's happening in virtual reality. So if I walk up to the strings, I get, I can almost hear the bow on the string if I stick my head really close. Um, or if I can stand in the middle of the room and I hear the strings all around me. Or if I go and stand in the corner of the room, I hear more of the acoustics and the spatial elements of Studio 2. As you would if you were actually walking through that room as the performance is happening in real life. Mm hmm. Um, so you can create these different sound environments in different zones and, um, I've got a representation of what Studio 2 looks like and, um, in Unity, for example, there's an asset store where you can download assets and someone had made graphics for violins and cellos, I don't know why, but they have and it's great, uh, it makes my life a lot easier, so I could download those assets and use those in my environment, I could even download things like Persian rugs, someone had made some Persian rugs, so you can make it look nice and, Mm hmm. And it's just a proof of concept, really, of where I think maybe this could go, you know. Artists and producers can, you know, represent their art, their work, in ways that just really weren't practical in any way. So, you know, you can walk through the environment, you can walk around the environment, you can, um, interact with things, and, and all the sound is head tracked and immersive, and it's a very unique way of experiencing music, I think. Yes, yeah. So, essentially, you, you create all these different outputs from all the different microphones, and the engine itself is mixing them in real time, as it senses you moving around in this, uh, Yeah, that's correct. Yeah. And going back to why we put the forum together in the first place, I mean, there are many different ways of doing this. Um, I mean, to make a recording engineer's analogy, I suppose, I mean, Unity, Unreal Engine, there's others. Um, it's the equivalent of like, I like Pro Tools, um, you like Logic sort of thing, they're essentially doing the same thing. Um, and then it's the same with the sound rendering scripts, they call them, these scripts, I guess we could call those plugins, for example, in the game engine. So these plugins that, um, allow you to control the sound and render the sound over headphones. Um, so there's various companies that do that as well. I personally think some are better than others and some give you more user options that I think a recording engineer would like to, to see. I mean, it's easy to forget that this whole world, if you like, of real time engines has come off the back of gaming. That's what they were designed for. So, um, they've kind of got that, that kind of gaming language and workflow if you like. And so to come. add it cold from a recording engineer's point of view, can be quite a steep learning curve. But having said that, I think it is getting better and I think it will get easier and, um, some of the language will kind of merge and it will make a lot more sense, but at the moment it's, it still feels a little messy. And then of course you've got, you know, the stage before that which, um, where you're in your door creating these stems to import into your game engine. And again, there's loads of different players there. in terms of plugins, ambisonic plugins, spatial audio plugins, some of those companies also provide plugins that work inside the game engine. So there is some sort of consistency, but also not necessarily. You might be using a set of ambisonic plugins in your door from one company and then be rendering those ambisonic files via another company within the game engine. So there's a few inconsistencies still going on. But like I say, there are companies like Blue Ripple, Dear VR, who do both. Um, so again, I think it's getting better. It's becoming a lot more streamlined. Like I say, it still feels to me like it's a bit of a steep learning curve at the moment, but I just want to encourage recording engineers and producers and artists to start maybe exploring this. stuff because I, I really do think it offers amazing opportunities for kind of, um, presenting your art from a better way of putting it. Like, you know, it could be the next step of concept albums, um, where you're in the artist's world. And, and it really, I mean, if you have a beautiful graphics and creative coding, uh, and of course with great sounding audio, only the imagination is the limit there, I think. Hmm, it's an interesting idea. Are artists taking this on, uh, as a way of presenting their material? Yeah, I mean, there's a few examples. I mean, Bjork's, um, she's got experiences in VR that have been absolutely fantastic. Suga Ross did something with, um, Magic Leap, so that there's early examples out there, but it's still, at the moment, it still feels like early days to me with regards to Creating music experiences using real time engines, um, but they are out there But um, I think we're going to see more and more but like I say, it's um, it's early days perhaps I remember when 5. 1 first came in and some mixes used to put the the main vocals alone in the center channel. And I remember some artists getting a bit upset with that because they felt very exposed. And it strikes me with this, that you're not necessarily presenting a finished mix that they can sign off on, because if people can move around, they can change the balance themselves. Which must have pluses, obviously, in terms of they can choose to focus on the bass, or the drums, or the keyboards, or guitars, whatever it might be. But it must also be a challenge for the artist, knowing that there isn't a single mix that people are going to listen to. Yeah, that's an interesting thought. I mean, it is a fixed mix from the point of view is that Um, you know, your assets aren't going to change, so you can still sign off on the stems, if you like, that are going to be used in the real time engine. Um, but yeah, I mean, you know, the audio is obviously going to be a lot more exposed, depending on where you decide. Or how you decide to interact with this experience. I mean, you can walk right up to the vocals, for example, and get a very, very intimate experience of that vocal recording. You'll still hear the other instruments around you, but because you're closer to the singer, you're going to hear more of that. And perhaps you'll hear imperfections, um, But that's kind of, I guess that's the nature of art. It's not, it's not perfect. Uh, I don't know, people are starting to be more comfortable with that, I think, maybe. I mean, artists, uh, are engaging a lot more directly with their fans anyway. There seems, like, everyone's guard seems to be down a lot more. Uh, anyway, I think it's the direction it's all going in. So, um, as long as you're happy with the assets that go into the real time engine, I don't see any, any, um, issues there. But yeah, like I say, it's not going to be for everyone. Yes, yeah. Um, some artists might be horrified at the idea of having their work presented in this way. But then again, other artists are going to love it, and they're going to embrace it. So, how much work is currently going on at Abbey Road that involves a major step towards capturing 3D sound? In terms of Derby Atmos, I mean, Abbey Road's been at the forefront of Derby Atmos for quite a while now. Uh, we obviously, we record a lot of film scores here. So there's always a need to capture height information for the dubbing engineers. Um, we actually have our own dubbing stage now, uh, Dolby Atmos, um, Premiere Accredited Mix Stage. So, we, I think we're the only facility in Europe that not only you can record your film score here, but you can do your final film mix in the same building as well. Um, and then we've got the, uh, Penthouse Studio, um, which is all set up for Dolby Atmos, and we're working with Universal Music, our parent company, uh, remixing a lot of the catalogue in Dolby Atmos. So Um, in terms of Derby Atmos, we're, we're full systems go, as it were, for the kind of more experimental things, uh, for the real time engine stuff, um, I mean, that, that's, that's pretty much me on my own, sort of in my spare time, um, dipping my toe in the water, as it were, um, I mean, not, not, Strictly my spare time, but you know, I mean, it's not my day job put it that way I mean, I work with companies like waves and China limited and spitfire audio and we create plugins and sampled instruments and hardware based on some of our historic intellectual property and Acoustics of our rooms. So yeah, I do a bit of this on the side to be totally honest It sounds like a fascinating job though It is! Um, yeah, No Day is the same, that's quite for sure. Going back to this concept of creating music in real time engines, it has been frustrating. All the information is out there, and I do encourage people to look into this, because you can download Unity and Unreal, you know, for free. Start playing with this stuff. So, so you can get your hands on Unicy, you can get your hands on Unreal, um, there is just a plethora of, of videos on YouTube, how to do this, how to do that. I mean, it is a steep learning curve, but once you start, it does slowly all make sense. Yes. Having said that, the information's out there, but it's just so, um It's all spread out, um, and also another, another issue I find is that even if you do find a fantastic tutorial video on YouTube about how to do whatever in a game engine, um, the game engines are updated quite regularly. So you might find that you're doing a tutorial and it just isn't working. Chances are it's because it was created with an older version of Unity, or an older version of the VR plugin you're trying to use. And it can be very frustrating, don't get me wrong. Um, but the information is out there, um, and I, I think I personally would like to, um, start working on ways to maybe get it all together in a nice neat package, um, but I'm still working it all out myself, like, so, so, but we will get there. Good, well that sounds very interesting. And we'll put some links to some of these things on the support page for the podcast here so that people can try and follow those up themselves. Yeah, I mean, yeah, fantastic. Um, I just definitely recommend, um, if anyone's listening to this and thinking all that sounds interesting, then if you go onto, um, Unreal or Unity, their websites are, they're really good. It's just, uh, it just might feel initially a bit alien, people coming from a traditional recording, um, background. Um, but yeah, the websites are really good and, and the tutorials are really good. So if somebody wanted to start experimenting with this, okay, they can, they can download the engines as you say, but in terms of, of physical recording equipment, what would you say was a minimum that people need to make some sort of headway into this? Do they need an ambisonic mic, for example? Well, this is the kind of interesting thing, really. I mean, you know, you could take any mono recording you've done, i. e. just stick a Shure SM58 or whatever in front of a guitar amp, record that, you could take that mono recording, put that into Unity, attach it to a graphic of a guitar amp, put on a pair of headphones, and you could walk around that guitar amp as it's playing. It would sound kind of basic, but you could, you could, you could set it up so that you get more high frequency content in the front than you do in the back. You could change how, how wide the sound is dispersed within the game engine. There's things you can do to kind of trick the ear into making it more immersive without having to spend loads of money on expensive ambisonic microphones, etc. Okay. Um, so you can make it work with. with pretty much any assets. I mean, for example, I took a four track tape recorded here in the mid 60s, um, just as an experiment. I think it was, uh, it was Rod Stewart singing in Studio 3, um, with Jeff Beck. Um, so it was a four track, so there was vocal on one track, guitar on another track, the bass and the drums were kind of on the same track, but I could just separate them out a bit of filtering, but I could still Take those assets and put them into a real time engine, attach them to graphic objects, and still walk through that audio. So, there was, there was, obviously that wasn't recorded using ambisonic microphones, so. But there are there's there's ambisonic microphones out there like first order ambisonic microphones that I think they come in a sort of you know Around that sort of just well under under a thousand pounds. It's still quite hefty I suppose I mean the third order of the high order ambisonic microphones are still very expensive But like I say, you don't necessarily need those. That sounds very encouraging. I hadn't realized that you could just start off with With a single microphone in effect and still put it into a space that it makes sense when you say it But I hadn't thought of it in that in that way so it's quite an appealing relatively easy thing to at least experiment in if if not to create your um, Your debut album. Yeah Yeah, I mean it like I say it's it's It's all out there, and it's, I think it's all within everyone's grasp. It's just finding the time and the, uh, the patience, if you like, to, to get over that, that initial hurdle. But I, I think it will get better, it will get easier. Um, yeah, still, still kind of early days in the grand scheme of things, really. And, and with your forum, you, you're pulling a lot of people together into this. Is that helping to people to, to use the same language and Make compatible plugins and technologies. Is that all you know, it's the forum doing what it set out to do? Yeah, absolutely. Um, I think it's helped everyone on the forum immensely. Sort of start using the same language, um, we do workshops, sort of show and tells, uh, kind of, you know, showing what experiments we're working on. And it is starting to come together. Um, it feels a lot more coherent than it certainly did a couple of years ago. Um, so, so yeah, it's working, absolutely. Um, I think probably, you know, the next step is to try and, you know, start, start getting some of this language out there, maybe via tutorials, that sort of thing, uh, to try and streamline it all a bit, so it makes more sense to people. But we're, we're still ironing out a few bits and pieces ourselves, so we don't want to do anything too early. Mm hmm. So, what's a realistic time scale, do you think, where this will become a mainstream thing? I really don't know. Um, I mean, it's, it's happening already, really. So, for example, the concept of, of audio, um, and music in, in real time engines, you know, it's, it's, it's been happening particularly more recently, um, in the game Fortnite, for example, where, where, you know, concerts, um, are happening in, in this virtual world, um, where users can go and, and see, um, like a concert from artists. So you're kind of in this virtual world experiencing music with, with avatars, um, from, from, you know, from artists. So that it's, it's kind of happening. Maybe the state of the world and the situation we find ourselves in has It's kind of amplified that and sort of brought it all forward a lot more quickly than anyone anticipated. So, but, but the foundations are being laid, definitely. Hmm. Excellent. Good stuff. Well, I'm going to go off and see if I can download the Unity engine and, uh, and see what funny noises I can make. It's been a pleasure talking with you, Merrick. Thank you so much for spending so much time with us today. Thank you very much. Cool. Thank you for having me. Thank you for listening, and be sure to check out the show notes page for this episode, where you'll find further information along with web links and details of all the other episodes. You can also find out what's playing on our other channels by going to soundonsound. com forward slash podcasts.