John Hubbard 00:00
Support for the blueprint podcast comes from the SANS Institute. Are you a new blue team member or SOC analysts looking to make sense of the vast variety of data that you see on a day to day basis? Are you working in a SOC that's plagued with alerts that are constantly bothering you with false positives or repetitive action? If so, I know that pain, I remember being an analyst and walking into the SOC and thinking everything was going to be great, and then realizing, oh, we have some serious problems here. I've dealt with those problems over the course of my long career and the results of what I've learned have been summarized and my brand new course sec 450. Blue Team fundamentals. If you're a new SOC analysts looking to get a grapple on all the network and host data that you interact with, improve your analysis technique. Learn how to reverse engineer malicious file types that you commonly see in phishing or downloaded from the internet. Or just get rid of those pain points from the average SOC like repetitive actions drowning in tickets alert fatigue, sec. 450 is the class for you. Check this course out at sansurl.com/450. Hope to see you in class. This is the blueprint podcast, bringing you the latest in cyber defense and security operations from top blue team leaders.
01:13
blueprint is brought to you by SANS Institute, and is hosted by SANS certified instructor, john Hubbard. And now, here's your host, john Hubbard.
John Hubbard 01:23
Hello, everyone, and welcome to the blueprint podcast. On this week's episode, we have Ryan Chapman, we are going to be discussing the very very interesting topic of malware. If you've ever wondered how to reverse engineer malware or why you might want to do so we cover that in so much more on this week's episode. So today in the podcast, we have Ryan Chapman, a SANS instructor, Blue Team course author and malware specialists. I'm super excited to have Ryan on the show today. We met a couple years ago, I think it was at a sim summit if that's correct.
Ryan Chapman 01:58
Yeah. It was SANS Scottsdale, yep!
John Hubbard 02:00
yeah. And met Ryan, before we got involved in SANS officially since then I've been fortunate enough to see you at a number of events, hang around, pick up some malware analogy along the way. And one of the reasons I wanted to have you on the podcast and one of the things I liked about Ryan for that, for the listeners that aren't familiar with him is he brings this intense energy and knowledge to the topic of malware. And if you've ever seen any of his discussions or talks, you know what I'm talking about. So I was hoping we could have some of that energy coming through and enlighten everyone on a topic today that I think is really scary for a lot of people that are considering getting into it, which is malware, reverse engineering and kind of malware in general. So thanks for joining me on the podcast today. Ryan, to start us off. Let's hear a little bit about you. What kind of brought you into the world of blue teaming and malware analysis specifically?
Ryan Chapman 02:49
Yeah, heck yeah. First off, thank you very much for having me. I don't think enthusiasm is going to be difficult to know from there, but man, I got an early start a very early start and a computing had a ti 85 Texas Instruments calculator, you know, back in junior high, and I realized that the assembly bass games were far better Oh, baby than the basic games. And so I took a concurrent enrollment class at the local college as soon as I hit ninth grade, to learn programming. And so I actually took an assembly course. And in doing that, I started developing even more of a fondness for just computers in general. throughout high school, I lived on my computer, I would work out after school with the jocks. And as soon as they go home, it was until midnight or later just sitting up, you know, I was in the what we call the Where's scene back then? Right? thinking I was cool, you know. But what really got to me in that era was how are these people able to do all this? You know, when I was living on IRC and hotline, I don't know if your hotline that was the blast back in the day. When I lived in those environments. It really I started realizing like people are putting these things together, these release groups, the courier groups, the way that they conduct themselves, how do they do all this? So I started falling like crack knees, and I started learning how to use soft ice, which at the time, and the windows 9x days, was a kernel level debugger and I just learned stupid stuff, really. But at the same time, it I started did it really sparked the passion like, Hey, this is stuff that even I can do, right? It's very interesting. So I ended up going to school, I got into as a technical trainer, a full time technical trainer while I was in school. So I was running my mouth professionally for six years. And while doing that, you know, towards the end of my graduate degree, I had a fella give me a call and he's a great dude. I'll throw his name Johnny. I love this guy. But he's calling and it's funny because really, he was bragging he was bragging about his new job. And he was like at the Security Operations Center and it's phenomenal. It's like right out of the movies are sick. I'm reading a book our day to day workflow is just zany as I so needless to say, a couple weeks after that, I was on boarded and I worked in my first SOC so working in a SOC environment it was just phenomenal i tried to dabble in each and every little thing i could possibly do working in a blue team you see malware all the time right you see signs of our artifacts of our you see all these things and i had that which i'm sure many of your listeners have now right they're looking at these artifacts and these things going how i analyze these like i don't like passing these off you know if you have a malware team or a third party that you hand things off to that's an achy feeling and they're like here can you tell me what this does i'm like no i want to do that so i started pulling things apart i started with malicious documents or mal docs as everyone calls them now i'll get into more of that at some point i'm sure but you know i took the grim the excuse me i took forensics 610 and it achieved the grim certification back in 2014 and it was just because i had malware it was pervasive it was all around me you know because it was all over the place and i wanted to know more and more and more about it so the more that i worked in the SOC the more that i would you know people would get malware and i would ask them just like can i see that like let me do it let me help you with that and i would just you know i pretend like i knew how to do it and then i learned how to do it i was looking at for most of it but we worked together and as a team we grew our SOC into more of a malware and threat intel oriented unit just because we were super nosy so i was at that organization for seven years and for two and a half or so i was on the SOC and i moved over to the cert on the cert i actually moved to the network security monitoring sub team first and that was because they had an opening there first and i looked at it and thought well i know deeper and i know malware and i know threat intel and i'm not gonna do now of course but still i was familiar with them i dabbled with them i don't know anything about running splunk i don't know anything about running an infrastructure so i took that opportunity while being there i eventually moved into a deeper role specifically which is obviously where i fit better and that experience all those things combined together allow me to have a far more holistic picture of dealing with malware you know so when i i'm pretty sure that you i have you to thank actually for getting my name at some point into the sand circles based on that that meeting in scottsdale and i recall the first reaching out on the sands part was hey let's you know we're gonna bring you on to the blue team if you're interested i was like yeah let's do it you know sims and yeah and then friends extol me way i was like i love our and then the rest is history so for me it was at a young age i was obsessed with computers obsessed with computing devices you know i was in advanced placement courses in senior year and i was failing out because i was busy programming my ti 85 you know like yeah i was programming the calculator for the other people and i didn't care about the physics course i just wanted to you know get the calculator to do what i wanted and that really evolved you know into what we really deal with so much and incident response and dealing with these threat actors in some way shape or form often deals with with malicious software right either how they're getting in the initial infection vector when they're moving around like all the above you know there are tools and resources and so i just really wanted to understand that better so for the past year and a half i moved over after seven years and that sucks certain vironment i moved to silence which is now blackberry so our security services formerly professional services consulting group we're consulting you know just seems weird crowd strikes and fire eyes we get cases each and every day they come in a lot of ransomware stuff and i've been talking a lot about ransomware these days getting all that i'll digress for now but you know in our role we actually have a threat research slash malware team my job specifically if i come across a legitimate malware sample is to pass it to them but that harkens back to what i was just talking i'm like oh i just get into them funny story a news sony have fantastic human he is a co author of forensics 610 and he is on our threat research team so day in day out he deals with malware that's his job right for me i come across it all day long and as time allows i rip it apart you know so i like to you know be in the trenches in that ir realm but i also love the fact that we constantly find new stuff and so yeah that's where i am now i'm loving consulting absolutely adore especially being in all these different environments seeing all these different types of attacks and learning how and teaching others how to support this stuff you know it's like hey this is silly let's just get
John Hubbard 09:40
rid of this sounds like we have kind of a pretty parallel path maybe i wasn't aware of but yeah i mean i was of course as anyone i think back in the day into the ti 8385 86 i think and 86 yeah the calculator games in math class i had like a mario clone and all that kind of stuff oh yeah i program this right and do and all that kind of stuff instead learning like i should have been and then going home and playing with computers and also in 2014 i also did the grimm which is kind of what brought me into this world as well that was my first science course i ever took and so i was like oh this is awesome right came back i'm like i got some malware superpowers now this is super cool so yeah i mean that kind of experience i think we definitely share that in your consulting career now what are you seeing as like some of the more prominent trends and stuff that's popping up now in terms of i guess the goals of the malware and styles and things like that
Ryan Chapman 10:32
so number one right off the bat and i've been doing i actually have a SANS webinar that just got scheduled for like a week or two from today apparently a better a better human operated ransomware is the term i believe microsoft coined that so i've been referring to it as hora h o ra it's not a great acronym but ransomware groups like maize and rebuke and derivatives of them and teams that are now joining up with them like the gandcrab folks who are joined up now officially to the maize cartel as they call themselves like that's not ominous right that is so so common it's ridiculous we're dealing with it day in and day out but whether it's in those attacks or whether it's in attacks that just you know whatever the wide net approaches and phishing someone clicks on something silly and then something dumb happens we're finding so much that we have the common info stealers or the banking trojans as they were originally known so you have imitate and trick bot and crackpot or cube if you want to call it that or iced id those types of things we're seeing those all over the place and then on top of that we're also seeing stuff that you download from github or could easily do so right now we see so much open source malware and the windows environment which is most of cases we get in because well duh it's just generally how we see so much powershell so much malicious powershell it you know was the new thing in 2015 16 and then ever since then it's just been so consistent so i find that i'm ripping apart malicious powershell d obfuscating as we like to call it probably like once a day if not multiple times a week and i'm finding that it's becoming easier and easier for us when we come into an ir the blackberry team we have you know our attitude is more like yeah we're gonna find it like don't worry but it's not just because hey we think we're good but it's because we also know we're seeing the same thing over and over and over and once the attackers are getting in and they're also leveraging of course tvs related to network appliances you know i'll try not to throw out some of the big names but if you name your three biggest most important network appliances or remote enablement without giving away names if you name them in the past six to 12 months how many cvs have they had and we are seeing those taken advantage of for initial entry and then you know we're seeing just like mimi cats in an environment it's so so common to see you know process hacker and maybe cats like over and over and over and over as far as the malware room in general we're seeing a lot of repetition over the past two or three years there's not a ton of evolution and while there is of course and i have some to speak to i we just ran across one yesterday i was like this is awesome it's perfect talk about what we're seeing a lot of that repetition so there are already so many articles out there there are already so many training videos or whatever dissections if you will order technical deep dives as we like to call them right because it just sounds cool on all these things imitating trick bot for example and then right after those are deployed we see cobalt strike so often so often blue team right how often do you see cobalt strike all the time right we see it with everything everything i don't know what percentage i can make one up but that's there's about that so we see the same tools over and over and over and while there may be of course adjustments and changes to maybe the initial vector or maybe one of their password dumping methodologies or what have you we still end up seeing very very similar payloads within environments and for me it's actually it gets a bit boring if you will because from the malware analysts side you identify a new family or an evolution of a family right and it gets named based on typically just indicators or strings and the darn malware you know like oh this is divergent which i want to talk about later coming up so that we're gonna call it divergent you know and when you start seeing them over and over as a malware analysts you build or use a framework that is able to chop that up and is familiar with it but then you're kind of waiting for the next thing you know like what's what's the new thing like what are we going to have some more fun here so and we're also seeing a lot more evolutions of previous families than we did in the past so for example i actually had a case this is awesome right this is like why i love ir one of the things and in the case we found this ransomware and it looked very very similar to a previous ransomware called brohn at least i'm calling it i don't know how to pronounce it officially but bu r a n but this was different and long story short and then ended up being an evolution of that it's called zeppelin you know the blackberry threat vector did a whole article on it and i was like that is cool man working on the blue team side i ran into it by mistake right i wasn't hunting for the client today we got hit with this i said hey that sucks we looked at it and it was like cool then we were like what's the differences then you learn to you know identify what's been changed and so on and so forth and that's what really keeps you know fuels my passion is that evolution so now that's what we're seeing a lot of
John Hubbard 15:51
it's interesting you see like there's not a whole well in terms of the families that you see like you're kind of seeing a lot of repetitive families of the same types of malware but those families themselves are evolving slowly for people who are wanting to get into this and are as excited about doing this i think we've probably made it clear that the benefits of doing so right we all know we're going to be seeing this forevermore right there's no end to this what are like some good first steps that someone can take to start to learn these different families and start to like get familiar with the tools and such to start walking down this path
Ryan Chapman 16:22
yes i personally believe that of the various realms of digital forensics and incident response right of overall deeper as we call it i think that malware analysis has the perceived highest steep in terms of learning curve and the reason i believe that is because a lot of times when people see our folks in a debugger or in a disassembler you know they're they see ida or they see ghidra or x 64 dbg debug right and they're like oh what is all that right like what's the register i don't know what's an ea x i would it was so it's all this however there is far more accessible far more accessible than that so the go to that i always recommend is practical malware analysis a textbook that most people are familiar with and have heard about the pma you know it was my bible for many years i absolutely adore it and it was published in 2012 minutes eight years ago right that's a while ago but it is still so relevant it's a phenomenal read so i really love pma there's a fellow named sam bone i'm not sure if you're familiar with sam he is out of the san francisco city college and he is another fantastic human being i've gushed over him i even have a youtube video where i just talk about his website and how cool it is like i'm a bit obsessed i can't wait for him to hear this and be like again again this guy's weird but he has a practical malware analysis based course on his website sam's class dot info and by the way along with literally hundreds of other classes that are completely free are completely documented including the labs and including in the past couple years youtube videos for all of his various sessions but anyway i'll take chris on my love for sam in his class he basically gives you a structure and labs to follow and it's like hey you know you read this part of the book and then you come to this or read this part of the book and then you go to this that is my number one i always tell people do that that is your first step i would prefer that people have some type of introduction to malware in some way shape or form via pma just because i just love the design and how well it's written of course i would be remiss not to say well done john you come take forensics 16 with us but you know of course accessibility wise you know of course these and such as they are that may not be the easiest method to dip your toe so to say so pma i think is phenomenal and there are a number of resources that you can use to set up a malware analysis environment which i assume will be a pivot question very soon for us and while setting that up one of the best things to do i find is to essentially go through the listing of tools as you're installing them identify and categorize you know what types of threats to these deal with are these four malicious documents in other words a word file or excel file which by the way i forgot to mention we see those constantly oftentimes as an infection vector via phishing it's probably will always be i don't know when that will change but you know you take these these tools put them into categories and then go okay as a blue teamer when do i see these things you know and if i have an example i run across just responding to an email thread then how do i rip it apart and rather than rip it apart just start very very slow and just open one of these tools and just throw your file at it and see like what is this you know having a foundation of something like pma or especially going through a more structured training kind of style, especially if it's forensics 610 every time we close 610 In fact, usually every day of the course, people are already asking like, Whoa, how are we going to keep this up? How do we continue to go forward? My answer to that is that in a blue team environment, you're constantly coming up against these random different threats, whether they be the exploit kits of yesteryear, I say of yesteryear, they're still common, like you still see exploit kits. But I mean, you know, it's not the same. It's not even close. So when you whether it's an exploit kit, or a mal document, or literally just a dot EFC, you know, a portable executable in Windows, whatever it is, just to take that set it aside if you don't even have hours at work to deal with it, or if you have research hours that you use that to try to analyze these things. So yeah, my stepping stones for myself literally was PMA Grimm. I mean, that's how I did it. And I love that resource. There are other great resources out there, you know, I have some courses on pluralsight.com. And there are some great guys, Tyler, who deck was over with a trusted sec group. He's phenomenal. And he has some malware courses on there. OEI labs has some fantastic stuff on YouTube, although it's a little more on the Advanced side, it's not so much dip your toe type of situation. But they're they have some great content there. So there are a lot of places to go to see malware being broken down, but I find that the initial, the first foundational type of stuff is not as commonly covered. And I see it more as you basically have your textbooks and I have a shelf full of them. I just prefer PMA as my number, even though it's sold as my number one. But then beyond that, you know, when you go to watch things on YouTube, it's like here's a complete breakdown of imitate, you know, in either the whole time and you're like, Whoa, Whoa, dude. So I think it's important to start with a solid foundation, like the PMA book. I don't get any proceeds for this, by the way, or oil for mentioning the name. I feel like 10 times already. I just love it.
John Hubbard 21:59
Yeah, no, that's, that's one of the books I always recommend as well. And you know, same exact progression as you like, I read that book, I was like, This is super cool. I think I can actually do this took the class. And that kind of set me off. One of the things I noticed and I still kind of tell this to students all the time when I'm when I'm teaching for 50. And I'll also have some labs on this is like most of the stuff you will do, at least in my experience, like every day is taken apart and malicious document like finding the link inside a PDF, right? Is there a malicious macro on this thing? Is there an embedded object that's a file that is supposed to double click whatever it is, there are certain the probably like five tasks that like 90% of what you will encounter day to day will fall into. And those things aren't really that hard to do. Like you run some kind of scripts that extracts the macro, you run some kind of tool that takes apart the PDF and tells you Hey, look, there's a link in this. And like you don't have to open it, you have to do any of that kind of stuff. Those are the things that I often point people towards and say like, don't worry about this assembly, you know, D compilation, all that. Yeah, like you can do most of what you need to do day to day on that kind of thing. I mean, I listed some there are the what are the things that you think are like the key if you were going to learn the most value in the littlest amount of time or the least amount of skills like what's the 8020, you know, of malware analysis for getting started for the most value in the shortest amount of time.
Ryan Chapman 23:16
For me, I always push malicious documents. And many reasons, one of the primary reasons being that my initial foray into hands on security was in a SOC, right? And we constantly are sent from users like What's this? Like? That's bad. Don't click that. That sucks, like, Well, why does it suck? Let's go find out. So I absolutely recommend learning how to deal with malicious documents to the point where this previous in person DEF CON, I got to run my first DEF CON workshop, and I ran a four hour workshop. I have it on GitHub, actually, it's a step by step PDF. So I have a couple workshops on GitHub, the one that's on there for the Cure, I call them carrier files, as opposed to Mel docs a lot personally. So it's called cer file workshop. But when you start pulling those apart and realizing these are not really difficult to analyze, going back to your question on what are we seeing, we're seeing so much of the same type of malicious documents, a dot doc file doc x, you know, what have you that just has malicious VBA. The levels of obfuscation, of course can differ. But once you learn to analyze it and to step through the debugger in Microsoft's own VBA editor, you know, I do 99% of my Word document analysis. In word, I literally double click the thing I Alt F 11 to open the developer, the VBA editor and I just step through the darn thing. That's right, that's right, I do most of it, you know, ripping those things apart becomes a lot easier when you start to understand their structures. So I recommend folks who are in for example, a blue team environment or specialist SOC environment that you grab those carrier files now docks, and you start to just Google around just research on how to analyze them, you know, whether you're looking at my workshop and big long PDF, you know, or whether you go whatever resource It is really, really eye opening to see, for example, the structure of a PDF. Many people don't realize that they are primarily ASCII based that they are textual that you can take a PDF, drop it on to a text editor, hopefully, within a secure environment, in general, by the way, more of that coming up, but it just be able to see where the objects are. And then once you start realizing how the trailer and the cross reference table point to the root object, the root object literally has a number often object one, not always, you know, one zero object. Okay, and what does that include? Well, it points to the pages in the document. How does it do that? Well, it literally says, like to find pages, slash pages, and then go to this object, you know, objects, five, zero are our means reference the object a lot, no, go number five pages are in there, you know, and then the streams where the binary data may be included. That is, it may be all it's all, it's all hexie. And crazy, it uses very, very common, there's 15 or so decoders and encoders. But often, it's the same two or three. And you'll learn how to decode those and how to utilize those. And there are many tools, you can just use PDF stream number, for example, take a PDF, drop it on there, and go just google PDF stream dumper analysis, right PDF file format, and you'll realize that within just a couple hours of research, you'll have a really good idea of what PDFs are all about. And usually how they're weaponized which these days, as you mentioned, is usually just a URL. You know, that's the quote unquote, weaponization, of course, JavaScript can be in a PDF, so you'll learn that too. But you get the idea. The idea is that when you have these documents that we come across all the time that are malicious, it may seem like I don't know anything about programming. I can't deal with that encoded VBA. You can, it's really not hard the basics of debugging, learning how to literally set a breakpoint on the first thing that happens, and then just start hitting step into step into step into and just trying to realize like, oh, okay, these variables are filled out as it's running in runtime. And I have no idea super obfuscated. The variable names are godmother, mernda, 572. Like, I don't know what that is, right? Who cares? What is Microsoft's VBA? editor tell you the value isn't that variable, right now. They're like, Oh, it's a 17. Okay, pick on it, it's really not that hard. So I always like to start with Val docs.
John Hubbard 27:38
So when it comes to, like, you mentioned, opening them in Word write, obviously, you're doing this in a secure environment, people who want to create this kind of thing, so that they can download these and kind of step through a debugger and all that kind of stuff. But what's the actual setup that you would typically use for bringing a malicious file into what I assume you're going to say, a virtual machine, and then making sure that there's like no, reasonable worry that it's going to escape the virtual machine? And you know, like, what are the precautions you take? What's the VM setup use?
Ryan Chapman 28:11
Yes. So I was I was waiting for this part, because I love showing folks that it's so easy to actually get this set up. So first off foundationally I prefer to do any and all malware analysis on a Macintosh computer. And there are many reasons to that, aside from the fact that the terminal and having new based tools at your fingertip and I don't like Windows subsystem for Linux and permissions are a blunder in that thing. Aside from all that, when you're moving malware between VMs they're going to hit your host, and if your host is a Windows machine, Mm hmm. If your host is a Macintosh machine, I mean obviously there it's your risk factor is far lower. Right on top of that I have officially a secondary Macintosh laptop that is just for our analysis. Now many people cannot go out and buy a secondary or tertiary I just want to use that word. I like that word. For no reason other than to hear me say I love that word. A third laptop or computing device just from our but if you have older laptops, even older windows laptops, cool dude on there, you want to avoid as best as possible infecting your personal machines that are daily drivers, if you will your gaming rigs or something right, let alone your work computer like don't be the infection doctor. how this happened was I was analyzing this thing, right? I double click that. Yeah, no good. So that's first and foremost, I love doing things on a Mac. And on top of that, I recommend at least one windows and at least one Linux virtual machine from our analysis. The Linux side is made easy by Lenny zeltser and a couple other folks who are working on the rim knucks distribution so rim ducks REM in ux.org It stands for reverse engineering malware knucks like you know Linux, so it is is a phenomenal distribution all kinds of amazing tools already baked in newer version just came out after a number of years with a primary you know point version so really great utility can't recommend it more on the windows side i recommend having at least one windows 10 enterprise machine now that is because most of the malware that we're going to be dealing with and my realm at least is going to run within an enterprise environment right so i literally want windows 10 enterprise now all you really need is windows seven or windows 10 i also recommend having multiple versions of windows at least a 10 or seven would be very very handy but in order to get a windows vm it may be easier than most people realize first off if you have access to an msdn license well then just grab a key you know and you can literally download windows 10 or seven from microsoft i'm sure folks where that is windows 10 iso microsoft and it takes you right to the link the legitimate link to download it right make sure it's at microsoft.com by the way i should say
John Hubbard 31:03
it come pre effected you can analyze that
Ryan Chapman 31:06
right there you have a base machine on top of that again back to linear zeltser if folks are not aware of lynnie he is the primary author of forensic 610 so he in a news there are the two concurrent developers right now for the course and maintainers liddy has a great article it is called how to get and set up a free windows vm for malware analysis i can't get more than that right i would read it straight out to you and in his article it's just on his website zeltser calm he shows you like hey you want to get a virtual machine and if you don't have access to a license microsoft provides test virtual machines and many folks are aware of this because you know we can get vms do fun stuff with so he gives you exact directions here on how to do it and how to choose which version you want and such they do have a 90 day limitation technically i'll leave it at that so when you have the windows vm you are then going to need to fill it with tools now remnants comes jam packed with a bunch of our analysis tools that's the purpose of the distribution by having a base windows 10 install basically what i recommend to everyone is that you grab the flare vm so fireeye which is a competitor of ours in terms of consulting but man do they have a solid outward facing public presence reverse engineering team the fireeye labs advanced reverse engineering team or flare they have a virtual machine they've put together at least they call it a virtual machine but in reality it is a bunch of tools that are installed via powershell script is what it is so you set up your new windows vm you know brand new and typically go with a brand new one make sure the regular windows updates are there and then just go check out the flare vm which is on github it's a powershell script it downloads and installs i don't have an exact count but if you were to ask me the top three tools to analyze the following types of malware and you gave me java and you gave me pe ease or whatever this thing has it already it comes with tools to mess around with those you know so i highly recommend just windows 10 pro or enterprise throw flare vm on top of it and then just start playing around and one of the things that i really like about the flare vm aside from there it's the i'm not gonna say it's the only of its kind for sure because it's not but it's the by far best known of its kind but on their github page they have a section an anchor to installed tools right the big name tools that are on here and they are categorized so earlier i was referring to categorize the tools on what they do for you right and keeping a running list so that you know oh hey look i just got an apk file you know down the line you may get into android reversing and you're like oh what tool do you use for this right well they have listed right here there's two tools they have listed right so for the various document types that you get you can even look here in the flare vm and go oh okay well i should i should probably try one of these you know you just open it up and then just get hands on in handy and just try to see what's up the under office for example they have like oily dump in office mal scanner like yeah those are tools we go to all the time so i really like the flare vm on top of the windows 10 with remnants on the side just for starters
John Hubbard 34:25
so one of the things that i've heard when i describe this stuff sometimes is you know yeah i can install this malware and do all this kind of thing but like don't i pay a vendor to do that like i have this malware sandbox can i just fill the sample into that and get my answer back how would you describe the difference between you know what you would expect to get from manual versus automated analysis and when is it appropriate to use which and how often do you find yourself using both of those
Ryan Chapman 34:50
yes okay so first up for the vendor side there's two parts i want to just real quickly throw out there and whenever i say real quickly it's a lie we all we all know that long winded here but you also have teams for example that you can send your samples to and they will literally do the analysis for you now they will most likely i mean they will use a combination of dynamic and static analysis including on the dynamic side using sandboxes but they will provide a write up and all that if you have an agreement with that type of vendor and if the hours required for that type of analysis are free with your contract which you know could look free free using you have x number of free a year whatever it is right then i recommend for the big stuff you throw it to them anyway no matter what but then taking that and abstracting it more to just general sandboxes or dynamic analysis in its very nature versus like static analysis sandboxes very many times can be tricked and some great examples are hanspeter and emotet at first we're really tricking sandboxes and i'll get to that in a moment on the static analysis side it takes way longer but the okay a quick static analysis triage which i can even run down coming up right how i flow through that and how you can too easily with just the flare beam tools and remnants but it's not that difficult and i want to explain how just finding strings or dapps getting strings with the flare teams floss tool which obviously is in their distribution you see imports that for example p e's which are just names for dot txt files right portable executables the official name you see the libraries that it's using and you go oh you know right away i know it has access to it's going to import these x number of libraries and within them these particular functions and that should really give you a heads up as to its overall capabilities right if it imports wind SOC well the right you know we're probably going to be doing something with networking right anyway looking at the dynamic analysis conundrum if you will when hensler and immitate first came out they were and they still do by the way almost every single one the downloaders usually were talks these days you know dogs will loop through a list of five different domains right oftentimes the vba for example will build powershell the powershell will literally just loop through five different domains it will try to download a sample from the first url right at a particular domain and if that fails it goes on sucks and then moves on to the next the idea behind that is that with one piece of what's the term well i guess i'll just call it one download sample right with one sample one compiled sample and sent out even if infrastructure gets burned up to four times that sample still gonna work as intended right but automated sandboxes at first they were running into the issue of in the powershell code that's often used for this looping what it does is it goes okay can i hit the first url cool can i download the file and is the file over a certain size in other words did it actually download and not just give me a 404 write the code checks length usually a bytes for the stored file and if it does it goes okay and it breaks out of the loop those other four urls oftentimes on other four completely different domains sat dormant and were not identified so the malware analysis sandbox is we're showing you network connectivity right a section of networking or whatever and it would have one domain with one url but in reality that sample had five now of course that was picked up and that's been adjusted such that if there are any breaks to certain types of loops sandboxes will go but if we don't break here why don't we just keep going right so there are certain ways around that but this is one example of you know when i was in the SOC certain environment you know we show our analysts you know go look at the dba like don't just take that particular i won't throw out which sandbox it was because it was a commercial one and otherwise it was phenomenal but it kept just showing us one you know so i really think that there's a lot to the static analysis side is very similar in fact it's the exact same concept but it's somehow similar in my mind and not the same but malware may do different things at different times it may do different things when executing it may sleep for a gratuitous amount of time as opposed to just x number of minutes and for that matter there are a ton and i mean a ton now of whether it's via hooking or whatever it is there are many ways that malware samples and families are avoiding sandboxes specifically so the anti analysis techniques have extended to anti sandbox techniques so looking at things like the user account under which the sample is being detonated right certain sandboxes were using certain usernames as a default right they just spin up a vm or environment and say hey this is joe user right that's not a real one but you get the idea but what always use certain artifacts that would always be the same so the malware was looking for those and going like oh hey look this is probably the fireeye sandbox, this is probably CrowdStrike, this is probably a Blackberry, we're not going to let's just not run right now. And they would just kill itself. And the sandbox would be like, yeah, that thing sucks. All it does is go to disney.com, or in whatever, you know, maybe it would quit. Or maybe it would do something else. Because it detects that it's in a sandbox, we run into those all the time, all the time, and we cover it in 610, we talk about not just anti analysis, but as part of the anti analysis and anti sandboxing doing something else completely just going like, Oh, I'm just a pretty little piece of software, and I do nothing bad. Like I do my stuff and then quitting, you know? So yeah, I think static analysis triage is very easy to do I start with static triage every time First, I want to know what type of file I'm dealing with, right? In general, like, What is this, and then based on that type, I will take a different triage method, obviously, from a PE, if it's dotnet, PE, there's a tool called dn spy, and dotnet. When it's compiled down, whether it was written in C sharp or VB, whatever, it compiles down to what's known as i l or intermediate language, and that is rude. Mostly easy to convert back to source code. So dn spy, if you have a tool, or soft malware, of whatever software that's written in dotnet, it's most likely obfuscated these days. But you just know you download DNS by the release from GitHub, or the code, build it yourself, whatever, it drop it on the DNS by and it D compiles it, oftentimes, within not even 1015 seconds, for the most part, and it goes, here you go, here's the source code, and you go, okay, just remember, like, there, you know, I might find strings, a lot of times threat actors will literally name things like SC for shellcode, or, you know, like all kinds of, you know, silliness. So that quick initial basic triage, I find that really easy to do, and I can get into how to do it with certain types of files. But once I do that, typically, what I'm doing right away is I'm looking for strings, it's usually a string bass game. And so or identifiers or indicators, and it's usually a string, it usually boils down to a string, an indicator, an IP, a domain something, and then you Google around you check buyers total, and you are looking for previous write ups that you may happen that is found it Oh, look, it's probably this and then you correlate further based on the analysis they already have done for you. You just go through it and go like this matchup that matches up to does that match up? I messed it up to, like, I think it says, right, but if it starts to get to the point where you're like, Alright, I'm gonna have to rip this apart, I go for dynamic analysis right away. And dynamic analysis is very, very easy. And a sandbox environment is very, very easy to use. And I think one of the problems with them, and going further into a difference between why I may prefer one or the other, I love them both, right? But I think that many people use a sandbox and don't really realize what it's telling you. What's critical in the output. For that matter. virustotal is the same thing. If you're using virus total intelligence, or excuse me, used to be intelligence. Now, enterprise, Russia, is my number one favorite tool again, I don't get paid by virustotal. But oh, my goodness, I I'm in love with virustotal Enterprise. But anyways, tools like that any run, even crowd strikes, hybrid analysis, website, binominal website, those sites will oftentimes show you hey, here's some behavior. And many people go okay wrote a file and it reached out to this IP, okay, moving along, but they're just not trained to understand that the mentions of pipes or mutexes, are critical, and that those can easily be used to correlate potentially what that malware is, or to further do research and find related samples, you know, and what have you. So I really, really think that they both go together, and you cannot have a proper analysis report, for example, without employing both of them, and hence, why in 610, we covered you know, both.
John Hubbard 44:03
Yeah, sure. You mentioned commercial sandboxes and some of these open source websites. What do you think of the open source options for a malware sandbox as compared to the commercial options that are out there? In my experience, I have seen some outstanding results from some of the free options, which is something I'm always mentioning in classes when people are like, well, we don't have a malware sandbox. I'm like, Oh, yes, you can do this. Right. And then in terms of also the websites like which ones are your favorite there,
Ryan Chapman 44:30
so I got a big stupid grin on my face. When you asked, I was like, Yay. So I love open source sandboxes. The two that come to just float to the top these days are cuckoo, and derivatives of cuckoo, specifically Cape caap. So cape is a malware configuration and payload extractor, if you will. It's built on cuckoo By the way, but CAKE CAKE what Nola okay
John Hubbard 45:01
so awesome yeah all right yeah
Ryan Chapman 45:03
i'd like to eat cake while using cake i got to do that take a picture of it and tweet that out so cape looks for very specific things on top of what the general kuku sandbox is looking for and some of the things it is designed to do is recognize given configurations of very common malware families so right on the github page for cape it says hey we will dump configurations and payloads of plug x server trick bot hanspeter you are sniff crackpot q boss all these thick drytex has decoders poison ivy decoders all these things that we come across so often if cape is in the mix cape does not only what cuckoo does and gives you like hey it does this in books these libraries i noticed these network call outs all that kind of stuff but it also may go hey by the way this is imitate i don't know if you knew that but but here's the configuration that it's using i decoded it here you go it's already programmed to do all that it's phenomenal i can't i just love it so people who maybe find it difficult to utilize these tools will often to not to utilize them it's very easy to utilize them what i meant to say is stand them up and folks who may find it difficult to for example stand up a proper sandbox may just simply not have the engineering background they may just not be you know jump into a spin of a brand new ubuntu box or whatever and be like oh install this or missing these packages they just may not be their forte in those cases a lot of times folks will turn to online based sandboxes i have a great level of respect and fear at the same time for online sandboxes and it's primarily because of operations security for opsec yeah so as we know especially the us government pushed objects so hard during world war two they had campaigns loose lips sink ships you know sailors would send letters their family saying you know hey we're going here tomorrow on the destroyer i'm on that would leak out and then the enemy would be waiting for him right like oopsie like shut up don't don't talk about that well opsec can be really harmed if you will or exposure just destroyed by using a site like buyers total hybrid analysis any dot run just a quick note any dot run is one of my new favorite favorite sites but you have to be very careful with it because it is actually run by a group in russia i'll leave it at that for your imaginations but you know you're literally sending your malware to russia is what you're doing but whether you upload it to virustotal hybrid analysis or what have you most of these systems allow folks like me researchers to download every single thing that you upload if you have an enterprise account and i actually have a researcher account i have a personal account that gives me access to enterprise via virustotal because i reached out to them and i was like i love you so much they can i drop your name everywhere and you give me cool fun stuff like so even without blackberry behind me i can pop in everything you upload i can grab forget about me i'm just a researcher said you know ir guy right but right actors can grab it and so you have many people who say well i just got this word document i don't know if it's bad or not let me just upload it to hybrid analysis which is crowdstrike falcon platform right let me just upload it there and run it there but now you are literally giving out all of that information whatever is contained in that sample is now being essentially spread around or shared with the world so you have to be very very careful about how you use them i highly recommend hashing whatever you're looking at and looking for file hashes first if you can't find via that first off that's a bad sign in general these days it could be a bad sign especially for something as common as you know trick bot cobalt strike type of stuff you know you're like oh it's not even on there yet like what
John Hubbard 48:58
what do you have the first person with this malware in the hallway
Ryan Chapman 49:01
right like oh you're something's but yeah sure you know also looking at the indicators that you may just see just by looking at strings that and looking in virustotal or hybrid analysis or just on google and looking for those things before actually uploading them there are also systems that you can get one of them is called erma i hope it's still an active development irma malware it is essentially it aims to be an on premises virustotal so its whole deal is hey you don't want to upload stuff to virustotal because the right you don't want to tip off a threat actors who are monitoring virustotal that hey i sent this to company a i only send it to them it just got uploaded to virustotal okay they're looking at it now right you're tipping them off aside from that you're spreading it to the world aside from well there's those that focus on those right herma is something in their projects very similar to it that runs on premises right you can put it in your own car Cloud but the idea is that you run it. And it just analyzes stuff based on what you provide it. It doesn't use outside resources, but it can use outside engines. For example, I think I want to top my head, I always remember renewing the license for it was McAfee. So McAfee has a EULA, just give it a license. And then it goes, Okay, whatever you give me, I will scan via these AV engines very similar to how buyers little does it. But oftentimes, you may have to actually license that engine, although it's usually a decent price for these types of environments. So if you have a team of SOC analysts, and you see them keep throwing stuff up on virustotal, like immediately be worried. I recommend that all blue teams monitor for traffic to virustotal, just virustotal.com, hybrid analysis.com, all those submission based sites, why are they being used? And if they're being used in the egressing is occurring outside of your SOC cert security teams? Like who's doing that? What are they saying there? Because there are many employees who receive documents from you know, joint ventures or clients or whatever, and they go, Oh, I wonder if this is bad. And they put it up in virustotal. And now whatever was in that document, right, the payroll information, all that stuff, I've seen it all man, it's all just sitting up there for anyone to
John Hubbard 51:14
go look at. Yep, that's actually exactly what I was just gonna say is, in my experience, I used to have intelligence enterprise, whatever you call it now, access. And that was the other thing I saw, right. Other than everyone else's viruses, it was everyone else's, not viruses. And I'm like, so there's a lot of invoices appear. And these look pretty real. And you probably don't want them here. But here they are. And now the whole world can see your potentially private info. Got so great. Yeah. So that's one of the other any other like big mistakes that people often make with any of these sandboxes or anything like that.
Ryan Chapman 51:45
I think the biggest mistake that I often see outside of our organization, hopefully not happening where I work, right? Yeah, that's just throwing it out there. Hopefully, we're not doing is opsec is just destroying opsec, sharing things out, they shouldn't be sharing, asking questions. They shouldn't be asking about different malware samples and things of that nature. If you have something that you know, is public, you know, I've said imitate what 20 times on this call already, right? On this podcast, imitate, imitate, imitate, right? If you get imitate, and you're like, wow, I'll just share it with the world even imitate can have specific campaign information associated with it and campaign IDs are for sure, I can guarantee you are monitored by the threat actors behind them. He has to be careful about about everything. Beyond that, I would say that assumptions and reliance on those assumptions is a huge problem in our analysis. So when you have a write up, for example, and you go Oh, what I have looks Oh, it looks just like this, you know, based on two paragraphs, and you go Okay, this is what this is the threat, this is what we have. And then you do stuff like okay, well, this article mentions that this does click fraud, that it does data Expo but only over FTP, and so on and so forth. And then you build that into your your threat hunting into your profile into your whatever incident write up without verification of some sort, you know, actual analysis yourself and ensuring and correlating that what you're finding online from, you know, Joe Schmo, heck, you may find a blog article I wrote, right, oh, I found this malware and you're like, Oh, it's the same thing right? here like, oh, yours, maybe, you know, it's kind of similar, but it's not the same. So I see that I see that a lot. You have to be very, very sure that everything kind of lines up. And, you know, recently when we ran into the divergent malware, A day or two ago with my coworker, you know, at first I was like, that's it. I was like, Okay, hold on, slow down. Okay, cowboy. Let's verify that's it. And then he eventually, you know, sent me some screenshots. He's like, no, that's it. Like literally the entire article, we were reading, you know, item for item file for file, everything he was finding that was the threat. But you have to be very careful not to make assumptions. And you also have to be very careful that when you're using tools, like just a standard, you have a standard dot e xe a PE file, and you drop it on to a tool like PE studio, which is one of my favorite tools for PE analysis. In fact, it's one of my top three that you don't take for granted. The fact that malware these days, its aim is to hide itself to obfuscate itself. Many times you'll run across packed malware and packed malware will not show you a fully built import address table. So which by the way, references the libraries. And often the functions are specific to those libraries that are pulled in during runtime. If you see a list of imports, and you say, Oh, this malware uses these 15 imports moving on. Whoa, be very careful because most malware these days will dynamically load certain libraries to do things it doesn't want you to be able to see during static analysis. So you just have to be very careful that whatever information you get, you run that to ground and again, a combination of dynamic analysis cape is awesome for static analysis is very important. And I'd be remiss if I didn't also mention that as far as dynamic analysis is concerned, I highly recommend that once folks have a Windows working VM, you grab Process Monitor, which is just a standard where the system internals. Yeah, so proc mon dot txt, right Process Monitor, when you run Process Monitor, and then you execute the malware Process Monitor is monitoring everything, not everything. But most of the things all the library calls all the file rights, all the registry reads and writes and all this fun stuff that it's doing, you can actually allow it to run for a couple minutes, stop the execution. And if you export the results by literally going to I think it's File, Save as a File, Export one of the two, and you choose to save it as a CSV, there's a tool called proc.so like process, but dot d o t. And that is provided by the Australian cert crt.au proc dot, you can load the proc mon CSV file, and then it turns through it for a couple minutes. You then tell it which executable or process was actually the malware, you say like this kicked off all the bad stuff, focus on this. And then proc dot goes okay, and it builds you this beautiful graph. And it's got these cute little color coded, I'm colorblind. So the color coding I'm like, I'm sure it's useful. But you know, it has all these shapes, certain shapes and sizes and colors for like registry entries or files. And you can literally even click play and watch it like a movie. And it's like the process starts and then it opens this rich key and then it writes this red value. And then it writes a file to this location. And then it moves itself to the app data directory, and it makes a new task and like you're literally just scrolling through a graph going, Okay, like, that's cool. So proc mon, plus proc.is, a very simple, very simple, dynamic analysis methodology that anyone can can implement with malware, again, you have to be very careful for the VM that you set up, that it's in an environment that is conducive to literally executing the malware. So if you have networking enabled in that VM, like, do you really want that enabled, right? If you have, you can just drag and drop enabled between the VM and your host like, should that be enabled while you're running the ranch, or while you're running the malware, you know? So yeah, Anywho, I just wanted to throw that out there, because that is such a great combination. And yeah, easy to get fun information. So that's it, we
John Hubbard 57:28
have a ton of awesome tools in the list. I'm sure all the listeners electronic, furiously write stuff down, I am getting make sure all of this stuff we've mentioned gets into the show notes, thank you for bringing up all that awesome stuff. Hopefully that gets everyone set off on the right path. Before we wrap up the episode for today. Final thoughts, if you were to look into the crystal ball going forward for maybe the next year to three years, is there any particular direction you think malware is going to potentially go and what kind of threats we might face in the future,
Ryan Chapman 57:56
I'm actually really worried because I see it continuing on the path that it's taking now. And that is malware as a service ransomware as a service, or wrasse, the human operator ransomware that I referred to earlier and that I've been so obsessed with, you know, for the past couple of months, is a great example of allowing folks who are not necessarily I shouldn't say maybe technical enough, but kind of mean that maybe don't have the wherewithal to know how or even just the time to design a threat to get in. But they're able to just lease it from someone else. I mean, that's how it imitates. That's how trick bot, that's how those are designed and how they work. They all have campaign IDs associated, if I want a particular thing, or I want to break into a particular company, or I want my like click fraud on a certain number of machines, I can go to you know, the dark web, I just connect to Tor, right, whatever, and go to some sites and forum and say, Hey, um, here's the thing, I need click fraud against this domain from this country, right? I can be very, very specific, right? Not me, I mean, I'm not gonna do it. I don't think I do. And, and it will get done. And it's just that easy. I'm worried that that model is going to continue to grow. So what we're going to see more and more and more are families very similar to trick bought and imitate. And they are going to be used more and more for the initial infection vector. And we're seeing that all the time. Now, we're also going to continue to see CDs for network appliances and external devices, VPN based things. For example, the one was fairly big over the past couple of DVDs. I actually love the company so I won't say their name, but we all know they are. So we're gonna see a lot of ease of getting into environments but made even easier by the fact that it's just like, what's the group that uses anonymous, a lot of quote unquote CDs or script kiddies are in anonymous, however, at the same time, there are some legitimate folks in that group. But when they were taken down so many sites via DDoS, much of it was just low orbit. ion cannon right there are literally just they say hey grab this tool put this apn or hostname and hit enter for us right that type of design is what we're seeing more and more with malware and entry vectors and what i see continuing to also be a problem and exacerbating the problem is the fact that while those malware families are being sold as services we're gonna see more and more and much more money being made from that we're also going to continue to see the proliferation of open source software for quote unquote pen testers or for researchers or things of that nature everyone is scrambling to come up with new ideas and it has enabled a new world for the blue team the red team but also the black hats i can't recall the last time i worked a large case you know on our professional services team in didn't see literally like cobalt strike a cracked version oftentimes cobalt strike you know merges tool and everyone loves and respects that tool meta sploit often we see meta split reverse with handlers and cobalt strike beacons and we see all the powershell based stuff you see powershell empire to this day even though it's officially been retired on github it's still being used all these tools as they come out more and more and more it's just going to open up more and more and more avenues for the threat actors for once they get into an environment at this point i do not believe that there's any way to prevent any threat actors from getting in you're not going to prevent them from getting in if they want to get in they're going to get in i don't care if they're going to have to stick to it they're going to get it once they're inside at this point they're going to get wherever they want it's not difficult tools like bloodhound oh bloodhound is awesome as as a blue team and understanding defense issues and things like that and of course red teams go you know crazy for well why because it's too good right so we're going to see this malware as a service model we're going to see a lot of people making money because of it and it's going to continue to grow it's going to end up being considered a national industries at some point i mean at some places already i guess maybe not reported as such but you know we're gonna keep seeing all these folks pushing out these awesome tools that can be used for good but are often being used for bad
John Hubbard 1:02:20
all right well very interesting thank you very much ryan for this amazing set of information tools and all that good stuff where can we follow you online to keep up to date with all the stuff you're doing and is there anything else you want to shout out well you have a moment here
Ryan Chapman 1:02:35
yeah my first shout out to the blackberry security services team we have a consultancy that takes on all kinds of cases i have to push my work because i was with a very very large corporation one of the biggest in the world for seven years and moving into consulting and being around the former silence you know as we came from team i couldn't be more in love so absolutely love the guys that i work with just shout out to them that so supportive management i love it i my whole life changed i love it for me personally i train you know forensics 610 so if you want to actually take the course fine with me we could chat make crack jokes throughout class all day long you know i am now the lead organizer for cactus con so cactus con is arizona's premier hacker and security we are coming up on cactus con number nine which will be february 6 and seventh potentially a three day we're looking into that so come check us out and also i have a website it's incident response dot training it is literally html 0.1 it is horrible and it's basically just a couple of links and the links if you go there you'll find all my other stuff all my stuff so i have some pluralsight courses if you already have a pluralsight membership and i'm not trying to push the byline courses that no if you have a membership then go watch them check them out i think they're really good right on github i'd love for folks listening to this particular podcast to check out my github page which is just rj dash chat typically i go by rj underscore chat but on my github page i have various workshops i have an exploit kit workshop i had the carrier file workshop and i have two different network forensics workshops they're pdf files along with associated data that you have to pay attention to and i designed each of them when i originally presented them so i'm going back to like besides las vegas 2015 on step by step by step instructions so it literally says grab this file from you know here in github and then follow these directions if you want to understand exploit kits i have a workshop there for you i love putting that material out but i spend so much time building them that when you give me opportunities like this i'm like just go look at them like they're still super useful so go check those things out and then i primarily use twitter i like to use twitter a lot so i keep up with the infosec community there so i'm rj underscore chap on twitter i have some stuff on youtube i started doing some twitch streaming but it's all on youtube all this is linked Done instant response dot train. So I will some, at some point enable at least WordPress or something silly. And so right now it's just a couple links literally on a white background. But go check that out. Yeah, before we sign out I just want to say thank you so much for having me here. You know I love to run my mouth but I also absolutely love the blue team. You come across malware and artifacts all day long, you know, you see a mutex or a pipe that was created Google it, go check virustotal. But you're like, what is that it can tell you something that you didn't know by just looking at that. So you know anyone who wants to reach out on a more personal level, I am completely gamed to do that. I love discussing malware and Bluetooth stuff in general. So yeah,
1:05:39
that's my little spiel. Awesome.
John Hubbard 1:05:41
Well, thank you for all the links, all the information. And I will be sure to get all of this in the show notes for everyone. I know. We had a ton of resources mentioned here. So yeah, thank you for joining me on the blueprint podcast and we will keep watching all the awesome stuff you're doing going into the future. Awesome. Thank you. Thank you. Yeah, have a good one. Hey, blue teamers. Hope you enjoyed today's episode of blueprint. If you've got a second and wants to help support the podcast, please subscribe and leave us a review on Apple podcasts. It would be really, really meaningful to us and if you have any ideas or suggestions, I would love to hear them. Your reviews are going to be one of the best ways to help others find this podcast so anything you could do would be a big help. As always, thank you for listening. You can connect to me on social at sec hub sec HQ, BB on Twitter, or on LinkedIn. So until next time, thank you for listening to the blueprint podcast.