Insight ON Inside Man: What Threat Actors Know About Your Recovery Plan That You Don’t

Audio transcript:

Inside Man: What Threat Actors Know About Your Recovery Plan That You Don’t

Jeremy Nelson:

So there's been a report of an organization that got hit by ransomware, took them down for two weeks. They came back up, big, big announcement, Hey, great news. We're back online. Within 24 hours, they were back offline because they got hit with the exact same ransomware, by the exact same threat actor group. So they just didn't do their right level of due diligence on understanding what the kill chain was, what the indicators of compromise were, and removing persistence, like truly expelling the threat actor out of the environment before going to a full recovered state.

Jillian Viner:

If you're making technology decisions that impact people, budgets and outcomes, you're in the right place. Welcome to Insight on the podcast for leaders who need technology to deliver real results. No fluff, no filler, just the insight you need before your next big decision. Hi, I am your host, Jillian Viner, and our last episode, we sat down with Insights Chief Information Security officer Jeremy Nelson, and he left us in a precarious moment right in the clutches of a security breach. Today, we're gonna walk through what happens after the incident takes place, and hopefully Jeremy can take us through completely unscathed. Let's go. Well, Jeremy, I'm really glad to have you back because you left me in

Jeremy

The cliffhanger, right? In

Jillian

Dire situation. Yeah.

Jeremy

I mean, you've now been breached. Yes. I imagine you wanna get outta that.

Jillian

I do, I do. So, just a refresher, and for those that maybe missed the last episode, first of all, you should definitely go back to it because that was really,

Jeremy

It was a lot of fun to record. It

Jillian

Was a lot of fun, but also terrifying and really, really informative. Uh, so today is really our part two of that conversation. The breach has happened. Someone, someone has infiltrated the co, the company. Uh, so you're gonna help me get out of this horrible situation, and I won't make you be the bad guy anymore. I love

Jeremy

It. Thank you so much. I was uncom know that's really

Jillian

Uncomfortable. Uncomfortable, uncomfortable for you.

Jeremy

Uncomfortable for, yes. I'm glad to be back on the good guy side of this

Jillian

Equation. And ironically, you're black today, so

Jeremy

I know we switched, switched up the uniform a little bit, <laugh>, so it doesn't exactly align, but

Jillian

All right. So again, we really left it at the moment when I realized that, oh my gosh, I've been hit mm-hmm <affirmative>. Something has happened. Yep. That's a very critical moment. So, first of all, let's start with the itty bitty baby step as a user. And I see something that I shouldn't see that indicates that I've been attacked. What, what do I do in that moment? What's the best thing to do and what's the worst thing that I could

Jeremy

Do? Yeah. So the best thing that you could do is you really wanna make sure that it's coordinated. So if you're a user and you feel like you have been the victim of some type of an attack, and you are patient zero, if you will, the, the number one thing you wanna do is you wanna reach out to your IT and your InfoSec team, identify them, let them or notify them that, Hey, something has happened, this doesn't look right. And give them that opportunity to do an investigation to understand exactly what's happening, and then to be able to incorporate that into their predefined response procedures. That's the best thing that you can do. The worst thing that you can do is to just ignore it, to be like, eh, it was just a system glitch, or I'm just going to, I'll deal with this some other time.

Jeremy

Or, you know, thinking that it's just gonna go away. Mm-hmm <affirmative>. Um, and we have seen that is where a lot of times these things will happen in the background, even though you've kinda got this suspicion, like something wasn't quite right. Mm-hmm <affirmative>. Um, or my computer's not behaving normally anymore after whatever, you know, this particular website that I visited. And so you just, but you just kind of put it in the background. You're like, I'll just deal with it. And you just kind of go about your day. And one of the biggest risks that we have, it's allowing persistence. And so allowing this threat actor to remain in the environment undetected, um, and to continue to scan. And really what they do is, once they get into patient number one, victim number one is they move laterally. They just start to scan. They look for other things, other targets within the environment.

Jeremy

They look for other ways that they can spread. And the deeper that they can get, the more pervasive that they can get embedded into your environment, the harder it is to expel them. And so never just let it sit, even if it's just suspicious. Mm-hmm <affirmative>. Nobody in InfoSec, nobody in it is going to chastise you for reaching out and being like, Hey, I think something bad is happening. I think I may have been the victim of some type of a cybersecurity attack or a cybersecurity event, and I think I should be looked at. No one is going to be like, oh, you know, boy who crawled wolf, right? Like, no, they're, they're going to take it seriously. They're gonna investigate and it gives them the greatest opportunity to identify, contain the risk early on, and help manage the overall exposure to the entire environment.

Jillian

You know, I'm, I think I'm like that panic person that wants to reach for the plug <laugh>. Like, if I suspect something's going on,

Jeremy

It's not terrible,

Jillian

By the way. I was gonna ask, does that even make a difference if I shut

Jeremy

Down my computer? It totally does. Shut it down. Like, turn it off, did get it disconnected from the network. All those things are great. Um, it, there's a good chance that, uh, your IT team will either want to boot it up to be able to look at some things, or they might just have you ship it in to them so they can put it in kinda like a clean room environment in an isolated space and be able to investigate it. Yep.

Jillian

Now you say it's gonna go in there and kind of see what else it can connect to. So that means if I'm connected to what A VPN, if I'm connected to my email, like what are the vulnerabilities in that moment that it's trying to, to seek out?

Jeremy

So it, it can go a couple of different ways. So the first one is, is it's gonna look at what it has access to just locally stored on your computer, right? Like, are there valuable assets that sit there? Um, it'll also try and harvest credential data. So there's a certain amount of credential

Jillian

Data. So that document that I've saved my usernames and passwords. Ah, I'm

Jeremy

Just kidding. I would never do that. Please, please. No. Uh, but yes, like it will look for those types of things. Like there are different pattern matchings that will go out and look for various different sensitive information that can sit on your, your hard drive. Um, but even just by nature of the systems that we use, right? Like you take your laptop and is it always plugged into the corporate network?

Jillian

Maybe? No,

Jeremy

It probably isn't. Right? Like, there's lots of times where you wanna use your computer and you're completely disconnected altogether. Mm-hmm <affirmative>. It could be someplace where connectivity's not available, like maybe you're on an airplane and, uh, the WiFi's not working, but you still wanna be able to get some things done. So there are things that exist locally on the laptop that allow that to function. Some of those are cashed versions of your credentials. And so there are tools that'll, that'll actually go into that and be able to harvest those credentials and be able to use those as a way to crack them, um, and to give them further access into other parts of the environment, right. That can be used either directly on your laptop or remotely to get access to other systems that use those same credentials. So obviously harvesting the valuable resources and assets that exist on the existing laptop.

Jeremy

The other piece is, is that immediately they'll usually start scanning like, what do I have access to? So it's connected to a network. Most of our devices these days are connected almost all the time. Right? So they're obviously, even when I asked you that question, you had to think about it. Uh, you know, maybe yes, maybe no. For the most part, yes, for the most part, our systems are connected to a network. And so what they wanna do is they wanna see what else is connected to that network. If I was able to exploit this particular attack and get access to this asset, are there other chances are this is a standardized set mm-hmm <affirmative>. Right tool set that, that you're using as an employee, I'm gonna go out and scan and see if there's any others out there. And if there are, chances are I can use the exact same attack to get access to them that I used to get access to. You

Jillian

Got it. Alright. So now a user, I've contacted it, it is on the case. Yep. Who's the first call?

Jeremy

So the first call, um, so, so I guess, we'll, we'll walk it this way. So there's usually differing degrees of, we'll call them, uh, cyber, cyber incidents, if you will. Right? So some of those are very isolated, right? So we catch it early. It's relatively, I'm not gonna say it's low risk, but it's low impact at the moment. So we have an ability to be able to isolate, identify, and contain it on that particular asset. Maybe it is just as much as like, Hey, let's just power that down. We don't see that it's spread, right? Let's bring it in. We're going to look at the, uh, indicators of compromise associated with this particular attack, and then go out and see if it's existing anywhere else in the environment. See if they move laterally, laterally. So that's the, usually the first thing that they, that they're going to do is they're going to try and isolate and contain on the device that's been identified, but also look for any other per, uh, any other activity, presence of activity across the entire IT environment.

Jeremy

So you use a variety of different tools for that. Threat hunting tools, sims or MDRs, all that kind of stuff to go out and look for other activities that might be embedded in your environment. So look to see, is this isolated or is this much broader? Um, then you start to get into like the, the middle tier if you will, where it does look like some level of spread has happened, or, um, they were able to exfiltrate some amount of data off of that particular asset. Then you start to identify like what's the overall risk of the credentials, whether or the data that's been, um, that's been basically shipped outside of the controls of your environment. Um, and then how do you actively respond to that? Then you get to the big one, and this is the one that people like, that's really makes the headlines because they're so big, they're so impactful.

Jeremy

And you're talking about like something that gets to the scale of like ransomware, where it literally takes down the entirety of your IT infrastructure. Um, and so the way that you would respond kind of depends on where it falls within those three different tiers. So usually at your IT SOC level can handle tier one, tier two, you're usually bringing in some leadership because there has to be an examination of sensitivity of data that's been leaked outside. What's the overall spread, really kind of extending your search a little bit further to see how far this particular threat actor was able to get into your IT environment. But still, the overall intent is to, you know, isolate, contain, and remove them because IT operations are continuing, you're continually able to operate your business. The third one is full on, you know, five alarm type of a situation.

Jeremy

The worst nightmare. You're the worst nightmare. Your business is literally offline. And so it's a much more severe type of a response. And so usually those calls you're bringing in the executive committee, um, and usually, uh, your legal team. So those are, that's typically where those types of conversations start. Now, I would imagine for the purposes of our exercise, right, the two smaller ones that can be typically handled by InfoSec and and your, your SOC team, those are pretty run of the mill. I think what we're probably gonna be focusing on a little bit more is that top tier, the major incident, the five alarm, the major incident mm-hmm <affirmative>. Exactly

Jillian

How many companies know who to call in that moment.

Jeremy

So that's where the incident response or crisis response plan comes in. Um, I think I shared this. You know, I, I, I've done a number of different recordings with Jillian and, uh, I always lean in really, really heavily into having that plan. So it always comes back to, um, I'm, I, I, I love, I love me, my sports. And when you're coming in and you're playing a game, like you want to have a plan, you never wanna just kind of come in and wing it, right? The, the more orchestrated that you're going to be, the more effective you're going to be able to carry out your plan and achieve your, achieve your objective. And so it's no different when it comes to incident response. You always wanna start off before the crisis to actually understand and build out a coordinated plan on how you're going to respond to these types of events. So I think that's step number one is making sure you have a plan. Step number two, making sure you're practicing that in peace time, right? Yeah. You don't want to, you don't want to, the first time you exercise your plan is when you're in the middle of a crisis because people are distracted, there's a lot of noise. It's really difficult. And so if you's, if it's not a part of your muscle memory on how to execute that response, you're just as bad as almost not even having a plan.

Jillian

Yeah. You and I have talked about this numerous times over the years. Yep. And I think you've told me before that you've been doing instant response for, for years. You've been in security space forever, you've been part of those teams. And I think you're always surprised at the number of companies who have a, maybe they have a

Jeremy

Plan, have a plan,

Jillian

But it's collecting dust in a chore.

Jeremy

Yeah. Yep. And, and it's, it, uh, it's easy. It's easy human nature, right? Because we've got a lot on our plates, and just like every organization, we're not getting more resources, right? We're constantly being asked to do more with less. And so annually, or semi-annually, whatever your review plan or your review process looks like or schedule, you come through and you're like, do we have an incident response plan? Yeah. Remember we developed that. You check the box, it becomes a check the box exercise as opposed to an integrated part of your business operations, which is really what an incident response is. Mm-hmm <affirmative>. And I'll even throw it out there that, um, I, I even said this a little bit earlier, is that it's not even just incident response through a cybersecurity lens, it's crisis response. And that could be through a number of different avenues where your communications or your operations is disrupted by a variety of different external factors, whether that's cybersecurity event, whether that is natural disaster, whether that's, you know, uh, uh, I'll even throw, you know, looking back at nine 11, like that, that was really a huge test on crisis response.

Jeremy

It was an act of terror that actually happened against a variety of different organizations that completely shut down operations. In fact, we saw a lot of disaster recovery solutions really spin up as a result of nine 11, because a lot of people just really didn't even foresee anything like that ever happening. Yeah. So, um, so it really does come back down to crisis response and building out what that plan looks like. And it's just good practice for business resumption, business operation and continuity. That's

Jillian

A good reminder though, that it's not just a security incident that's gonna trigger some of these actions. It could be anything. And I feel like that's particularly important right now, because we talk about the power grid Yes. And how sensitive the power grid is. We've got, you know, storms and monsoon season here in Arizona Yep. And all of these things that absolutely could cause issues. So you mentioned getting legal involved, but it sounds like you kind of need sort of your, your standing operator procedures. And maybe there's a different way that you phrase this for security teams, but like, who's on the first call list? Who's on your second wave, and how do you make sure the right people have the right communications at the right time? And that includes the public, right? Yes. Because you can only withhold this information part of that from the

Jeremy

Public for so long. Correct. Well, and especially with the new SEC regulations mm-hmm <affirmative>. As far as like, you know, the degree of impact on your organization and having to have, you have an obligation to be able to file a K eight that actually Right. Basically attests to, uh, cybersecurity event or major incident happening within your environment and the overall risk and impact to, to data and, and, uh, digital assets and, and your operations. So it's, there's a lot. It's, it's, and every year it gets more and more stringent as far as like what those communications look like. So you're spot on. So there's various different degrees where it starts, and then how do you bring more and more people into the understanding and visibility that something is happening? Mm-hmm <affirmative>. But I think it's also important to recognize that there's a certain level of compartmentalization that does have to happen. Not everybody gets all the information. Um, so why

Jillian

Is that? What's the importance of that?

Jeremy

So there's, there's sensitivity as far as like the information that's being leaked. Um, but I think one other thing to kind of consider is, is that when you get into these high pressure, um, highly complex types of response activities is that the role of leadership really shifts and can almost be a distraction because you'll end up having various different work streams that are happening. And the leadership team obviously has the greatest level of overall influence and responsibility to the business, but at the same time, they have an undue sway over the direction that technical people will be taking. And so when I talk about compartmentalization, is that one of the things that we highly encourage is separation between technical work streams and executive communications and directives mm-hmm <affirmative>. And so making sure that they're not combined together. Because some of the most damaging behaviors that I've seen is where you have, um, A CEO sitting on a tech bridge that's trying to give direction on how the technical team should be responding or where their priorities should be in certain areas, um, especially without necessarily knowing all the downstream impacts and technical considerations that have to go into these recovery efforts.

Jeremy

And what ends up happening is you end up, 'cause most employees won't say no to A CEO, right? So A CEO says, go do this right now. They'll drop whatever they're working on, they'll go and address that, but they won't realize the chain of events that just got set into motion and other things that are now going to be delayed. And so, um, one of the things that we really stress is creating a separate tech bridge and a separate executive bridge, but then really focusing heavily on a communication gap if you are basically a, uh, a bridge that allows those two to stay in sync with each other so that the executives are fully, always fully aware of the progress that's being made by the technical teams. And any decisions or activities that are happening within the executive space are getting appropriately communicated down to the tech team. So it's, uh, it's really, really important.

Jillian

Yeah, that makes sense. I mean, you're right. People aren't gonna say no to the ceo. No. So, no, let's talk about that bridge when you get on that phone call, and this is, I think, a scary part that you've, you've alluded to in a previous, uh, episode. What is the reality of that security call today that wasn't true, let's say two years ago

Jeremy

When we say the reality because

Jillian

Of ai?

Jeremy

Oh, because of ai. Okay. So, so yeah. So there's, there's a lot of things that, that AI has introduced that's, that's, um, that really amplifies some of these first kind of these first responses. So I think one of the things we're really starting to see is, you know, we talked about this a little bit on our previous conversation, is DeepFakes, right? So DeepFakes are a great way of not only getting your toe into an environment, but also how do you remain persistent. So typically what happens is, is that you assume, right? So this threat actor gets in and they get in through exploiting either one asset or one person. But the whole point is, is that they get credentials. Those credentials are usually uniform across all the various different systems that are integrated across your IT environment. And that includes your collaboration space. And so one of the default behaviors is, is everyone just falls back to using email communications.

Jeremy

They fall back to using the various different IM and video conferencing tools that they typically use. Well, guess what? Those credentials have been now compromised. And so that threat actor has the ability to potentially perpetrate as the person that they compromised. Mm-hmm. And it gives them access to be able to view into the various different response activities that are going on and being orchestrated by both the leadership team, the incident commander, your response team. And so there's a certain level of care that needs to be taken to make sure that the people that are participating in their response are just, the good people are just

Jillian

People, the organization. You've literal been in the midst of an instant recovery and realize that someone who you thought was on your team is actually, they're like, there's a rat in the room. Yep.

Jeremy

Yep. So that's, that's, that has actively happened. And we recognized, um, that something was, was off. What

Jillian

Was that moment like? Can you tell me what, what went down?

Jeremy

Yeah. So, so everyone jumps on and we're sitting there, we're having a deep con we're we're really in the, the throttles of the hole situation. And, um, we, we look over and we just look at the attendee list and we're like, man, this just doesn't, like, this doesn't feel quite right. Right? Like, something just seemed off, like some people were dialing with phone numbers. Some people were coming in with, uh, normal IDs. And so, uh, immediately just, it was just a gut reaction and instinct, Hey, everyone, go camera on. And as soon as the, and as soon as the mandate went out that no matter who you were or where you were at, you had to go camera on, all of a sudden you start seeing participants dropping off. And instantly we knew that the communication channels coordinating our incident response were being, were already being, um, the, the, the, the threat actors were already participating in them.

Jeremy

They were already being compromised mm-hmm <affirmative>. And they were actively listening to all the counter measures that we were putting into place to combat against them. And they were basically being, they were able to stay one step ahead of our response to recovery efforts because they were actively listening in on those activities. So that's another thing that, you know, when we talk about like recommendations, we often talk about what's your out-of-band communication plan? What's the tool? What's your method for connecting with the right people and ensuring that you're only connecting with the right people? Uh, because lots of times, whether it's because of availability, because they go through and they, uh, execute an attack that takes your email and your instant messaging and your normal paths of communication down, or you can no longer trust them. Yeah. You need to have some type of outand way of bringing the core team together to effectively manage communication.

Jillian

Even if you do that, I mean, we talked about earlier about DeepFakes, like DeepFakes is how they potentially get in Oh, yeah. How they get wire transfers. So how do you even know if a deep, like, if you go cameras on, how do you know you're look not looking at deep fakes?

Jeremy

So that's, that's absolutely risk. So, and that really comes back to, um, a, a big kind of talking point that I've had, or as far as like how do we battle deep fake, right? Yeah. And deep fake really comes back to really focusing in on what's your true identity. It, it, it's a, it's an identity verification problem that we're trying to deal with. And so what we start to see is typically it's just a username, password, and maybe MFA, and we check the box and we're like, okay, this person is who they are, who, who they say they are based off of these factors that we've been able to use to validate their identity,

Jillian

Multifactor authentication,

Jeremy

Multifactor authentication, username, password, all that kinda stuff, right? Mm-hmm <affirmative>. Um, but more and more we're starting to see where, you know, credentials are getting leaked and compromised. Um, even MFA tokens are being transferred over to different devices. So that's not necessarily a complete protection. And so really the whole concept starts to shift into how do we do true verified, ID like realtime video, like, uh, facial recognition, uh, at point of time of, of event, right? So you can't just log in, like you'll, you'll actually require extended verification that includes like realtime facial recognition Hmm. Or some other type of out band security, like, uh, like a, a identity wallet or something of that nature that is outside of the normal authentication flow that allows you to truly, I identify and validate the, the identity of the person on the other end.

Jillian

Terrifying. So even when you think you're safe with the people that you trust to get you out of a situation, you still have to be on guard. Yep. And, and work through different identity verifications, maybe some different security bridges, uh, to make sure that you're actually Yeah.

Jeremy

Speaking in confidence. So here, here's a great example. So coming back to that similar situation where we were talking about, uh, the threat actors actually joining into the incident response bridge mm-hmm <affirmative>. Um, this is a particular organization where the primary point of attack was their identity store. And so what we literally had to do is help them rebuild it from scratch. And the way that we did that is you, you called a phone number on file. We started with the CEO, and we got them to validate themselves. We created their account in a whole new directory environment, sent them the new password, made them, you know, we sat there on the phone with them, waited for them to change their password, and then once we had them validated, it was then, okay, who are your direct reports and what's their contact information that you have? And we literally had to build this out like a tree, like a call tree, starting at the very top of the pyramid and working our way down through and recreating only people that we could directly identify and verify who they were in person. Wow.

Jillian

Yeah. So that's the nightmare within the nightmare is realizing that your incident response is already infiltrated. Yep. And you have to have a switch plan so that you can actually safely and securely come up with a plan. What should it teams not do in that recovery model? And I'm thinking about things and tell me if I'm wrong, but like, how do they make sure that they're not so quote unquote, like compromising the crime scene? Or, um, maybe, I mean, you mentioned earlier like they're checking the scope, right? To make sure it has an infiltrated your, your face kind of lit up, so Oh, yeah. I'm sure

Jeremy

There's things that are coming mind for you. This is a question. Okay. This, this one's near and dear to my heart. So what we've seen is, is that what, so if you were to think about it, if you were to put yourselves in the shoes of an IT person, so you know that your system has been compromised, and let's say it's a ransomware, and so you've got this massive pool of data that is no longer accessible and you know that it's essential to keep the organization running, what's your gut reaction to do?

Jillian

Hopefully I have a backup so I

Jeremy

Can, there you go. A hundred percent. Yeah. Hopefully you had a backup. And then what do you do? You move to restore mm-hmm <affirmative>. Right? Oh, cool. That's what we do backups for. Let me just restore it. Right? The challenge is, and so I'm gonna come back. So your question was, what should you not do? Mm-hmm <affirmative>. Don't rush to re restore. So one of the biggest, so there's two different pieces to that. Number one is, is that you could potentially overwrite, uh, overwrite evidence that you would need in order to be able to identify this threat actor and be able to take the appropriate actions and things of that nature. But most likely what's happened is, is that, uh, either A, your backups have been compromised as well, or B, you've been backing up, like the threat actor's been persistent in your environment for a period of time, and you just have all their tools and all their indicators of compromise backed up.

Jeremy

And so what are you gonna do? You're just gonna restore fresh data that has the exact same vulnerabilities in it, and they're just gonna detonate again. But now they're going to know that you're actively in a recovery mode. And so you've actually tipped your hat your hand a little bit. And so it makes negotiating with them a little bit harder because they know that you're actively working to recover, and you might actually have good backups that they weren't able to get to. So the number one thing you don't want to do is rush into recovery. That's where forensics really comes in. So, you know, we've done, I haven't done a great job of kind of walking you through the individual steps here, but we talked about, you know, executive communications. We talked about getting legal involved. We talked about managing communications across the organization. The next one really comes into how do you get an incident response team engaged, heavily focused around forensics?

Jeremy

Because what you wanna do is you wanna partner with these experts to really go through and kind of with a fine tooth comb, identify the various different activities, log data files, and understand what is the kill chain. The kill chain is basically how did the threat actor actually penetrate your environment and land this malicious content inside of your environment? So understanding both how did they get in mm-hmm <affirmative>. But also what are the tools that they use to actually perpetrate the actual attack? Those are called the indicators of compromise. And the reason you want to do that is because, A, you wanna make sure that you effectively close the door so they can't get back in through the same means. And B, you wanna be able to contain and or, and remove those indicators of compromise, because both, you wanna get them out of persistence of living in the environment.

Jeremy

But as you're going through and doing restores, you wanna restore that into kinda like a clean room space where you can restore it, look for those indicators of compromise, cleanse the restore, and then promote it into production. And so all this comes back to your question. Don't ever rush too quickly into recovery. Make sure that you've really done your due diligence on understanding how did they get in, what is it that they used? What do those binaries look like? What are the changes that are made to the system? And then being able to appropriately contain and cleanse. So when you do the recovery, you can make sure that it's recovered in a good, healthy state. That doesn't mean you leave you vulnerable for future attack. So we've actually seen that happen before. So there's been a report of an organization that got hit by ransomware, took them down for two weeks, they came back up, big, big announcement, Hey, great news.

Jeremy

We're back online within 24 hours. They were back offline because they got hit with the exact same ransomware by the exact same threat actor group. So they just didn't do their right level of due diligence on understanding what the kill chain was, what the indicators of compromise were, and removing persistence. Like truly expelling the threat actor out of the environment before going to a full recovered state. Great lesson learned. And so when you think at ransomware, right? Mm-hmm <affirmative>. That's the first part of that. It's ransom. Mm-hmm <affirmative>. You're trying to get money. So if you think about like, if they were trying to avoid having to pay that ransom, if you get hit a second time,

Jillian

I mean, you're really at

Jeremy

Their mercy, you're a much harder negotiating place. Right?

Jillian

Anything else that it should not do?

Jeremy

Um, so I think another thing that it should not do is you, you really, and it, it's really hard to say this, but you wanna try and keep that the, the information that you've been compromised relatively contained, um, because one of

Jillian

The, the fact that you've been compromised or the data that's been compromised, the

Jeremy

Fact that you've been compromised. Okay. Yeah. So you don't necessarily want that being completely spread. You wanna make sure that you're really effectively managing the extent and the overall impact and, uh, how, especially the how mm-hmm <affirmative>. Um, you really wanna kind of keep that within a very, very tight group. And one of the biggest challenges is, is that we're starting to see that insider threat is a real thing. Um, well, you know, a lot of organizations are really kind of amping up the way that they're looking at insider threat. So people who have a business reason to be connected to their environment, but are for all intents and purposes, the threat actor that's embedded in their organization. And so, um, the, the more that we can kind of manage and obscure, uh, some of those communications to the broader overall populace of the organizations that we support is probably, you know, healthy until we get to a point where recovery is imminent. And, uh, even then, like still trying to manage those communications is extremely important.

Jillian

State the obvious. Why is that so important?

Jeremy

So the reason is, is because we still want to try and keep the threat actors out. We're, we're trying to avoid reinfection. Um, there, and there's a number of different reasons as far as, um, so we talked about ransomware, we talked about, and a big part of that is, is negotiation. Mm-hmm <affirmative>. So ransom negotiation. And the more that the more people that understand, the more people that are gonna potentially talk, especially with the presence of social media, things will tend to get out, um, that can get consumed by the threat actors. And it puts organizations in a tricky con, a tricky type of a situation when it comes to negotiating.

Jillian

So you're already vulnerable. Yep. You don't wanna make it worse. Correct. By letting others know that you're vulnerable. Yep. 'cause even though we would want to trust all of our teammates, that's

Jeremy

Sometimes

Jillian

The reality,

Jeremy

Right. Sometimes we can't. Right? Yep.

Jillian

But you definitely let your legal team know. You let your PR team know, and you limit your internal communications until it's absolutely necessary. Yep. And you mentioned earlier the, the eight K form mm-hmm <affirmative>. That has to be completed within four days, I believe it is, of an attack. Yep. And you have to know it's tight the right time, it's tight to do that.

Jeremy

Yeah. And you need to understand, like, and that that's really like a legal and executive really call as far as like understanding what's the overall extent and impact of the incident that's occurred. Like, does this actually trigger and require this, the filing with the SEC? So it's, uh, again, it's, it's a very delicate, it's a very, very delicate balance where you wanna do the right thing, but you also don't want to falsely alarm people or, you know, take a, I don't wanna say take accountability, but you don't wanna necessarily take blame for something that, uh, that isn't entirely within the scope of what you would be communicating out.

Jillian

Are there any final steps before we get to like, the after action of what happened?

Jeremy

So the big one is it really does come back to recovery. So I think that's one of the biggest things. I talked a little bit about this, but you know, one of the things that's super important is making sure that you are properly cleansing your environment so you're not just restoring good into bad mm-hmm <affirmative>. And so that's one of the, the big areas where, uh, we've seen organizations need a little bit more support. And so when I talk about recovery, it's not necessarily just purely a technical, it's also a people type of a conversation. So a lot of times, I'm just gonna throw this out there, um, it, people are very, um, they, they take a lot of pride of ownership in the systems that they construct and they support. Sure. And so when something like this happens, they're the first people that throw themselves at the bus.

Jeremy

And I have seen people work 60, 70 hours straight, no sleep, just continue to work, work, work, work, work. They feel like it is their personal obligation or responsibility to pull the company out of this, out of the, out of this, out of the fire, if you will mm-hmm <affirmative>. And so, um, they don't take any consideration for their own personal care or their own personal health. So, um, one of the big things that I really try and highlight to organizations and leaders that I meet with is as you're building out that plan that we talked about, making sure that you understand like, what is your flexible staffing strategy? How are you going to be able to bring people in at the right time in order, because this is not a sprint, sprint, this is a marathon. Like most of them, most of these incident response recoveries take 10 days.

Jeremy

Plus we've seen some that take up to 90 days. And for a lot of those, you're running in a 24 by seven operation where you got people around the clock that are actively working on recovery efforts. And so you can't do that with your existing staff. So number one is making sure that you have a really good flexible staffing model built into your incident response plan. And that's not just necessarily knowing where you can go to get labor, it's also making sure that you have good documentation so when you go out to bring that labor in, you can get them onboarded and familiar with your environment as quickly as you possibly can. That is so, so important. I, I literally can't stress that enough, is that recognizing who are going, who are those trusted partners in your ecosystem that, you know, bring good quality resources to bear, but providing them with the means so that they can get onboarded and effective as quickly as possible. We were, we were supporting another client in this same type of an event. It took us two weeks. Like we were able to amass a army of technical experts to be able to help them with their recovery efforts, but we couldn't get them busy because we had to go through and do a series of discovery in order to be able to get the familiarity that we needed in order to be able to properly augment the existing team mm-hmm <affirmative>. And continue to re the recovery efforts forward faster.

Jillian

It sounds like that documentation then should be part of your onboarding with a security team.

Jeremy

Yeah. And, and that's the thing. I think a lot of organizations today, they look at, you know, the need for documentation. It's, it's kinda like a, a running joke, if you will within it, is that we always let our documentation slide, and I'm guilty. I, I look back on my operations days, um, back a few years, and I was terrible at maintaining my documentation. Um, but not only is it good for just general recovery and operational best practices, but when you think about it through the lens of incident response and rapid resource ramp up, like it's it's absolutely critical. Yeah. Like, it's, it's almost like a life and death type of a situation.

Jillian

So what I'm gathering from you is, yes, there are technical considerations and there are processes to all this, but the other piece that I'm hearing is really the people impact mm-hmm <affirmative>. Like you mentioned at the beginning, like having that plan so that when you do have to act on it, it's, you're not figuring out on the fly, which causes just, you know, insurmountable stress on top of what you're already feeling. Don't act on impulse, so don't restore data in the moment. Yep. Um, and then having that, that that plan for, uh, contingency resources so that you don't have teammates working on the clock, you know, 24 hours for eight hours, like no one makes good decisions, no off of no sleep. No.

Jeremy

So no bad things happen. Yeah. Like, even somebody with the best of intentions, we're not. So we're talking about the good guys who are helping to try and recover. You go, you know, you hit like 40 hours of no sleep under probably some of the highest pressure moments of your entire career, your life. Like, you're, you literally probably feel like you're battling for your job. You feel like you're battling for the survivability of your company. Like it doesn't get much higher stakes than that. Yeah. And so when you're operating with that emotional element coupled together with just high intensity technical work, mistakes happen. Yeah. Mistakes happen a lot. And it's not out of malice, it's just out of just human nature and, and over exhaustion.

Jillian

Yeah. We're not built that way. No. So let's assume that we've done everything right. Okay. We had our incident response plan. Yep. We called the right people in the right order. We let the right people know the information. We kept our communication channels separate between our executive branch and our technical branch with really good communication. I love it. Between the two. Love it. Um, we've got everything under control and we avoided ending up in the news. Love it. That's a win. Best

Jeremy

Case scenario,

Jillian

Best set case scenario. Um, that's it, right? We just put it all in drawer and move

Jeremy

On. No way. No way, no way. So I'm gonna come back to, we talked about the incident response plan mm-hmm <affirmative>. And there is, you never let a good crisis go to waste. Right.

Jillian

<laugh> Okay. Says Jeremy Nelson. That's right.

Jeremy

Trademark <laugh>. Uh, so the, so the whole situation is, is that, you know, we talked about you build a plan and you wanna practice that plan. Mm-hmm <affirmative>. Well, there's no better way to practice a plan than in a real live incident major event response. And so the, the worst thing that you could do is to just come out the other side of it and go back to business operations. Doing an effective after action review and lessons learned event is absolutely essential, because that is going to directly reflect and get incorporated into your incident response plan, as well as your maturity of your cybersecurity operations. Like you should understand and learn, okay, how did this particular threat actor get in? What are the things that we need to incorporate into our cybersecurity best practices in order to make sure that this doesn't happen again? Whether that be through, uh, technical tool implementation, whether that be through, uh, organizational change management and employee training, whatever that might look like mm-hmm <affirmative>.

Jeremy

Um, but making sure that you're going to have lessons learned no matter how well you've practiced. What's Mike Tyson's? Right. Everyone has a plan until they get punched in the face <laugh>. Right? And so you had a plan, you executed it. There is not a single plan that goes off just as you'd scripted it, just as you'd practiced it, just as you table topped it. When you go into a real world event, you are going to learn things about yourself. You're gonna learn things about your organization, you're gonna learn things about your technology. You're gonna learn things about your partner ecosystem. And so the, the best thing that you can do is after you come out the other side, set aside some time while it's still fresh, get everybody get, get the key players back into a room and do that after action review. I would highly suggest bringing in almost like a neutral third party to run the after action review, kind of through a normal scripted process. Um, so that way they're not emotionally tied to any of the responses. They're, they can help facilitate a very open and honest view of the way that things went down and really capturing those and then incorporating those into your incident response plan and your future tabletop exercises.

Jillian

No organization too big or too small for good after action review? No,

Jeremy

Not at all. I mean, we do them all the time for normal operations, right? Mm-hmm <affirmative>. So when you think about like the day-to-day conducting of business, we always wanna learn the, the way that you get better is to learn from your mistakes of the past. And in this particular instance, and not even necessarily your mistakes, you should learn from your successes too. So you should after action review things that went well and say, Hey, we should do more of that. Mm-hmm <affirmative>. Like, why aren't we doing more of this? It's the same thing with an incident response. You're gonna do things that are really, really good and you need to amplify those, but you're gonna identify areas of opportunity and growth, and you wanna make sure that you put corrective measures in for those.

Jillian

Yeah. I'm glad that you said not just your mistakes, but also things that did well.

Jeremy

A hundred percent. Yeah.

Jillian

Um, here's a little bit of a curve ball. What happens when it's not your breach?

Jeremy

Ooh. So this is an interesting one. So you can mean a couple of different things. I'm gonna, I'm gonna lean into this one because basically one of the things that we're seeing a lot of organizations focus on heavily this, these days is third party risk management mm-hmm <affirmative>. And supply chain. And so really understanding what are the downstream impacts to you as an organization if somebody in your ecosystem is impacted by a major incident. Um, so, so yeah, I think that's really where third party risk management programs really come in super effective, because not only does it help you understand the security profile of the organizations within your supply chain, but it helps you understand like what's the overall impact. Like, what's the role that they play within your business? And then understanding where do you have overlap. So if they, so typically what you wanna establish is some type of notification so that, um, you are included in the notification program within, within an or within your third party partners incident response is that when it goes to notifying partners that you are part of that notification process and then building into your own response, like, what's the risk?

Jeremy

What does that mean as far as risk to your business? What are your contingency plans and what are potential controls that you need to put into place in order to be able to protect you from being the next victim in the chain from that particular third party or from the, the, uh, major incident that's happened within that third party.

Jillian

So that could potentially be, let's say there's a piece of software that most of your company uses. It's been, it's been impacted, it's got a security breach. Yep. You could tell your employees what to stop using that software for right now. Did you shut it off your network? Is that the kind of approach that you're looking at? So,

Jeremy

You know, we'll just use an example that's fairly current, right? So, uh, there's a network monitoring software package that was infiltrated where basically malicious code was embedded into the actual product itself that got pushed down to every organization that was consuming it through automatic updates. Right? So I think that's kind of what you're hinting at mm-hmm <affirmative>. So in that particular circumstance, yeah. Like that's, that's scenario where you need to, um, you, you do kind of go into incident response mode because that third party had a way of directly landing by executable binaries inside of your environment. So there's, there's other ones where there may be a little bit more cursory. So, uh, we'll, we'll just say like, say your logistics provider is hit by some type of ransomware, and so their operations go down, they can't actually direct vehicles. They can't actually load and deliver packages.

Jeremy

Tracking systems are down, things of that nature. There's one particular response that you would have as far as like operational and, you know, business continuity associated with that third party impact. It doesn't necessarily rec, uh, uh, represent a way for that threat actor to laterally move into your environment. Whereas with the one that we were just talking about, this is a lateral movement where you've actually allowed that third party to have the threat actor land executable and potentially expose your environment as well. So, yeah. So coming back to your point, that would actually turn into a similar incident response where you have to go through, you'd have to isolate that system. You need to pull it out of your environment, but then you also need to go start, and it's like a threat hunt. You're looking for the persistence and presence of threat actor activity in your environment that could have used that particular tool and a software supply chain as a launch point into your IT ecosystem.

Jillian

Got it. Um, Jeremy, I wanna end with you with five insights. Okay. It's kind of our speed round. Okay. Oh boy. Name one security red flag that leaders overlook or dismiss.

Jeremy

So, uh, the red flag that I think a lot of leaders overlook or dismiss, especially in the context of what we've been talking about today, um, I would come back to, uh, immutability of their backups, um, a lot of times for speed purposes, right? So they recognize as a lot of the backups and data protection systems that we use today are heavily focused around just what I would consider run of the mill IT operations outages, right? Some, some type of a failure that happens within the IT estate that you support. And so data protection is there to restore that quickly and get you back and running. It doesn't necessarily take into account threat actor activity. And so, uh, a lot of organizations are content with, you know, Hey, I've got backups, they run this long. Um, but not putting those in a protected area where those are immutable so that the threat actor wouldn't be able to get access to those and make them unusable when it comes time to restore. Um, because that's typically what we see. The behavior that we, that we see is that as soon as they get in, they start looking for where are the backups? Because they wanna attack those first and then go after the, the real data. Because as soon as real data gets impacted, their presence is now known. And so you immediately start falling back to your data protection. And so making sure that your data, your, uh, your backup datas, uh, your backup data is protected and immutable is extremely important. I don't see enough organizations incorporating that today.

Jillian

So assuming your backups are safe, red flag. Yep. The most cost effective action leaders can take today to increase their security posture is what,

Jeremy

Okay. So we talked about the plan. That's probably the most cost effective because you're using existing resources who know your business. Um, bringing them together, working on that plan and building out, and this is gonna seem so silly and so rudimentary. Build a culture, like start collecting cell phones, start collecting ways of connecting and communicating with key individuals out of band from a typical email and instant messaging platforms. So it's super cheap. It almost costs you nothing mm-hmm <affirmative>. Uh, but it is so highly and positively impactful when you're dealing with an incident response

Jillian

And then print it out and keep it

Jeremy

Everywhere. Print it out and keep it someplace. Yeah. Where it's handy. It's almost like the, uh, the football

Jillian

<laugh>. All right. I'd love to ask you this one, 'cause I know that you've fallen for this. What's the easiest type of email? I knew it to test employee vigilance.

Jeremy

Alright. So for me it was obviously a DocuSign request at the end of the month mm-hmm <affirmative>. Um, but I think one of the, one of the easiest ones to send out is just a, uh, and we see this, uh, it's, it's actually proven is that you just send out like, Hey, you actually send an email disguised as the, the platform that you use for communications, right. For identity services and say there has been, um, uh, there's been unexpected activity identified. Click on this link to check your activity and reset your password. Oh, yeah. So a, you're already instilling a certain level of concern mm-hmm <affirmative>. Fear, right? Mm-hmm <affirmative>. Personal accountability and responsibility. Um, and so there is a natural inclination to click on that and to help secure your identity, but what you've actually done is the exact opposite. So, so that's the, that's the most, most effective one that I've seen.

Jillian

That's a good one. What's one question that leaders should be asking their security teams but don't?

Jeremy

Uh, so I think one of the things, so I I, I'll come back to, we talked about this, like what's our, what's our process and procedure for flexible staffing if we become the victim of an event? Like how are we going to recovery if we have to move to 24 by seven operations for extended period of time? Mm-hmm <affirmative>. How do we do that? Like, what does that look like? Do we have the appropriate documentation to onboard resources quickly?

Jillian

I was gonna say, do you have the documentation done that right? Is it updated? That's right. It feels like every product right now is AI infused, right? There's AI in everything. There's AI washing, but it does almost feel like AI in security is probably an area that you need to pay attention to. Like, is there, in your opinion, if a security product doesn't have some sort of AI component is even worth putting in your stack?

Jeremy

It is. 'cause I think the interesting thing that's happening within AI right now is that there's a lot of ways to actually plug AI in through other means. So even if the tech stack, even if the particular technology doesn't have AI built into it, it probably has a way of like, whether it's through MCP integrations or some other way to be able to incorporate AI into it. Um, so, so yeah, I wouldn't necessarily rule it out if it doesn't already have an AI capability, but I would be looking for ways to incorporate that into whatever your AI strategy is. Because, and I say this all the time, security is a data problem. Like in order for us to be successful at what we do, everything, like I was talking about with forensics, everything from like the, an, the perspective of a SOC analyst, it all comes back to just get data gathering data, analyzing data, looking for trends, looking for anomalous behaviors, and being able to amplify those as events that warrant further investigation and response. And so AI is awesome at solving data problems. And so, yeah, it's, it's a, it's a space that is very uniquely ripe for integration and enhancement through the incorporation of ai. But just because the technology doesn't have AI built in doesn't necessarily mean that it's not a worthy component of your tech stack. And I would even go the opposite. Just because something has AI doesn't necessarily mean that it's an effective implementation of AI that's actually going to accelerate your ability to respond, increase overall security, or reduce operational complexity.

Jillian

And finally, because not every organization has a chief information security officer, especially one as talented as yourself. So if an organization does not have that role, you talked earlier about the importance of separating communications between IT leads and exec leads and how if you have them together, it can really cause more damage than good. So if you don't have a chief information security officer in place, who takes the lead during an incident response. Yeah.

Jeremy

So it's gonna depend on the organization. So I hate saying that that's such a consulting answer, right? <laugh>? It depends, but I'll, I'll kind of, but I'll actually get a little bit more precise than that. So, um, if it's a relatively small organization, usually like your lead IT person will kind of step up. Um, if it's a slightly larger organization, usually like a director of infrastructure, we'll kind of step into a role of this nature. Um, sometimes depending on the organization, it might be somebody in legal that ends up really kind of taking the reins and coordinating the technical activities that are happening just because of the way that they interact with both the cybersecurity insurer and potentially the incident response team. Um, but usually what ends up happening is, is that most organizations are gravitating towards having cybersecurity insurance. Like it's, if you go through and you talk to, uh, various different corporate insurance brokers or underwriters, it's the number one product that's out there right now.

Jeremy

Um, and so lots of co companies are kind of grappling with that and getting onboarded with some type of cybersecurity policy. And so a lot of times your insurers will help connect you with somebody that will provide a level of incident response and incident commander activities. So usually what ends up happening is, is that ends up being played by the role of a third party in those organizations that don't actually have an active CISO or an active InfoSec team, is that you'll have somebody who helps kind of coordinate and is a known entity to the organization and brings that level of familiarity, but it usually ends up being somebody from a third party.

Jillian

Thank you, Jeremy. I'm feeling a lot less nervous now.

Jeremy

Good. <laugh>, that's, that's what I'm here for.

Jillian

We're so glad that we were able to have you come back and explain both the breach and how to recover really great information for Cybersecurity Awareness Month. And if I don't see you before then happy Halloween.

Jeremy

Happy Halloween to you too. Thanks Jillian.

Speaker 3:

Thanks for listening to this episode of Insight on if today's conversation sparked an idea or raised a challenge, you're facing head to insight.com. You'll find the resources, case studies, and real world solutions to help you lead with clarity. If you found this episode to be helpful, be sure to follow insight on, leave a review and share it with a colleague. It's how we grow the conversation and help more leaders make better tech decisions. Discover more@insight.com. The views and opinions expressed in this podcast are of those of the host and the guests, and do not necessarily reflect on the official policy or position of insight or its affiliates. This content is for informational purposes only, should not be considered as professional or legal advice.