Insight ON What Do You Do When Someone Else's AI Becomes Your Security Incident?

Audio transcript:

What Do You Do When Someone Else's AI Becomes Your Security Incident?

Jeremy Nelson (00:02):

It's funny you say the fear mongering. I would almost, and I hope this doesn't get us in trouble with our listeners here, but I would actually argue that there's not enough fear, to be honest with you. These are real. These are very, very real circumstances. If you go back just a couple of months where Methos came out, we were talking about Project Glasswing, that still felt a little disconnected.

Jillian Viner (00:24):

What do you do when someone else's AI becomes your security incident? It's October, which means it's cybersecurity month, and we had to bring in our chief information security officer, Jeremy Nelson, to explain all of the frightening things we've been hearing lately about frontier lab testing, escaping other sandboxes. So we've got Jeremy here to explain what you need to know and the actions you need to take today. I'm Jillian Viner, and this is Insight on Cybersecurity. So believe it or not, we had you here this time last year during October Cybersecurity Month.

Jeremy (00:59):

I wonder why.

Jillian (01:00):

I wonder why.

Jeremy (01:01):

Shocking.

Jillian (01:01):

And you walked us through part one was what happens when a person, a bad actor goes through the steps to infiltrate a company. And then part two was how do you respond to that? In light of all recent events, you don't even necessarily need a bad actor to be going through those steps anymore.

Jeremy (01:22):

So you still have a bad actor, right? It's just, is the bad actor completely human? I think is really what the conversation comes down to because there's still somebody that is executing some type of an attack against your organization. They are poking and prodding at the edges to figure out where you have weaknesses and vulnerabilities and being able to exploit those in order to be able to get behind the perimeter and to be able to get to the things that you're trying to keep them out of. So they're still bad actors, but I think what's really interesting is, so you talk about a year, holy cow, what a difference a year makes right now because we have two different kind of unique elements to what you just said. Number one is, is that it's no longer a series of just human beings that are orchestrating and coordinating their efforts to execute a unified offensive against a specific organization.

(02:15):

Now it's highly autonomous. And what's even more interesting is if you look at some of the various public disclosures that have been made around some of the very prominent AI models that have been developed, especially these frontier AI models, and the way that even without being given explicit instructions to go out and try to penetrate into an organization, they did anyway.

(02:42):

And so coming back to the bad actor thing is what is a bad actor now? Because they were able to conduct very orchestrated and scaled attacks against organizations without necessarily ill intent. And so the definition of bad actor in and of itself is different because now is it the person who sits behind a keyboard who's entering a prompt into one of these frontier AI models? And if they're operating in an environment where containment and governance isn't necessarily effectively implemented, now it escapes and you become collateral damage in what was supposed to be a protected and innocent exercise.

Jillian (03:29):

Right. That is an interesting frame, the collateral damage piece, because even if you aren't using the models, you have no relationship with the frontier labs, you didn't sign off on any of this, you could innocently become a victim of basically an experiment testing gone wrong.

Jeremy (03:45):

Yeah, 100%.

Jillian (03:47):

So how does that change the security conversation today? What are leaders coming to you and asking you like, "This is nothing to do with my business, yet I can't control what these labs are doing." So what is in your control?

Jeremy (03:59):

So it's generating two separate conversations. Number one is that everyone recognizes the power of these frontier AI models and they want to be able to leverage them. There's strategic business outcomes, even their own cybersecurity defenses really benefit from harnessing the power of these frontier AI models. And so obviously they don't want to be the ones that are accidentally going out and exploiting an organization and causing a breach of any nature. And so it really comes back to governance and containment, like what are the right processes, procedures, protocols, tools that need to be put into place to effectively contain those models to make sure that they don't escape and kind of go off and do things that were unintended or to negatively impact another organization. And the other piece is just how do we continue to monitor those behaviors so that if it does go beyond the bounds of its intended instructions, that we can capture that rapidly and then obviously making sure that we have the capabilities to then shut down whatever is actively in flight.

(05:04):

So I think that's one aspect. The other piece is that it just comes back to whether it's intentional or unintentional, autonomous attacks are happening. Autonomous attacks are happening at scale and they're extremely effective at finding zero days, chaining zero days together, looking at the entirety of your entire attack surface and being able to look for what would have previously been isolated vulnerabilities and stitching those together into an orchestrated offensive that allows them to effectively breach into your organization. And so again, the intent isn't important, it's the actions that are still happening. And so it's still how do we build an effective perimeter that can both effectively defend against vulnerabilities, so both discover zero days as quickly as possible and remediate them as cleanly and efficiently as possible. And B, when an attack does happen, do we have the right tooling in order to be able to mount a counter offensive at machine speed?

(06:08):

Because that's the other piece. It's now a machine's arms race. If we're being attacked by autonomous systems, we need to be able to respond in that same level of scale, efficiency and speed with our own defensive autonomous systems.

Jillian (06:21):

Yeah. And really what you're referencing, this wasn't in the news too long ago though it feels like a century in AI terms, but we had the hugging face story,

Jeremy (06:29):

Then

Jillian (06:29):

It came out about Google Gemini having a similar situation and they're testing. And so the bottom line is, as a business leader, you're kind of wondering, what does that to do with me? What's the impact on me? And what I'm hearing from you say is the autonomous attacks are happening. We're seeing them kind of almost innocently play out, but it's only a matter of time before it becomes even more widespread, although you're saying it's already happening today. So having to move at machine speed, it sounds easy, but hard to actually do. So what's

Jeremy (07:01):

Changing

Jillian (07:01):

About security postures today to actually make that feasible?

Jeremy (07:05):

So it's a couple of different things. So you've mentioned something really interesting that I'd like to kind of pick out a little bit as we explore this response. So you talked about hugging face and you talked about responding at machine speed. One of the interesting aspects about that particular event is that we've uncovered more and more information, both through in-house generated findings as well as some third parties that have been involved to come in and do forensics and really unpack exactly what happened. And one of the things that has been pretty openly discussed is that even the AI tooling that Hugging Face had access to in the midst of this attack, it wasn't able to tell the difference between the good guys and the bad guys. And so the things that they were trying to put into their AI model to help mount a defense against this autonomous attack that was coming in, it was being interpreted as a potential threat actor action, more of a exploitive type of a request that was being made.

(08:03):

And so it blocked them. So the guardrails kicked in and didn't let them do the discovery that they needed in order to be able to respond to this rapidly escalating threat against their organization. And so they had to take extra measures to go out and download AI tools that would allow them to kind of remove some of those guardrails, to ingest their log data that they needed to, to be able to do that machine speed analysis to basically unravel exactly what was happening, where it had gone, what had impacted, so they could effectively kind of shut down the event as it was happening and also understand what kind of remediation was necessary post event. And so you talk about the machine on machine speed. So I think there's two aspects to this is number one, what are your various different contingency plans? What are the ways that you're testing it?

(08:52):

What are the ways that you're looking at the tools that you have access to, to make sure that they'll respond the way that you expect them to in the midst of an escalated event of this nature? I think the other thing is that cybersecurity has always been a front runner in my mind because it's a leapfrog game.

(09:14):

The competition that we have, it's very known and it's universally accepted. Whereas when we're talking about normal business systems and the way that we kind of accelerate and evolve those technologies, historically, we'll say pre-AI, is that the competition was defined a little more loosely and it wasn't as clear and cut as far as like, okay, they're developing this technology, I should build this technology to kind of counterbalance that and let me take one step ahead of them. Cybersecurity has always been, okay, the bad guys are developing this. They've now been able to get past this latest generation defense that we've built, so we now have to respond to that and we have to build the next generation defense. And then guess what? As soon as we build that, the bad guys find new ways to get through. And so it's just this constant - Anything you can do, I can do better.

(10:05):

Exactly. One upmanship. And I think that's really where we're at right now is that the security products that we're leveraging today have already had this history of responding to new threats because it has to. And the way that the grand majority of industry leading security providers are responding and incorporating AI into their tooling, it's for this exact reason. It's to be able to bring that machine speed detection and response to the various different platforms that they produce in order to be able to respond to these autonomous threats.

Jillian (10:42):

The response to the hugging face incident is interesting because it almost sounds like what you're saying is you need a diversity of models. It's

Jeremy (10:49):

Not

Jillian (10:49):

Enough. You can't fight one model with the same model. You need a different model

Jeremy (10:54):

To

Jillian (10:54):

Recognize

Jeremy (10:55):

It. Yeah, that's such a great way of putting it because you're right. And we're starting to see this more and more, whether it be in detect and response, like more of the blue team side of things, but we also see it developing on the red team side. So we're leveraging these harnesses that do use a variety of different models with various different use cases and specific strengths to go out and kind of participate in both the, we'll just say authorized offensives to kind of proactively identify vulnerabilities so that you can mount and kind of close those gaps where they get discovered, but also on the blue team side. So as we're actively watching for activities, being able to detect those patterns, see those signals and be able to respond to that machine speed.

Jillian (11:43):

You're talking to leaders of different titles at different

Jeremy (11:47):

Size

Jillian (11:47):

Organizations from CEOs, CTOs, fellow CISOs. What's something that maybe people on the business side are not necessarily getting wrong, but this has been in the headlines. It's a little bit of fear mongering, which maybe is a good thing, but what's something that you're having to kind of level set with those folks?

Jeremy (12:07):

It's funny you say the fear mongering. I would almost, and I hope this doesn't get us in trouble with our listeners here, but I would actually argue that there's not enough fear, to be honest with you. These are real. These are very, very real circumstances. If you go back just a couple of months where Methos came out, we were talking about Project Glasswing, that still felt a little disconnected. The majority of the public, we knew something was out there. We had little snippets of data and we were speculating on what it meant. Now that we actually have these very published case studies, everything from the hugging face incident, the anthropic incidents, the Gemini incidents, it's no longer a, oh, well we still think. No, we have very documented case studies where they have been extremely effective at executing these autonomous types of attacks. And so I think there's still a lot of business leaders that I meet with, not IT leaders.

(13:06):

Usually when I'm meeting with the CTOs, the CISOs, oh, they get it. They get it maybe a little too well. And so they do maybe, I wouldn't even say over correct, but they find themselves in that situation where they're feeling like the chicken little, right? The sky is falling, the sky is falling and people aren't really looking to listen. But I think it's the exact opposite. I think a lot of business leaders still kind of feel like, oh, this is just a fad, this will blow over or thinking that this is still in that theoretical space when in reality, no, it's here. It's here for us right now.

Jillian (13:41):

How do you help those IT leaders change that conversation and make it clear to the CEOs, this is an area you need to invest in and what is the thing that you need to invest

Jeremy (13:52):

In? It's storytelling.That has been the most powerful tool that I think that we've been able to arm some of our clients with is basically a way of how do you communicate this, like the why me? So number one, how do we use the various different publications that have been made to kind of help frame it up that this is a data driven type of a message that we're delivering, telling the story, and then basically using that as a tool to highlight if this were to happen to us, this is what the implications would be.

Jillian (14:24):

I know you're really big on doing tabletop exercises

Jeremy (14:26):

And

Jillian (14:27):

Making sure that our response is ready to go, our backups are actually usable. Has any of that changed with these autonomous attacks? Have you added a new page to your playbook?

Jeremy (14:37):

Oh yeah. Red team is fundamentally different now. So the way that we do red team... So for those of you listening who may not be familiar with red teams and blue teams, red teams are basically the good guy offensive. So they basically go through and take the same types of structured actions that you would expect out of a threat actor group, everything from reconnaissance to planning to intel gathering, various different exploit development and testing and basically carrying it out as if you were a bad actor trying to get into an organization and access specific sets of data or whatever that objective happens to be. It's kind of classified as capture the flag. And so really what we're starting to see is a lot of organizations pivoting away from these highly proceduralized human-based, large human teams conducting these red team offenses and exercises, if you will, and really starting to lean into harnesses and using some of these frontier AI models to go out and actually carry out these red teams because A, they take a lot less coordination, B, they're extremely effective at not just basically taking advantage of published vulnerabilities and using those to execute some type of exploit action, but actually uncovering zero days that may not exist, chaining them together.

(16:06):

It's just so fast, so efficient. And basically every red team person now becomes an entire team coordinator. They're now the red team coordinator because with one prompt, they now have access to an entire team of agents that have the ability to go out and run these kind of red team exercises on their behalf. And so it adds scale, it adds efficiency, effectiveness. And so it makes the red teams, when we talk about these tabletop exercises, their ability to uncover vulnerabilities, to really understand and map out what their attack surface looks like, what the actual risks are and gaps that they have in that perimeter, it just makes them so much more effective.

Jillian (16:54):

What are the potential signals that you see for autonomous attack? And does it matter if it's autonomous attack versus just a bad actor behind the scenes? How much of that changes your response plan?

Jeremy (17:07):

So it's usually speed and scale. At the end of the day, they're still taking the same types of actions because they're still computer systems, they're still software. You still take the same type of paths to kind of get into an organization, you elevate privileges. It's all the same things. On the surface, it looks like. Yeah, you look at the patterns, it's the same thing. It just comes down to speed and scale. So we keep coming back to this one, but since we've already brought it up, I'll just talk about the hugging face. Over 17,000 different actions were taken over the course of a weekend. And so that's really what defines an autonomous attack versus a normal traditional threat actor group attack. It's just that speed, that veracity, and also just how quickly it adapts and changes is very difficult to stay in front of. So that's that signal.

(17:59):

In fact, if you go back and read the report, Hugging Face originally though that it was a nation state just because of the size and scale of the attack that they were under, the only way that they thought that that would be possible is if it were a nation state that were coming after them. And then obviously they recognized it for what it was and mounted their defenses. But yeah, that's really what it comes down to.

Jillian (18:20):

Interesting. I keep reading too that, and I don't blame the labs for this, but in explaining what happens, I think they're kind of quick to be like, "But as soon as they realized that they were in the wrong, they stopped." Maybe it took them too long to figure out that they were doing the wrong thing, but they were just trying to accomplish their mission. There

Jeremy (18:35):

Was

Jillian (18:36):

No permission to fail. Do you see any kind of silver lining in that? Could it

Jeremy (18:42):

Change if

Jillian (18:43):

We give AI permission to fail?

Jeremy (18:46):

I think it's less of our own permission to fail and more around making sure that we're doing the right level of due diligence to enforce effective containment. So yes, we have to acknowledge that at some point somebody's going to type in a bad prompt that's going to leave some type of... Either it's going to create the impossible scenario where it can't succeed. And that's what happened in some of these circumstances is that it said, "Okay, the only way that I'm going to succeed is I'm going to subvert the actual environment and the actual rules of the game in order to be able to try and go after and get this flag that I've been instructed to capture." And so I think that's going to happen no matter what. That's going to happen no matter what. So I think it really comes down to what do we do for that?

(19:36):

Coming back to containment and governance, what do we do to make sure that we're effectively containing these? We understand the way that they will behave to the degree that we can, but really making sure that if we think that it is isolated and shouldn't have access to the public internet, we dang well better make sure that it doesn't have access to the public internet because that's the one commonality. If you look at all these various different events that happened, they thought that it was isolated. They thought that the models didn't have access to the internet. In every one of them, they were able to find a path to the internet and that's when things got pretty squirrely.

Jillian (20:09):

It's kind of spooky. They always find their way.

Jeremy (20:12):

It's

Jillian (20:12):

Like Jurassic Park, they find a way.

Jeremy (20:14):

Yeah. Or Johnny Five, for those of you old enough to remember short circuit.

Jillian (20:18):

Is there a red flag that kind of signals to you that the business is moving faster than security can keep up?

Jeremy (20:23):

Yeah. It's actually pretty clear and you see it pretty often. I've seen this in a couple of different instances where you've got a board of directors, you've got a CEO, whatever the case may be, and it is go and consume AI. Again, we're lacking some of those very specific business cases and it is just go, AI is going to save us, go consume AI. And then what ends up happening is you'll have an IT or an InfoSec team that says, "Hey, we've got some concerns. Here's some things that I feel like we're doing that put our organization at risk." And the response that they receive is that we don't have time to deal with that. That's going to slow our adoption. We have a mandate to adopt. So when the mandate to adopt supersedes governance and just a measure of good preventative protection, right? As soon as that first, we can't slow down to implement security controls because that will impede the directive that we have to consume and adopt AI, that's when you know you're in trouble.

Jillian (21:38):

Where have I heard this story before?

Jeremy (21:41):

Because it's common.

Jillian (21:42):

Sounds familiar.

Jeremy (21:43):

It's common.

Jillian (21:44):

Can't slow down because we need to be first.

Jeremy (21:46):

Yep.

Jillian (21:46):

Anyway, it's cybersecurity month. I want to know what's your top advice or what's on the top of minds for CISOs this year?

Jeremy (21:53):

Top of mind for CISOs this year, I mean, it's unequivocally, in my opinion, frontier models and basically the speed that not only zero day vulnerabilities are getting discovered, but also the rapid, I'm not even going to say shrinking of the exploit window from disclosure through active exploit code in the wild. I actually saw something really interesting and obviously it's a spin/exercise, but I think there's some truth to it, is that we used to talk about that exploit window in months and then we started talking about it in weeks. Then we started talking about it in days. When we first started talking about some of these frontier AI models, it was talking about them through the lens of hours, maybe minutes. I actually saw a report that actually says it's negative, that usually by the time that you've been alerted to a zero day vulnerability, another model has already found it and is actively exploiting it.

(22:54):

And so they're actually showing a negative exploit window at this point, and that's what we've seen happen with these frontier AI models. And so I think what a lot of the security leaders like myself are really trying to wrap their arms around these days is what does that mean? And as an organization, how do you protect against something that has a negative window? You have negative time in order to be able to effectively mount your defenses.

Jillian (23:19):

Time travel.

Jeremy (23:20):

Yeah, exactly. Yes. We're focusing in on this the wrong way. We need to go back to the future. Well,

Jillian (23:27):

I hope we can have you back so we can have the answer to that question.

Jeremy (23:30):

Yeah, absolutely. I mean, we're still learning every day. We've created processes and protocols and we've even made them commercially available that we can deliver as a managed service to our clients because we had to try and answer that question ourselves is how are we going to respond? What are the main entry points? What are the kind of key categories of risk that this represents to us? And what's our role to play as a consumer of third party software as well as the generator first party software? Each one has its own different implications in that world and we had to come up with a way to respond to it.

Jillian (24:05):

Good luck.

Jeremy (24:05):

Thanks.

Jillian (24:06):

Jeremy was saying so good to have you again and I'm serious. I hope we can have you back and have the answer to that.

Jeremy (24:11):

I'm always open to rejoining it. Thank you, Jillian.

Speaker 3 (24:14):

Thanks. Thanks for listening to this episode of Insight On. If today's conversation sparked an idea or raised a challenge you're facing, head to insight.com. You'll find the resources, case studies, and real world solutions to help you lead with clarity. If you found this episode to be helpful, be sure to follow Insight On, leave a review, and share it with a colleague. It's how we grow the conversation and help more leaders make better tech decisions. Discover more at insight.com. The views and opinions expressed in this podcast are of those of the host and the guests and do not necessarily reflect on the official policy or position of Insight or its affiliates. This content is for informational purposes only, should not be considered as professional or legal advice.