Getting Real About Remediation w/ Cyentia Institute

Media Thumbnail
  • 0.5
  • 1
  • 1.25
  • 1.5
  • 1.75
  • 2
This is a podcast episode titled, Getting Real About Remediation w/ Cyentia Institute. The summary for this episode is: We discuss the second report in our multi-part dive into the Prioritization to Prediction research series by Kenna Security and The Cyentia Institute. Prioritization to Prediction, Volume 2: Getting Real About Remediation picks up on the overall vulnerability landscape analysis from Volume 1 and dives deep into the vulnerability landscape from within actual enterprise networks (a little over 500 of them to be exact).
Michael Swiped Right On Jay
01:31 MIN
Summary of P2P Volume 1
01:17 MIN
P2P Volume 2 Looks At The Subset of Vulns That Impact Real Enterprises
03:53 MIN
Less Than 35% of All CVE's Exist In Enterprise Networks
01:47 MIN
Jay Backs It Up
00:50 MIN
Focus On The Tangible Decisions Businesses Need to Make
00:54 MIN
The Number of Assets Impacts By CVEs
01:18 MIN
Three Vendors Represent 69% of Vulns, Guess Who?
05:34 MIN
An Aside on Analyzing Large Data Sets
01:08 MIN
Comparing CVE Publish Dates
03:11 MIN
Coverage vs. Efficiency and Unbalanced Data Sets
06:16 MIN
Randomness and Remediation Strategies
01:23 MIN
Normalizing Patch Performance Measures Based On Realty
02:16 MIN
What Is A Confusion Matrix?
04:07 MIN
Hindsight Is 2020
02:04 MIN
Strategies Tend To Bias For Coverage vs. Efficiency
01:53 MIN
VM Maturity and Patch Strategy Tradeoffs
01:28 MIN
Diving Deep Into Survival Curves
04:33 MIN
There is Massive Variance Between Enterprise Remediation Performance
02:34 MIN
A Primer on Area Under the Curve (AUC)
02:50 MIN
Keep Pulling the Research Thread
05:43 MIN

Dan: Today on Security Science, we analyze the vulnerability landscape of real enterprises. Thank you for joining us as we discuss the second report in our multi- part dive into the Prioritization to Prediction research series by Kenna Security and Cyentia Institute. In Prioritization to Prediction volume two, Getting Real About Remediation, we pick up on the overall vulnerability landscape and analysis that we did in volume one, and really dive deep into the vulnerability landscapes as it applies to real enterprise networks, just around 500 of them to be exact. With me today, I have the prima donna of risk based vulnerability management, Kenna Security co- founder and CTO, Ed Bellis. Ed, thanks for actually joining. I thought you were going to lead the podcast today.

Ed Bellis: I did as well, but thank you for making me attend, Dan.

Dan: Awesome. He almost prima donnad out of here because of the color coordination of Jay's books in the background that we're seeing right now. It's very nice inaudible. Anyway, so quick intro here. We also have a special guest that I just implied was on the line here. So I don't really know many people who are more qualified to discuss this topic, but our guest today is a security data scientist that enjoys digging into the data. Probably best known for his contributions to Verizon's Annual Data Breach Investigations Report series. And he also has a book, Data- Driven Security: Analysis, Visualization and Dashboards. He's a founding member of the Society of Information Risk Analysts, co- creator of the Exploit Prediction Scoring System, and finally, partner and co- founder of the Cyentia Institute, Jay Jacobs. How's it going, Jay?

Jay Jacobs: Really good. Great to be here.

Dan: Well, we're glad to have you as well. I know that we already did round one of this series with your cohort Wade. And I believe we came to the conclusion that how we all started working together was Michael Roytman swiped right on your profile. Is that correct?

Jay Jacobs: I am going to say, yeah. I think that pretty much sums it up.

Dan: So two data scientists meet in a bar?

Jay Jacobs: Yeah, pretty much. And the feeling is pretty much mutual there. Anybody who knows Michael, it's really hard not to be affected by his passion and a wonderful sense of humor.

Dan: Do we call it charm, maybe?

Jay Jacobs: Sure.

Dan: Yeah. I would stick with passion.

Jay Jacobs: Energy? Is his enthusiasm for the whatever topic is at hand.

Dan: That's very true. Pretty articulate as well. I think we did epidemiology with him on the last episode, and that was a little bit over my head, but he delivered with a lot of passion. There we go. Well, I wanted to kick things off today with a quick recap of our last episode and PTP volume one report just to give everyone a little bit of context for this one. So overall, we looked at the vulnerability landscape for... essentially through the lens of CVEs that existed at the time. This report is a little bit different because we're actually looking at enterprise environments, so the getting real portion of the title. In that report, we found that 23% of CVEs have exploit code published somewhere, and of those 2% had exploits observed in the wild. So the things you should really care about, and again, this is the overall gamut of every CVE that had ever been published at the time. From a timeline perspective, 50% of the exploits published within two weeks plus or minus of a vulnerability being published to the CVE, we defined a couple of key measures of coverage and efficiency, which I'll let Ed and Jay dig into a little bit deeper as it comes up in this episode. We also looked at some of the strategies. So a strategy is CVSS seven plus was one of the most effective yet it only achieved some efficiency of 32%, coverage of 53%. And then we proposed a new model to predict vulnerabilities and prioritize from there. So I think that's where we left off. Ed, I wanted to hand this over to you because in volume two, we really started looking through this realities as it pertains to enterprise data, and we had a couple of questions for volume two that we wanted to address. Do you mind covering off on this?

Ed Bellis: Yeah, sure. So Dan, as you mentioned, one of the things that we really wanted to take a look at given the model that we had come up with in volume one, and we covered a little bit around coverage in efficiency, and compared and contrasted a lot of different remediation strategies in volume one, but in some ways, it felt a little academic in the sense that we weren't dealing with any of the real world data. We were looking at the kind of encompassing the definition of every vulnerability that's out there in the universe and saying, okay, if you had all of these, how would you go about prioritizing your remediation or using a particular remediation strategy? So with this volume in volume two, really, we want to do say, okay, so what do we actually see in these various enterprise environments across all of these different verticals? What are companies actually dealing with? And does that change anything when we look at these remediation strategies? And how are they doing against them? And we started to look at things like... I know we'll talk about this a little bit later, but Cyentia introduced the concept of survival curves around some of the remediation. And I think we saw it for the first time here in volume two. But that really was a great way to visualize some of the velocity around the remediation and also that there's really a long tail that occurs when remediating vulnerabilities at any level across these organizations because there's just so many of them, but they never really eradicate any particular vulnerability typically out of their environment completely. So what we wanted to do is say, okay, so of the vulnerabilities then that they actually have that are being identified, and again, there are some caveats even within the data and volume too. So keep in mind, we're mostly looking at the results coming from various vulnerability scanners. So what does that mean? Well, first off, what vulnerability scanners are they using? What signatures are available for these particular vulnerability scanners because they're not going to identify something that they don't have a particular signature for? And then what are they actually scanning? Are they scanning their entire environments? Are they looking at a subset? Are they looking at particular platforms? Are they looking at their desktops and laptops? Are they looking at their data centers, their cloud? You name it. So all of the caveats apply there. But really, we wanted to understand. And so of the coverage and efficiency, so of things that either have... that we see an exploit for in the wild or that we know of particular weaponized exploit code or POC code that is out there to exploit that, what should they be remediating? So there's a certain subset that is much different than what we saw in volume one. So first off, probably one of the big takeaways is the vast majority of those vulnerabilities that are in the national vulnerability database aren't in these environments at all. Either because they're not being identified or because they simply don't exist, those companies aren't running that software, it doesn't matter. So being able to inaudible all of that off, you can really start to refine what it is you're looking at. That was one of the really big early takeaways is there is a big difference between the theoretical universe of vulnerabilities versus what people actually have.

Dan: That's a really good point. I just want to let everyone on the line who's listening and know that there'll be a link to download the report if you want to follow along in real time and it will be on the podcast page, kennaresearch. com. But what you're referring to is at the time, roughly, so this was what? Late 2018, I want to say. So of all the CVEs that existed at the time, it was roughly 108,000, only 34. 2 of them, so 37, 000, were actually observed in the enterprise. So we're talking about a much smaller subset that actually were observed in enterprise environments. Jay, I don't know, have we done any updates on that since then to see if that proportion has roughly held through?

Jay Jacobs: We did, but I think we updated it in inaudible. But I don't think we dwelled on it because once you know that stat and then you hear it a second time, you're like, well yeah, of course. I mean, that's the way it is.

Ed Bellis: Yeah, that makes sense. Okay.

Dan: I was just curious if that proportion is crosstalk.

Jay Jacobs: It didn't shift.

Dan: It did not. It's been inconsistent.

Ed Bellis: There's definitely some things that we saw shift over time, but that was not one of them.

Dan: Right. Interesting. Very cool. So we already laid out basically of the entire gamut of CVEs that exist roughly 35%, we'll just round up a little bit, exists within enterprise environments. And of that subset, only about 5% of all CVEs have exploit and/ or observe exploits in the wild. So that's the stuff you really care about. So we start to really lay down the proportion of vulnerabilities observed. Jay, what were some of your takeaways from this kind of fingerprinting?

Jay Jacobs: I want to back up a little bit because in the first one, we looked at from the CVE perspective, you start with a long list of CVEs from MITRE through NVD and things, and you look at that and you think, all right, how do I prioritize this? And that's not really what organizations are doing. They get the output from the vulnerability scanner and they say, all right, out of this stack, what should I focus on? They don't start with CVEs, but I think it's useful for volume one to look at these CVEs and just lay a groundwork for what we're talking about. And so I think volume two, it's really great to say, all right, what do organizations actually care about? What are they looking at? And then let's focus on that. And I think that's a great perspective. And I think it also sets up, again, that foundation on which the subsequent volumes also expand from.

Dan: Interesting. So getting back to what Ed was saying a little bit earlier is almost like theoretical. So it's kind of the first report was a theoretical foundation. So this is what CVE landscape looks like, and this is more of the practical, I guess.

Jay Jacobs: I don't know if I'd make that separation because I think CVEs, there's nothing theoretical about CVEs. They exist, the vulnerabilities exist, the products exist, and they're out there. And just the fact that we're not seeing them in corporate environments doesn't make them any less real. But it's just not the things that a lot of the commercial entities are going to care about or have to deal with. And so we just wanted to focus in on actually, I mean, the decisions. What decisions do the companies have to make? And they don't have to make a decision about some obscure piece of PHP that will never be in a corporate environment. And so that's what we wanted to focus on, were the things that are very tangible and the decisions that need to be made.

Ed Bellis: Yeah. I'd agree with all of that. I will throw out one kind of caveat here where we're talking about, in these enterprise environments, you're dealing with much less in terms of the number or the unique CVEs that you're dealing with. But that's not to trivialize that to say, they're not dealing with a lot of vulnerabilities. You could have that one CVE across as we've seen across millions of assets. And you're talking about tens of millions or even hundreds of millions of vulnerabilities. So there's a lot of work ahead for these companies. Even if it's just a handful of CVEs, there's a lot proliferated across the environment.

Dan: That's a really good point. Just to call out the numbers here, it looks like 8% of CVEs affect less than 10 assets a piece. And that was kind of the bottom bound on this figure four in the report. But 3% of CVEs affect more than 1 million assets, and there's just this super wide distribution. You can see. crosstalk.

Ed Bellis: And everything in between there.

Jay Jacobs: Yeah.

Dan: Exactly.

Jay Jacobs: It's pretty amazing. I mean, that was really the focus to say, hey look, we could talk about this one CVE encountered among the 100,000 CVEs that we looked at, but in reality, that one CVE could be, like we're saying, over a million different assets may have that one vulnerability repeated over and over and over again.

Dan: And so how does that play? And it looks like when we looked at the overall sample, roughly 40% of vulnerabilities remained opened or unpatched out of the sample. Was that a surprise?

Jay Jacobs: I would say, yeah. It was a little bit lower than I think I expected. I knew we wouldn't see all of them, of course, so I knew we wouldn't see 100%. But I expected a little bit more, but really, I mean, I didn't have any skin in the game. And so I was just looking for what's in the data. And so whatever the data was going to say, I was going to be like, oh, okay. Of course, that's what it is. Just try to pretend I expected it, but I didn't. I didn't have anything set beforehand.

Dan: Jay, so what you're saying is you were actually expecting them to have more open vulnerabilities after a year or longer?

Jay Jacobs: I expected more of these CVE vulnerabilities to be observed in the enterprise environment.

Dan: Got it.

Jay Jacobs: Than what we saw.

Ed Bellis: And I know, Dan, you're going to get to this, but one of the things that we did is in volume two, start to look at it by vendors and things like that, and who is responsible for the percentages of open vulnerabilities. And it was a very telling, not only that there's three very large vendors in there, but obviously, that encompasses a lot of different products across the enterprise.

Dan: I mean, we can jump into that right now. It looks like of the vendors we looked at, three, which people can probably guess who they are, are responsible for 69.1% of all the volumes in enterprise networks. So those being Oracle, Microsoft, and Adobe in that order. I don't think that's surprising to anyone though.

Ed Bellis: No, I don't think so.

Jay Jacobs: I don't think the top three is a surprise. But I mean, for seven out of every 10 open vulnerabilities are coming from one of those three vendors, that to me was pretty shocking. So if you aimed your vulnerability management program at just those three vendors to try and get really good at dealing with vulnerabilities from those vendors, man, you could knock seven out of 10 out of there, out of the open vulnerabilities, which is going to be better than the average rate as we found.

Ed Bellis: And spoiler alert, some people did much better on at least one of those vendors than the others in a later volume of B2B.

Jay Jacobs: Even in this one.

Dan: Yeah, even in this one. I was going to say we've dug into this quite a few times as split out performance of vendor- led remediation efforts. And I think this is what spurred that line of research later on. But when we look at performance of those top three, there's one clear winner, and it's Microsoft. And then there's one clear loser, and that's Oracle. And there's someone who's in between, which is Adobe. And they all have various levels of vendor- led remediation programs.

Ed Bellis: Yup. And none of those actually surprises me as a former practitioner. And obviously, we talk quite a bit about all the efforts that Microsoft's done to put into their remediation process. Adobe benefits actually from some of the Microsoft processes actually, especially obviously the Adobe on Microsoft. Oracle is all over the board, and also, Oracle owns a lot of technologies that are notoriously difficult to remediate and patch. Just Java alone. Imagine everything that's got to go in. If I'm using Java in my application and I've got to recompile my application and all of the dependencies and all of the regression tests and everything that goes along with it, suddenly, upgrading Java is a very big deal versus deploying a patch for Microsoft Office as an example.

Dan: Right. And Java, I mean, there's a figure in there, figure 11, that calls out the proportion of open vulnerabilities associated by product. And Java has a little over 18% of open vulnerabilities are in Java. So almost one out of five open vulnerabilities that we know of are coming from Java. And so anybody who's tried to patch Java is like, oh yeah, of course. That makes sense.

Ed Bellis: Of course, crosstalk.

Dan: Try being the keyword there.

Ed Bellis: Oh yeah, of course.

Jay Jacobs: Yep.

Ed Bellis: Of course.

Jay Jacobs: Yep. That makes sense. Yeah.

Ed Bellis: Of course.

Dan: Well, actually, I'm curious, should there be a little more effort going into trying to figure this stuff out on the front end before it's rolled out? Just because it's so hard to patch on the back end, I'm just-

Jay Jacobs: Well, I think anybody who has worked with Java and tried to patch Java, I think what that serves is like, you're not going to look at that and be like, oh, I think we can figure out how to fix Java. It's going to be like, hey, when we build the next thing, let's not do it like Java. I mean, that's really the lesson there. When you're trying to figure out how to write a language and update it, you do not want to copy what Java is doing.

Ed Bellis: That's fair. I would say Java has more vulnerabilities certainly that we see than like Python or Ruby or any of these others. But I would also say that if, as an example, that a new vulnerability comes out in Ruby, it's not simple to upgrade Ruby and then everything just works either. Right?

Jay Jacobs: Yeah.

Ed Bellis: The nature of vulnerabilities in your actual coding language of choice is a fundamental problem for you.

Dan: On that, just to lay out the performance overall just so people have the actual numbers here. So out of all the Oracle volumes, 69% of their CVEs were still open. So 70% were open, not plugged in for all the reasons we mentioned earlier. With Adobe, they split that. So they're 35% open, to 64% closed, roughly, depending on how you round. And then Microsoft, they actually have 77% of their vulnerabilities were closed. So much crosstalk a lot of them. A whole lot of them. That 77% that was closed was roughly 1 billion vulnerabilities on assets. Yeah, that is impressive.

Jay Jacobs: I just want to add too, it's really fun to work on data that you count in billions doing data analysis. It's just lickety- split as you could imagine.

Dan: Says the data scientists. What are some of the challenges behind that? How long does it take you to run some of this stuff?

Jay Jacobs: So let's say you just want to do a count of something, just look up... Let's count the vendors, for example. In a normal data set, you do that and you get the result. And when you're working with large data sets like this, you set it up and you run it and then you go fill up your coffee or tea and maybe grab a sandwich or something. Come back, and then it's done. So I mean, the larger data sets, that's really the biggest challenge is just the interrupt driven because you kick something off and then you might get distracted by another project, checking email, and then the job finishes, and then 10 minutes later you come back. And so it just draws out that analysis process a little bit with the larger data.

Dan: Quantum computing can come soon enough?

Jay Jacobs: I have no idea what that will do to our daily work.

Dan: crosstalk security or anything in general.

Jay Jacobs: Yeah.

Dan: Let's figure it out inaudible. Speaking of time, we did some digging into the age of open vulnerabilities on systems. So in volume one, we looked at timelines for exploitation or timelines for when an exploit was developed, plus or minus two weeks, we mentioned. How long do these vulnerabilities stay open?

Jay Jacobs: Oh man, it is such a complex topic. So essentially, I think you're going after figure 17 in here where we have a survival curve.

Dan: Well, we were looking at some of the age of open vulnerabilities in figure six where we're laying out this crosstalk. Yeah, we're still laying out this foundation, like what does the fingerprint crosstalk look like? Right?

Jay Jacobs: Yeah. So this is actually interesting because we're looking at the year that a CVE was published, which might actually be different than the year that it was disclosed publicly because sometimes, some CVEs will get reserved and then they get published like two years later or something. But according to the published year, there are open vulnerabilities from 2013 and beyond. I think what it looks like, about 15% or so of vulnerabilities are from 2013 and before. So I mean, there are some really old vulnerabilities that just linger and linger and linger in environments. And so it's rather impressive actually that they're still around.

Ed Bellis: But at the same time without looking at what those vulnerabilities are, the risk of those vulnerabilities, we might actually think that could be okay depending on which vulnerabilities they are. Right?

Jay Jacobs: Right.

Dan: Well, then we've got another chart though that breaks it up by vendor. And I think 2013, it looks like 80%, 90% are from Oracle. So I think that tells you how hard Java can beat a patch of 80% of the vulnerabilities from 2013. Even before, it was mainly Oracle.

Ed Bellis: I don't want to pick on Java too much because that's clearly one of the ones that was very difficult to patch, but Oracle has an awful lot of software. And a lot of very difficult software to patch, frankly, whether it be Java or RDBMS database or WebLogic, which is weird. It's ugly head many times in the past. There's a lot of difficulties around a lot of that stuff.

Dan: Yeah. And we should point out that to your point, Ed, we're not saying, look how terrible Oracle is or anything like that. I mean, there obviously could be some arguments either way on that. But I mean, this is basically calling out what we're seeing. This is the reality. I mean the reality is that organizations have a ton of open vulnerabilities, and a lot of those are coming from Oracle, and Adobe, and Microsoft. And that was one of the reasons we looked at the remediation rate. Even though Microsoft has a ton of vulnerabilities out there, they are closing a ton. And as we'll see later, they close them pretty fast.

Ed Bellis: Yeah, remediation. I mean, one of the lessons learned throughout several of these reports is making it easier for the users, the practitioners to actually remediate these vulnerabilities makes a big difference later on.

Dan: Yeah.

Jay Jacobs: It's very true.

Dan: So I mean, let's jump into that. The second research question that we looked at with these reports was how comprehensive and efficient or organizational vulnerability remediation practices. So what can we measure primarily using coverage and efficiency? So I don't know who wants to take... Jay, you haven't done crosstalk.

Jay Jacobs: Sure.

Dan: Would you like to do a quick definition of coverage versus efficiency?

Jay Jacobs: Yeah. If anybody has worked around modeling or information theory, there are concepts in there called precision and recall. And these are very standard. I think they even have their own Wikipedia page if you search up precision and recall. And the problem though is that these words are not intuitive. So precision is kind of intuitive and actually, we use it in the definition in volume two for that concept. But then recall is... I don't even know how you begin to internalize what recall actually means. So we decided to rename them to be more applicable to patch management, which is efficiency. In other words, we define it as it measures the precision of remediation. So out of the things that you choose to do, how efficient were those choices? Are you putting your resources to the most efficient use? And then coverage is saying, out of all the things you should be fixing, all the things that we know that either are being exploited in the wild or have exploit code and weaponized or available on the internet, those are the things that you want to focus on. So the coverage is saying, what proportion of those did your decisions cover? And so we've got efficiency for our efficient as your decision and coverage for how much you're covering and what you should do. So those two measurements are the two that we use to talk about strategies that people are employing to remediate all the vulnerabilities. And the thing with these two measurements is that they are essentially trade- offs with each other. You could go for 100% coverage, but your efficiency is probably going to be terrible. And one of the reasons to talk about efficiency and coverage or precision and recall is because this is such, what they call an unbalanced dataset. It's not like 50% of the things you should fix and 50% you shouldn't that you get very unbalanced. So between 2% and 5% are actually exploited in the wild, and roughly about a fifth have exploits available. And so that unbalanced aspect means that you can't just talk about a normal statistic like accuracy, which is something I don't even want to talk about. So if you focus on efficiency and coverage, you get a much better sense of this unbalanced data problem. And so if you focus on efficiency and you're like, all right, I want to fix just the ones that I think really matter, you probably won't get good coverage. And if you want good coverage, you're probably going to have terrible efficiency. And so the trick is how do you balance those two? And essentially, it's going to be organization- dependent because you may have a small company with a small budget for vulnerability management who's going to say, " You know what, we can only spend some small proportion, so we want to probably focus on efficiency. The work that we put in here, we want to make sure that we're focusing at correctly." And an organization that has a ton of resources might say, " We're going to focus on coverage. We want to be sure that we're getting really good coverage, that we're fixing the things that matter, and we're going to accept that lower efficiency." And so depending on the organization and the resources available, we're going to have different strategies.

Ed Bellis: I'll talk about accuracy, Jay. Another way to put it is, if I had a vulnerability out there and I had a prediction system that said, this will not be exploited in the wild, and I say that every single time about every single vulnerability, well, I'd be right 95% of the time. And therefore, I'd have 95% accuracy, and boy, am I doing well? Except for those 5% crosstalk.

Jay Jacobs: That's an A crosstalk.

Ed Bellis: Yeah, exactly. I get an A, even though I've fixed zero vulnerabilities.

Jay Jacobs: So that's why accuracy is not a good stat in an unbalanced dataset like this.

Dan: Yeah, because it only takes that 0. 01% to happen. But the repercussions of said thing happening are so bad, it doesn't matter if you were right 99.9% of the time. Yeah. Interesting. Well, and this is all referring to figure 12 as well is a nice coverage versus efficiency plot. So you can see the trade- offs that people would have to make if they're trying to take different strategies, and it follows this curve along a 3D plot. And what's nice is, Jay, you guys were able to basically model out coverage and efficiency on this chart for different strategies. And we did that this in volume one, and it's a really cool way to visualize the overall work and where you would sit along this axis.

Jay Jacobs: Right. And I do want to point out that everything that we did for figure 12 is this is definitely theoretical because as a company, it looks at what to fix. We grabbed CVSS and said, 10 and above, nine and above, eight and above, and so on and so forth. And I don't think anybody actually, when we looked at it, we couldn't see evidence that companies are like, all right, we're going to fix 10 and above and nothing else. Nobody does that. Everyone is like, oh, it's a 10. Okay. Well, that's interesting. Let me see what else I can look at. And every strategy is going to have some mix of these things, but we looked at CVSS, we looked at where is the vulnerability published? So if it's on Bugtraq, maybe if you had a strategy, fix everything that Bugtraq talks about. Or the full disclosure list, or Microsoft publishes in their disclosure list. And then the other thing was just looking at vendors. So if you looked at, from a prevalence perspective, the top five, top 10, top 20 vendors, just try to fix everything from them, what kind of outcome would you have from that strategy?

Ed Bellis: And clearly, by looking at the real world data in volume two, nobody took the vendor strategy.

Jay Jacobs: Right. Yeah.

Ed Bellis: Hence the big three.

Dan: Well, what I find interesting as well for these coverage versus efficiency plots is most of these prioritization methods, I guess you could say, we'll just call them that for sake of having a descriptor, they tend to prioritize coverage versus efficiency, if anything at all. So it seems like it's a little bit easier to say go patch every CVE that's ever existed than it is to be like, here are the ones that you should take care of.

Ed Bellis: It's more conservative kind of risk averse approach. So in theory, in everybody's mind, they're thinking that is what I would like to do. Especially as a security practitioner, I don't want all of this risk. But when you look at it in practicality, that becomes an impossible task, especially at the enterprise level where they have so many of these.

Jay Jacobs: Well, I think as far as these strategies, I would rather talk about it in terms of randomness because essentially, if you take a strategy like CVSS eight or nine and above or something like that, essentially what you're seeing is that the efficiency is on par with a random selection. So if you just randomly pick out vulnerabilities and say, we're going to fix that one, and then you roll the dice and oh, we're going to fix this one and you keep rolling dice, then the coverage is just a factor of how many times you roll that dice. So if you're going to fix 10, your efficiency is going to be constant because you're rolling a random dice and getting a uniform distribution. So it's going to be constant. So if you wanted more coverage, you just keep rolling the dice and keep on fixing more. So then the point is going to move towards higher coverage, but the efficiency is going to stay right at the random level. And I think that's what we're seeing for a lot of these strategies that they might go above a little bit or below a little bit, but largely, everyone gets coverage by essentially brute force.

Dan: Yeah. No, and I believe, what was it, CVSS seven plus was one that was right around random chance. That was one of the better strategies. It had slightly better than random chance. And then when we looked at actually all the enterprises here, if they were to pick a volume to handle it random or roll the dice, they'd be correct 15.6% of the time.

Ed Bellis: Yes. And I think we'll look at this a little bit later in the report and it's really important to call this out, is coverage and efficiency when you're looking at it and an individual vulnerability level, which people don't do in the real world versus at the patch level. So I go out and I deploy a patch, oftentimes, I'm fixing more than one vulnerability, and it might be that I'm fixing something that's a CVSS 10, but I'm also fixing three of them that are CVSS six or something like that. And we did look at this, I think it was figure 16, where we sample inaudible a number of different orgs that participated and said, all right, so here's what they looked like in terms of coverage and efficiency during remediation. But then we looked at, and Jay, I believe you looked at it, you were the one that actually pulled us by patch, which was a significant improvement because then you're looking and saying, " So really, why they deployed this patch or at least the theory is they deployed this patch because there was a risky vuln in that patch along with a bunch of non- risky vulns.

Jay Jacobs: Yeah. And the thinking is that organizations apply a patch, they don't fix a vulnerability. And so the decision then is that a patch layer, do I apply this one or not? And so what we didn't want to do is penalize them for deciding to apply a patch that fixed a whole bunch of things. If anything in there is something that they should prioritize, then the patch got a yes flag, a label on there. And so if they applied that, that was a good thing. And if they didn't, that was a bad thing. And so that's how the efficiency improved greatly because we weren't penalizing for these extra vulnerabilities being fixed in one specific patch.

Ed Bellis: Yeah. And it's important because also, it represented no additional work for them. So why not fix those other five crosstalk.

Jay Jacobs: Exactly.

Dan: And what we're grounding out is Microsoft Patch Tuesday. You typically get a patch and it's going to fix a whole gamut of things. One may be critical, it's very rare for there to be actually nothing to really worry about in a Patch Tuesday list, but you're going to be handling a whole lot of other things with that one patch. So that's by nature going to bring your efficiency down even though artificially, basically.

Jay Jacobs: Right. Yeah.

Dan: So getting into this as well, we use what you call a confusion matrix and a little piece of fun report trivia here. I think this is the first and only time we use a confusion matrix chart in any of the report series. Jay, what's a confusion matrix.

Jay Jacobs: Well, I want to clarify, it's the only time we've published it. These occur quite a bit behind the scenes. So confusion matrix is essentially, if you build a decision machine, if you will, you want to know how confused does it get. And that's what a confusion matrix is. And so if you think about like think of flipping a coin heads or tails, and so I'm going to flip it and I'll say, "Head or tails?" And you say, " Heads." And I flip a heads, then hey, they agreed that is a true positive. You said heads and it turned out to be heads. And so you get these four quadrants, if you will. Basically, like when you say heads and it is heads, it's a true positive. When you say tails and it's a heads, assuming that you always want to go after a head, you have a positive class in a negative class. So you get false positives, false negatives, and then true negatives when both agree on the negative class. So you have these four quadrants. And then basically, can put in a model that makes a decision like this, and then you can say, all right, what proportion were true positives, what proportion were true negatives that you can safely delay? And the real problem, I think isn't the false positives. And that's what we see in a lot of the models that we created, that a lot of the approaches, like I said, sort of this brute force approach, so you get a lot of false positives. Hey, we want to fix these things, but they aren't really being exploited. There's no exploits out there. So you're sort of saying, I want to prioritize this, but you don't need to. So that's the false positive aspect.

Dan: Good description. I'll try to... Let me see if I can read over the actual results of this confusion matrix as well here. So this is figure 14 for anyone following along at home, but basically, the top left quadrant falls into the, you remediated something and you should have. It has an exploit for it, and out of the entire list, that was 381 million actions essentially. Is that how that breaks down?

Jay Jacobs: Yeah.

Dan: Cool. And then the top right quadrant was you remediated it, but you probably should have delayed. This was not an action that needed to be taken necessarily, so it was wasted effort, I guess you could say. And so that was 1.7 billion. So in total, people took, or enterprises or organizations, a little over 2 billion actions essentially out of this chart, correct?

Jay Jacobs: Yep.

Dan: Cool. And then when you break down, bottom left is you delayed and you probably should have taken care of this, so that was a miss. There was 163 million actions. And then bottom right was you delayed, you should have delayed, good call. And that was 1. 2 billion. So people are taking a ton of actions.

Jay Jacobs: Yeah. And the false positives are a debatable topic because I mean, like we talked about, if you have a patch that fixes 20 things but you think you only need one out of the 20, just fix the whole thing. But the other 19 might be called a false positive then because inaudible, they're not exploited, they're not exposing you or whatever. And so there's a little bit of debate on whether we want to focus on the false positives. But I think when you're talking about resources of a team, if you have high false positives and you still have false negatives, you're still missing them, then obviously, the prioritization effort is not aligned here.

Dan: One of the takeaways for me with this is people are... If we're looking at amount of work being done, there's enough work being done to actually cover all the stuff that should be taken care of if you know what to do.

Jay Jacobs: And a lot more. crosstalk.

Dan: And more, yeah. A lot more. So you can hit efficiency and then start to go to town on coverage. Right?

Jay Jacobs: Right.

Dan: And what the data suggests is that these companies actually have the resources to do exactly that if they know how to do it.

Ed Bellis: That's the positive. But again, I'll throw out the big caveat as I always do, which is, it's not every vuln is the same amount of work and effort to fix. We talked about Java earlier versus deploying a Microsoft patch. Sometimes you just do it because it's simple and it's easy. This represents 15 minutes of my time. This represents a month and a half of my time. The theory versus the reality.

Jay Jacobs: And to add to that caveat, I think it's super easy for us to look going back and say, " Of course, this is what people should do." But that's obviously not reality. I mean, everyone gets that output from the vulnerability scanner they laid on the desk with a thump because it's a huge stack and-

Ed Bellis: And then they run away crosstalk.

Jay Jacobs: And that's the tricky part. But I mean, the reason that we're doing this is not to say, " Look how bad people are," it's to say, " Look how hard this is." This is a very complex environment, and there's no easy strategy that we found that would work well. So there's no easy answer. There's no silver bullet. It is a very complex thing. And so I just wanted to throw that out there as another caveat. It's super easy to look back and be like, " Look how wrong everybody is," when in reality it's just super hard.

Ed Bellis: We can all just be prima donnas like Ed. Right?

Jay Jacobs: We can all be prima donnas like Ed.

Dan: Well, I mean, I think he used this to calculate the overall coverage and efficiency across this entire data set, so all the enterprises. So the coverage was 70% and efficiency is 18.5%.

Jay Jacobs: Right, roughly. So, I mean, it does give the impression that the strategy of throwing a dart at a wall seems to be what we're seeing, but I mean, obviously, there's companies doing better and companies doing a little bit worse. And so it is a complex problem. So we've got to be careful about making judgment calls on anything.

Dan: Yeah, absolutely. Well, and it also shows that a lot of companies seem biased towards coverage versus efficiency, which is not a wrong approach to take, right?

Jay Jacobs: No, that's actually a good approach. And I think in later work, we actually ourselves started to de emphasize the efficiency thing saying, hey, I mean, if the attacker is coming at you and you talk about the weakest link approach, you want to make sure that the whole chain is strong along the way and that's coverage.

Ed Bellis: Yup, agreed. I would say some of the things that we actually see typically in these customers is they grow and mature over time. And when they're early on and they're a little bit more immature in this state, oftentimes, you see companies that will start emphasizing efficiency because when they start to put the effort into remediation, they want to make sure that they're making as much of it count as possible, and they're really driving down their risk as quickly as possible. And then over time, they start to mature and they build in more processes and they get better at it. And then they start to shift more towards coverage so that they can get to what we've been talking about, which is to say, " I want to really broadly reduce my risk and make sure that I'm covering myself off."

Dan: Yeah, that's interesting. So just for people listening, Kenna Security, our platform, we have a thing called top fixes. And, Ed, I think you're referring to a lot of companies when they first start up, they go after these top fixes. The biggest inaudible for the buck, which would be in theory, the most efficient use of your time and really drive down the risk score within the platform. But then at some point, you get to a place where you're operationalized, you're handling things as they're coming in more or less, and you have a steady state where there's always going to be some inherent risk and now you're trying to tackle the broader landscape, I guess.

Ed Bellis: Yep.

Dan: Cool. Well, so we go from one published chart first and only in the confusion matrix to something that I think is in every single report since then?

Ed Bellis: I hope so.

Dan: Maybe, maybe not, but it's almost all of them for sure. But survival analysis. So Jay, could you give us a breakdown? What is that?

Jay Jacobs: Yeah. I've really grown to appreciate survival analysis. And so survival analysis is a collection of techniques. It's not just one technique, but it's a collection of techniques that's focused on essentially time to an event. So starting at some period of time, time passes, and then an event occurs. And you want to understand what that is. And so simple way to think about is like, imagine a row of light bulbs and you want to know how long light bulbs last, you could plug a whole bunch in and watch them in time to an event and then use survival analysis and figure out the expected life of your light bulb or whatever. And so it's the same concept. And survival analysis obviously comes from healthcare type things when you're looking at disease and rate of survival for people or subjects. But this applies in so many other things. And actually it has, I don't know, at least a dozen different names depending on the field that you're dealing with. So you could think of it here as the survival rate of vulnerabilities in an environment, which is how we're using it. And so the thing that it measures, there's two things that it measures. So when people think, what is the average time to patch something or remediate something? If people aren't familiar with the survival analysis, they're going to take a look at their data and see that they've got a bunch of open vulnerabilities still, and they've got a bunch in close. So the average time to close, let's take a look at those closed and look at the dates. But if you have 100 vulnerabilities and you close two and they were closed in two days, and you say the average rate of closure is two days when you've got 98 still open after three months, that's not a very accurate picture. And so survival analysis accounts for everything that's open. And so the open is put into the models like, it's been open at least this long, as long as we observed it. And so what the survival curve represents is at the time of discovery of a vulnerability, what is the probability it is going to still be open after some time? And that time is the horizontal axis in the charts. So after three months, for example, there's about 50% chance that it will still be open or 50% chance it'll be closed if you want your glass to be half full. But after 30 days, we still see roughly about a two thirds still open. As you get to one year, we still have roughly 18% still open. And so, I mean, it's pretty amazing actually. I mean, what we see in a lot of these curves and efforts like this is that you have a really steep incline of people seeing this and they're like, we got to go after these. And they fix a whole bunch. Then you have a steep decrease in the line, and then it starts to smooth out and flatten a little bit. But the other thing I really like about the data here is that if you look closely, you see sort of a wiggle in the line, and that wiggle is every seven days because companies tend to scan weekly. And so what you see is like it's discovered whatever day of the week they scan, and they scan seven days later and all companies generally scan seven days later, so you see a decrease at seven days. So it's kind of cool to see that scanning rate. So even if they closed it in day two, they don't scan till day seven, and that's when it's recorded as close because that's the interval of time that we get recorded. So you see the line go and then it drops at day seven inaudible. It's really subtle because not everybody's doing that, but it's sort of cool to see that scanning rate in there.

Dan: And this is figure 17 in the report just for people following along. And I don't think I ever noticed that little wiggle, actually. It looks like a nice curve to me and my brain just rounds it crosstalk.

Jay Jacobs: It's seen a lot. Across all of the curves, it shows up. It's pretty cool.

Ed Bellis: It's interesting too, and we'll do this, I think in some later volumes, looking at comparing different, whether it's tech stacks or vulnerabilities or specific things, it's interesting to just look at the steepness or lack thereof of the curve. It's a really easy visual to go, oh man, it's hard, or people are really slow about remitting this vulnerability, or it's easier super fast just looking at the... How quickly that curve drops off is a great visual.

Dan: Yeah. And I did want to note as well, so for calculating this, and I believe the patch efficiency where we were talking about basically compensating for patches that are remediating multiple vulnerabilities, we actually looked at a sample of 12 firms. So it wasn't the entire gamut for those nor are they survival inaudible because I think we were talking about number crunching and time when we're talking about crosstalk.

Jay Jacobs: That comes later.

Dan: Yeah.

Jay Jacobs: And so I remember actually, in volume two here, we did a sample of 12 firms because of the large data problem. And I think we were under a time crunch and decided to go with 12 firms to first make it a little bit lighter, and it was a little bit lighter at 190 million vulnerabilities. And so I think later, the curves actually drop and look a little bit better from remediation times now.

Ed Bellis: Yeah, we've continued to refine and dig into it, but I like this first one. So you guys were referring to figure 17, which has just the overall sample of these 12. But then you go and look at the survival analysis by each individual firm, and one of the fun things when we're reviewing these reports is typically, we're looking at all these lines showing the timelines for remediation ultimately, and we'll see one company or one vendor or one data point where it just drops really fast. And that always starts a conversation for every single report that we've done. And so that's figure 18 here. You can see dotted lines, different colors, and it shows that there's a lot of variation between these kinds of remediation programs.

Jay Jacobs: Yeah, and this actually is subtle compared to what we actually see. So, I mean, if we look at the hundreds of companies that we're dealing with now, essentially, you look at the box that you've got the full plot area, it essentially fills up with lines when we do it by organization. So you've got organizations that stay at the top that roughly keep a lot of things open for some reason, and then some orgs that drop almost straight down right away fixing a whole lot of stuff. And everything in between and every different direction in between. And so you just get this spaghetti- looking plot of lines going every which direction. So it's really hard to talk about the average company because there's so much variety, there's so much variance across the companies. It's a really interesting space.

Dan: Hey, Jay, maybe now would be a good time to tell us what area under the curve means and how that might apply.

Jay Jacobs: Sure. Do we talk about it in here?

Dan: Not in this one, but it'll show up in the next report.

Jay Jacobs: Essentially, so if you look at the figure 17 and survival curve, and for those who aren't looking at it, it starts in the upper left and it drops quick, and it's just this gentle curve that slopes to starting to flatten out to the left, and at the area under the curve, if you think of calculus and you're doing the integral and you're looking at the space under a function, that's exactly what we're talking about. How much space is under that line? So if you have a company that is slow to fix and doesn't fix a lot, their line is going to stay relatively high. And if you just had a line that went straight across the top, the area under the curve would be one. It'd be a 100% of the plot areas under the curve. If you had something that went straight diagonal to the lower right corner, it would be 50% because 50% of the plot area would be under the curve. And so it's a really easy way because it's really hard to take a line. Maybe some company will go attack vulnerabilities quick, but then flatten out and maybe come down again. So you have these wavy lines that are going down at different rates. And so it's hard to say which one's faster. So one way to do that is to say, what is the area under the curve, which will obviously reward companies that go faster quicker because if they go fast, they're going to bring that line down right away and reduce a lot of the area under the curve. And so it's one way to compare survival curves and the rate of remediation across companies, and we do that later across industries in a whole bunch of different ways. And so it's one way to compare these survival curves and say which one's actually faster.

Dan: And I think that makes a lot of sense to compare something like this, where in theory or actually, in reality, no one's ever going to get to 100% for the most part, all intents and purposes, unless it's Dan LLC and I patched my only laptop that exists in my company, which is mine. And so it's a good way to compare these data sets that have large numbers and very long tails, especially when you probably shouldn't be tackling a lot of this stuff in general as well.

Jay Jacobs: And talking about average time to close or something is really difficult with this before that problem where you can't say, well, you never get 100% closure, so what is the average when you don't have 100% closure? And so talking about survival analysis, there's probably at least three major different ways to talk about a typical time for closure here, and I think we talked about a lot of them throughout the reports actually. We inaudible them into the conversation. Anyway, we may not have explicitly called out the details behind them, but we brought them into the conversation. One of them is actually area under the curve.

Dan: Yeah, I think area under the curve, we start really looking into that in the next report series. In this one, we broke down these intervals. So like it took 30 days on average, almost 69% of vulnerabilities were open. And so we can compare that against some of the exploitation timelines from the first report and be like, yeah, not ideal. Realistic, not ideal. 50% of vulns were closed after 90 days, and then 32 and a half after 180, and then like you said, 18% were open after a year. So very long tail crosstalk. And I think that's where we start to lead into our next research report. So we'll have to get you guys on the line for that one again, but Jay, what are some of your takeaways from this report?

Jay Jacobs: I want to go back to the complexity because I mean, anybody who has not worked directly in vulnerability management will think, why don't you just go patch everything. I get a notice on my PC that says patch stuff and I patch stuff and it's patched. Why don't companies just do that across the enterprise? The fact is you can't. It just doesn't happen. And so you have to talk about prioritization, and even that gets tricky because there are so many factors. There are technical aspects of the vulnerability, the asset it's on, how important is the asset? How much effort is it going to take to actually apply that patch? Is a patch going to break anything? You've got all these things to consider, and you might have downtime and all this stuff like that trying to pick a window. There's so much stuff going on that the complexity here is amazing, and I would say interesting from maybe a sadistic point of view. But I think that there's so much going on here. And I think volume two is like our second step obviously. And digging into that complexity and as we'll see in some of the future volumes that will be discussed, there's so much going on. And I don't see a bottom to this research. There's so much going on. There are so many different areas to look at. It's a really fascinating area to study.

Dan: Awesome. Ed, how about you?

Ed Bellis: I mean, I agree with Jay on the bottom or lack thereof. But it's funny because we go out and we'll finish one of these research reports and we've set out to answer a particular question, and then we come away with five new questions with every question that we answer. And there's certainly no exception here in volume two. Volume two is great because we said, " Hey, all right, we've looked at everything in NVD, but how real is that? Let's actually start applying some of these real world found in the wild vulnerabilities within these environments and see what we get." And we came up with some interesting results sets. There are certainly some things that we're like, as Jay talked about, oh yeah, of course, that's exactly what we were expecting. And including, there's a subset of vulnerabilities that actually companies have to deal with. But this was interesting. Actually, one of the things that I really walked away with because we started talking about coverage and efficiency in volume one was looking deeper at efficiency as a patch efficiency versus a vulnerability efficiency. So when we started to pluck out these sample set of companies and say, how are they doing? It's like, wait a minute, we can't measure them on the individual vulnerability level because they're actually going out and fixing 10 vulnerabilities with this patch. And that was an eyeopening graph, I think. And I forget exactly which figure that was within the report, but we actually show the lift that you're actually getting from the patch was one of the cool takeaways. And of course, we ended up with, oh God, now there's like 15 other questions that we have to go out and answer. Hence, we created volume three and so forth.

Dan: That's figure 16, by the way, for people following at home, the patch non- inefficiency. Double negative. Yeah, no, it's interesting. And as we dig into this, I think the coolest thing for me is I get to see us... When we're working through these reports in real time, it's easy to get lost in the data sets and analyzing these really deep very long Google Docs, right?

Ed Bellis: Right.

Dan: Looking through the numbers and validated its accuracy. But it's kind of cool because we're figuring out, especially looking back on these, how to measure some of this stuff, how to define this stuff, what the challenges are, and trying to figure our ways around those or how to explain them, which I just found interesting. That's why I brought up the whole confusion matrix use for this one time and then survival curves. I'm like, oh, this was the start of that. I've spent so much time looking at these. Anyway, we will continue to dig through this series, and for everyone listening at home, again, I will link Cyentia Institute, I will link, Jay, your book, and I will do a plug for Saira because I think it's a really, really cool organization. And they actually have Saira Kahn going on right now, so a lot of cool information popping out there. And then of course, we'll have the reports and all that good stuff. And for anyone else listening, check out Cyentia Institute's research library, they have a ton of industry research there. I think it's probably the largest collection of security industry documents on the planet, if I had to guess. So anyway, thanks everyone for listening, and I will see you later.


We discuss the second report in our multi-part dive into the Prioritization to Prediction research series by Kenna Security and The Cyentia Institute. Prioritization to Prediction, Volume 2: Getting Real About Remediation picks up on the overall vulnerability landscape analysis from Volume 1 and dives deep into the vulnerability landscape from within actual enterprise networks (a little over 500 of them to be exact).