In Search Of Assets At Risk w/ Cyentia Institute

Dan: Today on Security Science, we go in search of assets at risk. Thank you for joining us. As we discuss the fifth report in our multi- part dive into the Prioritization to Prediction research series by Kenna Security and the Cyentia Institute. Prioritization to Prediction volume five in search of assets at risk. With me today, I have the squire of risk based vulnerability management, Kenna Security co- founder and CTO Ed Bellis. How's it going Esquire Bellis?

Ed Bellis: It's going excellent, Dan. Thanks for having me.

Dan: I liked the poppers you're wearing. You need a few more metals, I think on your chest though for the squire.

Ed Bellis: I know. I got that reference, but I don't really know what an esquire is.

Dan: And so our next guest as well, so rumor has it that our next guest today served as the prototype DNA for a new generation of bioengineered data scientists. Of course, I'm talking about Cyentia Institute partner and co- founder Jay Jacobs. How's it going, Jay? Awesome. So, we're doing a picking up on volume five, which is our most recent right ahead of a new one that's coming out. Tease, tease. But we launched volume five on April 21st, 2020. So it came out this year and it's pretty fresh for all of us. If you've been keeping track at home, the past four volumes of the Prioritization of Prediction series have been primarily based in a very vulnerability centric view of remediation. So we're looking at individual CVEs, we're looking at the NVD list. We've tried to do some work to look at like number of patches across assets, things like that. But in general, this time we took a very different approach and started with the asset and took a very asset centric approach to looking at the vulnerability landscape. So, just a really quick recap slash these numbers were actually updated for this report. So we just had the most accurate from the four leading up to this one, but there's a ton of vulns, roughly 130, 000 in the NVD right now, remediation takes a ton of time. So 45% of vulns are remediated typically in the first month, two thirds within three months. And then just under 20% remain open after one year. We've discovered that orgs can not fix all vulnerabilities. So the Bellis rule, only 10% are typically closed in any given month. The top quartile, the top performers in this will do what, 25% any given month, I think. Right?

Ed Bellis: Give or take. Yeah.

Dan: Yeah. Not all vulns need to be fixed right away. So only five percent of vulns actually exist as far as we can see within corporate environments and have an exploit developed. So you don't need to handle everything right away. And then one of the cool things that shifted since the first time we measured this, but orgs can fix all their high- risk vulns within a given timeframe. So current stats at the time of this report was 51% of organizations were reducing their high risk vulnerability debt, 16% were kind of treading water or maintaining. And then 33% were actually falling behind the number of vulns, high- risk vulns that were open versus what they're remediating every month.

Ed Bellis: Which is definitely exciting. Again, centering that chart was almost flipped a year before.

Dan: Yeah, yeah. That's exactly right. So let's jump in. Again, we looked at things from the perspective of assets this time. So Jay, I'll actually kick off with you. I know we've done so much work looking at the vulnerability centric view. Why did you guys want to go look at the asset basis and kind of flip things around?

Jay Jacobs: Well, so if you think of what it takes for a company to remediate vulnerability to prioritize vulnerability, it's not only what we talk about in the first four reports about the high risk, what's being exploited. Is there some exploit code published and indicators like that. There's all these things are going to help drive that decision, but ultimately it's going to be, what is this asset? Is it important and all these things about it. In volume four, we talked about the patch management systems. Is this a platform that is generally involved in a patch management system? Is it a one- off? And so looking at the asset, we wanted to focus on that asset to try and understand more about it. Like, what are we seeing? Do we see certain platforms doing better, doing worse, having different indicators, having more or less vulnerabilities generally speaking and such? So trying to get that a more full picture of the vulnerability landscape, I think is what really drove us to talk about the assets here.

Dan: Absolutely. Well, and I know when we started reviewing the data on this one as well, it kind of opened up more questions than normal. So we wanted to dig for, I know Ed is super keen to dig into a couple of the different stats that we found. So it kind of opened up a whole new world because we've done I would say a pretty good job of setting some of these baselines that we can now apply to our thinking and kind of making things a little more consistent with the way we think about the vulnerability landscape. But the asset was a big gap, I think for us. Ed yeah, I wanted to see if you had any comments there as well.

Ed Bellis: Yeah. And I'm just going to add on top of that. And what Jay is saying is right, and is super important. So vulnerabilities are one piece of the puzzle when it comes to how an org makes a decision to remediate them. And some of that has to do with the things that we've discussed before. Like are there exploits, or is it being exploited in the wild and all these different things about the vulnerability. But the asset is also a huge piece in their determining should I go fix this or not? Part of it is what Jay said is how important is this asset? Where's this asset located? Do I have upstream controls around this asset? How easy is it to patch this asset? Because that, as we find out in this report, makes a huge difference. So all those factors kind of go into your prioritization decisions.

Dan: All right. Well, let's kind of jump in with asset prevalence. So I know Ed's really into baseball and way to open this section with a Pete Rose quote. See the ball, hit the ball. It's a lot more difficult when you've got a million balls that you need to hit, but asset prevalence, just some highlights right here. So we're trying to basically provide kind of an overview of what we're looking at. What's the size of the problem? How do we classify different assets? What's the best way to classify them, at least right now with the data that we had? So the median org has about 800 active assets, and I think that's not as interesting as the full range is huge. So, we will have the report available to download the link to it from the kennaresearch. com/ podcast, in case you want to follow along, I will mention the figures here. So not as interesting as figure six, which shows that the full range is just huge. So starts at less than 10 assets and goes to over a million assets per organization. So it's just a massive spread. From there, we kind of broke things down as platforms. So the software that basically underlying the operating system. Jay, could you go into some depth on why do we choose to split things out that way?

Jay Jacobs: Yeah. So this beginning section is basically trying to we're walking into this absolutely pitch black room and we have to figure out what are we standing in? Are we looking at companies with 10 versus a million versus 1000? So, we want to try and figure out what's going on. And then once you look at the prevalence of just the sheer number of assets, you want to break down what are these assets? And so there's no surprise. The number one platform out there is Windows 7. And we see that in about a quarter of the assets that we looked at were Windows 7. And then, because we're getting this from vulnerability scanners, basically the vulnerability scanners will fingerprint an asset, and then they report to the best of their ability and bless them to the variety that they choose to report these. And we get a large variety of reporting, especially around Linux. And so some of these are able to identify that it is Linux, but they are not able to separate out as a Red Hat, Ubuntu, et cetera, et cetera. And so we get reports about just generic Linux and a kernel version. And so second place is Linux at 13%, just generic Linux. And then we started to get into the devices. So you have Cisco in there. Later on, we see HP devices, typically printers, F5, big IP devices, VMware we listed as a device, but it was really hard to say, is VMware server or device? What is that? But then also at the top, we see like Windows 7 is in fourth place, nine percent of the assets were Windows 7 and down on the list. And so it's just interesting like I said, we're walking into this room, we want to know what are we looking at? Are we looking at, is it 100% Windows 7? Is it two percent? And we want to try and understand what are we looking at here?

Dan: And then when we look at the macro level, I know you guys did a really good job of kind of splitting these out. So Microsoft platforms accounted for 50% of assets that we looked at in this study here. Linux slash Unix devices, 22.4, and then the appliances slash devices. Because we all know how secure printers are. 26.7% of all the assets looked. And I would encourage people to go and follow along. Because there's some really pretty charts in this report.

Ed Bellis: Jay is the master of pretty charts.

Dan: Yeah. Well, because we're looking at a lot of proportions, literally millions of devices, individual devices, which is it can be harder to visualize.

Jay Jacobs: And there's a fourth category too. So you mentioned, we've got Microsoft platforms, Linux and Unix and then appliances and devices and the fourth one are basically Apple Mac, Mac OSX. But that was like, I think three percent or something like that overall.

Ed Bellis: And obviously this is a call- out to the bias in the data and Jay touched on it, but this is all data coming directly from vulnerability scanners for the most part. So it's just a representation of what people are scanning.

Jay Jacobs: Yeah. That's a pretty good clarification. And then we get into how do these spread across companies? So we may see a company that is 100% Windows 7, or we should say what you just clarified, that they're scanning 100% of what they're scanning is Windows 7. And then you see some where it's like, they have one in 10,000 assets as a Windows 7. And so we tried to create this chart and what's really interesting if you look at that chart, it's actually on a log scale, but it's sort of hidden with how the labels are generated and stuff. So we talk about like Windows 7, the majority are about one in 10, maybe one in 20. And then things like generic Linux, generally about one in 100 assets is a generic Linux on the environment. But that goes up to one in 10 depending on the organization. So we created this plot, it's figure eight in there. And so one point is essentially an organization for that asset platform. And so we looked at what does that spread look like? What is the proportion of assets in that company and what does that proportion spread out across these asset types? And what does that look like?

Dan: And what does that really tell us in this case? Because I always found this, it's like super... it's one of the prettier charts just out of my personal aesthetic. But looking at that hidden log scale, I find it hard to look at figure eight in here and there's just a bunch of dots. And a lot of the Ubuntu, VMware appliance driven sections have a really high spread that's pretty consistent from one in 10,000 to roughly one in 10. Whereas more of what I would consider end user devices, Windows devices, some Linux devices are much more... they have a lower proportion of distribution.

Ed Bellis: It should be said that Jay is also a master of a bunch of dots.

Dan: I thought he was crosstalk.

Jay Jacobs: Actually, that's an influence of Ben and David Seversky. They brought those in when they came to the company.

Ed Bellis: Everything makes much more sense now.

Jay Jacobs: If you sort of look at this plot that you can walk away. The top four on there are Microsoft platforms. So it was 10, then 2012, 2008, and 2016. And that basically is what is in most organizations. And that's what the data is showing us here. And then as you go down, you get more spread. But where you see one in 100, which is in the middle of the plot, that's one percent of a company's assets, one in 100. And so when you see something like Ubuntu, essentially most organizations have less than one percent Ubuntu. And then you see the dots going to the right. Of course there's one over there. It's probably 90% of their assets as Ubuntu. And so there's that one Ubuntu shop way off to the right. But you can sort of get that feel. So when we say, " Hey, companies mainly run Microsoft." That's mostly true, but obviously there's going to be there's that one Ubuntu shop out there and there's some other Ubuntu shops and some Red Hat shops, certainly a lot of Red Hat in there. So it's just a way to say there's a lot of complexity and we're not just showing... this isn't just a Windows 7 study.

Ed Bellis: Yeah, yeah, yeah, no, I agree. And I'd say it would be interesting to cut it up, but I don't think that the data is available to us as how much of this. You could almost look at it a client versus servers as well. Based on operating system, there is some caveats there crosstalk imagine.

Jay Jacobs: A lot of caveats.

Ed Bellis: And it comes down to, again, what are people scanning? So a lot of people are... we're scanning all of our Windows desktops, and they're pretty good about that. And we see later about the remediation patterns as well.

Dan: Yup. Interesting. I don't think it'll come as a surprise to anyone that, I mean the majority of scanned assets within most businesses are Windows assets basically.

Ed Bellis: And I think the next section will say why.

Dan: Yeah. Yeah. Well, let's jump in. Vulnerability density. So this is a measure of the number of crosstalk. This is a measure of the number of vulnerabilities that exist in a given asset. So you're looking at just to call out some of the definitions, because there's similar to things that we've talked about in previous reports, but slightly different as it pertains to the asset itself. So, for an asset, this is base operating system. So Windows, Ubuntu, Red Hat, VMware. Other software installed on the asset as well. So third- party, or installing Adobe on top of your Windows system and then any misconfigurations identified by a vuln scanner. So that's what we're talking about with the summation of all of that per asset.

Jay Jacobs: Yep. Definitely. And again, it's really hard because a lot of people will just want a single number. Like, " Hey, how many vulns does Microsoft have or Microsoft platform?" And it's like, well some have a few and some have a whole lot and that's what we're trying to show on the spot. Again, we put a lot of dots on this one. But essentially, when we talk about Microsoft platforms, we try to draw a line. And I think this is the 25th to 75th percentile. So on Microsoft we see the first number in there's a 20. So that means 25% of the Microsoft platforms have less than 20 vulns on their asset. And this is in a given month. So you may have them come up and they might get remediated quick and they might not stay on there long. But if they're seen during a month period, like the 25% have less than 20 vulnerabilities. Now you get to the median. 50% of the assets have less than 119 is that, and this is by far the most heavily populated from a vulnerability density perspective as Microsoft platforms. So we see that median about 119 and then it goes, keeps going up. And so the 75th percentile is 280 vulnerabilities on an asset per month. And that's saying essentially 25% of the assets have more than 280 vulnerabilities on that asset during a given month.

Ed Bellis: Yeah. And if you go back a little bit earlier where we broke down the Microsoft platforms, for example, you can kind of see, I think it was Windows 7 that we said was the most, which tends to be, it's client side and server. I think when we look at these four platforms based on their distribution, there's a few things that come to mind. One is obviously Microsoft dominates in a lot of different ways, including scanned assets, but you're also looking and comparing multipurpose platforms to single purpose platforms. And there's a big difference and crosstalk density there as well.

Dan: So a printer versus a Windows 7 laptop.

Ed Bellis: Yep.

Dan: Right. Yeah.

Jay Jacobs: Or even servers, typically Linux and Unix are typically more on a server environment. And so those are generally single purpose or in some cases, dual, but you don't have that multi- purpose platform as you do on like a Windows 7 desktop where you've got I don't know how many different vendors have their fingers on that part.

Dan: Yeah. That makes a lot of sense. And just to go over the figures, so we're looking at the distribution that Jay was talking about for Windows, but we also looked at Mac OSX. So that had what, an average of 32 vulns per month per asset. Linux was seven and just the appliances slash devices. So printers, routers, things like that was four. So small in terms of average. And also worth noting, Microsoft has the bulk of the volume. So that's 119 and it's also across millions of devices in this specific sample. And then we kind of jumped into the percentage of assets. So I mean we talk about, here's the number, average number of vulnerabilities that exist, which ones are high risk? So which ones bear prioritization? So ideally that's kind of the real where the work starts from a security standpoint. And we looked at the percentage of assets in each category that had at least one open high- risk vuln. And again, Microsoft kind of leads the pack here.

Ed Bellis: And just another caveat there, when we say high risk in this case, we're talking about vulnerabilities that are either exploited in the wild or have an exploit.

Dan: Yeah, very true. So Microsoft, 71.6% have at least one.

Jay Jacobs: And this is open at the time we pulled the data. So at that point in time, 71% of the Microsoft platforms had reported at least one either vulnerability that's being exploited, known exploited in the wild, or has an exploit code published that can exploit that vulnerability.

Dan: And this chart struck me. So we're looking at figure 11. It kind of, so Microsoft no surprise here again, you're going to see this repeated through all the different metrics that we look at, both good and bad actually, as Microsoft's pretty much going to be at the top of the list. And we break down into some nuance and we'll probably bear some different analysis going forward on this stuff as well. This chart in particular though is interesting because now you see Linux take the second place. Which it's traditionally fallen into like the third place spot over and over here. With 40% of the assets in the Linux slash Unix category have one open high- risk vulns, Mac OSX drops in a third at 31%, and then the appliances devices are close last at 30.5% of the devices. And I'm just curious as just what Windows is easy to target. People like to try to find exploits for it, write exploits for it. Microsoft is very good itself of discovering and putting up patches for all that stuff. Makes sense. Why the flip from like Linux Unix and Mac OSX and appliances? Any insights there?

Ed Bellis: I'll add there and let Jay talk about this as well, but things that come to mind here, obviously we talked about the volume or the density of these platforms. So the odds of having a high risk vulnerability on the Microsoft platform is much, much higher just because of the number of peer open vulnerabilities that are on those platforms. It's interesting to me when you look at things like this. It tells me that even though there's not a lot of vulnerabilities on the appliances and devices in the Linux machines, but the proportion of those that are high risk are much, much higher. Which I think when we get into some of the remediation velocity and some things around that, we'll probably see at least part of the why. But that was a very telling story for me.

Jay Jacobs: The data doesn't often tell us why things appear like they do. So it's a little bit hard to say, but I think this is basically a popularity contest and not just from the prevalence of these platforms, but also the prevalence of the valuable data that attackers may want to go after generally on a Microsoft platform these days. And if it's not there, then it's generally on Linux or Unix. The other two appliances and devices and Mac to some extent are more of like a stopping point, a launch point, if you will. You get at these assets to go after something else too or to have just an IP address or something. So I think those first two, the Linux Unix and the Microsoft platforms are generally where the data is and they're prevalent.

Ed Bellis: Yeah. Very prevalent. And the density prevalence. We talked about the Microsoft platforms on average have 119 open vulnerabilities. So the odds of one of those being high- risk seem quite-

Dan: 71.6%.

Ed Bellis: Yeah.

Dan: Awesome. Any other thoughts on the vulnerability density piece before we move on?

Ed Bellis: No, but again, we're trying to paint the picture. Like I said, we're looking around this room and so we've looked at what are the platforms out there and then what is the density of that? And so I think the next step that we get into is essentially how are people treating these? How are these things being remediated? And how are the vendors supporting that remediation?

Dan: Cool. Yeah. We jumped into remediation coverage. So I think what it's described as the comprehensiveness of remediation, AKA, what percentage of high- risk vulns have been remediated on these assets is the measure overall. And figure 13 in the chart is interesting because you're comparing these platforms again. But when you look at the difference between raw volume, again, no surprise. Microsoft just, it has so many. Everything else is like a little tiny sliver. And then there's Microsoft at like what, 200 million different assets here. But you guys did a good job and kind of normalized the percentage. Jay, do you mind going over that?

Jay Jacobs: Yeah. So this is another thing where you kind of have to look at it from a multiple different perspective to get a feel for it. So you can talk like, " Hey, Microsoft has 36 million high risk vulnerabilities open." And that sounds terrible. Compare that to Linux and Unix where there's 3.5 million open. But we really want to understand how are companies and organizations addressing these and going after them. And so to do that, you look at it as a proportion. So you might say, " Hey, on the Microsoft platforms, out of all of these high risk vulnerabilities that we've seen on there, there's only 17% of those still open." And so obviously something is going fairly well there. And then you compare that to like Linux and Unix where 37% of the high risk vulns are still open and then appliances 36% and Mac has 21% still open. So Microsoft is closing the majority from their massive pile that they begin with. They're closing from a proportional standpoint, most of them. But again, like you said, it's pretty much just dwarfing everything else. But from a sheer numbers perspective, from a proportion, they're doing better than any other platform out there.

Ed Bellis: Yeah. And frankly, not that big of a surprise. So one, not a big surprise that they lead in terms of number of platforms and the density of vulnerabilities, but also in kind of the remediation coverage. Everyone knows that they've made tons of effort into kind of the patching process and patch Tuesday and automating a lot of that and making it just a lot easier for their customers to fix the great number of vulnerabilities that they're introducing into those environments in the first place.

Jay Jacobs: I think if somebody has been dead asleep for 20 years and just woke up and the Microsoft they knew from 20 years ago, this would completely shock them. But yeah, like Ed said, just massive focus and improvements around remediation and security.

Dan: Yeah. It seems like the first half of this report was almost like, okay, Microsoft, you're very popular in the enterprise. So you got a lot of devices. You also have a ton by far the most vulnerabilities density and total number. And like you dwarf all the others. But the back half is like, and you're also the best at enabling people to remediate, it seems like. So it's kind of interesting. We do have a little chart that shows the percentage. So the coverage of high- risk vulnerabilities that are closed and it breaks out Windows instead of just Windows as a platform, which is good. But you look at Windows 7, 2012, 2016, they're really high in terms of their coverage. The highest out of all the individual operating systems that we looked at. So they in some respects are really strong top performers overall, which they kind of have to be given their volume I guess.

Ed Bellis: Certainly on the remediation side.

Jay Jacobs: Yeah. But if you take a step down and I think you're talking about figure 14, you've got the Microsoft platforms in the upper right. Clearly head and shoulders above everybody else. But then there's three green dots for Linux and you've got Red Hat and Ubuntu essentially at polar opposites, which I think is kind of interesting. So Red Hat and Linux has roughly about, I don't know, probably six million high- risk vulnerabilities and they've closed 58% estimating based on the chart. But Ubuntu sort of flips that around and they've got maybe 800, 000 high- risk vulnerabilities, but they're closing close to 80%. And so, they've got a whole lot less, but they're closing a much higher proportion of them. Maybe it's easier because they've got less, I don't know. But it's just interesting seeing that there's Red Hat and Ubuntu appear to be at somewhat polar opposite ends here.

Ed Bellis: Yeah. And I would add, unlike the Microsoft stuff, this one was a little bit of a surprise to me, especially given some of the efforts that I see Red Hat put into security specifically in patching and their advisories and different things. I do wonder, and we don't know this for sure. And it's just a guess, but what proportion of those are also kind of the single purpose versus multipurpose machines? How many of them, although I would think if anything, it would be the opposite. In general I would think both of them are probably heavy server versus client. But if anybody had more percentage on the client side, I would've thought it would've been Ubuntu, so I'm not really sure on this one. It's a definite surprise.

Dan: Yeah. Something to dig into the maybe a Linux specific asset report.

Ed Bellis: That's going to be dependent on all the caveats that Jay said earlier about what the scanner reports back, does it even know the flavor of Linux at all?

Dan: So the next section right now. So now we looked at remediation coverage. So kind of like the overview. We kind of set out the number overall, like the volume of assets and kind of their split. And so we start to move into velocity, so speed and progress of remediation. So, how quickly are issues addressed and how long do they persist across the different asset categories? And this is where a lot of this stuff is in a total surprise because we've done some of this work from the other side in the previous reports. But just looking overall velocity, Windows is just crushing it. So 50% of their vulnerabilities are remediated within 36 days. The next closest is OSX. So Mac at 70 days. And then we go to Linux Unix, 254 days, and appliances slash devices at 369 days. So over a year to get half of far fewer volume of vulnerabilities on appliances and devices.

Jay Jacobs: Yeah. I think the Linux line really surprised me on this one because like you said, it takes 254 days to get to 50% closure rate. Which is I thought an amazingly long line to talk about. And I always pictured back when I was doing system administration and the world of VNX 20 years ago there was systems set up like you log into a Linux. It's like, " Hey, you got to apply 30 patches." And it tells you when to log in. And that's if it's not automated. So that really surprised me just seeing that really long line on Linux and Unix. The appliance devices did not surprise me. I don't know who patches printers, but-

Ed Bellis: No one.

Dan: Yeah. Apparently no one, or one guy once every year.

Ed Bellis: Yeah, somebody crosstalk. Or apparently Cisco firewalls and routers and you name it.

Jay Jacobs: Right. Those do get updated, but yeah, it's just really long. And so it's interesting, Microsoft and Mac at the 50% mark Mac is twice as long as Microsoft, but they merged around seven months. They're about the same. They both get to about what, 75% closure rate at seven months? And then Mac actually overcomes them a little bit. And so at one year Mac is closed about 86%, whereas Microsoft is closed about 84%. So it's interesting seeing those two sort of go at it. But Microsoft definitely coming out of the gate by far head and shoulders above everybody else. So it's pretty cool to see. And that's probably due to some lessons learned and some hard lessons learned about that timeline and how fast they have to get off that starting block to remediate these.

Dan: Yeah. Two weeks plus or minus, right?

Ed Bellis: Totally. And I would say obviously when you're looking at like comparing the Microsoft and Mac platforms and Microsoft getting out of the gate much quicker, I think a lot of that has to do with all of the managed aspects of the Microsoft environments. Whereas I would guess the MicroS environments are probably a little bit less managed and I don't know when they catch up around that six or seven month mark, I think that's just more of a manual patching effort more than it is the automated process that gets kicked off right away.

Jay Jacobs: Well, I know for me, and I use max daily. And so, I would say maybe not seven months, that seems long. But it's sort of like nagging patch ware. And so like, they're like, " Hey, you got to apply a patch." and a few days later, "No, no, no, really you got to apply patch." And so you can only keep snoozing that for so long before you're like, " All right, I'm just going to fire this up on Saturday morning and let it go." Yeah. I can see that being the difference that out of the gate, Microsoft I'll attack it through the patch management, through automated processes. And then Mac is a little bit more naggy and still relies on that manual application to some extent.

Dan: I'm curious, what's the OSX kind of refresh schedule from just an OS perspective? Do they typically do pretty big OS refresh right around what WWDC every year ish within a few months, plus or minus?

Ed Bellis: Only like the major revs. Yeah. But they do minor revs in between as well. Probably not as often as the, I don't know that they do the... certainly do the average 30 days like Microsoft does or every month like Microsoft does. I'm not sure.

Dan: Yeah. What struck me as interesting as the 36 days for 50% for Microsoft seems to adhere to a patch Tuesday cadence. And we actually there might be more to this story a little bit later in this podcast as we look at supported versus unsupported Microsoft platforms that may be dragging down that other 50 percentile. So it sounds like they're doing a decently good job with their patch Tuesday, that whole process of plugging a decent chunk of these vulnerabilities, at least on their more, I guess, their newer operating systems overall. And then Apple's decently good at nagging us into doing a massive restart and taking a little bit to do an update. I know the next box figure 16. So Ed, you wanted to dig really far into this and we kind of ran out of time when we're doing this research. But we also looked at splitting out just of the Microsoft devices, the Microsoft vulnerabilities that were native to the operating system. I'm assuming as well office suite, all the first party vulnerabilities, which are very popular and then the third party vulnerabilities on their system. So Ed here, you want to describe this chart?

Ed Bellis: I had a sneaking hunch that Microsoft is actually even better than we're reporting here in the sense of vulnerabilities that Microsoft is actually responsible for patching things that get funneled through SCCM, et cetera, are even better than the overall average of Microsoft. The third party apps, as we saw on figure 16, there was a definite lag in terms of the velocity. It's still quite good when you compare it to some of the other platforms, but the Microsoft has specific vulnerables or even much, much faster in terms of the remediation velocity, which didn't surprise me, but I wanted to kind of dig into that because I suspect if we could make it easier for those third party apps to also go through, and that's where you probably see some of the more generic patch management systems, like the big fixes and things like that, that patch all of the vulnerabilities, including the third party apps that comes to benefit velocity, I think overall.

Dan: Yeah, it's interesting. I'm just eyeballing figure 16, Microsoft vulnerability rate. And it looks like at a month time, it looks like they probably double the 50% mark. So it was 36 days, so roughly a month to get to 50% vulns. It looks like there's 70 plus percent of the vulns remediated within a month. And then when we look at the year mark they're what, 97, 98%?

Ed Bellis: Pretty good. Yeah.

Dan: Yeah. That kind of like blew me away a little bit. And then I hinted at this as well. So figure 17, we also looked at what I have just decided to call the Microsoft end of life effect. So we looked at the comparison of remediation velocity, but across Windows platforms. And we looked at things that were still supported by Microsoft officially. So what, Windows? Well, if people even still use Windows 8, but Windows 8 2016 server, 2012 server Windows 7. And then we looked at newly unsupported operating systems or versions. So Windows 7 in 2008 server. And then also compare that with unsupported. So you do not get Microsoft support for Windows Vista, Windows 2000, XP, good old XP, and 2003 servers. So Jay, you want to do a little breakdown?

Jay Jacobs: I don't think anybody is surprised by this chart at all in any way. I think anybody's still running Windows XP, they have no plans to do anything with it anyway. They're not going to replace it. They're going to wait until it starts smoking or something. So it's not a surprise that the remediation rate on Windows XP is terrible. I mean after two years, they've closed 20% of the vulns across this data. So it's pretty sad. And then you just sort of scale down from there down into the newer systems. And then the thing I think is kind of interesting is that all of the supported systems are pretty much just grouped right together. They're all right there. Windows 7 isn't any different than 2016 server. So I think that's kind of interesting.

Ed Bellis: I'd like to correct Jay. There was one surprise and the fact that Windows Vista existed on this chart at all, just because even when it came out, I don't think anybody crosstalk, let them know now.

Jay Jacobs: Well maybe there was like two assets behind the chart or something. The two Windows Vista assets.

Dan: It was awesome. It had that whole arrow interface, remember? Transparency and glass. That broke GPU.

Ed Bellis: I don't remember.

Jay Jacobs: I don't know.

Ed Bellis: Everybody hating it. I don't think I ever actually thought in a while, but there is one thing to call out here too. I think when we're talking to Jerry Gamblin about this, he was looking at this chart and for the last, the bottom two lines, the fastest two, isn't that also when Windows or Microsoft introduced auto update by default, as opposed to logged in which made those even faster?

Dan: Yeah. There's a big hubbub about Windows eight forcing security updates. And people getting kicked off of their conference call meetings because their Windows machines like, " Nope, we're going to do this right now. This is happening. You have no say." Yeah, no, that makes sense. And I just thought it was interesting as well that it's not a surprise, but it's a clear gradient almost. So like Jay, you mentioned all the currently supported devices, almost all track, same remediation rates differ right around three months or so. But they still are very tightly grouped. And then the next area in terms of lessening performance overall is the newly unsupported operating systems. And then there's just a gradient between there and the one Windows Vista machine. That's some guy's maintaining somewhere in Oklahoma and scanning for some reason.

Jay Jacobs: That's the other thing, is you're actually not only do you have Vista, you're scanning it.

Dan: What if this is like Windows museum or something? It could be like a technology museum somewhere, just keeping it up and connected to the internet, which is scary.

Ed Bellis: This chart would have been more interesting I think if Jay would have actually made the time to break this out also by third- party versus Microsoft only.

Jay Jacobs: So much to do.

Dan: And infinite amount of reports to go. The next section we looked at for velocity was also the Unix slash Linux platforms. I know there was some interesting, like what the free BSD and some interesting kind of like jumps here, but I don't think anything was super out of band. There wasn't anything crazy, but was there any takeaways from you Jay?

Jay Jacobs: Well, no, like you mentioned free BSD and that the line for free BSD jumps around a lot, which indicates that it's a pretty small sample. So I probably wouldn't read too much into that, but something like Ubuntu that does sort of stick out ahead of the pack as, and of course, it's nowhere near Microsoft, but it is sort of ahead of the pack of the different Linux flavors and how they get off the block there. And so they're way ahead. Probably about the same at three months where most other platforms are six to nine months. So it's kind of interesting to see that, but then one year they're right there in the pack. So people who don't patch Red Hat also don't patch Ubuntu.

Dan: Yeah, it just seems to me to speak to, I guess, the challenge may be of trying to fully patch and, or update these systems. Because as a whole group, it just takes a lot longer to hit 50% overall and then even at a year. Even the best among are at only 70%. I don't think that's surprise to Ed either.

Ed Bellis: No, nothing surprising here.

Jay Jacobs: Same thing for the devices and network appliances. They're all sort of bunched in there and you can see some do a little bit better, do a little bit worse, but they're all generally pretty poor, especially having looked at Microsoft and how they're so great at getting off the start and patching a whole bunch of stuff. And so I was looking at this as like, " Oh, these are all sort of terrible."

Ed Bellis: Yeah. I would say, as we kind of progressed through these charts too, we looked at Microsoft, which was very fast, but also had to deal with the most amount of vulnerabilities and the most amount of high- risk vulnerabilities. Then we get into the Linux flavors and they had to deal with less vulnerabilities, but probably more than these appliances and they're slower and it takes them longer. And then you get into this the network appliances and there's very few vulnerabilities and very, very few vulnerabilities actually being patched.

Dan: Yeah. Yeah. I thought to me, you see these large steps and they're happening across the different vendors that we see in this appliances section. So I mean to your point earlier, Jay, it's probably a lower sample size, but I could also seeing this being one vendor decides to put out an update to their actual devices, which may happen once a year. And people are like, " Okay, I'm going to go ahead and do this right now. They finally issued a new update to this firmware. Let's go firmware update all these, this is our weekend project."

Ed Bellis: And people are running links in the corporate environment, who knew?

Jay Jacobs: And you made a good point though, Dan, that when we talk about these devices and network appliances that we say that people aren't patching them, but there's also that flip side of are the vendors producing patches when these vulnerabilities are announced and publicly disclosed, or do they sort of sit on them and like, " Eh, we'll get to that firmware at some point." And those access cameras, they're running fine for the most part. We're just going to wait on that firmware upgrade. The vendor saying that. So the consumer might sort of have their hands tied here.

Dan: I think we did a little digging too when we looked at it just to see if that was a case, if there was some aligned from a time perspective, with a big patch or a news item. Something that hit the airways.

Jay Jacobs: Yeah. I think the patch stuff is going to be a topic to dig into, one of these iterations. It's always been on our list and we want to look at that patch generation application.

Dan: Awesome. Cool. All right. And then I guess the last area that we look into in this report is remediation capacity. So this is the whole ratio of new and closed vulnerabilities within a given timeframe. So are you gaining ground? Are you treading water, or are you falling behind in vulnerability remediation, particularly high risk? And so we kick off with the median proportion of vulnerabilities closed per month. And Microsoft again at the top 25% within a given month, Mac OSX is 22.6%, which makes sense as well, looking at the velocities earlier. Linux Unix follows at 10% and then appliances devices at 9. 4%.

Ed Bellis: But when you think about this chart then and apply the raw numbers in the background, it becomes even more impressive for Microsoft. Because not only are they leading on percentage, but the 25% of Microsoft on vulns is a lot more vulnerabilities than 10% of Linux vulnerabilities as an example.

Dan: What was the total number of devices for Windows roughly? It's like crosstalk I'm trying to look back. There's like three, 200 million or something like that in the sample I think. So to your point-

Ed Bellis: It was lots.

Dan: Yeah, it was lots and 25% of lots is a whole lots still I think is your point Ed. And then we start looking at the high- risk remediation capacity across the asset categories. So figure 21. What's the percentage of assets that are improving for lack of a better word, I guess? And Microsoft still, they're actually not number one as a percentage, but keep in mind, this is a massive, massive number here. Jay, do you mind doing a little overview for figure 21?

Jay Jacobs: Yeah. And these were probably my least favorite charts in the whole thing. Measuring capacity as a super hard thing to do because month to month companies will push ahead one month and lag another month. Or trying to look across, I think a two year period on these assets and some of them haven't been around for two years, so on and so forth. And so it's hard to get at these numbers. And so, these are looking specifically at assets where we talk about going after 10% of their vulnerabilities a month.

Dan: The overall capacity kind of Bellus law. Right?

Jay Jacobs: Right.

Dan: So the average organization can patch roughly 10% of their vulnerabilities in a given month.

Jay Jacobs: Yeah. And so if we go back to that figure 20, we're seeing 25% of Microsoft platforms, or I should say on Microsoft platforms are closing about 25% of their vulnerabilities per month. So it's interesting to see that shift like that, but again, this is like per asset per vulnerability. And then you get into 21 and 22 where we're looking at platforms and then specific asset operating systems. For the most part, most of them are improving. You do see that, I don't even know how to describe these, I guess, but most of these charts are blue saying that they're improving across the-

Dan: Blue means good. Red means bad. Gray is somewhere in the middle.

Ed Bellis: And we see in these two, it's kind of the opposite is of most of the things that we saw in terms of the remediation. And the fact that the network devices are improving the most or are staying ahead the most. Versus Linux next and then Windows kind of trailing a bit there. Everybody seems to be doing better in terms of, at least I'm looking at this kind of figure, what is this 21 now? Everyone seems to be gaining ground versus losing ground, I guess, which is good. But Microsoft's gaining less ground, I guess, than the others. But also again, caveat that with everything we talked about previously, which is the number of assets and the number of vulnerabilities way higher for Microsoft than the others.

Dan: Yeah. Well, I'm thinking about the velocity and how just the sheer volume that Microsoft had, they kind of have to be that fast, given the overall volume of vulnerabilities and the prevalence in the environment. Like just to me looking at these kind of, if you're falling behind staying ahead, roughly 61% are improving of the assets. They're reducing their vulnerability debt on those and 31% are falling behind. It would be really, really easy for that number to invert without Microsoft's help just thinking about the speed that they're already hitting right now. Right?

Ed Bellis: Yeah. If Microsoft's velocity was quarterly patch updates instead of monthly patch updates, I would imagine that this chart would be very different.

Dan: Yeah. So fun stuff. So just kind of going back over everything different approach, we looked from an asset centric view this time. And basically we started out by trying to classify assets and how prevalent they were within enterprise environments. We then created a brand new metrics, so vuln density per asset. So which ones out of them actually had the most vulnerabilities per asset on them. And then we started applying some of the other metrics that we've defined over the course of our research. So coverage, remediation velocity, and remediation capacity thus far. And then it's fun because you guys built out this kind of overall chart that almost looks like a heat map based off of these different metrics. So vuln density, assets at risk, high risk, high risk coverage, all this fun stuff. And it almost shows you could probably map some of your asset based strategies to some of this stuff as well. So Jay, do you mind just giving us a quick overview of this last kind of uber chart, which has been a nice staple of scientific research for the last few reports?

Jay Jacobs: Yeah. And so essentially we look at the four platforms that we've been talking about and we looked across the seven topics that we covered here and tried to show the quantities that we derived in each of these seven categories for each of the four platforms and then essentially ranked them and how they're performing. So you can see like Microsoft does not do well for vulnerability density assets at risk, that's the high risk in forms. And then the high risk density per asset, they are at the bottom of the pack, but then they do great at their high risk coverage. The half- life in the remediation cycle, their fixed rate is doing great and the net capacity, again, they drop way down. They're in fourth place on that. So it's just sort of interesting to see obviously Microsoft has a ton of vulnerabilities, but they're attacking them well. And then the appliances devices and Linux are essentially the opposite. They don't have a huge density, but they do rather poorly at going after them, fixing them and the half- life of them. And so, here's one plot to essentially look at these four platforms, seven metrics and see how everything stacks up.

Dan: Awesome. And that kind of wraps up this report here. I think the overall takeaway that we stated here was assets matter, ultimately not a surprise to anyone involved. But I think this was a good first start at being able to really ground out with some baseline data that we can start to use to build on to inform our remediation strategies from an asset perspective. So Ed, any final takeaways before we hop off here?

Ed Bellis: No. Overall, I do love these charts at the end that Cyentia has been adding in terms of looking at all of the various things that the report included in terms of the remediation metrics and things like that. Overall, not tons of surprises here, Microsoft dominates in terms of assets, they dominate in terms of volume of vulnerabilities and volume of risky vulnerabilities, but they also dominate in terms of remediation capacity and velocity. Which are all things that we kind of thought were right going in, but definitely confirm that here.

Dan: Awesome. Jay, any final takeaways from you?

Jay Jacobs: The only thing I can think of is we talked about these different platforms and one of the things, actually, we had a discussion early on to breakout these assets, and we realized that a lot of these assets are maintained by different teams. You have the Linux team paneling that, the Windows team handling that. And so to break it out, we could essentially talk about how these teams are doing. And we ended up talking about way more than that, of course, but that was our initial thought there. So I think it's just cool to see this difference. And as we look at things like the overall remediation timeline now, now after doing this, especially, we're like, " Hey, let's break out Microsoft and let's break out the vendors of the responsible parties for their vulnerabilities." And let's just not assume it's one big bucket that these differences do matter and that can affect the outcomes and conclusions that we draw.

Dan: Awesome. Well, I think that was a good summation and I appreciate Jay and Ed your time as always. For anyone listening in, again, you can go find links to all the resources and figures and everything we mentioned on this podcast at kennaresearch. com/ podcast. It'll be in the show resources section here. I do want to tease a little bit. So this is the last of the currently published P2P reports that we're currently doing a podcast on. Should have a new one, volume six, coming out within the next few weeks as well, depending on when we publish this episode. So getting it out right in time for a new one. So should be some cool analysis. So stay tuned here and thanks for listening to us, have a nice day.