Patch Tuesday Rituals with JCran

Dan Mellinger: Today on Security Science, a security researcher walks us through his Patch Tuesday ritual. Hello, and thank you for listening to today's podcast. Today we're going to be discussing both the bane and salvation of security teams, and the start of a month- long fix cycle for IT teams, Microsoft's Patch Tuesday. And my guest today is going to walk us through his monthly Patch Tuesday rituals and giving us some best practices along the way. The internet's most intriguing security researcher, JCran. How's it going?

JCran: Really good. How are you doing, Dan?

Dan Mellinger: Awesome. Oh, I'm doing good. Doing some of the research is pretty interesting. I didn't know a lot of these tidbits. I think I just took it for granted that Patch Tuesday existed.

JCran: It's a long tradition of not fun.

Dan Mellinger: Hey, well it seems to be providing some actual good benefits overall, but here let's kick it off with a history in case anyone doesn't know what I'm talking about. Patch Tuesday is essentially Microsoft Windows. They conglomerate a bunch of patches over the course of a month, so fixes, primarily security, bugs, things like that, into a monthly cycle of updates. So Patch Tuesday occurs on the second and sometimes fourth Tuesday of each month in North America, and then basically they push these updates to Windows update function on 1800 or 1700 UTC, which maps out to 10:00 AM Pacific time, no matter if we're daylight or savings or not.

JCran: Yeah, it's pretty... I thought that it was really interesting that it's always at the same hour. And one interesting thing about the Microsoft patches being released is by that point you've got the Adobe patches, so you're already kind of like digging through those because they almost always get released earlier. So it kind of goes from Adobe into Microsoft, and then if there's other patches during the day you have the data to kind of dig through them after noon.

Dan Mellinger: Oh, that's fun. Well, I think it kind of makes sense, right? Adobe, their past security performance warrants them having a little bit of a head start.

JCran: I don't think they always did it on Patch Tuesday, too. It's been a while, but at some point they lined them up so that they would be at the same time. We'll dig into it, I'm sure. But this idea of the cadence is really key to security these days.

Dan Mellinger: Yeah, absolutely. Well, it's kind of interesting because what, it kicked off October 2003, so I did not know that was when it got formally... I guess Microsoft formalized this whole process and really kicked off in Windows 98. Microsoft started including Windows update that you actually had to install it at the time. It wasn't part of the operating system, but it would check for patches to Windows and all the components, so Microsoft Office, the whole subset, if you're running server, any of that stuff intermittently, and it would just hold all these patches and then all of a sudden release them every Tuesday. And you know what? They actually put a little bit of thought into the date as well. So Tuesday was chosen, apparently, because it was the optimal day of the week to distribute software patches. It maximizes the amount of time available before the upcoming weekend, so hopefully IT and security teams can get a weekend and gives them a little bit of time to correct any issues that might arise with those patches and it gives Monday free to address any unanticipated issues, server downtime, things like that that may have popped up from the preceding weekend. So, yeah.

JCran: I mean, Monday is always a scramble, right? And you need to get back into the office or I guess back into the home at this point, and Tuesday is actually a really nice day to do it just because it does give you that time to deal with it over the week. And we get a lot of questions on Tuesday itself of should we accelerate this patch or that patch and do you guys have any information about this bug or that bug? And so Tuesday is always a little bit hectic around kind of, but I like those questions because it shows people are digging into the details of it and trying to figure out should we create a fire drill or not?

Dan Mellinger: Speaking of, I know there's a term, Exploit Wednesday, so that would denote, right, Patch Tuesday hit. Exploit Wednesday was when ostensibly all the bad guys on the internet started looking at the patch notes and figuring out what they could exploit a lot faster.

JCran: Bad guys, bad guys.

Dan Mellinger: Is that still a thing?

JCran: I mean, do people inaudible the batches day of or on Wednesday ahead of trying to understand what the actual bug is? Sure. Yeah, absolutely. I think the mitigations have gotten good enough inside Windows especially that you're not going to see an exploit released on Wednesday in general, especially for memory corruption bugs that can take weeks or months or even years. Some of the research kind of points to this idea that you can still be exploited from a bug that's never been seen three years down the road. But in general, I would say it takes a little bit longer with Microsoft just because of all the mitigations that are in place.

Dan Mellinger: Huh. Interesting. So not necessarily time based but more dependent on the actual vulnerability and/ or bad thing itself.

JCran: Yeah. They should really call it Patch Diffing Wednesday, right? You're spending all that time digging into it to try to understand where the vulnerable code path is. And by the way, like you mentioned, October 2003 is when they formalized it, but I think the trustworthy computing memo was like 1995. That was a long time before that. Was there a Patch Tuesday before 2003 in your digging? I don't know.

Dan Mellinger: Yeah. Yeah. It just wasn't actually like a formal process. So it did exist leading up into them. October 20, 2003 was when Microsoft was like, " Oh yeah, this is Patch Tuesday. This is how we're going to do it from now on", and they've kind of stuck with that. The only exception is they've done a few out of band...

JCran: Right.

Dan Mellinger: ...Patch days, and that's typically something really bad. Otherwise, they might also if they have a big enough, I guess, group of patches, they might also push to the fourth Tuesday of the month and those are typically not security- related updates. So when they get a big batch of security- related updates, they might just put all those in the initial second Tuesday of the month cadence like normal, and then they might push a second one on the fourth. So pretty interesting.

JCran: Yeah. It is super interesting. I think they committed right before everything hit earlier this year kind of as it was happening, that they would only do security patches. In your digging, did you find anything about that?

Dan Mellinger: No, I did not see any of that, but we can look it up.

JCran: We should. Yeah. There was an announcement made earlier this year that they would really focus on security patches just for stability during this time, and that's why I think you're seeing more security patches or at least it's a big component of why the latest Patch Tuesdays have been pretty heavy.

Dan Mellinger: Oh, absolutely. Yeah, definitely. So Microsoft has committed to really doubling down on their security investigations and reporting for patches, especially with COVID and everything going on. So if anyone's listening decades down the line, we're going through the COVID 2020 pandemic/ 2020 the year that the world lit on fire and we're all just kind of sitting here and dealing with it. But anyway, yeah, Microsoft has been pushing record numbers of security patches. The last few, I think April was the first record and then June, right? The last one was also a new record- setting Patch Tuesday in terms of total number of patches issued, right?

JCran: Yeah. I don't know the numbers off the top of my head, but that sounds right.

Dan Mellinger: Yeah. That stuff we'll get into a little bit later, but it does yield better security overall. So with that, JCran, do you mind running us through what's your ritual? What do you do? Start prepping Monday night, put the aspirin on the bedside table for when you wake up.

JCran: No, no. So I'm central time zone so it hits a little later for me, which is nice. And so the biggest thing is we just try to get the information as soon as available on Microsoft sites so we have scrapers that go out and grab that information. You know, MBD often ends up being a little delayed. The information will be available on Microsoft site. In theory, they are released at the same time, but what we find is that sometimes miter and MBD can lag a little. And so we'll pull that information down, dig through it, and I'm really looking for things at this point that are particularly scannable or particularly wormable, partially because that's my interest but also partially because those things tend to present risk faster than other things just because if you can find the software, fingerprint it, detect it, and find it's vulnerable, those things tend to be particularly interesting.

Dan Mellinger: How do you determine if something's wormable versus not?

JCran: Well, you look for something that's remote code execution in general. If you can run code on the server, then that's usually wormable. And in inaudible it really just comes down to the type of software, so like Exchange, SharePoint, those types of software. And anything in Office is pretty interesting, right? Office has been under attack for the last two- three years, especially macro- related things, and any bug that you can use to sort of get user side execution through a file is particularly interesting. So you look for the attack patterns that you know attackers are using, and if there's new bugs in that those are things that are probably going to pick up pretty quickly. And so we don't do a ton of patch diffing. If there is a particularly interesting thing, we will potentially look through it to pull up in diff and try to understand the underlying fundamentals of what's changed. Otherwise, we're looking at things like ZDI, the Zero Day Initiative, for kind of advanced analysis pretty quickly. Those folks tend to put some really good detailed analysis out there. And if there's been anything particularly exploited in the wild, that's also stuff we're looking for. We'll go back and dig through our own intelligence and see if we can figure out if these are things that we've seen already or not.

Dan Mellinger: Yeah. On a month- to- month basis, how many times do you typically see something and you think, " Oh, we should probably go patch that immediately."

JCran: Well, that's a good question. Yeah, so usually there's one or two. This whole concept of the fire drill. I think programs and teams are getting sufficiently advanced to where they're just knowing that they need to make those decisions, and so they're ultimately trying to put things into one of two boxes. It's either delay this and do it as part of the normal cadence or we need a fire drill around this, we need all hands on deck, we need to find where the software is and we need to get it patched. And so how many fire drills? The one that most recently comes to mind is the DNS bug that was last month or this month I guess at this point. We're still in July. That was effectively if you can make a DNS server make a request to a malicious server through a recursive DNS, which is pretty low barrier to entry, that's a wormable bug and so that's the thing you're going to need to accelerate. And so things like that. Anything that needs to be put in the fire drill bucket is stuff that you're going to want to patch right away.

Dan Mellinger: Hmm. It seems like pretty clear cut what those are most of the time looking just at descriptions for the vulnerabilities themselves, the patches.

JCran: Yeah. It's a combination of things. It's like you really do have to dig a level deeper than just the software, just the version. Is it a default configuration? Is this something that is very prevalent? Really what you're looking for is prevalence. Is this something that's particularly prevalent in our environment? You're trying to understand what's the impact to us, and there's always limited information. I mean, Microsoft does a really fantastic job in comparison to other vendors about providing detailed information that helps you understand what the impact is going to be, but every little bit helps and this is why I'm all for patch diffing. If you can give me better information about what's really vulnerable and I can use that to decide whether I'm going to get folks out of bed or not, that's super helpful.

Dan Mellinger: To that end, do you use any kind of tooling? I mean, Intrigue.io, I use that all the time for scanning, fingerprinting, stuff like that. What are some of the tool sets you use when you're trying to do these kind of Patch Tuesday diffing?

JCran: Yeah. So I mean, the biggest thing is prevalence, so actually I spend most of my time kind of going through Kenna's database trying to understand inaudible is this, and it takes a while for the scanners to pick up. So you've got to go through this process of the information has been released, it needs to be loaded into the various vulnerability scanners, and then it eventually makes its way into Kenna and makes its way into the risk- based tools. And so there can be a delay in us understanding the prevalence, but what we can do is go back and look at the software that's running and look at the kind of like most common using CPEs. CPEs are common platform enumeration, basically versions of software. Using that information, we can kind of understand what the scenario is going to look like for a particular piece of software. Again, it's missing some information, but it's enough to kind of give us a sense of it. And that tends to be the bulk of the analysis that we're doing is just trying to understand the impact from how many systems are vulnerable, and then what does it actually mean to be vulnerable for this thing? There's a lot of analysis around what does it mean to be vulnerable. There's not great information always about how many of these systems are actually vulnerable. And that's what we end up doing with both Kenna and Intrigue data.

Dan Mellinger: Kind of fingerprinting which ones are facing the internet, which ones are actually accessible, which ones can we hit, that kind of thing.

JCran: And you can use stuff like Shodan and Census and things like that to understand as well. Increasingly, there's great fingerprinting around DNS data or DNS servers. There's increasingly great fingerprinting around network services, that sort of thing.

Dan Mellinger: Well, that's fun stuff. And I guess we can take us to the next stage of does this actually help? Is Patch Tuesday ultimately beneficial or a lot of IT teams spend a lot of time parsing this data, trying to apply patches, testing, making sure things don't bring their servers down, don't crash the CEO's laptop on accident, things like that. So there's a lot of risk and a lot of effort that goes into these monthly cycles of patching.

JCran: Well, you bring up a really good point. I mean, like the GRUB bootloader vulnerability that just came out last week, which affects Microsoft systems and Linux systems, and a lot of the discussion around that bug is like this is a very serious bug in certain circumstances. If you've got administrative access on the computer or you have physical access to the computer, this is game over.

Dan Mellinger: So if you chain it with this other thing, that's already pretty bad.

JCran: But at the same time, that's such a fundamental component of the system that if you mess up the patch and you rush out a patch, you can brick a system easily, and that's far worse consequence than somebody running adware on the box, right?

Dan Mellinger: Self- inflicted ransomware.

JCran: Absolutely. And so it's worth understanding what those systems are going to do, and you'll see more advanced teams testing out patches in a specific test environment before they roll them out. And so I think does the Patch Tuesday help? Absolutely. But there's some caveats to that. One thing it's done is get teams on a regular cadence and it kind of forces this idea of every month for this week, we're going to be pushing out patches to systems and it at least helps create that routine that helps people maintain systems, because maintaining systems is not fun. This is not fun work.

Dan Mellinger: Not the sexy stuff.

JCran: It is not the sexy stuff at all, but is some of the most fundamental and important stuff. And one thing, I remember the PDP. I think this was before where we were digging into what are the most effective ways to ensure that your vulnerability management program is more successful than others relatively, and I'm sure you remember this: It's the automated patch deployment systems. That was a huge factor. And that makes sense, right? You get these patches. They come into your system. It takes a little bit to get information. Maybe you do some quick testing across all your different environments, and then you push the button and you roll out patches to each of those systems either simultaneously or on a kind of a rolling schedule. And that's the ideal scenario, right, is you've tested it, you know it's going to work, and then you roll it out automatically. Does it always work like that in practice? Heck, no. And you've always got those kind of systems that sit in the corner and the exchange server, of course, gets patched manually, but in the end does Patch Tuesday help? I think those are a couple of data points that just really reinforce the point that it's very helpful. I know you had some other ideas and points around this as well.

Dan Mellinger: Yeah. I mean, it's interesting because we've kind of touched on Microsoft a few different times in the prioritization prediction series. The last one we did really an asset centric view, so we were looking at software versions versus Linux versions versus embedded systems, right? Printers, network devices, all that good stuff. And Windows, I should say Microsoft, by far has the largest amount of vulnerabilities of any of the classes, but they also made up 70% of...

JCran: The assets, right?

Dan Mellinger: Of the assets. Yeah. They're so prevalent in business machines that, yeah, of course there's going to be more because: ( 1) Microsoft is putting in concerted effort to have people pen test and submit bugs and find holes in software and are really on the leading edge of that and, they're just so prevalent. Of course, if I'm going to target something I'd go after something that people are using a lot, right?

JCran: Yeah. I mean, I think it's a case study in scale. You look at other companies that push patches on a regular basis. Oracle. Google, also, is definitely a case study in scale, though all of that would be centered around the browser or Microsoft has many patches and many sort of manual deployment mechanisms as well and has kind of grown into this leader in how to do this, right? And to be clear, you need a certain amount of scale to do this really, really well, because if you think about those patch servers and the value of those patch servers to an attacker, you probably don't want Joe arbitrary company running patch servers which are pushing patches to all your whatever widgets. Patch servers in and of themselves are kind of interesting targets for an attacker given they provide binary code to systems to run. Just kind of an interesting tidbit there, but Microsoft has done that really, really well. And to my knowledge, there's never been a particular incident around those servers, and thus the scale that they're operating at... I know you mentioned this earlier, 119 vulnerabilities per month on a given Windows asset, and they get patched within 36 days. That's incredible.

Dan Mellinger: That's incredible. Millions and millions worldwide that are patched literally every month; 36 days is just impeccable, is the fastest by far of any of the device classes we saw.

JCran: Yeah. Network devices were much lower in terms of vulnerabilities and much slower to patch for obvious reasons, right? It's very manual, but it was something that was kind of astounding in the difference. It was like three, four vulnerabilities a month, and an average of a year to fix.

Dan Mellinger: Yep. It's crazy. So that's... Yeah, I was joking, I think, earlier with Ed and I was like, " Yeah, I better go patch my router. It's been a couple years." Right?

JCran: I mean, you might as well throw it away, right? Can you even run the latest firmware on it? And by the way, it's storming. I don't know if you can hear it in the background.

Dan Mellinger: Yeah. Wow. That's was... stay safe. I hope you don't lose your internet connection.

JCran: No, no. It's nice to get a little rain. You might as well go open source and just go grab the latest...

Dan Mellinger: WRT?

JCran: Yeah. There's a couple of them now.

Dan Mellinger: Yeah. But what about my printer?

JCran: Yeah. crosstalk Yeah, that's nice. I don't know of any open printer firmware.

Dan Mellinger: Me neither, actually.

JCran: If it gets that old, can you still buy cartridges? I mean, I have a laser printer here from 10 years ago, maybe 15 years ago.

Dan Mellinger: Well, I think... Don't they even make the cartridges proprietary?

JCran: Yeah.

Dan Mellinger: Right? Isn't that the whole thing. They're all electronically coded with chips, so you can't use knockoffs as easily.

JCran: I don't know. I know that they have the little printer codes that come on every page. This printer might be so old that it doesn't actually print printer codes.

Dan Mellinger: Wow. Yeah.

JCran: That was a big thing a couple years ago when that was used to basically find the people that were distributing information.

Dan Mellinger: Oh, yeah. The watermarks.

JCran: Yes.

Dan Mellinger: It just reminds me of when you use Reddit posts now. They always have a little embed when you download the pictures so you know who it's from. I was like, " That's awesome. Way to go, Reddit."

JCran: Oh, wow.

Dan Mellinger: Yeah. You know, I think when we're getting back to the conversation of the efficacy of Microsoft Patch Tuesday, what's interesting about it is you can actually really clearly see the impact when you start looking at the end- of- life operating systems that they stopped supporting. And so this was super interesting because when we're looking at the overall remediation rates, it varies wildly for these unsupported operating systems. So I think Windows XP was by far the worst, and they still had 75% plus of their vulnerabilities unpatched after two years. And then when you look at all of the current gen systems, so Windows 8, right? Windows 10, all the brand new Windows server packages that are still within support. They're all right on top of each other in terms of remediation rates, so 50%. Within roughly just over a month, they're hitting 75- 80% and beating out Apple products, actually, up until month four. So very, very interesting. Clearly, they're driving remediation of their vulnerabilities at, just like you said, spectacular scale.

JCran: Yeah.

Dan Mellinger: Awesome, JCran. Well, what are the three things, right? What are the last considerations people should be keeping in mind next time Patch Tuesday rolls around?

JCran: Yeah. Good question. I would say get your timelines so you can be aware of the releases and buy yourself as much time as possible. So talking on Pacific time, by 8: 00 AM you should be looking for the Adobe release, by 10:00 AM the Microsoft release. And by the way, make sure to subscribe to the Microsoft updates because sometimes they'll send additional updates after the fact, and keep an eye on the Microsoft exploitability index. That's a thing. It's kind of a one- time thing. I don't believe that they keep it continuous like inaudible does, but it's at least a quick check to where they think this is going to be exploitable or not or has it been exploited in the wild. And that's really a good indicator of whether this is something that might need to be accelerated or not, and obviously keep an eye out for keywords like RCE and exploited in the wild. And then you can always go to kennaresearch. com and go to the EPSS calculator, plug in the information that you have, how many references, who is the vendor, in this case Microsoft or Adobe, and get a rough estimation of how likely this is to be exploited in the wild. So if you need a tiebreaker, you got an argument on your team," Oh, we should fix this one or this one", I highly recommend going to kennaresearch.com and hitting that EPSS calculator.

Dan Mellinger: Awesome. That sounds like a fun argument to have. Awesome. Well, appreciate the time, Jake Gran. Like he said, I will be linking some of these, what, Zero Day Initiative. We'll link some of the Microsoft subscription services. I'll link Intrigue, Shodan, and EPSS along and some of the resources on our podcast page, which is also on kennaresearch. com. And other than that, stay safe and have a fun month- long cycle patching.