There are always 0 days. There are some though, where the tendency is to eat your life and take over the news cycle. All of them are important and some (due to a combination of media, reach, and impact criticality) are significantly urgent. When the news of Log4J began to make the rounds it almost instantly fell into the second category. One of those takes over your brain in the middle of the night kind of items.

I have the privilege of being enough of a trusted source on these kinds of items that both clients and industry friends have asked me about it. The thing is, like most of you, I kind of have some other things to do. Between the standard business operations, responsibilities of our team supporting clients, and our own internal business needs; anything else feels like a distraction. Even with that, being a trusted business advisor for our clients around their technology puts us square in the middle of having to give answers to news stories like this.

The core framework I like to work in is less security, and more resiliency. What happens when the worst happens? With so much dependence on IT, how do we serve businesses in a way they would still function if we weren’t there? While security is a part of that, it’s not the entirety. With that framework, the question for me usually is how much do I do, say, or show?

In this instance we’ve already built in some help to mitigate so the majority of the doing is done. The next step is to say and show. So, I’m taking the rare opportunity to give a peak behind the curtains at what we do in these instances, in hopes that it adds some peace of mind for everyone.

Breaking News

This is always the first step. With our industry contacts that are both official and personal, we at least hear about this rather quickly. Probably in the first few minutes to an hour, we’re aware of the situation and must quickly decide what to do. Partner reliance is a big deal in these situations. We have some preferred security providers who do an excellent job of jumping in front of these. Between Network Box USA, Sophos, and Bitdefender, we’re generally covered on the communications front. There are times we’ll write up our own alert, and many times we’ll let our security partners handle this due to their expertise. There are times it makes sense for us to step in and communicate more so that it doesn’t feel like you’re a character in a Batman movie where Pennywise the clown just showed up.

What to do ourselves?

Like you, we have our own business, with our own stack of technology internally. Step 1 is always to make sure all our stuff is covered. It’s the lifeguard scenario. If you’ve never heard that until now, here it is. During a rescue the lifeguards always protects themselves first. The logic being you’re no good to the person you’re rescuing if you die in the process of the attempt. Clients are our priority, and we protect our own environment first because of that.

Step 1 is to confirm we’ve patched all the things in terms of hardware. Basically, anything that touches the network. Here’s a non-inclusive list of some items I checked to give you an idea of the scope I’m talking about:

Servers
Laptops
Desktops
UPS
Firewalls
Routers
Switches
Access Points
Printers
Thermostat
Cameras
TVs
Streaming Devices
Toasters

A side note on this: Many of the IoT (Internet of Things such as cameras, doorbells, lightbulbs, etc.) things out there do not have a manual update procedure. You’re simply supposed to trust the manufacturer to do it. Without launching into a commentary on our industry, I’ll just leave it at deeply concerning and frustrating in moments like this.

Step 2 is the harder question. How to find out the scope of something like this? First thing is to read….a lot! I personally scoured for anyone talking about the situation on the internet. Starting with the source. https://nvd.nist.gov/ and https://cve.mitre.org/ are great places to read the actual articles on any CVE (Common Vulnerability and Exposure) hitting the news. Reviewing these articles alongside a review of the response from trusted partners will give a sense for what is at actual risk vs what concerns the media is throwing out that are on the level of Matthew Broderick from War Games.

Step 3 is the reactive place where we look for the vulnerability.

First, we reached out to all vendors to confirm any possible actions needed. We were clean by all accounts. (I still ran them all through the Huntress Tool as a double check.)
Second, I confirmed that our DNS/WAF provider already had rules in place to account for Log4J to cover my bases on any external facing web property that we manage and house ourselves. They had in fact already added this. Thank you Cloudflare!
Third, we layered further tools on top of our standard vulnerability management.

Some clients utilize our Vulnerability Management as a Service (VMaaS). For those clients and ourselves I’m less worried. However, I wanted that extra 1% assuredness for environments moving forward. So, I looked to our partners for their response to search. I found two partners who had done some excellent work on finding and testing for the vulnerability. So, we used Microsoft’s guide in building our own process and utilized Huntress’ tool to scan our own software. I’ll go into more detail further down on this article about what we did technically.

Microsoft’s Guide

Huntress Tool

ConnectWise Log4J Update Page

Log4J

A quick update on Log4J. There is still information coming out on this. As of this writing, Apache has already issued a 3^rd patch yesterday (December 18^th) and there are at least 4 CVE’s related to this discovery. As I said earlier, I did some research. At the end of the day this is a big one in terms of reach, and the reach keeps growing. We’re definitely feeling a bit surrounded over here. 😅

At minimum, anything with a web interface is concerning.

To quote ConnectWise from their reddit post:
“TL;DR Update, or bug your vendor until they update, Log4j to 2.16, or if you use Logback, version 1.2.9. Yes, Log4j 1.x is now vulnerable. No, the initial 2.15 patch didn’t fix everything. No, the configuration changes originally recommended don’t fix everything. Yes, there’s Ransomware operators exploiting this vulnerability (Conti in particular). Yes, you can be exploited even if you don’t have a publicly facing vulnerable server. No, we haven’t slept yet this week.”

What should you do now?

I’m going to move forward and cover the immediate response for those of you who are concerned and simply wondering what to do next. If you’re in any way interested in your business tech stack, I’m going to go into the technical side of this below. It will possibly give you a bit more peace of mind.

For the first group, here’s an immediate actions list:

Talk to your vendors, all of them. If you don’t have a list of who all your software providers are, then you need one. Use this as an opportunity to start making that. We run reports for our clients on all software installed on their systems. Reach out to us for help.
(Note: In this, beware of Shadow IT. There is a high likelihood that one of your staff is paying for a tool off the books that they’re utilizing to do their job. That unknown software has possible hooks into your business and your data. There are ways to figure out where these items are, and we can certainly help find those areas and guide you through that process. Reach out to us for help.)
- If your vendors are not helping fix this, then you need to be clear that this is very specifically in their responsibility area. The only way software manufacturers will change their security posture is because their clients demand it.
Be sure all your devices are patched. Use the list mentioned earlier in this article. For things like this some of ours are covered under our standard service agreement. If you’re wanting that and don’t have it, let us know. If you have that and want all your IoT devices covered, then we have a VMaaS service that does this. You don’t have to do this on your own, just let us know.
Use the Huntress Tool and the Microsoft Guide listed in the reference links section of this article to do some searching and validation. We can also do this for you.
Setup a Web Application Firewall (WAF) for any web property that you own. (e.g. our tools are all sub-domains of unicom-tech.com). So, anything we host falls under that domain and any traffic that goes to that domain has to pass through a WAF before it hits the site. It’s like having a big barricaded front door for any potential attackers. We have a preferred vendor for this and we can help put this in place.

The Technical Side of Finding (for the geeks out there)

If you’re a client of ours, you’ve noticed some changing of tools over the years. My tendency with tools is to be agnostic. My leader in decision making on this is customization and flexibility that allow us to do what we need to service our clients better and faster. In this instance our Remote Monitoring tool (Automate) wins the day a lot of times. Especially if we’re looking at having to touch 1000’s of machines that are geographically dispersed across the planet all at the same time.

Setup

There are times my job is to let the team run. We have a solid team who handles business and that lets me do the other parts of my job. When the issue is big enough, my job becomes dive in and build the system to hand back off for others to run with and serve people. In this case I was looking for any place where a Log4J filename showed up and on Friday night we had a solid example case to test with.

To be clear, changing things like this, even if it’s the right decision, CAN break things. We don’t want to break business operations to protect the business unless it’s absolutely a must. Communication with the client in this situation is key. We spoke to the client and were cleared to test on a few machines. So off to the races.

Goal

In this one I had a few goals for mitigation. First, we need to be able to find any place where a file with log4j exists on a windows workstation or server. Second, we needed to be able to make the file useless to malevolent actors. Third, we needed to be able to roll-back the change if necessary for Line of Business applications to function. Fourth, we needed a way to see all machines where something like this exists.

Find All the Files

PowerShell is a great tool and incredibly powerful. Without going into tons of detail on specifically how the thing works, I was able to create a PowerShell script that parses through every internal drive to search the file system for files with Log4J in the name. At this point I tested it on one of the machines for the client and it ran perfectly. From here our Automate comes into play. It has an option to utilize its script engine to run PowerShell commands and return the results as a variable to use in the tool itself.

I set it to return Vulnerable or Not Vulnerable depending on the results. From here Automate would send me an email with the results.

Make Files Unusable

From here the next step was to rename the file extensions with .txt instead of .js or any other file extension. Equivocally making them a non-usable library. Also using PowerShell for this, however it wasn’t as easy said as done. I had to split the file name from the extension and then put them back together with a different extension. Then use both the old filename and the new filename in a command that renamed the file itself. Fun times.

Have a rollback plan

Finally, I had to be able to roll back from .txt to the original file name. This was a bit better since it was a simple matter of taking the previous script and reversing the file extensions.

Deploying in Mass for tracking

The final goal was being able to deploy and apply in mass. Here’s where most of the work went to the Remote Monitoring tool. Up to this point, the power had been PowerShell utilized by Automate. As I said, it’s a flexible tool. So from here the next step was to build out something that could be used over and over again no matter the scale we use it at. I certainly don’t want several thousand extra emails to try and figure out what to do about. (While email is great for communication, it’s a terrible monitoring tool.)

First, Automate allows us to create our own fields in the interface. So I made certain there was a Log4J Detected box. I also added a Log4J Remediated yes/no drop down to track when it’s been remediated after discovery.

Second, I updated the search script to check the box rather than send me an email.

Third, Automate has powerful searchability. I created a search against the Log4J Detected box being ticked while the Remediated field is not Yes to find all machines where this was discovered and not fixed yet.

Finally, you can create groups that automatically pull machines in based on a search.

So, at this point it’s a simple matter of running the search on all clients to get a group where any machine in our client base with a Log4J named file exists. Completely deployable to 1000’s of geographically dispersed machines. While this doesn’t fix 100% of the situation, it certainly helps mitigate it. In this instance, we’ve only run it on one client where we know it exists. We’re scheduling this to run on the entire client base over the next week and then we’ll alert if there are any files found. The nice part is that we already have the remediation and a rollback plan in the event of trouble.

Hopefully, this technical walkthrough gives you a little glimpse into all the areas that you don’t normally see. This is what we do when you aren’t looking. We dig in and think about this for you. We help in visible ways through support calls and invisible ways that are much more proactive and impacting, such as 0 Day vulnerabilities. This is the power of a great team and having the expertise to rapidly create and deploy items like this. Does it make all the problems go away? No. Does it help mitigate the pain? Yes. Have we slept yet? Not much.