When the Hell are you Going to Patch?

178481370Growing up in a small town, it isn’t uncommon to find livestock (or farm animals) on the road. When this happens, the proper and neighborly thing to do is to get the animal off the road so everyone is safe.

Driving late one night, I saw a black shadow by the side of the road.  Assuming the shadow was a horse, I quickly stopped.  Sure enough, there were six horses on the road.  We found a gate and carefully herded the horses into the pasture.  By the time we accomplished this, there were three cars stopped, along with a sheriff.  None of us knew who the owner was, or even if it was the correct field, but soon the horses were safe in the field and everything was secured. .

The sheriff told us all to move along so we didn’t create a hazard, but having had experience with escaped livestock, we did a quick U-turn and drove slowly back up the road.  Quickly checking the rest of the fence, we found a gate that was ajar in the far corner.  We stopped and closed that gate.   While closing the gate, the sheriff pulled up and wanted to know why we hadn’t left, we had to explain that by just remediating the current issue and not checking for the hole in the fence, would have allowed the horses to be back on the road in a matter of minutes.

This story came to mind the other day while I was discussing a security issue with an enterprise admin and some consultants.  They were discussing how they had been fighting a virus for two weeks. It was polymorphic and was morphing faster than the AV definitions could be released.  The AV definitions were not keeping up with the transformations. The network was being brought to its knees, network outages had been plaguing the business due to the virus and the traffic generated.  They were frustrated with their AV vendor.  We discussed the challenges and technologies that could be implemented to stop this virus.  They had a comprehensive plan that included new AV definitions, application blocking to block the files that were known bad, leveraging network devices blocking ports, poisoned routes and custom layer 7 switch rules, etc.

After agreeing that they had been using a good plan, I had to ask, “When the Hell are you going to patch the vulnerability?”  They looked at me like I was in the wrong meeting.  I explained my thinking.  If the virus could propagate like this, there almost certainly had to be a vulnerability that it was leveraging.  We just needed to find the vulnerability and patch it.  If there wasn’t a patch, we needed to have the vendor create one.  As it turned out, there was a patch.  Of course with every patch comes some risk, the risks of application compatibility, the unexpected glitches caused by a failed patch, or just plain old operating system issues.  After analyzing the risk factors, we were able to deploy the patch.  By the end of the day, we had the virus squashed to the level of an annoyance with no more network outages.  All of the best laid plans were not needed if we just patched the hole.  This incident triggered a more robust patch testing cycle that has allowed them to shorten the patch deployment time.

When I was a kid, there was an old cowboy that asked me if I was good at math. I told him I was of course, so he asked: “If there are 10 sheep in a field and one gets out, how many are left in the field?”  I knew the answer and quickly responded with “nine.”  He chuckled and said, “Not if you haven’t patched the hole in the fence.  By the time you did the math, more sheep got out.”    Patching is the best tool to stop the spread.