Patch Management Crisis? How to Get Over It.

Organizations of all sizes often find that the process of updating software has been halted by various business reasons. Two of the biggest stumbling blocks are the need to reboot after installing an update and having no guarantee that the system will work properly afterward. Any downtime can disrupt vital processes, so these concerns can lead to a business decision to stop the update process altogether — taking a company into the patch management crisis.

This crisis can drag on for years unless the company has skilled and proactive personnel who are willing to sort through all the accumulated mess and develop and implement a set of new patch management procedures that will be executed regularly rather than remain only on paper. For example, Peiter Zatko, Twitter’s former head of security, accused Twitter of a litany of poor security and privacy practices that constitute a national security risk; poor patch management was among his claims.

If your company is in a patch management crisis that puts its services at risk, where do you start to resolve the problem? How do you get the software upgrade process back to an optimal level after years of issues? This article provides a detailed plan for solving this challenge, step by step.

Plan Overview

The first thing you need is a plan for implementing a new patch management process. Here are the steps to take:

1. 1. Write down all the steps for implementing the process.
   2. Inform everyone involved that the process is designed and what role each participant plays.
   3. Set up a patch management solution.
   4. Test the plan on selected test devices. Resolve any issues you uncover and test again until the process is successful.
   5. Implement the plan on all devices in the company using maintenance days and months.
   6. After the successful upgrade phase, monitor and troubleshoot raised problems, after that each stakeholder must report that everything is working as usual.

Let’s take a look at all the steps of the plan in detail.

1. Create a Clear and Simple Process

The process has to be simple and each participant has to understand it clearly. Follow these guidelines:

• ^ Divide assets into logical groups according to criteria such as security groups in Active Directory, system type, or geographical location. Choose the design that fits your organization best.
• ^ Create an update task for each group.
• ^ For each group, assign a stakeholder who knows the assets in detail, as well as a system administrator who will apply updates to the assets using a patch management solution.
• ^ Notify users working with assets from each logical group about the maintenance time.
• ^ Assign each task to a stakeholder and a sysadmin to monitor the update and restart process and troubleshoot any problems that appear after the update.
• ^ Repeat this process at least every quarter, and ideally every month.

2. Talk with Stakeholders

Before going to system stakeholders, you have to get approval for the patch management process at the board level by explaining all the risks entailed by lack of patches on critical systems. Ideally, the patch management plan should be signed off at CEO level.

After getting approval from the board, talk to all system owners to find out how their systems work and if they have a process for testing updates before they are installed on production servers. Negotiate a schedule for rebooting servers as well as the sequence of reboots; this is relevant on database servers, for example. You can build this system in any project management system, such as Jira, Trello, or Project. It is also necessary to discuss the process of reacting to failures that may occur after the restart of a server, such as rolling back updates or restoring the system from backup.

It is also important to work with department heads to assign at least one person (and their device) from each department to the test group. It is important to root out compatibility and performance problems with any department-specific applications, such as accounting software or development tools.

3. Set Up a Patch Management Solution

To apply updates to your assets, you need an appropriate tool. For Windows OS patch management, Microsoft provides two popular options: Windows Server Update Services (WSUS) and System Center Configuration Manager (SCCM). SCCM can also handle third-party software updates but you’ll need to purchase an additional license. There are also other automated patch management tools available in the market, such as Action1.

4. Create a Test Environment

The two most common approaches to testing updates are:

• ^ Testing in a full test environment
• ^ Testing on local virtual machines, usually created using VMware or Oracle VirtualBox

To test using VMs, be sure to:

• ^ Create virtual machines for each OS version you have in your environment.
• ^ Use test servers for every system in the environment, divided by type: front end, database, etc.
• ^ Gather a group of test workstations that includes one device per department.

First, apply updates to the virtual test perimeter. If that process is successful, move on to the test workstations and then to the test servers.

5. Schedule Maintenance Days and Months

After you have successfully updated the test environment, proceed with the real updates on production servers and user workstations. You’ll need a “maintenance day” for servers in order to reboot them painlessly. Such days may happen several times per month or per quarter; each logical group will have its own maintenance day, as specified by the system stakeholder, so there is no downtime in business processes. For example, you will have one month in a quarter with a schedule of maintenance days for each group of servers.

It is a good practice to update your workstations after you update all your servers, as a simultaneous update may increase the load on your technical support and system administrators.

Here is an example of simplified maintenance schedule that shows the tasks for each day of the month:

• ^ 2 – Virtual test environment update
• ^ 5 – Workstation test group update

• ^ 10 – Infrastructure servers test environment update
• ^ 12 – Infrastructure servers production environment update

• ^ 13 – ERP test servers environment update
• ^ 15 – ERP production servers update

• ^ 17 – Accounting test servers environment update
• ^ 19 – Accounting production servers environment update

• ^ 21 – Web servers test environment update
• ^ 23 – Web servers production environment update

• ^ 25 – Security servers test environment update
• ^ 27 – Security servers production environment update

• ^ 28 – All workstations update

6. Monitor Update Effects and Troubleshoot Problems

You need to always keep in mind that any update could disable the system or disrupt application settings. For this reason, it is very important to make a system backup before the upgrade phase, which can be run on the day the updates are installed on the test group.

After the updates are installed, the stakeholder need to make sure that everything works as usual and there is no need to restore or roll back the system to the previous update. If problems arise, it’s important to fix them quickly and record the solution in the knowledge base so that all stakeholders can see what problems may arise after the update and how to solve them.
The same goes for user workstations: Make sure that tech support is aware of updates and is ready to help users if something goes wrong.

Conclusion

The article outlines a simple way to get out of the crisis of update management. The main prerequisite is that the CISO and board are committed to solving the problem and are ready to allocate the necessary resources, both human and technical. Otherwise, it will be very difficult to make the patch management process work as it should.

Remember that the update process for each system type should be a task in your project management system, all problems and their solutions should be recorded there, and solutions to common problems can be written in the corporate knowledge base. This will enable you to monitor the process and set clear deadlines.

After the first month of running this process, the next rounds will be easier, since the process will be fine-tuned and it will be necessary only to polish it. Don’t worry if the process drags on for more than a month; you can allocate two or three months for it, depending on your infrastructure. It is very important that this process is in place and that it is executed properly, even if that means going slowly.

After a while, you can choose to automate patch management tasks such as scanning endpoints for missing updates, installing all necessary patches, and reporting on installed ones. Action1’s automated patch management tool can help you streamline your patch processes. Action1 empowers IT professionals to efficiently manage updates and automate patch deployment to secure IT assets while saving time, resources, and budgets.