Troubleshooting Patch

If Patch is not performing as expected, you might need to do some troubleshooting or change settings.

To prevent issues that might occur due to endpoint health issues, ensure that endpoints have:
  • At least 5 GB of free space
  • A healthy Windows Update Agent
  • An up-time of less than 90 days

Collect a troubleshooting package

For your own review or to assist support, you can compile Patch logs and files that are relevant for troubleshooting.

  1. Get the Patch log.
    1. On the Patch Home page, click Help .
    2. Click Collect Troubleshooting Package.

    The log zip file might take a few moments to download. The files have a timestamp with a Patch-YYYY-MM-DDTHH-MM-SS.mmmZ  format.

  2. (Optional) On the endpoint, copy the <Tanium Client>\Patch\scans folder, excluding the CAB file.

Configure endpoint logging

Distribute the Patch - Set Patch Process Options package to your endpoints to change the default logging type and log rotation settings.

  1. Target the systems on which you want to configure logging.
  2. Click Deploy Action.
  3. (Windows) Select the Patch - Set Patch Process Options package.
  4. (Linux) Select the Patch - Set Patch Process Options - Linux package.
  5. Configure the logging type and log rotation settings.
    By default, a new log is created when the log size reaches 1 MB. For example, you might have patch0.log, patch1.log, patch2.log, and so on, up to 10 log files.

Patches are not listed in the Patches view

If you are having difficulty getting patches to appear:

  1. Verify that the Patch - Is Process Running sensor returns Yes for your endpoints.
  2. Check the scheduled actions for Patch.
    1. From the Main menu, go to Administration > Actions > Scheduled Actions.
    2. In the Action Groups section, click Patch.
    3. Review the issue details of the Patch - Ensure Patch Process and Patch - Distribute Deployment # (name) actions.
  3. Check the endpoint log at <Tanium Client>\Patch\patchx.log.
  4. For offline CAB file scan configurations, check that a CAB file is available at <Tanium Client>\Patch\Scans\Wsusscn2.cab.
  5. For WSUS or Microsoft Online scan configurations, check the c:\Windows\WindowsUpdate.log for details.
  6. In the Scan Configuration, change the Random Scan Delay setting.

Troubleshoot scan errors

Investigate endpoints with scan errors that have scan results older than two days and resolve the errors for each endpoint. For more information about how to find and resolve common scan errors, see How to Clear Those Pesky Patch Scan Errors: Troubleshooting Common Errors to Enable Successful Scans.

Scans are not completed on Linux endpoints

Patch 2.3.5 supports Red Hat and CentOS Linux endpoints. Patch 2.4.3 also supports Oracle and Amazon Linux endpoints. If you are having difficulty getting scans to run on Linux endpoints:

  1. On the Patch Home page, click Settings and then click Operating Systems to verify that the RedHat, CentOS, Oracle, Amazon option is enabled.
  2. Verify that the Patch - Is Process Running sensor returns Yes for your Linux endpoints.
  3. Verify Contact your TAM to verify that the Tanium Server can reach the repomd.xml file by appending /repodata/repomd.xml to the configured baseurl value.
  4. Check the endpoint log at /opt/Tanium/TaniumClient/Tools/Patch/logs/scan-process.log for errors.

Sensors return Could not get results on Linux endpoints

If your sensors return Could not get results on Linux endpoints, the Patch tools might not be installed on your Linux endpoints.

  1. On the Patch Home page, click Settings and then click Operating Systems to verify that the RedHat, CentOS, Oracle, Amazon option is enabled.
  2. Verify that the Patch tools are installed on your Linux endpoints.
    If the Patch tools are not installed on your Linux endpoints, the Patch - Tools Version sensor returns:
    Not Installed
    Linux Package Required
  3. To install the Patch tools on your Linux endpoints, Initialize Patch endpoints.

Red Hat Linux endpoints stuck in Waiting for Initial Scan status

If you configure a scan that includes both Red Hat Enterprise Linux 6 Server (RPMs) and Red Hat Enterprise Linux 7 Server (RPMs) repositories, your targeted endpoints might appear to be stuck in the Waiting for Initial Scan status.

  1. Verify that each major operating system is configured in separate scan configurations.
  2. Target Linux endpoints by major operating systems.

Linux repository snapshots issues

If you encounter issues creating or using snapshots, review the following solutions.

Patch Home page warning

The Patch Home page displays a warning message to indicate that a snapshot failed. To modify the snapshot, click Manage Repository Snapshots. To remove the warning, click Dismiss warning.

Connectivity issue

On the Yum Repositories tab of the Patch Settings , the following error message appears: Repo URLs found but unable to snapshot metadata files, likely due to connectivity issue. This message indicates that Patch could not access the source repository for download. To resolve this issue:

  • Ensure that the Tanium Server can access the repository upstream URL.
  • For Red Hat Enterprise Linux repositories, ensure that you have properly configured the Red Hat certificate. For more information, see (Red Hat endpoints) Configuring TDownloader to use certification authentication.
  • If the problem persists, contact your TAM for assistance. Provide logs from the time period during which the problem occurred. For more information, see Collect a troubleshooting package.
  • For Red Hat Enterprise Linux repositories, contact your TAM to ensure that the Red Hat certificate was properly configured.
  • Contact your TAM to ensure that the Tanium Server can access the repository upstream URL.

Patch process is not running

Start the Patch process:

  1. In Interact, ask the Get Operating System from all machines question.
  2. From the grid, select the Linux operating systems.
  3. Click Deploy Action. Choose the Patch - Start Patch Process - Linux package.
  4. Click Preview and then click Deploy Action.

Snapshot controls are disabled

On the Yum Repositories tab of the Patch Settings , Create Snapshots is disabled and the following message appears: Selected repo <repository_name> is missing required yum variables. This issue indicates that Patch could not gather Yum variables for the operating system type of the selected repository. This issue might occur after you import a new version of Patch. To resolve:

  1. Ensure that the Patch process is running on Linux endpoints. For more information, see Patch process is not running.
  2. Re-create the Yum variables by clicking the Interact icon in the message.

Yum variable mismatch

On the Yum Repositories tab of the Patch Settings , the following error message appears: Repo URLs were found but the snapshot could not be created, likely due to a Yum variable mismatch. This issue indicates that Patch could not resolve the listed Yum variables in the repository upstream URL. To resolve:

  1. Ensure that the Patch process is running on Linux endpoints. For more information, see Patch process is not running.
  2. Ensure that custom Yum variables in the URL are accurate.
  3. Ensure that you have defined the custom Yum variables on your Linux endpoints.
  4. Reinitialize Patch, which runs the Patch - Yum variables sensor to gather Yum variables from Linux endpoints. For more information, see Initialize Patch endpoints. Alternatively, wait up to three hours for normal Patch processing to gather the variables.

Check and update the Windows Update Agent

You can use Tanium to check which Windows Update Agent versions are installed on your Windows endpoints.

  1. In Interact, ask the Get File Version["C:\Windows\System32\wuaueng.dll"] from all machines question.
  2. Update any endpoints that have a version earlier than 6.1.0022.4. See the Microsoft article Updating the Windows Update Agent.

Monitor and troubleshoot Patch coverage

The following table lists contributing factors into why the Patch coverage metric might report endpoints as Needs Attention or Unsupported, and corrective actions you can make.

Contributing factor Corrective action
Gaps in Patch action group membership Ensure all operating systems that are supported by Patch are included in the Patch action group.
Gaps in scan configuration coverage

Review each scan configuration and which computer groups are targeted by each configuration.

Ensure that every endpoint that is supported by Patch is targeted by at least one scan configuration.

Scan frequency value is set too high Review each scan configuration to ensure that the Frequency value is set to less than three days for all scan configurations.
Scan windows are too restrictive Scan windows are optional. If you decide to use them, the Override option should be set to less than two days.
Unmitigated scan failures

Investigate endpoints with scan errors in scan results that are older than two days.

Remediate the error conditions on each endpoint. See Troubleshoot scan errors.

Patch process is not running Ensure that there are no conditions that could prevent the Patch process from running on endpoints that are included in the Patch action group.

Monitor and troubleshoot endpoints missing critical or important patches

The following table lists contributing factors into why the endpoints missing critical or important patches metric might be higher than expected, and corrective actions you can make.

Contributing factor Corrective action
Selective patching as a practice Avoid choosing specific patches based on vulnerability reports. Instead, use dynamic, rule-based patch lists. These lists should be cumulative. For example, do not create any rules that prevent patches that are older than a specific date from being included in a patch list.
Patches are deployed only for the current month Use dynamic, rule-based patch lists. These lists should be cumulative. For example, do not create any rules that prevent patches that are older than a specific date from being included in a patch list.
Having an n-1 patching policy

Include patches for the current month.

Expand endpoint diversity in patch testing groups to increase the changes of identifying newly-released problematic patches for deploying patches to production.

Not enforcing post-deployment restarts

Use the Restart option within deployments.

(Windows) Use the Notify User option and set the Deadline for restart value to less than a few days. To decrease both the endpoints missing critical or important patches and the mean time to patch metrics, the optimal value for this setting depends on your patching cycle. Successful customers find that setting the Deadline for restart value to less than three days is optimal.

Monitor and troubleshoot mean time to patch

The following table lists contributing factors into why the mean time to patch metric might be higher than expected, and corrective actions you can make.

Contributing factor Corrective action
Delayed testing cycle Begin the process of testing new monthly patches the day they are released, typically Patch Tuesday (second Tuesday of each month).
Delayed start of production cycle Avoid waiting longer than two weeks after a patch release to start patching production systems. The longer you wait to start patching production systems, the more aggressive the subsequent deployments need to be to complete the patching cycle in a reasonable time.
Staggering deployments to distribute the load on the network

Do not stagger deployments in an attempt to distribute the load on your network or Tanium. The more endpoints that are being patched simultaneously, the more efficient Tanium becomes with overall WAN usage.

For bandwidth-constrained locations, you can implement site throttles. For more information, see Tanium Console User Guide: Configure site throttles.

Staggering deployments to distribute the load on the Tanium Server or Patch Do not stagger deployments in an attempt to distribute the load on your network or Tanium.
Endpoints do not have enough time to install patches

Ensure that deployment windows are at least four hours and properly overlap with maintenance window times.

Ensure that maintenance windows are at least four hours long, repeat at least once each month, and properly overlap with deployment times and change control process timelines.

For deployments that are scheduled for the future, select the Download immediately option. If you find that endpoints are not completing patch installations within the specified windows, schedule the deployments even further in advance.

Attempting to minimize disruption to users with maintenance windows Use Tanium End-User Notifications instead of restrictive maintenance windows.
Not enforcing post-deployment restarts

Use the Restart option within deployments.

(Windows) Use the Notify User option and set the Deadline for restart value to less than a few days. To decrease both the endpoints missing critical or important patches and the mean time to patch metrics, the optimal value for this setting depends on your patching cycle. Successful customers find that setting the Deadline for restart value to less than three days is optimal.

Unmitigated installation failures

Identify endpoints with patch installation failures.

Investigate the specific error codes.

Remediate the conditions that caused the failures.

Endpoint health issues Ensure that endpoints have at least 5 GB of free space, a healthy Windows Update Agent, and an up-time of less than 90 days.

Uninstall Patch

If you need to uninstall Patch, first clean up the Patch artifacts on the endpoint and then uninstall Patch from the server.

  1. Clean up patch artifacts from the endpoints.
    1. Use Interact to target endpoints. To get a list of endpoints that have Patch, you can ask the Patch - Is Process Running question.
    2. Click Deploy Action. Choose the Patch - Clean Up Patch 2 Processes and Files package.
    3. Check the status of the action on the Actions > Action History page.
  2. Remove the Patch solution from the Tanium Module Server. From the Main menu, go to Administration > Configuration > Solutions.
    1. In the Patch section, click Uninstall and follow the process.
    2. Click Proceed with Uninstall.
    3. The uninstaller disables any actions.
    4. Return to the Tanium Solutions page and verify that the Import button is available for Patch.
      If the Patch module has not updated in the console, refresh your browser.

Restore the state of the Patch database

You can import the patch.db file to restore the Patch configuration.

  1. Stop the Patch service on the Tanium Module Server.
  2. Copy your patch.db file into the <Module Server>\services\patch-service\ directory, replacing the existing file.
  3. Restart the Patch service.
  4. In the Tanium Console, refresh the Patch workbench.
  5. Reset the service credentials. Click Set your service account and enter your user name and password.
  6. Any existing data, including patch lists, deployments, and associated patches and actions are displayed in the Patch workbench.

    If a deployment scheduled action is missing, you might need to wait up to 5 minutes for it to show up.