Troubleshooting

If Threat Response is not performing as expected, you might need to troubleshoot issues or change settings. For more information, see Contact Tanium Support.

Resolve issues with Threat Response components

The Health Status page displays issues that have occurred in a Threat Response environment, and provides actions that you can take to remediate. Any encountered issues display in the order of their significance to help prioritize remediation actions. The results displayed on the Health Status page refresh every 12 minutes (720 seconds).

  1. From the Threat Response menu, click Management > Health Status.
  2. If multiple issues have been encountered, start a the top of the list and expand the issue you want to remediate.
  3. The Threat Response component where the issue has occurred appears along with a description of the issue, the number and types of endpoints on which it occurs, and a remediation action.
  4. Click the remediation action button in the Actions column.

Collect logs

Use Threat Response to compile a collection of logs relevant for troubleshooting.

  1. From the Threat Response overview page, click Help , then click the Troubleshooting tab.
  2. Click Create Package.
  3. When the status of the package creation is complete, click Download Package.

The log zip file might take a few moments to appear in the download folder.

Collect troubleshooting information from endpoints

You can collect logs and other artifacts from individual endpoints to help resolve issues. Collecting logs from endpoints requires a live connection to the endpoint from which you want to gather troubleshooting information.

  1. From the Main menu, click Administration > Client Management to open the Client Management Overview page.
  2. From the Client Management menu, click Client Health.
  3. In the Direct Connect section, create a live connection to the endpoint from which you want to collect troubleshooting information. For information on creating a live connection, see Connecting to live endpoints.
  4. Click the Gather tab. In the domain section select Threat Response.
  5. Click Gather from Endpoint.
  6. A package is gathered from the endpoint. The name of the link is the time stamp of the troubleshooting package. Select the Must Gather that you want to download and click Download to download a ZIP file that contains troubleshooting information.

View notifications

From the Threat Response menu, click Management > System Notifications. These notifications show non-match alerts, including when alert throttling is enabled on any endpoints. To delete a notification, select the row and click Delete.

View task status

You can view the status of and review other historic data for Threat Response tasks.

From the Threat Response menu, click Management > Tasks.

For each task that you have initiated, you can view the status, the start, stop, and creation time, and the task ID. The types of Threat Response tasks for which you can view this detailed information include:

  • Deploy Profile
  • Remove Profile
  • Deploy Intel
  • Deploy Tools
  • Start Index
  • Group Migration
  • Content Migration
  • Suppress Alerts
  • Response Action

In the event of an error, you can locate a specific task and expand the cell to view results and metadata from log files. The data that appears in the advanced details view provides information that is useful for troubleshooting and saves the time of trying to locate debugging information in lengthy log files or in formats that are not intended to be human readable.

Get Threat Response tools status

Ask the question: Get Threat Response - Status from all machines.

The results returned are divided by operating system, component versions, and the endpoints that do not have the tools installed. The Threat Response tools status is evaluated when the Threat Response service restarts or is updated.

For more information, see Tanium Interact User Guide: Asking questions.

Tune alert throttling

Adjust the throttling settings to control how many alerts you are getting on the endpoint. Both endpoint and service alert throttling are enabled by default.

Configure throttling for Signal alerts on endpoints

You can configure throttling of Signal alerts on the endpoint when you create an engine configuration. By default, Signal alert throttling on an endpoint is enabled and occurs when five events on a single piece of intel occur within five minutes.

  1. Go to Management > Configurations. Click the Engine tab. Edit a configuration.

  2. In the Advanced Engine Settings section, update the settings for Signal throttle rate.
  3. Save the engine configuration, and deploy any profiles that use that configuration for the endpoint throttling to be deployed.

  4. If Signal alert throttling occurs, notifications about the event and the endpoint that has throttling enabled display on the System Notifications page.

Configure match alert throttling

You can configure match alert throttling for the engine. This service-level throttling can apply for quick scan alerts and all types of intel. By default, match alert throttling is enabled and occurs when 100 events on a single piece of intel occur within 20 minutes.

  1. From the Threat Response overview page, click Settings then click the Service tab. Click Intel.
  2. Edit the settings for match alert throttling. You can adjust when throttling occurs, and adjust the cooloff period, which controls how long alerts continue to be throttled.
  3. Click Save.
  4. If alert throttling occurs, you get notifications that throttling is enabled display on the System Notifications page.

Configure the engine manually

You can change most engine settings in the advanced settings of an engine configuration. You might need to manually run scripts from the Tanium Client directory on the endpoint to edit the engine configuration.

  • get-config provides all of the available options and their current values.
  • set-config changes the value or clears it, returning to the default behavior.

You can also find more information about engine configuration options in the engine documentation.

File, network, or security events are not displayed on Oracle Linux server

If you are not seeing file, network, or security events in the recorder results, you can disable SELinux. When SELinux is enabled and the auditd fallback is disabled on Oracle Linux, only process information is returned. Alternatively, ensure that the Client Recorder Extension configuration parameters are set as follows:

  • CX.recorder.AuditdStopAuditdService is set to 0.
  • CX.recorder.AuditdEnableAudispdFallback is set to 1.

For more information, see Client Recorder Extension User Guide: Configuring recorded events .

Identify Linux endpoints that are missing auditd

If Linux endpoint events are not being recorded, they might be missing the audit daemon and audispd services. Ideally, the audit daemon is installed and configured before installing the Threat Response module, but it is possible for endpoints to come online at a later time.

  1. (Optional) Create the auditd package.

    You can either create a general installation package and put the logic in the scripts or you can have a simple script and put the logic in the Tanium query. See Tanium Core Platform User Guide: Creating and managing packages.

  2. Ask the question: Get Installed Application Exists[audit] from all machines with Is Linux containing "true".
  3. Deploy the appropriate auditd package to the identified endpoints.

    If you need to distribute the package to a large number of endpoints, spread the changes out over time to avoid a negative impact on the network.

Resolve issues with legacy Client Recorder Extension installations

If Tanium Endpoint Configuration detects endpoints that have legacy versions of the Client Recorder Extension installed, it reports the endpoint as Unsupported in the recorder column of the results grid when you ask the question: Get Endpoint Configuration - Tools Status. If Client Recorder Extension version 1.x exists on a targeted endpoint, you must remove it before you install Client Recorder Extension version 2.x tools. To target endpoints where Client Recorder Extension version 1.x exists, ask the question: Legacy - Recorder Installed. If the Supported Endpoints column displays No, you must remove Client Recorder Extension version 1.x from the endpoint before you install Client Recorder Extension 2.x tools. To remove Client Recorder Extension version 1.x, deploy the Recorder - Remove Legacy Recorder [Operating System] package to targeted endpoints.

Change the Module Server address

When Threat Response is installed, the installation process fills in the Module Server IP address that the endpoints use to connect. If this address changes, you might need to update the Service Settings.

  1. From the Threat Response overview page, click Settings , then click the Service tab. Click Misc.
  2. Enter the IP address of the Module Server.
  3. Click Save.

Start or stop the recorder

You might need to manually start or stop the recorder.

Resolve the underlying issue and restart the recorder. Or, if you find that the recorder is using more system resources than expected, you can stop the recorder and troubleshoot the issue.

  1. Identify the computer groups that contain endpoints on which you want to stop the recorder.
  2. Deploy a profile that does not contain a recorder configuration to the computer groups you want to target.
  3. You can optionally create a live endpoint connection to specific endpoints to troubleshoot any issues.

To start the recorder, deploy a profile that has a recorder configuration to the targeted computer groups.

Monitor and troubleshoot Threat Response coverage

The following table lists contributing factors into why the Threat Response coverage metric might be lower than expected, and corrective actions you can make.

Contributing factor Corrective action
Tools not deployed

Ensure the Threat Response Action Group is set to All Computers.

Ensure the Trends Action Group is set to All Computers.

Ensure that all endpoints belong to a computer group that is defined in a Threat Response profile.

Tools in an unhealthy state

From the Threat Response overview page, click Management > Health Status and use the workbench to help take corrective actions to ensure Threat Response is healthy in your environment.

Monitor and troubleshoot mean time to investigate threats

The following table lists contributing factors into why the mean time to investigate threats metric might be higher than expected, and corrective actions you can make.

Contributing factor Corrective action
Alert status not being properly updated in the Threat Response workbench

Ensure analysts are accurately updating alerts from New to In Progress and then from In Progress to Resolved.

If alerts are sent to a SIEM or another location, ensure that your workflow uses the Tanium Threat Response API to appropriately set alert state in Tanium to be able to track this metric correctly.

Monitor and troubleshoot mean time to remediate threats

The following table lists contributing factors into why the mean time to investigate threats metric might be higher than expected, and corrective actions you can make.

Contributing factor Corrective action
Alert status not being properly updated in the Threat Response workbench

Ensure analysts are accurately updating alerts from New to In Progress and then from In Progress to Resolved.

If alerts are sent to a SIEM or another location, ensure that your workflow uses the Tanium Threat Response API to appropriately set alert state in Tanium to be able to track this metric correctly.

Remove Threat Response tools from endpoints

You can deploy an action to remove Threat Response tools from an endpoint or computer group. Separate actions are available for Windows and non-Windows endpoints.

  1. In Interact, target the computers from which you want to remove the tools. For example, ask a question that targets a specific operating system:
    Get Endpoint Configuration - Tools Status from all machines with Is <OS> equals True , for example: 
    Get Endpoint Configuration - Tools Status from all machines with Is Windows equals True
  2. In the results, select the row for Threat Response, drill down as necessary, and select the targets from which you want to remove Threat Response tools. For more information, see Tanium Interact User Guide: Drill Down.
  3. Click Deploy Action.
  4. On the Deploy Action page, enter Endpoint Configuration - Uninstall in the Enter package name here box, and select Endpoint Configuration - Uninstall Tool [Windows] or Endpoint Configuration - Uninstall Tool [Non-Windows], depending on the endpoints you are targeting.
  5. For Tool Name, select Threat Response.

  6. (Optional) By default, after the tools are removed they cannot be reinstalled. To allow tools to be automatically reinstalled, clear the selection for Block reinstallation. Re-installation occurs almost immediately.

    If reinstallation is blocked, you must unblock it manually:

    • To allow Threat Response to reinstall tools, deploy the Endpoint Configuration - Unblock Tool [Windows] or Endpoint Configuration - Unblock Tool [Non-Windows] package (depending on the targeted endpoints).

    • If you reinstall tools manually, select Unblock Tool when you deploy the Endpoint Configuration - Reinstall Tool [Windows] or Endpoint Configuration - Reinstall Tool [Non-Windows] package.

  7. (Optional) To remove all Threat Response databases and logs from the endpoints, clear the selection for Soft uninstall.

  8. (Optional) To also remove any tools that were dependencies of the Threat Response tools that are not dependencies for tools from other solutions, select Remove unreferenced dependencies.

  9. Click Show preview to continue.
  10. A results grid displays at the bottom of the page showing you the targeted endpoints for your action. If you are satisfied with the results, click Deploy Action.

If you have enabled Endpoint Configuration, tool removal must be approved in Endpoint Configuration before tools are removed from endpoints.

Test Live Response connections manually

To test Live Response connections manually, see Generate known_hosts and test connections.

Uninstall Threat Response

You might need to remove Threat Response from the Tanium Module Server for troubleshooting purposes.

  1. From the Main menu, go to Administration > Configuration > Solutions.
  2. Under Threat Response, click Uninstall. Click Proceed with Uninstall to complete the process.

Contact Tanium Support

To contact Tanium Support for help, sign in to https://support.tanium.com.