Troubleshooting

If Threat Response is not performing as expected, you might need to troubleshoot issues or change settings. For assistance, you can also contact your TAM.

Collect logs

Use Threat Response to compile a collection of logs relevant for troubleshooting.

  1. From the Threat Response home page, click Help , then click the Troubleshooting tab.
  2. Click Create Package.
  3. When the status of the package creation is complete, click Download Package.

The log zip file might take a few moments to appear in the download folder.

Collect troubleshooting information from endpoints

You can collect logs and other artifacts from individual endpoints to help resolve issues. Collecting logs from endpoints requires a live connection to the endpoint from which you want to gather troubleshooting information.

  1. From the Threat Response menu, click Live Endpoints.
  2. Create a live connection to the endpoint from which you want to collect troubleshooting information. For information on creating a live connection, see Connecting to live endpoints.
  3. Select the endpoint from which you want to collect troubleshooting information.
  4. Click Troubleshoot.
  5. Confirm that you want to collect troubleshooting information. A package is gathered from the endpoint and a link to the download package is displayed. The name of the link is the time stamp of the troubleshooting package. Click the link to download a ZIP file that contains troubleshooting information.

View notifications

From the Main menu, click Management > System Notifications. These notifications show non-match alerts, including when alert throttling is enabled on any endpoints. To delete a notification, select the row and click Delete.

Get Threat Response tools status

Ask the question: Get Threat Response Status from all machines.

The results returned are divided by operating system, component versions, and the endpoints that do not have the tools installed. The Threat Response tools status is evaluated when the Threat Response service restarts or is updated.

For more information, see Tanium Interact User Guide: Asking questions.

Tune alert throttling

Adjust the throttling settings to control how many alerts you are getting on the endpoint. Both endpoint and service alert throttling are enabled by default.

Configure throttling for signal alerts on endpoints

You can configure throttling of signal alerts on the endpoint when you create an engine configuration. By default, signal alert throttling on an endpoint is enabled and occurs when five events on a single piece of intel occur within five minutes.

  1. Go to Management > Engine. Select a configuration.
  2. In the Advanced Engine Settings section, update the settings for signal throttling.
  3. Save the engine configuration, and deploy any profiles that use that configuration for the endpoint throttling to be deployed.

  4. If signal alert throttling occurs, notifications about the event and the endpoint that has throttling enabled display on the System Notifications page.

Configure match alert throttling

You can configure match alert throttling for the engine. This service-level throttling can apply for quick scan alerts and all types of intel. By default, match alert throttling is enabled and occurs when 100 events on a single piece of intel occur within 20 minutes.

  1. From the Threat Response home page, click Settings then click the Service tab. Click Intel.
  2. Edit the settings for match throttling. You can adjust when throttling occurs, and adjust the cooloff period, which controls how long alerts continue to be throttled.
  3. Click Save.
  4. If alert throttling occurs, you get notifications that throttling is enabled display on the System Notifications page.

Configure the engine manually

You can change most engine settings in the advanced settings of a group configuration. You might need to manually run scripts from the Tanium Client directory on the endpoint to edit the engine configuration.

  • get-config provides all of the available options and their current values.
  • set-config changes the value or clears it, returning to the default behavior.

You can also find more information about engine configuration options in the engine documentation.

Identify Linux endpoints that are missing auditd

If Linux endpoint events are not being recorded, they might be missing the audit daemon and audispd services. Ideally, the audit daemon is installed and configured before installing the Threat Response module, but it is possible for endpoints to come online at a later time.

  1. (Optional) Create the auditd package.

    You can either create a general installation package and put the logic in the scripts or you can have a simple script and put the logic in the Tanium query. See Tanium Core Platform User Guide: Creating and managing packages.

  2. Ask the question: Get Installed Application Exists[audit] from all machines with Is Linux containing "true".
  3. Deploy the appropriate auditd package to the identified endpoints.

    If you need to distribute the package to a large number of endpoints, spread the changes out over time to avoid a negative impact on the network.

Resolve live endpoint connection problems

If you are having difficulty making a live connection to an endpoint, diagnose the issue with this workflow.

A configuration that has the recorder enabled is required to make a live connection.

  1. Ask the Get Threat Response - Status saved question and verify that the endpoint status is No Issues found.

    If the status is Install Needed, run the Distribute Tanium Trace Tools action.

  2. Ask the Get Trace Endpoint Certificate Installed saved question and verify that the endpoint status is True.

    Run the Trace - Install Endpoint Certificate action from the Tanium Threat Response action group action if necessary.

  3. Click Settings , then click the Service tab. Select Misc. Verify that the Module Server IP address is correct.
  4. Verify that a firewall is not blocking the connection from the endpoints to the Module Server:
    1. From a remote computer, browse to https://<module server IP address or FQDN>:17444/status.
    2. If you get a message that the site cannot be reached, update firewall rules.
  5. Verify that you are using the correct endpoint identification to initiate the connection.  

    If initiating a connection from Threat Response, you must type in the IP address or Computer Name, usually FQDN, that is recognized by Tanium or the connection fails. Alternatively, you can:

    1. Use Interact to ask the Get Computer Name and IP Address from all machines question.
    2. From the results grid, select an endpoint and deploy the Start Tanium Trace Session package.
  6. Review the Start Tanium Trace Session action to verify that it has completed.
  7. Access the endpoint and review the Trace Websocket Client file.
    Operating systemFile path

    Windows

    <tanium_client_directory>\Tools\Trace\TraceWebSocketClient.log
    Linux<tanium_client_directory>/Tools/Trace/TraceWebsocketClient.log
    Mac<tanium_client_directory>/Tools/Trace/TraceWebsocketClient.log

    If needed, increase the log level to debug:

    1. Create an .ini file with the same name and location as the websocket client location.

      For example, <tanium_client_directory>\Tools\Trace\TraceWebsocketClient.ini.

    2. Add the line: logging.loggers.root.level=debug.
    3. Try to connect again and review the results in the log file.
  8. If you cannot resolve the problem, contact your TAM with the following information:
    • Trace service logs
    • Action logs
    • Trace Zone Proxy and Hub logs, if available
    • Trace Websocket Client log

For configurations with the Trace Zone proxy service, see Troubleshoot the Trace zone proxy service for more information.

Change the Module Server address

When Threat Response is installed, the installation process fills in the Module Server IP address that the endpoints use to connect. If this address changes, you might need to update the Service Settings.

  1. From the Threat Response home page, click Settings , then click the Service tab. Click Misc.
  2. Enter the IP address of the Module Server.
  3. Click Save.

Update the endpoint certificate

Certificates authorize live connections between Threat Response and endpoints. The certificate must be in PEM format. In the certificate signing request, enable both web server and web client authentication.

As certificates near expiration, a warning message is displayed.

  1. From the Threat Response home page, click Settings , then click the Certificate tab.
  2. Select an option:
    • Click Generate self-signed certificate.
    • Upload a certificate and private key.
  3. Click Install.

Start or stop the recorder

You might need to manually start or stop the recorder. The recorder does not restart automatically.

For example, if the database size or CPU usage limits are exceeded, the recorder is automatically stopped. Resolve the underlying issue and restart the recorder. Or, if you find that the recorder is using more system resources than expected, you can stop the recorder and troubleshoot the issue.

  1. Use a question to target the affected endpoints. For example, ask Get Tanium Threat Response Status from all machines.
  2. Drill down to the specific endpoints.
  3. Deploy the Disable Tanium Recorder [Operating System] package as an action to disable the recorder.
  4. Deploy the Enable Tanium Recorder [Operating System]package as an action to enable the recorder.

For more information, see the Tanium Core Platform User Guide: Managing and creating Packages or the Tanium Interact User Guide: Using Deploy Action.

Recreate the endpoint database

If the database is identified as corrupt, you might need to clear the endpoint database.

  1. Use a question to target the affected endpoints.

    For example, ask Get Trace Invalid File Operations from all machines.

  2. Drill down to the endpoints that return true.
  3. Deploy the Trace - Recreate Database [OS] or Threat Response - Recreate Database [OS] package as an action.

For more information, see the Tanium Core Platform User Guide: Managing and creating Packages or the Tanium Interact User Guide: Using Deploy Action.

Uninstall Threat Response

You might need to remove Trace from the Tanium Module Server for troubleshooting purposes.

  1. From the Tanium Console, click Solutions.
  2. Locate Threat Response, and then click Uninstall.
  3. To confirm, return to the Solutions page and check that the Import button is available.

Last updated: 4/16/2019 3:14 PM | Feedback