Troubleshooting Trace

If Trace is not performing as expected, you might need to perform troubleshooting or create workarounds. For assistance, contact your Tanium technical account manager (TAM).

Identify Linux endpoints missing auditd

If Linux endpoint events are not being recorded, they might be missing the audit daemon and audispd. Ideally, the audit daemon is installed and configured before installing the Trace module, but it is possible for endpoints to come online at a later time.

  1. (Optional) Create the auditd package.

    You can either create a general installation package and put the logic in the scripts or you can have a simple script and put the logic in the Tanium query. See Tanium Core Platform User Guide: Creating and managing packages.

    Create saved actions that periodically check for and deploy this package in the future.

  2. Ask the question: Get Installed Application Exists[audit] from all machines with Is Linux containing "true".
  3. Use your preferred method to deploy the appropriate auditd package to the identified endpoints.

    If you need to distribute the package to a large number of endpoints, spread the changes out over time to avoid a negative impact on the network.

Change the endpoint database configuration priority

You can create multiple configurations and assign them to computer groups. The order of the endpoint database configuration decides its priority. If an endpoint is a member of multiple computer groups with conflicting endpoint database configurations, the first configuration on the list is applied to the endpoint.

The default configuration is always applied and does not apply to the prioritization of other groups. Add filters that apply to other configurations to the default configuration.

When there are three or more endpoint database configuration groups, you can move the groups up or down as needed.

  1. Go to the Trace home page.
  2. Click Settings .
  3. Click the Groups tab.
  4. Use the arrows to reorder the configurations.

  5. Click Publish All.

When changes are saved but have not been deployed, a Changes pending message appears. If you need to change the settings, see Configure endpoint database settings for more information.

Resolve live endpoint connection problems

If you are having difficulty making a live connection to an endpoint, diagnose the issue with this workflow.

  1. In Interact, ask the Get Tanium Trace Status saved question and verify that the endpoint status is No Issues found.

    If the status is Install Needed, run the Distribute Tanium Trace Tools action.

  2. Ask the Get Trace Endpoint Certificate Installed question and verify that the endpoint status is True.

    If it is not, then run the Install Tanium Trace Certificate action.

  3. In Trace, click Settings , then click the General tab. Verify that the Module Server IP address is correct under the Update Service Settings heading.
  4. Verify that a firewall is not blocking the connection from the endpoints to the Module Server:
    1. From a remote computer, browse to https://<module server IP address or FQDN>:17444/status.
    2. If you get a message that the site cannot be reached, update firewall rules.
  5. Verify that you are using the correct endpoint identification to initiate the connection.  

    If initiating a connection from Trace, you must type in the IP address or Computer Name, usually FQDN, that is recognized by Tanium or the connection fails. Alternatively, you can:

    1. Use Interact to ask the Get Computer Name and IP Address from all machines question.
    2. From the results grid, select an endpoint and deploy the Start Tanium Trace Session package.
  6. Review the Start Tanium Trace Session action to verify that it has completed.
  7. Access the endpoint and review the Trace Websocket Client file.
    Operating systemFile path



    If needed, increase the log level to debug:

    1. Create an .ini file with the same name and location as the websocket client location.

      For example, <tanium_client_directory>\Tools\Trace\TraceWebsocketClient.ini.

    2. Add the line: logging.loggers.root.level=debug.
    3. Try to connect again and review the results in the log file.
  8. If you cannot resolve the problem, contact your TAM with the following information:
    • Trace service logs
    • Action logs
    • Trace Zone Proxy and Hub logs, if available
    • Trace Websocket Client log

For configurations with the Trace Zone proxy service, see Troubleshoot the Trace zone proxy service for more information.

Update the Trace service URL

The Trace service URL is used to upload snapshots to Trace. It is unlikely that you would need to change this URL; however you might need to update it if the host name or IP address of the Tanium Module server changes.

  1. Go to the Trace home page.
  2. Click Settings , then click the General tab.
  3. Update the Trace Service URL under the Update Service Settings heading.
  4. Click Save.

Change the Module Server address

When Trace is installed, the installation process fills in the Module Server IP address that the endpoints use to connect. If this address changes, you might need to update the Service Settings.

  1. Go to the Trace home page.
  2. Click Settings , then click the General tab.
  3. Enter the IP address of the Module Server under the Update Service Settings heading.
  4. Click Save.

Update the endpoint certificate

Certificates authorize live connections between Trace and endpoints. A warning message displays on the workbench as certificates near expiration. You can expand the message for details about the specific certificate. Errors display in the same area in a red box. See the following image for an example of a warning message.

  1. Go to the Trace home page.
  2. Click Settings , then click the General tab.
  3. Scroll to the Endpoint Certificate Settings heading.
  4. Select an option:
    • Click Generate self-signed certificate.
    • Upload a certificate and private key.

      The certificate must be in PEM format. In the certificate signing request, enable both web server and web client authentication.

  5. Click Install.

Change the live connection export limit

By default, exports are limited to 10,000 rows. You can adjust the limit if needed, as a lower limit reduces the impact on endpoints.

  1. Go to the Trace home page.
  2. Click Settings , then click the General tab.
  3. Under the Update Service Settings heading, edit the maximum number of rows.
  4. Click Save.

Start or stop the Trace event recorder

You might need to manually start or stop the event recorder. For example, if the database size exceeded the limit or if the CPU usage exceeded the threshold, which automatically disabled the event recorder, you must resolve the underlying issue first and then manually restart the event recorder. Or, if you find that the event recorder is using more system resources than expected, you can stop the recorder and troubleshoot the issue without risk of additional resource consumption.

After stopping, you must manually restart the event recorder, as it does not restart automatically.

  1. Use a question to target the affected endpoints.

    For example, ask Get Tanium Trace Status from all machines.

  2. Drill down to the specific endpoints.
  3. Deploy the Disable Tanium Recorder [OS] or Enable Tanium Recorder [OS] package as an action.

For more information, see the Tanium Core Platform User Guide: Managing and creating Packages or the Tanium Interact User Guide: Using Deploy Action.

Recreate the endpoint database

If the database is identified as corrupt, you might need to clear the endpoint database.

  1. Use a question to target the affected endpoints.

    For example, ask Get Trace Invalid File Operations from all machines.

  2. Drill down to the endpoints that return true.
  3. Deploy the Trace - Recreate Database [OS] package as an action.

For more information, see the Tanium Core Platform User Guide: Managing and creating Packages or the Tanium Interact User Guide: Using Deploy Action.

Collect a troubleshooting package

Use Trace to compile a collection of logs relevant for troubleshooting.

  1. On the Trace home page, click Information then click the Info tab.
  2. Under Troubleshooting, click Collect.
  3. Under Download Package, click the package file link.

The log zip file might take a few moments to appear in the download folder.

Collect troubleshooting information from endpoints

You can collect logs and other artifacts from individual endpoints to help resolve issues. This information is useful to debug and configure signals with Tanium Detect. Collecting logs from endpoints requires a live connection to the endpoint from which you want to gather troubleshooting information.

  1. From the Trace menu, click Live Endpoints.
  2. Create a live connection to the endpoint from which you want to collect troubleshooting information. For information on creating a live connection, see Connecting to live endpoints and taking snapshots.
  3. Select the endpoint from which you want to collect troubleshooting information.
  4. Click Troubleshoot.
  5. You are prompted to confirm whether or not you want to collect troubleshooting information. If you select Yes, a package is gathered from the endpoint and a link to the download package appears below the Troubleshooting button. The name of the link is the time stamp of the troubleshooting package. Click the link to download a ZIP file that contains troubleshooting information.

On Windows endpoints, a download package can contain:

  • Tanium Client\Trace\trace_*.txt
  • Tanium Client\monitor.db
  • Tanium Client\filters.json
  • Tanium Client\watchlist.json
  • Tanium Client\Tools\Detect3\detect.db
  • Tanium Client\Tools\Detect3\intel.db
  • Tanium Client\Tools\Detect3\version
  • Tanium Client\Tools\Detect3\logs\detect.log
  • Tanium Client\Tools\Detect3\logs\detect.log.*
  • Tanium Client\Tools\Trace\TraceWebSocketClient.log
  • Tanium Client\Tools\Trace\TraceWebSocketClient.log.*

On Linux and Mac endpoints a download package can contain:

  • TaniumClient/Tools/Trace/recorder.json
  • TaniumClient/Tools/Trace/recorder.log
  • TaniumClient/Tools/Trace/recorder.log.*
  • TaniumClient/Tools/Trace/filters.json
  • TaniumClient/Tools/Trace/watchlist.json
  • TaniumClient/Tools/Trace/monitor.db
  • TaniumClient/Tools/Trace/TraceWebSocketClient.log
  • TaniumClient/Tools/Trace/TraceWebSocketClient.log.*
  • TaniumClient/Tools/Detect3/detect.db
  • TaniumClient/Tools/Detect3/intel.db
  • TaniumClient/Tools/Detect3/logs/detect.log
  • TaniumClient/Tools/Detect3/logs/detect.log.*
  • /etc/audit/audit.rules
  • /etc/audit/auditd.conf
  • /etc/audisp/audispd.conf
  • /etc/audisp/plugins.d/trace.conf

You can use Tanium™ Incident Response Copy Tools to retrieve additional files from endpoints. See the Tanium Incident Response User Guide: Copying IR data to a central location for more information.

Manually back up and restore Trace configurations

You can create a backup or restore a backup to an existing Trace configuration. For example, you might use a backup and restore when you are moving from a testing to a production environment.

  1. On the Tanium Module Server, stop the Trace service.
  2. On the source host computer, go to Program Files\Tanium\Tanium Module Server\services\ and copy the trace-files directory.
  3. On the target host computer, paste the directory to another Trace installation to restore the state.
  4. Restart the Trace service.

For a Linux host machine, replace the backslash (\) with a forward slash (/).

Remove Trace tools

You can deploy a preconfigured package to remove Trace tools from an endpoint or computer group.

  1. Remove the target endpoint or computer group from the Trace Setup Action Group.
    1. From the Tanium Console, go to Actions > Scheduled Actions.
    2. In the Action Groups pane, click Trace Setup.
    3. Click Edit.

      The Trace Setup Action Group page opens.

    4. In the Computer Groups section, deselect groups as needed.
    5. (Optional) In the All machines currently included in this action group section, review the included machines.

      This grid might take a few moments to populate when you change your selections.

    6. Click Save.
  2. Using Tanium Interact™ or a saved question, run the Tanium Trace Status sensor.
  3. From the endpoints that have Trace tools installed, drill down and select the targets.
  4. Deploy the Remove Tanium Trace Tools or Remove Tanium Trace Tools [Mac-Linux] packages to the targeted endpoints.

For more information, see Tanium Interact User Guide: Questions and Tanium Interact User Guide: Using Deploy Action.


Uninstall Trace

You might need to remove Trace from the Tanium Module Server for troubleshooting purposes.

  1. From the Tanium Console, click Solutions.

    The Solutions page opens.

  2. Locate Trace, and then click Uninstall.

    The Uninstall window opens, showing the list of contents to be removed.

  3. Click Proceed with Uninstall.
  4. Enter your password to start the uninstall process.

    A progress bar is displayed as the installation package is removed.

  5. Click Close.
  6. To confirm, return to the Solutions page and check that the Import button is available.

    If the Trace module has not updated in the console, refresh your browser.

Last updated: 12/3/2019 2:56 PM | Feedback