Troubleshooting Trace

If Trace is not performing as expected, you might need to do some troubleshooting or create workarounds. For assistance, you can also contact your TAM.

Identify Linux endpoints missing auditd

If Linux endpoint events are not being recorded, they might be missing the audit daemon and audispd. Ideally, the audit daemon is installed and configured before installing the Trace module, but it is possible for endpoints to come online at a later time.

Trace appends its own rules to the end of the existing audit rules. If the existing audit system is in an immutable mode using the -e 2 flag, Trace on Linux is not supported.

  1. (Optional) Create the auditd package.

    You can either create a general install package and put the logic in the scripts or you can have a simple script and put the logic in the Tanium query. See Tanium Core Platform User Guide: Creating and managing packages.

    Create saved actions that periodically check for and deploy this package in the future.

  2. Ask the question: Get Installed Application Exists[audit] from all machines with Is Linux containing "true".
  3. Use your preferred method to deploy the appropriate auditd package to the identified endpoints.

    If you need to distribute the package to a large number of endpoints, spread the changes out over time to avoid a negative impact on the network.

Change the endpoint database configuration priority

You can create multiple configurations and assign them to computer groups. The order of the endpoint database configuration decides its priority. If an endpoint is a member of multiple computer groups with conflicting endpoint database configurations, the first configuration on the list applied to the endpoint. When there are three or more endpoint database configuration groups, you can move the groups up or down as needed.

  1. Go to Trace home page.
  2. In the right toolbar, click filter .
  3. Use the arrows to reorder the configurations.

  4. Click Deploy.

When changes are saved but have not been deployed, a Changes pending message appears. If you need to change the settings, see Configure endpoint database settings for more information.

Resolve live endpoint connection problems

If you are having difficulty making a live connection to an endpoint, diagnose the issue with this workflow.

  1. In Interact, ask the Get Tanium Trace Status saved question and verify that the endpoint status is No Issues found.

    If the status is Install Needed, run the Distribute Tanium Trace Tools action.

  2. Ask the Get Trace Endpoint Certificate Installed question and verify that the endpoint status is True.

    If it is not, then run the Install Tanium Trace Certificate action.

  3. In Trace, on the right toolbar, click Settings > Service Settings and verify that the Module Server IP address is correct.
  4. Verify that a firewall is not blocking the connection from the endpoints to the Module Server:
    1. From a remote computer, browse to https://<module server IP address or FQDN>:17444/status.
    2. If you get a message that the site cannot be reached, update firewall rules.
  5. Verify that you are using the correct endpoint identification to initiate the connection.  

    If initiating a connection from Trace, you must type in the IP address or Computer Name, usually FQDN, that is recognized by Tanium or the connection fails. Alternatively, you can:

    1. Use Interact to ask the Get Computer Name and IP Address from all machines question.
    2. From the results grid, select an endpoint and deploy the Start Tanium Trace Session package.
  6. Review the Start Tanium Trace Session action to verify that it has completed.
  7. Access the endpoint and review the Trace Websocket Client file.
    Operating systemFile path



    If needed, increase the log level to debug:

    1. Create an .ini file with the same name and location as the websocket client location.

      For example, <tanium_client_directory>\Tools\Trace\TraceWebsocketClient.ini.

    2. Add the line: logging.loggers.root.level=debug.
    3. Try to connect again and review the results in the log file.
  8. If you cannot resolve the problem, contact your TAM with the following information:
    • Trace service logs
    • Action logs
    • Trace Zone Proxy and Hub logs, if available
    • Trace Websocket Client log

For configurations with the Trace Zone proxy service, see Troubleshoot the Trace zone proxy service for more information.

Update the Trace service URL

The Trace service URL is used to upload snapshots to Trace. It is unlikely that you would need to change this URL; however you might need to update it if the host name or IP address of the Tanium Module server changes.

  1. Go to Trace home page.
  2. In the right toolbar, click settings .
  3. Click Service Settings .
  4. Update the URL.
  5. Click Save.

Change the Module Server address

When Trace is installed, the installation process fills in the Module Server IP address that the endpoints use to connect. If this address changes, you might need to update the Service Settings.

  1. Go to Trace home page.
  2. In the right toolbar, click settings .
  3. Click Service Settings .
  4. Enter the IP address of the Module Server.
  5. Click Save.

Update the endpoint certificate

Certificates authorize live connections between Trace and endpoints. As certificates near expiration, Trace warns you with a message on the workbench. You can expand the message for details about the specific certificate. Errors are also displayed in the same area in a red box. See the following image for an example.

  1. Go to Trace home page.
  2. In the right toolbar, click settings .
  3. Click Endpoint Certificate Settings .
  4. Select an option:
    • Click Generate self-signed certificate.
    • Upload your own certificate and private key.

      The certificate must be in PEM format. In the certificate signing request, enable both web server and web client authentication.

  5. Click Install.

Change the live connection export limit

By default, exports are limited to 10,000 rows. You can adjust the limit if needed, as a lower limit reduces the impact on endpoints.

  1. Go to Trace home page.
  2. In the right toolbar, click settings .
  3. Click Service Settings .
  4. Change the maximum number of rows.
  5. Click Save.

Start or stop the Trace event recorder

You might need to manually start or stop the event recorder. For example, if the database size exceeded the limit or if the CPU usage exceeded the threshold, which automatically disabled the event recorder, you must resolve the underlying issue first and then manually restart the event recorder. Or, if you find that the event recorder is using more system resources than expected, you can stop the recorder and troubleshoot the issue with risk of additional resource consumption.

After stopping, you must manually restart the event recorder, as it does not restart automatically.

  1. Use a question to target the affected endpoints.

    For example, ask Get Tanium Trace Status from all machines.

  2. Drill down to the specific endpoints.
  3. Deploy the Toggle Recorder or Toggle Recorder (Linux) package as an action.
    • Select the Enable Trace Recorder checkbox to enable the recorder.
    • Clear the Enable Trace Recorder checkbox to disable the recorder.

For more information, see the Tanium Core Platform User Guide: Managing and creating Packages or the Tanium Interact User Guide: Using Deploy Action.

Recreate the endpoint database

If the database is identified as corrupt, you might need to clear the endpoint database.

  1. Use a question to target the affected endpoints.

    For example, ask Get Trace Invalid File Operations from all machines.

  2. Drill down to the endpoints that return true.
  3. Deploy the Recreate Tanium Trace Database or Recreate Tanium Trace Database (Linux) package as an action.

For more information, see the Tanium Core Platform User Guide: Managing and creating Packages or the Tanium Interact User Guide: Using Deploy Action.

Collect a troubleshooting package

Trace can compile a collection of logs relevant for troubleshooting.

  1. On the Trace home page, in the toolbar, click Information .
  2. Under Troubleshooting, click Collect.
  3. Under Download Package, click the package file link.

The log zip file might take a few moments to appear in your download folder.

You can use Tanium™ Incident Response Copy Tools to retrieve logs from endpoints. See the Tanium Incident Response User Guide: Copying IR data to a central location for more information.

Manually back up and restore Trace configurations

You can create a backup or restore a backup to an existing Trace configuration. For example, you might use a backup and restore when you are moving from a testing to a production environment.

  1. On the Tanium Module Server, stop the Trace service.
  2. On the source host computer, go to Program Files\Tanium\Tanium Module Server\services\ and copy the trace-files directory.
  3. On the target host computer, paste the directory to another Trace install to restore the state.
  4. Restart the Trace service.

For a Linux host machine, replace the backslash (\) with a forward slash (/).

Remove Trace tools

You can deploy a preconfigured package to remove Trace tools from an endpoint or computer group.

  1. Remove the target endpoint or computer group from the Trace Setup Action Group.
    1. From the Tanium Console, go to Actions > Scheduled Actions.
    2. In the Action Groups pane, click Trace Setup.
    3. Click Edit.

      The Trace Setup Action Group page opens.

    4. In the Computer Groups section, deselect groups as needed.
    5. (Optional) In the All machines currently included in this action group section, review the included machines.

      This grid might take a few moments to populate when you change your selections.

    6. Click Save.
  2. Using Tanium Interact™ or a saved question, run the Tanium Trace Status sensor.
  3. From the endpoints that have Trace tools installed, drill down and select the targets.
  4. Deploy the Remove Tanium Trace Tools or Remove Tanium Trace Tools (Linux) packages to the targeted endpoints.

For more information, see Tanium Interact User Guide: Questions and Tanium Interact User Guide: Using Deploy Action.

Uninstall Trace

In certain situations, you might need to remove Trace from the Tanium Module Server for troubleshooting purposes.

  1. From the Tanium Console, click Solutions.

    The Solutions page opens.

  2. Locate Trace, and then click Uninstall.

    The Uninstall window opens, showing the list of contents to be removed.

  3. Click Proceed with Uninstall.
  4. Enter your password to start the uninstall process.

    A progress bar is displayed as the installation package is removed.

  5. Click Close.
  6. To confirm, return to the Solutions page and check that the Import button is available.

    If the Trace module has not updated in the console, refresh your browser.

Last updated: 3/20/2018 1:28 PM | Feedback