Troubleshooting

This section identifies resources that you can use when troubleshooting issues with the Tanium Client deployment.

Basic tips

Review the Tanium Client installation log to troubleshoot installation on Windows

If you encounter issues with your installation on Windows endpoints, examine Install.log in the Tanium Client installation directory to identify actions that failed during the installation. On Windows endpoints, the Tanium Client installer generates this log file to record a chronology of the actions that the installer performed. Each time the installer runs (that is, for each installation and upgrade), it appends the actions for that execution to the end of the existing log file.

Review Tanium Client logs to troubleshoot connections and other client issues

Review Tanium Client logs to help you troubleshoot client issues. For example, a client might not answer questions or appear in the Tanium Console (Administration > Management > Client StatusAdministration > Configuration > Client Status) because that client cannot connect to a TaaS instance the Tanium Server or Zone Server. In this case, you can review the client logs to determine whether the connection failed due to an invalid instanceserver IP address, DNS resolution failure, missing Tanium public key file, or firewall rule.

The Tanium Client writes new client logs to the file log0.txt. The default maximum log file size is 10 MB. When log0.txt reaches the maximum size, the client renames it log1.txt and then creates a new log0.txt. When log0.txt again reaches the maximum, the client renames log1.txt as log2.txt, again renames log0.txt as log1.txt, and again creates a new log0.txt. The process of rolling logs whenever log0.txt reaches the maximum size continues until 10 logs exist: log0.txt to log9.txt.

When log0.txt reaches the maximum size again after that, the client compresses log9.txt as a file named log10.zip. When log0.txt again reaches 10 MB, the client renames log10.zip as log11.zip and again compresses log9.txt as a file named log10.zip. The ZIP file rollover process continues until 10 ZIP files exist, log10.zip to log19.zip. When log0.txt reaches 10 MB again after that, the client creates a new log10.zip without renaming log19.zip as a new file, effectively dropping the old log19.zip information upon renaming log18.zip as the new log19.zip.

The location for log files is configurable (see LogPath). The default is <Tanium Client installation directory>/Logs.

You can use client health functionality in Client Management to directly connect to an endpoint and retrieve logs. For more information, see View detailed client health information for an endpoint.

Review action logs and associated files to troubleshoot actions and packages

When a package does not seem to work after you deploy it through an action, review action logs and the files associated with the action to help troubleshoot. Each time the Tanium Client receives an action message with an instruction set to execute, the client creates an action log file named Action_<ID>.log, where <ID> is the action identifier. The action log contains the CLI output associated with the action command. The Tanium Client stores any files that are required to deploy an action package in Action_ID directories.

Both action logs and Action_<ID> directories are in the <Tanium Client installation directory>/Downloads directory. The Tanium Client removes action logs from its host after a configurable interval (see Action log and package cleanup).

The Tanium Console displays the Action ID in the Action > Action History and Action Summary pages (see Tanium Console User Guide: Deploying actions). The Action Summary page provides options for accessing action log information from multiple endpoints: see Tanium Console User Guide: View action summary and status.

Action history logs provide a longer history of which actions a managed endpoint has run, but without the CLI output and other details.

You can use client health functionality in Client Management to directly connect to an endpoint and retrieve action logs. For more information, see View detailed client health information for an endpoint.

Action_<ID> directories

Each Action_<ID> directory contains all the files that are required to deploy an action package. For example, if you deploy a package that has five files, the Tanium Client places each file in the Action_<ID> directory after it finishes downloading. After all five files download, the action status changes from Preparing Files to Running on the Action Summary page. Even if a deployed package has no associated package files, the Tanium Client creates an empty Action_<ID> directory for it. The Tanium Client removes Action_<ID> directories from its host after a configurable interval (see Action log and package cleanup).

Action log contents

Action logs record each phase of an action:

  • Downloading Files

    During this phase, the action log entry indicates the files are downloading:

    2016-11-28 14:12:30 +0000|Downloading Files.
    2016-11-28 14:12:30 +0000|Files Failed Verification

    Although it appears to be an error condition, the message "Files Failed Verification" indicates simply that the client does not have the necessary files in its local cache, so it asks for the necessary files from its peers. This indicates normal behavior.

  • Running

    During this phase, the action log notes that the action is currently running. Following this entry, the log displays anything echoed from the package:

    2016-11-28 14:12:37 +0000|Files Verified, running action.

  • Completed

    When the action finishes running, the log records a completion entry under the standard output capture of the action.

    2016-11-28 14:12:37 +0000|Command Completed

Completion does not indicate success. For example, an action to execute a command might complete even if the command itself fails. For example, the command line for the package might not match the name of the distributed file or the command might fail to distribute a file. Managed endpoints show that the action completed, even though nothing occurred. Optionally, consider adding a validation query to the package to have the action status indicate success or failure.

Action log and package cleanup

The Tanium Client checks hourly, or immediately upon resetting (every two to six hours), whether any Action_<ID>.log files are over seven days old and deletes them if they are. The Tanium Client also checks hourly, or immediately upon resetting, whether any corresponding Action_<ID> directories have expired, and deletes them if they have. This process ensures that the endpoint does not consume more disk space than necessary for Tanium actions. Contact Tanium Support if you want to preserve action logs or action directories for a longer time.

Review action history logs to troubleshoot or audit actions

When you troubleshoot or audit actions on managed endpoints, review the action history logs to see which actions ran, their start and run times, and associated commands. Although the Action logs record more details, the Tanium Client preserves action history logs for a longer period (their individual log files are smaller) and therefore they provide a longer chronology of actions. The Tanium Client archives the first 10 MB of action history logs as plain-text files. After reaching the 10 MB threshold, the client archives the oldest logs as ZIP files before adding new logs as plain-text files. The log rollover process is as follows:

Plain text logs files

The Tanium Client creates a new action-history0.txt file whenever an action runs. When that file reaches 1 MB in size, the client renames action-history0.txt as action-history1.txt and creates a new action-history0.txt. When action-history0.txt again reaches 1 MB, the client renames action-history1.txt as action-history2.txt, again renames action-history0.txt as action-history1.txt, and again creates a new action-history0.txt. The process of rolling logs whenever action-history0.txt reaches 1 MB continues until 10 logs exist: action-history0.txt to action-history9.txt.

ZIP log files

After recording 10 MB of plain-text action history logs, the Tanium Client compresses action-history9.txt as a file named action-history10.zip. When action-history0.txt again reaches 1 MB, the client renames action-history10.zip as action-history11.zip and again compresses action-history9.txt as a file named action-history10.zip. The ZIP file rollover process continues until 10 ZIP files exist, action-history10.zip to action-history19.zip. When action-history10.zip reaches 1 MB again after that, the client creates a new action-history10.zip without renaming action-history19.zip as a new file, effectively dropping the old action-history19.zip information upon renaming action-history18.zip as the new action-history19.zip.

The Tanium Client stores action history logs in the <Tanium Client installation directory>/Logs directory.

Review sensor history logs to troubleshoot or audit sensor activity

When you troubleshoot or audit sensor activity on managed endpoints, review the sensor history logs to see the following information about each sensor that ran:

  • Sensor identity, by name and hash value
  • Start and run times
  • Size in bytes
  • Parameter values (the logs identify parameterized sensors as temp sensors)
  • Number of answer strings and associated hash value

The Tanium Client archives the first 10 MB of sensor history logs as plain-text files. After reaching the 10 MB threshold, the client archives the oldest logs as ZIP files before adding new logs as plain-text files. The log rollover process is as follows:

Plain text logs files

The Tanium Client creates a new sensor-history0.txt file each time a sensor runs. When that file reaches 1 MB in size, the client renames sensor-history0.txt as sensor-history1.txt, and creates a new sensor-history0.txt. When sensor-history0.txt again reaches 1 MB, the client renames sensor-history1.txt as sensor-history2.txt, again renames sensor-history0.txt as sensor-history1.txt, and again creates a new sensor-history0.txt. The process to roll the logs whenever sensor-history0.txt reaches 1 MB continues until 10 logs exist: sensor-history0.txt to sensor-history9.txt.

ZIP log files

After recording 10 MB of plain-text sensor history logs, the Tanium Client compresses sensor-history9.txt as a file named sensor-history10.zip. When sensor-history0.txt again reaches 1 MB, the client renames sensor-history10.zip as sensor-history11.zip and again compresses sensor-history9.txt as a file named sensor-history10.zip. The ZIP file rollover process continues until 10 ZIP files exist, sensor-history10.zip to sensor-history19.zip. When sensor-history10.zip reaches 1 MB again after that, the client creates a new sensor-history10.zip without renaming sensor-history19.zip as a new file, effectively dropping the old sensor-history19.zip information upon renaming sensor-history18.zip as the new sensor-history19.zip.

The Tanium Client stores sensor history logs in the <Tanium Client installation directory>/Logs directory.

Review or reset the public key to troubleshoot connection issues (Tanium Client 7.4 only)

You can review or reset the public key to help resolve connection issues that are related to an invalid key.

  1. Access the operating system CLI on the endpoint and change directory (cd) to the Tanium Client installation directory.
  2. Enter the following command:

    • Windows: TaniumClient pki show
    • Non-Windows: ./TaniumClient pki show

    The output displays information about the current public key.

  3. (Optional) Reset the key with a new tanium-init.dat file.

    1. From the Main menu in the Tanium console, go to Administration > Shared Services > Client Management.
    2. From the Client Management Overview page, download the installation package for the OS of the endpoint.
    3. Extract the tanium‑init.dat file from the downloaded bundle, and copy it into the Tanium Client installation directory.
    4. From the Main menu in the Tanium console, go to Administration > Configuration > Tanium Server > Infrastructure Configuration Files.
    5. In the Clients v7.4+ and Zone Server section, click Download.
    6. Copy the downloaded file into the Tanium Client installation directory.
    7. From the CLI on the endpoint, enter the following command:

      • Windows: TaniumClient pki reset tanium-init.dat
      • Non-Windows: ./TaniumClient pki reset tanium-init.dat

    Be careful not to allow the tanium-init.dat or tanium.pub file to be distributed or stored outside of your organization, such as in a publicly accessible source code repository or any other location accessible from the public internet. Limit the distribution to specific use in the deployment of Tanium Clients.

    Though these files do not contain private keys and cannot be used to provide control over a Tanium environment, a user with malicious intent could use them to connect an unapproved client and use this unauthorized access to learn how your organization is using Tanium.

Review and manage sensor quarantines to troubleshoot sensors

Enforcing sensor quarantines prevents sensors from running on an endpoint for the current question or action if those sensors exceeded the runtime timeout during a previous question or action. Quarantines are useful for limiting the impact on endpoint resources, such as CPU utilization, when questions and actions use excessively long-running sensors. The non-configurable timeout is set to one minute.

By default, quarantines are not enforced: after a sensor exceeds the timeout and stops running, the sensor has quarantined status but still runs for future questions or actions until it completes or times out. In this case, the Tanium Client uses the quarantined status just to record that the sensor timed out.

Regardless of whether you enable enforcement, the Tanium Client stops any sensor at the moment it exceeds the timeout. Each client quarantines sensors and enforces the quarantines independently. Consequently, a sensor might be quarantined on some endpoints and not on others.

When a Tanium Client quarantines a sensor, the Tanium Console displays the following message in the Question Results grid: TSE-Error: Sensor evaluation timed out. When you issue a question that uses a sensor that is already quarantined and enforcement is enabled, the Question Results grid displays TSE-Error: The sensor is quarantined. The Tanium Client adds entries to the client logs and sensor history logs when it quarantines a sensor or prevents an already quarantined sensor from running.

If temporary sensors exceed the one-minute timeout, the Tanium Client quarantines the original sensor as well as all current and future temporary sensors that are based on the original sensor.

When enforcement is enabled, quarantined sensors do not run when you use them for targeting endpoints, even if the sensors are members of computer groups. However, quarantined sensors might skew the targeting of a question that has a vague from clause, such as from all machines with Is Windows not equals true. In this case, Windows endpoints on which the Is Windows sensor is quarantined would match the condition not equals true because their response would be TSE-Error: The sensor is quarantined rather than true. To avoid such outcomes, make the target clause as specific as possible and do not use negative matching conditions such as not equals true.

View quarantined sensors

If the Tanium Client does not answer a question, you can determine whether the associated sensors are quarantined. To see a list of all the quarantined sensors on all endpoints, see Tanium Console User Guide: Manage sensor quarantines. To list all the quarantined sensors on a specific endpoint, perform the following steps:

  1. Access the operating system CLI on the endpoint and change directory (cd) to the Tanium Client installation directory.
  2. Enter the following command.

    • Windows: TaniumClient quarantine list
    • Non-Windows: ./TaniumClient quarantine list

    The output lists the quarantined sensors by name and associated hash value.

Remove all sensors from quarantine

In some cases, enabling the Tanium Client to answer questions that use quarantined sensors might be more important than limiting the impact that long sensor run times have on the resources of an endpoint. Note that even after you remove the sensors from quarantine, if they exceed the timeout in a future question, the Tanium Client will then stop the sensors and quarantine them again without answering the question. To remove sensors from quarantine through the Tanium Console, see Tanium Console User Guide: Manage sensor quarantines. To remove sensors from quarantine through the operating system CLI on the endpoint, perform the following steps:

  1. Access the operating system CLI on the endpoint and change directory (cd) to the Tanium Client installation directory.
  2. Enter the following command:

    • Windows: TaniumClient quarantine clear
    • Non-Windows: ./TaniumClient quarantine clear

    The output displays the number of sensors removed from quarantine.

Remove a single sensor from quarantine

To remove a sensor from quarantine through the Tanium Console, see Tanium Console User Guide: Manage sensor quarantines. To remove a sensor from quarantine through the operating system CLI on the endpoint, perform the following steps:

  1. Access the operating system CLI on the endpoint and change directory (cd) to the Tanium Client installation directory.
  2. Enter the following command to see the hash values associated with quarantined sensors.

    • Windows: TaniumClient quarantine list
    • Non-Windows: ./TaniumClient quarantine list
  3. Enter the following command, where <sensor_hash> is the hash associated with the sensor that you want to unquarantine:

    • Windows: TaniumClient quarantine remove <sensor_hash>
    • Non-Windows: ./TaniumClient quarantine remove <sensor_hash>

If you modify a sensor, Tanium Clients that receive its new definition automatically remove that sensor from quarantine.

Add a sensor to quarantine

You can manually quarantine a sensor on an endpoint if you anticipate that running the sensor will negatively affect the endpoint.

Quarantining a sensor does not automatically enable quarantine enforcement.

  1. In the URL field of the browser that you use to access the Tanium Console, enter https://<TaaS instanceTanium Server>/hash/<sensor>. For the <TaaS instanceTanium Server>, enter the TaaS Tanium Server FQDN or IP address. The <sensor> must match the sensor name that the Tanium Console displays with respect to capitalization and spaces.

    The browser displays the hash value associated with the sensor.

  2. Access the operating system CLI on the endpoint and change directory (cd) to the Tanium Client installation directory.
  3. Enter the following command.

    • Windows: TaniumClient quarantine add <sensor_hash>
    • Non-Windows: ./TaniumClient quarantine add <sensor_hash>

Enable or disable enforcement of quarantined sensors

After you enable quarantine enforcement, Tanium Clients do not answer questions that use quarantined sensors and those sensors do not run for actions. After you disable enforcement, clients still quarantine sensors and log quarantine events, but do not prevent those sensors from running.

Your user account must have a role with the Global Settings write permission to enable or disable quarantine enforcement. Users with the Administrator reserved role have this permission.

The first time you enable enforcement, you must add the EnableSensorQuarantine setting to the global settings on the Tanium Server as follows. By default, enforcement is disabled and the setting does not appear in the Tanium Console. After you add the setting, the Tanium Server applies it to all Tanium Clients.

  1. Access the Tanium Console.
  2. From the Main menu, go to Administration > Management > Global Settings, and click New Setting.
  3. Enter the following values and click Save.
    • Setting Name = EnableSensorQuarantine
    • Setting Value = 1
    • Affects = Client
    • Value Type = Numeric

Perform the following steps if you want to change the enforcement setting after adding it to the global settings:

  1. From the Main menu, go to Administration > Management > Global Settings.
  2. Select EnableSensorQuarantine, click Edit, set the value to 1 to enable enforcement or 0 to disable enforcement, and click Save.

If you want to change the enforcement setting in specific clients instead of all clients, add or edit the EnableSensorQuarantine setting in the local configuration of those clients.

Add or edit the EnableSensorQuarantine setting on the Tanium Clients for which you want to enable or disable quarantine enforcement.

Troubleshoot Client Management

To send information to Tanium for troubleshooting, collect logs and other relevant information.

Collect logs

The information is saved as a ZIP file that you can download with your browser.

  1. From the Client Management Home page, click Help , then the Troubleshooting tab.
  2. Click Download Debug Package.
    A tanium-client-management-support.zip file downloads to the local download directory.
  3. Attach the ZIP file to your Tanium Support case form or contact Tanium Support.

Tanium Client Management maintains logging information in the client-management.log file in the \Program Files\Tanium\Tanium Module Server\services\client-management-files directory.

Download deployment information

You can download a JSON file that includes deployment settings and endpoint details for a deployment.

  1. From the Client Management menu, click Deployments.

  2. In the Name column, click the name of a deployment.

  3. Click Download to download the JSON file.

Troubleshoot deployment problems

Problem: A new deployment instantly switches to the Completed status with no attempted deployments to endpoints

Cause: The Module Server is having trouble downloading the client binaries.

Solution: Check the TDownloader log for download errors. For information about where to find this log, see Tanium Core Platform Deployment Reference Guide: TDownloader logs.

Problem:Endpoint Installation Status = ERROR_CONNECTION_FAIL with "no response"  log message

Log messages for the deployment contain the following message:

Deployment Result Generated: All n connection attempt(s) resulted in no response from the target.

Cause: The Tanium Module Server cannot communicate with the endpoint, or cannot authenticate with the endpoint.

Solution: Check the following items.

  • Check the user name provided with the credentials. Credentials must be active and not disabled. Check that the domain is added correctly, for example: domain\username for a domain account, or username for a local endpoint account.
  • Check the password provided with the credentials to ensure it is not disabled or expired.
  • Check both the target endpoint firewall and network device firewalls. The Module Server might be blocked from initiating a connection to the target endpoint by a firewall. WMI port 135, SMB port 445, and SSH port 22 must be open. Use the following testing techniques to check the ports: 
  • If the endpoint is not joined to a domain, and either you use a non-default Administrator account or you use the default local Administrator account with the Admin Approval Mode for the Built-in Administrator account policy setting enabled, User Account Control (UAC) remote restrictions prevent access to administrative shares and remote installations. These administrative tasks are necessary for deployment of the Tanium Client using Client Management, and you must disable UAC remote restrictions to allow this deployment. To disable UAC remote restrictions, add the following value to the Windows registry and restart the machine:

    Subkey: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies\System
    Data type: REG_DWORD
    Value name: LocalAccountTokenFilterPolicy
    Value data: 1

    Administrative shares are not available in Home editions of Windows operating systems.

    After you deploy the Tanium Client, remove the LocalAccountTokenFilterPolicy registry value or set it to 0 to restore UAC remote restrictions. These restrictions help prevent malicious users from accessing the endpoint remotely with administrative rights.

Problem:Endpoint Installation Status = ERROR_CONNECTION_FAIL with SSH connection log message

Log messages for the deployment contain the following message:

Command resulted in error: Error: Connection to 'SSH Client for '192.168.24.11'' was not established

Cause: The Tanium Module Server is attempting an SSH deployment and cannot communicate with the endpoint, or cannot authenticate with the endpoint.

Solution: Check the following items. 

  • Verify the client configuration and deployment settings. You might be targeting a Windows endpoint with a deployment while only using SSH as a connection method.
  • Verify that the targeted Linux endpoint has SSH enabled and is configured on port 22.
  • Check the user name provided with the credentials. Credentials must be active and not disabled. Check that the domain is added correctly, for example: domain\username for a domain account, or username for a local endpoint account.
  • Check the password provided with the credentials to ensure it is not disabled or expired.

Problem:Endpoint Installation Status = ERROR_ACQUIRE_LOGS_FAIL with "necessary file missing" log message

Log messages for the deployment contain the following message:

Deployment Result Generated: Necessary file(s) missing on disk: C:\Program Files\Tanium\Tanium Module Server\services\client-management-files\deployment-runner-data\bc6bf6fd-0388-4f2d-9120-860cac75e8d4\tanium.pub

Cause: When you are deploying a version of the Tanium Client earlier than 7.4, the public key is missing.

Solution: Upload the tanium.pub file. See (Tanium 7.2.x, 7.3.x only) Upload Tanium public key.

Problem:Endpoint Installation Status = ERROR_ACQUIRE_LOGS_FAIL with "cli_rpc_pipe_open_noauth" log message

Log messages for the deployment contain the following message:

Error creating/starting the installation bootstrap service on the target: Error: cli_rpc_pipe_open_noauth: rpc_pipe_bind for pipe svcctl failed with error NT_STATUS_CONNECTION_DISCONNECTED Could not initialise pipe svcctl. Error was NT_STATUS_CONNECTION_DISCONNECTED

Cause: The Tanium Server could not establish WMI or RPC communication with an endpoint.

Solution: Verify that the firewall allows WMI, RPC, and SMB traffic between Tanium servers and endpoints. For more information, see Network connectivity, ports, and firewalls.

Firewalls with application-based control might not allow this traffic for Tanium by default.

Problem:Endpoint Installation Status = ERROR_ACQUIRE_LOGS_FAIL with "'mkdir' command exited" log message

Log messages for the deployment contain the following message:

SMB 'mkdir' command exited with exit code 1.

Cause: A Tanium Client might have been previously installed on the endpoint and not fully removed.

Solution: Verify that you are not trying to deploy to an endpoint that already has the Tanium Client installed. The endpoint could have a Tanium Client that was not fully removed, or a Tanium Client installation that points to a different Tanium Server.

Uninstall Client Management

Uninstalling Client Management also uninstalls Endpoint Configuration and affects all Tanium solutions. Contact Tanium support before you uninstall Client Management.

  1. From the Main menu, click Administration > Configuration > Solutions.
  2. In the Content section, select the Client Management row.
  3. Click Delete Selected . Click Uninstall to complete the process.

Contact Tanium Support

Tanium Support is your first contact for help when troubleshooting the initial deployment and for optimizing the speed and scale of your deployment as the number of managed endpoints grows. As necessary, Tanium Support can help adjust Tanium Client-related settings, including:

  • Tanium Client registration frequency
  • Connections between Tanium Clients and TaaSTanium Core Platform servers
  • Client-to-client connections
  • Bandwidth
  • File caching

If you require further assistance from Tanium Support, include Tanium Client and, if applicable, Tanium Client Management version information for Tanium Core Platform components and, if applicable, Tanium Client Management. Also include specific details on dependencies, such as the host system hardware and OS details. Finally, indicate if your installation uses a non-default installation directory for the Tanium Client.

To contact Tanium Support for help, sign in to https://support.tanium.com.