Troubleshooting the installation

1. Find precise version information.
2. Run the Health Check.
3. Check whether a Tanium service or networking needs to be restarted.
4. Review Tanium Core Platform logs.
5. Review solution module logs.
6. Review the configuration.
7. Run Tanium Support Gatherer.
8. Examine OS processes and files written to the file system.
9. Perform a TanOS reset (software/factory).

Display version information

When discussing issues, it is important to communicate precise version information with colleagues and with your TAM.

1. Log into the TanOS console as a user with the tanadmin or tanuser role.

   The TanOS console displays the tanadmin menu. View screen

   ------------------------------------------------------
   
                   >>> tanadmin menu <<< 
   
            1: Tanium Installation Menu
            2: Tanium Operations Menu
            3: Tanium Support Menu
            4: Status Menu
   
            A: Appliance Configuration Menu
            B: Appliance Maintenance Menu
            C: User Administration Menu
   
            P: Password change (current user)
            Q: View End User License Agreement (EULA)
   
            F: Flexible Menu Search
            @: About this Appliance
   
            H: Help
            Z: Log out
   
   ------------------------------------------------------

2. Enter @ to go to the About this Appliance menu that displays version information, system time, uptime, and basic network configuration details. View screen

   ----------------------------------------------------------------
   
                >>> About this Appliance <<< 
   
   Hostname:                         appliance-160
   Domain Name:                      tam.local
   
   System Time (MM/DD/YY):           05/21/20-17:23:46-UTC
   Uptime:                           up 1 hour, 18 minutes
   
   TanOS Version:                    1.6.0.0270
   Shell Version:                    1.6.0.0270
   
   Alternate TanOS Version:          1.6.0.0270
   Alternate Shell Version:          1.6.0.0270
   
   Appliance Model:                  TV-110
   Serial Number:                    VMware-564d0181aa8ea7af-2cece30817ed026f
   
   IP Address:                       10.10.10.60
   Gateway address:                  10.10.10.2
   
   Name Servers:                     10.10.10.10,
   NTP Servers:                      0.pool.ntp.org,
   
   Tanium Version:                   7.4.2.2063
   Server Role:                      Tanium Server (All-In-One)
   Server Addons:                    No add-ons installed
   Tanium Cluster Role:              Not configured
   
   Number of Processors:             2 X 2901 MHz
   Total Memory:                     1G
   
   Appliance Activation State:       No license or activation key found
   
   Tanium Server Pub Key MD5:        88da5655a7f03950077d9e3681dffba7
   Tanium Server Init Key MD5:       78fc1a51b0779f89f13d341f633f1f9e
   Tanium License MD5:               Not Available
   
   ---------------------------------------------------------------
   
   
   Press enter to continue

Run the Health Check

1. Log into the TanOS console as a user with the tanadmin role.
2. From the tanadmin menu, enter 3 to go to the Tanium Support menu. View screen

   ------------------------------------------------------
   
                   >>> Tanium Support menu <<< 
   
            1: Tanium Log Files
            2: Tanium Module Log Files
            3: Database Operations
            4: Run Network Diagnostics
            5: Run Health Check
            6: Display Last Scheduled Health Check Results
            7: Appliance Hardware Report
   
            A: Run TSG (Tanium Support Gatherer)
            B: Run Tcpdump
            P: Performance Monitoring
   
            X: Advanced Support
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

3. Enter 5 to run the health check. View screen

   >>> Tanium Support -> Run Health Check <<< 
   
   Report time: 2020-04-09 20:59:04 UTC
   Report Uptime:  20:59:04 up 52 min,  1 user,  load average: 0.00, 0.01, 0.05
   
   >>> Is EULA Accepted <<<
       [email protected] accepted June2019 EULA on Fri Feb 14 13:58:15 UTC 2020
       [email protected] accepted June2019 EULA on Mon Mar 23 19:22:43 UTC 2020
       [email protected] accepted June2019 EULA on Tue Mar 24 13:48:46 UTC 2020
   Is EULA Accepted: pass
   
   >>> Operating System health (will take 7-10 seconds) <<<
   CPU: pass
   Memory: pass (35% avail)
   Swap: fail (swap space is in use) 
   Partition /: pass (5%)
   Partition /boot: pass (24%)
   Partition /var: pass (5%)
   Partition /var/log/audit: pass (1%)
   Partition /opt: pass (1%)
   Partition /tmp: pass (1%)
   Partition /home: pass (1%)
   Partition /cores: pass (1%)
   Partition /var/log: pass (1%)
   Boot Check: Pass (BIOS Boot with virtual machine)
   Active partition: pass (VolGroup1-root)
   
   >>> Hardware health (will take 1-12 seconds) <<<
   hardware type: pass (TV-110)
   
   >>> User health <<<
   user tanium: pass
   user tanadmin: pass
   user tanuser: pass
   user tancopy: pass
   user tanfactory: pass (enabled)
   system user shells: pass
   system user policies: pass
   
   >>> Network health (will take 5-7 seconds) <<<
   default gateway: pass
   generic name resolution: fail (could not resolve content.tanium.com) 
   own hostname resolution: pass (Hosts)
   L2 check ens160: pass
   mount /opt/mounts/connect: pass (not configured)
   mount /opt/mounts/detect: pass (not configured)
   mount /opt/mounts/trends: pass (not configured)
   Connection tracking: pass
   
   
   >>> Service health <<<
   ntpd service: pass (running)
   ntpd service: fail (time not syncd) 
   rsyslog service: pass
   syslog delivery: fail (217146 messages have not been delivered) 
   iptables service: pass
   sshd service: pass
   ipsec service: pass
   local auth service: pass
   
   >>> Application health <<<
   taniumserver.service: pass (service is running)
   taniumserver.service: pass (iptables)
   taniumserver.service: pass (database connected)
   taniumserver.service: pass (database size: 1 GB)
   TMS connectivity: pass (127.0.0.1:17477)
   taniummoduleserver.service: pass (service is running)
   taniummoduleserver.service: pass (iptables)
   taniumzoneserver.service: pass (does not exist/not installed)
   
   >>> TanOS <<<
   BIOS Version: pass (not checked)
   PERC Version: pass (not checked)
   iDRAC Version: pass (not checked)
   Database Backup: fail (no backup files found) 
   Local sync: pass
   TanOS key material: pass
   
   >>> Miscellaneous <<<
   misc 1: pass
   core files: fail (core files exist) 
   shell keys: pass (no db)
   
   >>> Database Replication Health <<<
   
   Database Replication: pass (Not configured)
   
   >>> RAID Controller Security Key <<<
   
   RAID Security key check: pass (N/A for this platform)
   
   >>> Tanium Application file Permissions <<<
   TaniumServer file permissions: pass
   TaniumModuleServer file permissions: pass
   
   >>> Postgres SSL Health Check <<<
   Postgres SSL: pass
   SSL CRL file: root.crl.pem
   
   >>> OVA Health (TV-2xx) <<<
   RAM Requirements: fail (991) 
   CPU Requirements: fail (2) 
   
   executed checks: 63
   failed checks: 8
   
   >>> End Health Check <<<
   Press enter to continue

If your health check report prompts you to accept the end-user license agreement (EULA), go to the tanadmin menu and enter Q to display the EULA. Follow the prompts to accept it.

Restart services or networking

Check whether a Tanium™ service needs to be restarted. You can use the TanOS menu to stop a service whether or not it is enabled. You can use the TanOS to start a service if it has been enabled.

Restart services

1. Log into the TanOS console as a user with the tanadmin role.
2. From the tanadmin menu, enter 2 to go to the Tanium Operations menu. View screen

   ------------------------------------------------------
   
                >>> Tanium Operations Menu <<< 
   
            1: Tanium Service Control
            2: Tanium Configuration Settings
            3: Change Tanium Port
            4: Install Custom SOAP Cert
            5: Manage Custom Signing Keys
            6: Download Public Key
            7: Download SOAP Certificate
   
            A: Configure Remote Module Server
            B: Configure Cluster
            C: Manage Content
            M: Module Operations
   
            I: Import public key to Tanium Zone Server
   
            X: Advanced Operations
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

3. Enter 1 to go to the Tanium Service Control menu. View screen

   ------------------------------------------------------
   
    >>> Tanium Operations -> Tanium Service Control <<< 
   
            1: ipsec               enabled   started
            2: postgresql-11       enabled   started
            3: postgresql-9.5      enabled   started
            4: slapd               enabled   started
            5: taniummoduleserver  enabled   started
            6: taniumserver        enabled   started
            7: taniumzoneserver    enabled   started
   
            A: Restart ALL enabled Tanium services
            B: Stop ALL Tanium services
            C: Start ALL enabled Tanium services
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

4. Enter the line number of the service you want to manage to display the service commands. View screen

   >>> Tanium Operations -> Tanium Service Control -> Service <<< 
   
       Service         State      Status 
       taniumserver     enabled    running
   
            1: Start service
            2: Stop service
            3: Restart service
            4: Disable service
            5: Enable service
            6: Status Details
   
            R: Return to previous menu
   
   ------------------------------------------------------

5. Type the number of a service control command to issue it.

Restart networking

1. Log into the TanOS console as a user with the tanadmin role.
2. From the tanadmin menu, enter A to display the Appliance Configuration menu. View screen

   ------------------------------------------------------
   
              >>> Appliance Configuration <<< 
   
            1: Hostname/DNS Configuration
            2: Networking Configuration
            3: NTP Configuration
            4: Syslog Configuration
            5: SNMP Configuration
            6: Module File Share Configuration
            7: Reset all NICs to DHCP (VM only)
   
            A: Security
            C: Appliance Array
            X: Advanced Configuration
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

3. Enter 2 to display the Networking Configuration menu. View screen

   ------------------------------------------------------
   
        >>> Appliance Configuration -> Networking <<< 
   
             1: Network Interfaces
             2: IPSEC Configuration
             3: Routing Configuration
             4: Restart Networking
   
             T: NIC Teaming
   
             H: Help
             R: Return to previous menu
   
   ------------------------------------------------------

4. Enter 4 to restart networking.

Review Tanium Core Platform logs

If you are diagnosing issues with the Tanium Core Platform installation, review the logs.

1. Log into the TanOS console as a user with the tanadmin role.
2. From the tanadmin menu, enter 3 to go to the Tanium Support menu. View screen

   ------------------------------------------------------
   
                   >>> Tanium Support menu <<< 
   
            1: Tanium Log Files
            2: Tanium Module Log Files
            3: Database Operations
            4: Run Network Diagnostics
            5: Run Health Check
            6: Display Last Scheduled Health Check Results
            7: Appliance Hardware Report
   
            A: Run TSG (Tanium Support Gatherer)
            B: Run Tcpdump
            P: Performance Monitoring
   
            X: Advanced Support
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

3. Enter 1 to go to the Log Files Access menu. View screen

   ------------------------------------------------------
    
         >>> Tanium Support -> Log File Access <<< 
   
            1: TanOS Appliance
            2: Tanium Server
            3: Tanium Module Server
            5: PostgreSQL
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

4. Select an item to view its submenu. View screen

   ------------------------------------------------------
   
   >>> Tanium Support -> Log File Access -> taniumserver <<< 
   
            1: Tanium Server Log (log0.txt)
            2: Tanium Server TDL Log (log0.txt)
            3: Tanium Server RBAC Log (rbac0.txt)
            4: Tanium Server PAM4 Log (tanium_pam4.log)
            5: Tanium Server Workbenches (workbenches_manager.log)
            6: Tanium Server Signature Verifier (signature_verifier.log)
            7: Tanium Server DB Upgrade Log (database-upgrade0.txt)
   
            R: Return to previous menu
   
   ------------------------------------------------------

5. Select an item to view the log, follow its growth, delete it, or copy it to the /outgoing directory. View screen

   ------------------------------------------------------
    
   >>> Tanium Support -> Log files -> View -> log0.txt <<< 
   
          Size: 3.6K   Last Update: Apr 9 20:21
    
            1: View log (exit by pressing 'q')
            2: Follow log file growth (exit by pressing Ctrl-C)
   
            A: Delete content of logfile
            B: Copy to outgoing (sftp)
   
            R: Return to previous menu
   
   ------------------------------------------------------

When you view a log, you can use commands similar to ex editor commands to search for patterns (keywords).

Review Tanium solution module logs

If you are diagnosing issues with expected behavior for solution modules, examine the module logs.

1. Log into the TanOS console as a user with the tanadmin role.
2. From the tanadmin menu, enter 3 to go to the Tanium Support menu.
3. Enter 2 to go to the Module Log Files Access menu.
4. Select an item to view its submenu.
5. Select an item to view the log, follow its growth, delete it, or copy it to the /outgoing directory.

When you view a log, you can use commands similar to ex editor commands to search for patterns (keywords).

Review the configuration

1. Log into the TanOS console as a user with the tanadmin role.
2. From the tanadmin menu, enter 2 to go to the Tanium Operations menu. View screen

   ------------------------------------------------------
   
                >>> Tanium Operations Menu <<< 
   
            1: Tanium Service Control
            2: Tanium Configuration Settings
            3: Change Tanium Port
            4: Install Custom SOAP Cert
            5: Manage Custom Signing Keys
            6: Download Public Key
            7: Download SOAP Certificate
   
            A: Configure Remote Module Server
            B: Configure Cluster
            C: Manage Content
            M: Module Operations
   
            I: Import public key to Tanium Zone Server
   
            X: Advanced Operations
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

3. Enter 2 to go to the Configuration Settings menu. View screen

   ------------------------------------------------------
   
    >>> Tanium Operations -> Configuration Settings <<< 
   
   Note: Some settings may require a service restart to take effect.
   
             1: Edit Tanium Server Settings
             2: Edit Tanium Server TDL Settings
             3: Add Tanium Server TDL Auth User
             4: Add Tanium Server TDL Auth Cert
   
             5: Edit Tanium Module Server Settings
             6: Edit Tanium Module Server TDL Settings
             7: Add Tanium Module Server TDL Auth User
             8: Add Tanium Module Server TDL Auth Cert
   
             9: Edit Tanium Zone Server Settings
             11: Edit isolated subnets list
             12: Edit separated subnets list
   
             13: Control RedHat CA Cert
   
             H: Help
             R: Return to previous menu
   
   ------------------------------------------------------

4. Use the menu to view and edit Tanium server configuration files.

Run Tanium Support Gatherer

The Tanium Support Gatherer (TSG) collects system status, process status, network interface status, and so on, to help your TAM evaluate possible appliance or Tanium server issues.

Each Tanium server component has a predefined list of files and commands to gather relevant data. The TSG output files are placed in the SFTP outgoing directory. The output files are ZIP archives, named with their collection or module name and a datestamp. The files remain in the outgoing directory until a daily cleanup task removes them. From the TSG menu, you specify a single item (a module or collection of files) or a comma-separated list of items.

1. Log into the TanOS console as a user with the tanadmin role.
2. From the tanadmin menu, enter 3 to go to the Tanium Support menu. View screen

   ------------------------------------------------------
   
                   >>> Tanium Support menu <<< 
   
            1: Tanium Log Files
            2: Tanium Module Log Files
            3: Database Operations
            4: Run Network Diagnostics
            5: Run Health Check
            6: Display Last Scheduled Health Check Results
            7: Appliance Hardware Report
   
            A: Run TSG (Tanium Support Gatherer)
            B: Run Tcpdump
            P: Performance Monitoring
   
            X: Advanced Support
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

3. Enter A to go to the Tanium Support Gatherer menu. View screen

   ------------------------------------------------------
   
   >>> Tanium Support -> Tanium Support Gatherer (TSG) <<< 
   
    **NOTE** TSG runtimes will vary depending on collections selected.
             The hardware inventory command can take 12 minutes on large appliances.
   
    Please select a module, platform or appliance feature for which to gather support data below.
   
    #   Module
    1   taniummoduleserver
    2   taniumserver
   
    #   Collections
    3   Appliance Support Files (OS,Hardware,Network,Auth,DB)
    4   Platform Support Files (regardless of role)
    5   Database Support Files
    6   All Module Support Files
    7   Everything
   
    R   Return to main menu
   
    Please select a menu item or comma separated list:

4. Specify a single line item number or a comma-separated list of item numbers.

   TanOS runs the report and indicates the path to the zipped report file. View screen

   ------------------------------------------------------
   
             >>> Tanium Support -> TSG -> Run <<< 
   
    Run TSG with --taniumserver
   2020-09-04 21:34:30 +0000 Starting Tanium Service Gatherer (TSG)
   2020-09-04 21:34:30 +0000 Starting taniumserver Collection
   2020-09-04 21:34:30 +0000 Completed taniumserver Collection
   2020-09-04 21:34:30 +0000 Collection Result is 0
   2020-09-04 21:34:30 +0000 Tanium Service Gatherer Complete
   2020-09-04 21:34:30 +0000 Zipping results to sftp outgoing /opt/home/tancopy/outgoing/2020-04-09-2134-appliance-156-tsg_taniumserver.zip
   
    Press enter to continue

5. Press Enter to progress through the reports you selected.
6. Use SFTP to copy the archive to your local working directory.

Examine OS processes and files

In rare cases, you or your TAM might need to examine OS processes and files written to the file system.

Open read-only restricted shell

1. Log into the TanOS console as a user with the tanadmin role.
2. From the tanadmin menu, enter B to go to the Appliance Maintenance menu. View screen

   ------------------------------------------------------
   
                >>> Appliance Maintenance <<< 
   
            1: Backup
            2: Alerting
            3: Upgrade TanOS
            5: Activation Keys
   
            A: Clean directories
            B: Reboot/Shutdown
            C: Maintenance mode
   
            X: Advanced Maintenance
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

3. Enter 5 to go to the Activation Keys menu. View screen

   ------------------------------------------------------
   
      >>> Appliance Maintenance -> Activation Keys <<< 
   
        Current shell authorization for tanadmin is expired 
        Virtual Activation key is expired 
   
            1: Validate Response
   
            O: Open Read Only Restricted Shell
            W: Request Read Write Restricted Shell Activation
            F: Request Full Shell Activation
   
            L: Shell Key Listing
            A: Revoke All Shell Keys
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

4. Enter O to open a read-only shell. View screen

   >>> Appliance Maintenance -> Launch Read Only Restricted Shell <<< 
   
   RO access is still valid. A read only restricted shell will be opened.
   
   The shell will be forcibly closed after 10 minutes of inactivity.
   
   Are you sure you want to open RO shell? [Yes|No]: yes
   ([email protected])/
   (0 13:43:24 1) ->

5. Enter exit to close the shell.
6. When you are finished troubleshooting, go to the Activation Keys menu and enter 2 to remove shell access.

Request read-write restricted shell or full shell access

You must follow a special procedure to request read-write restrictive shell access or full shell access.

1. Log into the TanOS console as a user with the tanadmin role.
2. From the tanadmin menu, enter B to go to the Appliance Maintenance menu.
3. Enter 5 to go to the Activation Keys menu.
4. Enter W or F, and follow the prompts to generate a shell access request package. The package is written to /outgoing folder. View screen

    All previous keys or requests have been deactivated.
   
    The request file TanOS-key-request.tgz has been copied to tancopy outgoing.
   
    Provide the request file to your Tanium TAM and they will provide a response
    for verification. 
    The response file will be valid for 24 hours after processing.
   
    Press enter to continue

5. Use SFTP to copy the request file from the /outgoing directory to your local computer.

6. Email the file and TanOS version information to your TAM.

Your TAM will send you a response file.

7. Use SFTP to copy the response file to the /incoming directory.
8. At the Appliance Maintenance > Activation Keys menu prompt, enter 1 to validate the response. View screen

   >>> Appliance Maintenance -> Feature Request -> Validate Response <<< 
   
    Signature verified, genuine response
   
    Verifying DB matches
    DB Match has been found
    Request is still valid
    Shell access timeout has been updated.
    Return to the menu to open a shell session.
   
    Press enter to continue

The Activation Keys menu now has additional options. View screen

------------------------------------------------------

   >>> Appliance Maintenance -> Activation Keys <<< 

     Current shell authorization type is Read Write Restricted Shell 
     Current shell authorization expires in 1 hour 59 minutes 

     Virtual Activation key is expired 

         1: Validate Response
         2: Remove shell access
         3: Launch Read Write Restricted Shell

         O: Open Read Only Restricted Shell
         W: Request Read Write Restricted Shell Activation
         F: Request Full Shell Activation

         L: Shell Key Listing
         A: Revoke All Shell Keys

         H: Help
         R: Return to previous menu

------------------------------------------------------

9. Enter 3 to launch the shell. View screen

   >>> Appliance Maintenance -> Shell Access -> Laynch Unrestricted Shell <<< 
   
   Access is still valid. A unrestricted access shell will be opened
   
   All connections (regardless of shell access) will be forcibly closed
   after 60 minutes.
   
   ([email protected])~
   (0 20:02:08 1) ->

10. Enter exit to close the shell.
11. When you are finished troubleshooting, go to the Activation Keys menu and enter 2 to remove shell access.

Perform a TanOS reset (software/factory)

The Appliance Maintenance > Reset menu has two options:

• Perform a software reset to erase the Tanium application software and data. A software reset does not delete the Appliance Array.
• Perform a factory reset only if you want to erase the Tanium application software and data, as well as the appliance configuration.

1. Log into the TanOS console as a user with the tanadmin role.

   The TanOS console displays the tanadmin menu. View screen

   ------------------------------------------------------
   
                   >>> tanadmin menu <<< 
   
            1: Tanium Installation Menu
            2: Tanium Operations Menu
            3: Tanium Support Menu
            4: Status Menu
   
            A: Appliance Configuration Menu
            B: Appliance Maintenance Menu
            C: User Administration Menu
   
            P: Password change (current user)
            Q: View End User License Agreement (EULA)
   
            F: Flexible Menu Search
            @: About this Appliance
   
            H: Help
            Z: Log out
   
   ------------------------------------------------------

2. Enter B to go to the Appliance Maintenance menu. View screen

   ------------------------------------------------------
   
                >>> Appliance Maintenance <<< 
   
            1: Backup
            2: Alerting
            3: Upgrade TanOS
            5: Activation Keys
   
            A: Clean directories
            B: Reboot/Shutdown
            C: Maintenance mode
   
            X: Advanced Maintenance
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

3. Enter X to go to the Advanced Maintenance menu. View screen

   ------------------------------------------------------
   
       >>> Appliance Maintenance -> Advanced Menu <<< 
   
   Attention: The options in this menu can affect performance/stability 
   Please consult your TAM before utilizing the below options.
   
            1: Install Firmware Updates
            2: Reset (Factory/Software)
            3: Re-Apply ACLs
            4: View TanOS Upgrade Logfile
            5: View TanOS Partition Sync Logfile
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

4. Enter 2 to display the Reset menu. View screen

   ------------------------------------------------------
   
           >>> Appliance Maintenance -> Reset <<< 
   
            1: Software reset (remove all Tanium application software)
            2: Factory reset (remove software and configurations)
   
            H: Help
            R: Return to previous menu
   
   ------------------------------------------------------

5. Enter 1 to perform a software reset or 2 to perform a factory reset.
6. Follow the prompts to initiate the reset. View screen

    ------------------------------------------------------
   
      >>> Appliance Maintenance -> Reset -> Software <<< 
   
    Removing Tanium Software from this appliance
    The appliance will be rebooted after the removal 
   
    Would you like to continue with the software reset? [Yes|No]: yes
    Starting removal of software
   
    Removing installed software
    Working on Module Server and Modules...
    Removed Tanium Module Server
    Working on Tanium Server...
    Removed Tanium Server
    Removed database configuration
    Finished removing software
    Finished reset of software!
   
    Rebooting appliance