Monitoring and maintaining the Tanium Appliance

Perform regular maintenance tasks to ensure that the Tanium Appliance infrastructure remains in good health. For maintenance tasks to be performed on a routine basis, see Perform monthly maintenance and Perform quarterly maintenance. If an appliance is not performing as expected, you might need to troubleshoot issues or change settings. See Support and Troubleshooting for related procedures.

Use the Appliance Maintenance menu to perform most maintenance tasks, such as backups, system resets, TanOS upgrades, and system reboots or shutdowns. The Appliance Configuration menu contains some of the configuration options for ongoing monitoring and maintenance. The Tanium Support menu contains several tools to monitor status and run diagnostics.

Configure Tanium Appliance monitoring

Perform any of the following tasks to facilitate monitoring the health of your Tanium deployment and Appliance infrastructure. For example, if your organization has a syslog server or SNMP manager, you can integrate it with the Appliance for monitoring. If these monitoring solutions reveal issues that require resolution, see Support and Troubleshooting.

Configure TanOS alerts

TanOS can send alerts to a syslog server or to an email recipient. For optimal results, configure an SMTP email recipient. If the syslog server fails, the SMTP recipient receives a failure notification every 15 minutes until either the failure is resolved or syslog forwarding is disabled. See Configure syslog alerts.

Configure syslog forwarding

You can forward Appliance logs to a remote syslog server. The syslog forwarding configuration is separate from the syslog alert configuration. For the differences, and the steps to configure syslog forwarding, see Configure syslog forwarding.

Configure SNMP

You can configure integration with an SNMP manager to collect and analyze Appliance information. After you configure credentials, the user tansnmp can make a remote SNMP connection to the Appliance or to the Integrated Dell Remote Access Controller (iDRAC) interface of a physical Appliance to conduct SNMP polling from a remote host or SNMP manager. See Configuring SNMP.

Perform monthly maintenance

If these tasks reveal issues that require resolution, see Support and Troubleshooting.

Review the Health Check report

The Health Check report provides information on the health of the Appliance operating system, hardware, users, network, services, applications, database replication, RAID security, Postgres SSL, and virtual machine (if applicable).

  1. Run the report. See Run the Health Check.
  2. Review the output for actionable items, which are summarized at the end of the output.

    For example, the output might indicate that the End User License Agreement (EULA) is not accepted.

Monitor Appliance performance (optional)

See the following tasks for the steps to run commands for viewing Appliance performance information:

  • Run a SAR command to view statistical information such as CPU load, memory paging, memory utilization, swap usage, and network input/output (I/O).
  • Run the iotop command to view I/O utilization by process.
  • Run the perf top command to view CPU usage by function.
  • Run the htop command to view detailed information about each running process, such as memory and CPU consumption. The output provides an interface whereby you can navigate among values and tabs by keyboard and mouse.

Perform quarterly maintenance

Verify the grub key backup

You can use the grub key during the boot sequence to diagnose and recover from failure conditions. During recovery, you must provide the key to Tanium Support for a technician to extract the grub password.

  1. Verify that a backup of the latest key resides in a safe location off the Appliance.

    A new backup is required whenever the key password is regenerated. See Change the grub key password.

  2. Export the key and save it in a safe location if no backup exists or if the current backup is not the latest. See Export the grub key.

Review and update TanOS user accounts

  1. On each appliance, review the TanOS system users to ensure that they can access the Appliance operating system and that they have the appropriate authentication settings. For example, users who authenticate through passwords must comply with the password policy of your organization. See Modify the local authentication service security policy.

    The predefined roles for TanOS system users include:

    • tanadmin: Users with this role can access all TanOS console menus. It is useful to have more than one tanadmin user in case you forget the password for the initial tanadmin user that is created during Appliance setup.
    • tancopy: Users with this role can copy files to and from the /incoming and /outgoing directories on the Appliance.
    • tanuser: Users with this role can access only status menus in the TanOS console.

    For details and procedures, see Managing users.

  2. Verify that the predefined tanremote user account is present if you configured an Integrated Dell Remote Access Controller (iDRAC) interface on the physical Appliance. The account provides remote access to the iDRAC virtual console. This is useful for diagnosing hardware and network interface issues if the TanOS system becomes unavailable. For details and procedures, see Manage the iDRAC interface (physical Tanium Appliances only).

Backup overview

TanOS contains the options to perform core and comprehensive backups. physical Tanium Appliances and virtual Tanium Appliances with inactive partitions also have the option to back up the active partition to the inactive partition. On virtual Tanium Appliances and cloud-based Tanium Appliances, you can also take a snapshot of the appliance image. For core and comprehensive backups, you can schedule automatic backups or perform a manual backup.

Manage the iDRAC interface (physical Tanium Appliances only)

Use the tanremote user account to sign in to the iDRAC virtual console to diagnose hardware and network interface issues in the event the TanOS system becomes unavailable. The tanremote user is not a TanOS user or a Tanium Console user.

Before you begin

  • You must use a cable to connect the iDRAC interface to your network and use TanOS to configure the iDRAC interface before you enable the tanremote user.

Configure the iDRAC interface

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-I (Appliance Configuration > Manage iDRAC).

  3. Enter N and follow the prompts to configure the iDRAC interface.

Set password for the tanremote user

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-I (Appliance Configuration > Manage iDRAC).

  3. Enter P and follow the prompts to change the password of the tanremote user. ClosedView screen

Enable the tanremote user

Set the password for the tanremote user before you enable the user.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-I (Appliance Configuration > Manage iDRAC).

  3. Enter E and follow the prompts to enable the tanremote user.

Disable the tanremote user

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-I (Appliance Configuration > Manage iDRAC).

  3. Enter D and follow the prompts to disable the tanremote user.

Close all iDRAC sessions

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-I (Appliance Configuration > Manage iDRAC).

  3. Enter C and follow the prompts the close all iDRAC sessions.

Reset iDRAC

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-I (Appliance Configuration > Manage iDRAC).

  3. Enter R and follow the prompts to reset the iDRAC interface.

Access the iDRAC virtual console

You can access the iDRAC virtual console at http://<iDRAC interface IP address>. Sign in with username tanremote and the password that was set with this procedure.

Configure syslog forwarding or alerts

TanOS can forward appliance logs to a remote syslog server or send alerts to a syslog server or to an email recipient. For optimal results with alerts, configure an SMTP email recipient. If the syslog server fails, the SMTP recipient receives a failure notification every 15 minutes until the failure is resolved or syslog forwarding is disabled.

Severity level is a global setting that applies to both Syslog and SMTP alerts.

Syslog alerts versus forwarding

The syslog forwarding configuration under Appliance Configuration is separate from the syslog alert configuration in the Appliance Maintenance menu. Note these key differences:

  • Syslog configuration for alerts sends events that match the specified alert threshold severity (info, warn, and error).
  • Syslog forwarding configuration sends all messages located in /var/logs/messages to a syslog destination.

Check the syslog status

The syslog status shows the last five log entries and the current syslog forwarding configuration.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-4 (Appliance Configuration > Syslog Configuration).

  3. Enter 1 to view the status. ClosedView screen

Configure syslog forwarding

Import a syslog server trust certificate

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-4 (Appliance Configuration > Syslog Configuration).

  3. Enter 2 to view the trust certificate, 3 to paste it (PEM format), or 4 to remove it.

Enable syslog forwarding

Syslog Forwarding sends the same data that gets logged in /var/log/messages to the destination.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-4 (Appliance Configuration > Syslog Configuration).

  3. Enter 5 and follow the prompts to specify the settings for the remote syslog server. ClosedView screen

    If you do not enable RFC5424 output format, TanOS defaults to RFC3164 syslog output.

Configure syslog alerts

Set the alert severity level

Use the Configure Alerts menu to set the alert severity threshold to info, warn, or error.

  • Info: Includes all alerts
  • Warn: Includes all error and warning alerts
  • Error: Includes error alerts
  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B-2-3 (Appliance Maintenance > Alerting > Configure Alerts).

  3. Use the menu to set a severity level and enable/disable alerting.

Configure a syslog server destination for alerts

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B-2-1 (Appliance Maintenance > Alerting > Configure Syslog Destination).

  3. Enter 2 and follow the prompts to configure a syslog destination. ClosedView screen
  4. Enter 1 to enable syslog alerts. The Configure Syslog Destination menu updates to show the current status. ClosedView screen
  5. Enter 3 to send a test alert to the syslog server.

The test alert appears in the syslog server logs.

Configure an SMTP destination for alerts

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B-2-2 (Appliance Maintenance > Alerting > Configure SMTP Destination).

  3. Enter 2 and follow the prompts to configure the SMTP destination. ClosedView screen
  4. Enter 1 to enable SMTP alerts. The Configure SMTP Destination menu updates to show the current status. ClosedView screen
  5. Enter 3 to send a test alert to the mail recipient.

Configuring SNMP

Tanium Appliances support SNMP v3, though the SNMP service is not enabled by default. You can configure SNMP credentials and start the service to allow remote SNMP connections to the appliance or to the iDRAC interface of a physical Tanium Appliance. The default user name for SNMP connections is tansnmp. A remote host or SNMP manager can use the configured credentials to conduct SNMP polling on the appliance. Tanium Appliances only respond to SNMP requests; they do not send SNMP traps.

There is not a Tanium-specific MIB. Tanium Appliances report a specific SNMPv2 sysObjectID and include the following standard MIBs:

  • SNMPv2-MIB
  • IP-MIB
  • IF-MIB
  • TCP-MIB
  • UDP-MIB
  • HOST-RESOURCES-MIB
  • UCD-SNMP-MIB

For a physical Tanium Appliance, see Dell Technologies: SNMP Reference Guide for iDRAC and Chassis Management Controller for information about the MIB used with iDRAC. Some limitations apply for the iDRAC implementation in the Tanium Appliance. For example, the Tanium Appliance does not support SNMP v1 or v2, nor does it send SNMP traps.

Example of SNMP polling using snmpwalk

Set password and start the SNMP service

Passwords must contain at least 8 characters.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-5 (Appliance Configuration > SNMP Configuration).

  3. Enter S , enter the desired SNMP password at the prompt, and hit enter to save the password and enable the SNMP service. ClosedView screen

Change the SNMP user name, location, or contact

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-5 (Appliance Configuration > SNMP Configuration).

  3. Enter U to change the user name, L to change the location, or C to change the contact and follow the prompts to enter the new value.

View SNMP service status

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-5 (Appliance Configuration > SNMP Configuration).

  3. Enter V to view the SNMP service status details.

Stop the SNMP service

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter A-5 (Appliance Configuration > SNMP Configuration).

  3. Enter D to stop and disable the SNMP service.

Upgrade TanOS

See Upgrade TanOS.

Request a shell access key

You can request OS shell access to examine OS processes and files written to the file system. See Examine Tanium and TanOS files.

Clean up generated files

Clean directories to clear up disk space or clear logs to make it easier to work with new entries in the log viewer.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B-A (Appliance Maintenance > Clean directories).

  3. Use the menu to delete files that have been generated in the SFTP /incoming and /outgoing directories, core dump files, application logs, and so on.

Reboot or shut down

Tasks that you complete with TanOS menus typically do not require you to reboot the system. A reboot might be required during troubleshooting workflows.

Shutdown turns off the system and powers down the appliance.

On a physical Tanium Appliance, you must have physical or iDRAC access to the appliance to power it on. Do not perform a system shutdown unless you are prepared to power the appliance back on.

Reboot

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B-B-1 (Appliance Maintenance > Reboot/Shutdown > Reboot).

  3. Follow the prompts to reboot the appliance.

Shut down

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B-B-2 (Appliance Maintenance > Reboot/Shutdown > Shutdown).

  3. Follow the prompts to shut down the appliance.

Increase storage

On cloud-based Tanium Appliances and virtual Tanium Appliances, you can add a disk to the virtual image or increase the size of the existing virtual disk to increase the amount of storage that is available to TanOS.

This action is not reversible. Storage that you add to the appliance is permanently allocated. Do not attempt to remove disk storage from an appliance, as the appliance becomes unusable.

  1. Modify the virtual image to add a disk or increase the size of the existing virtual disk.
  2. Sign in to the TanOS console as a user with the tanadmin role.
  3. Enter B-I (Appliance Maintenance > Increase Storage).

  4. Follow the prompts to add the disk storage. ClosedView screen

    If your virtual Tanium Appliance has an inactive partition set, any new storage is evenly allocated across the active (/OPT) and inactive (/ALTOPT) partitions.

Manage OS services

Use this menu to start, stop, restart, enable, and view status details for the network time protocol daemon (chronyd) and SSH daemon (sshd) services.

  1. Sign in to the TanOS console as a user with the tanadmin role.

  2. Enter B-S (Appliance Maintenance > OS Services).

  3. Select a service to open the Service Control menu:

    • To manage chronyd, enter 1.ClosedView screen

    • To manage sshd, enter 2.ClosedView screen

  4. Use the menu to select an action to start, stop, restart, enable, or view status details for the service.

  5. Follow the prompts to perform the action.

View the overall status for TanOS, Tanium, and the appliance

View system status

System status shows OS and network status.

View system status with the tanadmin role

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 4-1 (Status > System Status).

  3. Enter 1 to view OS status, or enter 2 to view network status.

tanuser: View system status with the tanuser role

  1. Sign in to the TanOS console as a user with the tanuser role.
  2. Enter 1 (System Status).

  3. Enter 1 to view OS status, or enter 2 to view network status.

View Tanium component status

View Tanium status with the tanadmin role

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 4-2 (Status > Tanium Status).

View Tanium status with the tanuser role

  1. Sign in to the TanOS console as a user with the tanuser role.
  2. Enter 2 (Tanium Status).

  3. Use the menu to view Tanium service status.

View appliance status

Appliance status shows appliance version information, OS status, or hardware status.

View appliance status with the tanadmin role

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 4-3 (Status > Appliance Status).

  3. Use the menu to view appliance version information, OS status, or hardware status.

View appliance status with the tanuser role

  1. Sign in to the TanOS console as a user with the tanuser role.
  2. Enter 3 (Appliance Status).

  3. Use the menu to view appliance version information, OS status, or hardware status.

Review Tanium support information

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 3 (Tanium Support).

See Support and Troubleshooting for information on the reports available in this menu.

Perform database operations

If you encounter issues with the Tanium deployment, Tanium Support might direct you to perform database operations.

Database operations are available on the Tanium Server and Tanium Module Server. In an All-In-One deployment, database operations apply only to the Tanium Server.

View the Postgres log file

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 3-3-1 (Tanium Support > Database Operations > Select File).

  3. Enter the line number for the postgres.log file, and use the menu to view the log or copy it to the /outgoing folder. ClosedView screen

View Postgres configuration files

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 3-3-2 (Tanium Support > Database Operations > Select File).

  3. Use the menu to review or modify the configuration.

View Postgres control data

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 3-3 (Tanium Support > Database Operations).

  3. Enter 3 to view Postgres control data. ClosedView screen

Enable full Postgres audit log

Postgres logs are very rarely useful in troubleshooting appliance or platform issues. Audit logging is disabled by default. When enabled, Postgres logging can consume inordinate disk space. For best results, enable audit logging only when debugging.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 3-3 (Tanium Support > Database Operations).

  3. Enter 4 and follow the prompts to enable audit logging. ClosedView screen

Manage the database memory plan

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 3-3-D (Tanium Support > Database Operations > DB Tuning).

  3. Use the menus to view or make changes to the database memory plan.

    You must select the same database memory plan for both Tanium Servers in a cluster, or for both an active and standby Module Server. A Tanium Server and a Module Server are not required to have the same memory plan.

Run the Postgres top command

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 3-3 (Tanium Support > Database Operations).

  3. Enter M to view results of the top command. ClosedView screen
  4. Enter Q to return to the Database Operations menu.

Query the tanium database

The Manage Queries menu includes predefined queries that can be useful during troubleshooting.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 3-3-Q-S (Tanium Support > Database Operations > Manage Queries > Select Query).

  3. Use the menu to select a predefined query and return to the Manage Queries menu.
  4. Enter X to run the query and save the results to the /outgoing folder. ClosedView screen
  5. Enter Q to view query results. ClosedView screen
  6. Enter Q to return to the Manage Queries menu.

View replication status

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter 3-3 (Tanium Support > Database Operations).

  3. Enter S to view the status.

Initiate database server failover

  1. Sign in to the TanOS console of the appliance with the secondary database as a user with the tanadmin role.
  2. Enter 3-3-F (Tanium Support > Database Operations > Database Server Failover).

  3. Follow the prompts to initiate the failover.

Monitor performance

Use the Performance Monitoring menu to view resource usage.

Run a SAR command

  1. Sign in to the TanOS console of the appliance with the primary database server as a user with the tanadmin role.
  2. Enter 3-P-1 (Tanium Support > Performance Monitoring > SAR command).

  3. Use the menu to issue a command. The results of the command are returned to the screen. ClosedView screen

Export a SAR snapshot

  1. Sign in to the TanOS console of the appliance with the primary database server as a user with the tanadmin role.
  2. Enter 3-P (Tanium Support > Performance Monitoring).

  3. Enter 2 to take a five second snapshot of SAR data and export it to the /outgoing folder. ClosedView screen
  4. Use SFTP to copy the snapshot file from the /outgoing directory on the appliance to your management computer.

Export a SAR performance data

  1. Sign in to the TanOS console of the appliance with the primary database server as a user with the tanadmin role.
  2. Enter 3-P (Tanium Support > Performance Monitoring).

  3. Enter 3 to collect the complete set of SAR data files for the last 30 days and export it to a ZIP file in the /outgoing folder. ClosedView screen
  4. Use SFTP to copy the snapshot file from the /outgoing directory on the appliance to your management computer.

Export all SAR files

  1. Sign in to the TanOS console of the appliance with the primary database server as a user with the tanadmin role.
  2. Enter 3-P (Tanium Support > Performance Monitoring).

  3. Enter 4 to collect all SAR files into a single file and export it to the /outgoing folder. ClosedView screen
  4. Use SFTP to copy the snapshot file from the /outgoing directory on the appliance to your management computer.


Run the iotop command

  1. Sign in to the TanOS console of the appliance as a user with the tanadmin role.
  2. Enter 3-P (Tanium Support > Performance Monitoring).

  3. Enter I to monitor input/ouput usage for the appliance.
  4. Enter Q to return to the Performance Monitoring menu.

Run the perf top command

  1. Sign in to the TanOS console of the appliance as a user with the tanadmin role.
  2. Enter 3-P (Tanium Support > Performance Monitoring).

  3. Enter P to monitor CPU usage for the appliance.
  4. Enter Q to return to the Performance Monitoring menu.

Run the htop command

  1. Sign in to the TanOS console of the appliance as a user with the tanadmin role.
  2. Enter 3-P (Tanium Support > Performance Monitoring).

  3. Enter T to monitor processes for the appliance.
  4. Enter Q to return to the Performance Monitoring menu.

Perform advanced maintenance tasks

Consult with Tanium Support before you use advanced options. For more information, see Support for Tanium Appliances.

Install a firmware update

When you Run the Health Check , you might see messages alerting you to perform a firmware update.

Use the Advanced Maintenance menu to stage and apply firmware updates. On a physical Tanium Appliance, the updates include iDRAC firmware updates, PERC firmware updates, NIC firmware updates, and BIOS firmware updates.

Updating a firmware update is a major task. The process can take from 10-30 minutes, depending on model. Allow the firmware update to complete before attempting any other tasks with the appliance. Do NOT manually power off or reboot the appliance.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B-X-1 (Appliance Maintenance > Advanced Maintenance > Firmware Update).

  3. Follow the prompts to update the iDRAC, PERC, NIC, and BIOS firmware. ClosedView screen

Perform a TanOS reset

The Appliance Reset menu allows you to perform a software reset which erases the Tanium application software and data.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B-X-2 (Appliance Maintenance > Advanced Maintenance > Reset).

  3. Enter 1 to perform a software reset.
  4. Follow the prompts to initiate the reset. ClosedView screen

Re-install ACLs

If you experience issues copying to or from the tancopy/incoming or /outgoing directories, you can use this menu to reapply the access control lists (ACL) for those directories.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B-X (Appliance Maintenance > Advanced Maintenance).

  3. Enter 3 to reapply the ACLs.

View the TanOS partition sync log

This menu only appears if the appliance contains an active partition and an inactive partition. This menu is not available on cloud-based Tanium Appliances.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B-X (Appliance Maintenance > Advanced Maintenance).

  3. Enter 5 to view the TanOS partition sync log file.