Reference: TanOS backup and recovery

Use this guide to learn about available backup and recovery options in TanOS. Topics include:

Concepts and terminology
TanOS backup options
TanOS backup recommendations
Testing a planned failover
Backup automation
TanOS restore options

For backup procedures, see Maintaining the Tanium Appliance.

Concepts and terminology

There are several concepts and terminology that you need to know to plan for backup and recovery on a Tanium Appliance.

Tanium cluster

You can configure Tanium Appliances into a Tanium cluster, where the Tanium Server is active-active, and the database component is active-passive. The Tanium Servers read and write to the database co-located on the first appliance.

The primary server is the server that hosts the read/write database.
The secondary server contains a read-only database that replicates from the database on the primary server.

For more information on how to use Tanium Appliances with a Tanium cluster, see Installing and managing an Appliance Array.

Partitions (physical Tanium Appliance and virtual Tanium Appliance only)

Each Tanium Appliance contains two partitions, a primary partition and a secondary partition.

A virtual Tanium Appliance contains only one partition by default. You can add a secondary partition to perform a partition sync, or you can take a snapshot of the virtual image.

Boot options (physical Tanium Appliance and virtual Tanium Appliance only)

When the Tanium Appliance boots, you are prompted to select a partition. Your choices are TanOS Active and TanOS Inactive.

The TanOS Active option boots into the active partition. By default, the primary partition is the active partition.
The TanOS Inactive option boots into the inactive partition. By default, the secondary partition is the inactive partition.

Partition swap (physical Tanium Appliance and virtual Tanium Appliance only)

If you have multiple partitions, you can swap partitions through the Active Partition menu (A-X-3). For more information, see Change the active partition.

When you change the active partition, the partition that was set as the inactive partition becomes the active partition, and the partition that was set as the active partition becomes the inactive partition.
For example, in the default configuration, the primary partition is the active partition and the secondary partition is the inactive partition. After you change the active partition, the secondary partition becomes the active partition and the primary partition becomes the inactive partition.

Mounting states (physical Tanium Appliance and virtual Tanium Appliance only)

In a normal mount, the active partition is set to the primary partition (root is mounted on /).
In an inverted mount, the active partition is set to the secondary partition (root is mounted on /altroot).

TanOS backup options

TanOS offers multiple options for backup. You can find the available options in the Backup menu (B-1). The following sections describe the backup options in detail.

Backup options are also available to schedule on a regular basis. See the Backup menu (B-1) for options around this, or refer to Configure and run automatic backups.

On a cloud-based Tanium Appliance or virtual Tanium Appliance, in addition to the backup options that TanOS provides, you can also take a snapshot of the virtual image.

Partition sync (physical Tanium Appliance and virtual Tanium Appliance only)

TanOS can have two partitions: an active partition and an inactive partition in case of failover or troubleshooting. A partition sync is a backup procedure that uses the rsync utility to copy the active partition to the inactive partition.

A virtual Tanium Appliance contains only one partition by default. Appliances with only one partition do not contain the option to perform a partition sync. You can add a secondary partition to perform a partition sync.

Perform a partition sync before you upgrade TanOS or a Tanium Server component, so that you have an alternate partition in case issues occur during the upgrade process or the Tanium Server update. You can also use the inactive partition if the active partition fails to boot. During the TanOS boot process, you have the option to select the inactive partition if needed.

To protect data consistency, the partition sync job disables (shuts down) the Tanium Server, Tanium database server, and other related services for the duration of the partition sync. Make sure to set a partition sync schedule that does not disrupt solution processes.

Core backup

Perform a core backup to back up essential files that can help you quickly recover from failures. Tanium services do not stop during a core backup. For additional information and backup procedures, see Maintaining the Tanium Appliance.

A core backup produces a core recovery bundle, which includes the following content:

Appliance, array and network settings
LDAP database contents (locally managed Tanium accounts)
Critical Tanium Server, Tanium Module Server, and Tanium Zone Server configuration and key material
LDAP and root CA certificates
Tanium Server database contents (primary Tanium Server only)

Sample contents of a core recovery bundle

$ ls -laR
total 40
[email protected] 12 john.doe  staff   384 Nov  2 14:18 .
[email protected] 45 john.doe  staff  1440 Nov  2 14:17 ..
[email protected]  1 john.doe  staff  6148 Nov  2 14:19 .DS_Store
[email protected]  1 john.doe  staff  2238 Nov  2 14:05 README.txt
drwxr-xr-x  11 john.doe  staff   352 Nov  2 14:18 TaniumServer
drwxr-xr-x   4 john.doe  staff   128 Nov  2 14:18 TaniumZoneServer
drwxr-xr-x  13 john.doe  staff   416 Nov  2 14:06 config
drwxr-xr-x   3 john.doe  staff    96 Nov  2 14:05 database
-rwxr-xr-x   1 john.doe  staff  1493 Nov  2 14:05 decrypt-payload.sh
drwxr-xr-x   5 john.doe  staff   160 Nov  2 14:05 enc_ssl
-rw-r--r--   1 john.doe  staff  1067 Nov  2 14:06 manifest.txt

./TaniumServer:
total 288
drwxr-xr-x  11 john.doe  staff    352 Nov  2 14:18 .
[email protected] 11 john.doe  staff    352 Nov  2 14:24 ..
drwxr-x---   4 john.doe  staff    128 Nov  2 10:31 Backup
-r--------   1 john.doe  staff   1111 Oct  9 14:55 SOAPServer.crt
-r--------   1 john.doe  staff   1704 Oct  9 14:55 SOAPServer.key
drwxr-x---   5 john.doe  staff    160 Oct 24 18:10 content_public_keys
-rw-r-----   1 john.doe  staff  90112 Nov  2 14:05 pki.db
-rw-r-----   1 john.doe  staff  20480 Nov  2 10:31 server.db
-rw-r-----   1 john.doe  staff    158 Oct  9 14:55 tanium.pub
-rw-r-----   1 john.doe  staff  20480 Oct  9 14:55 tdownloader.db
-rw-r-----   1 john.doe  staff   1133 Oct  9 15:01 trusted-module-servers.crt

./TaniumServer/Backup:
total 352
drwxr-x---   4 john.doe  staff    128 Nov  2 10:31 .
drwxr-xr-x  11 john.doe  staff    352 Nov  2 14:18 ..
-rw-r-----   1 john.doe  staff  90112 Nov  2 10:31 pki-backup.db
-rw-r-----   1 john.doe  staff  90112 Oct  9 15:12 pki.db

./TaniumServer/content_public_keys:
total 0
drwxr-x---   5 john.doe  staff  160 Oct 24 18:10 .
drwxr-xr-x  11 john.doe  staff  352 Nov  2 14:18 ..
drwxr-x---   3 john.doe  staff   96 Nov  2 10:31 console
drwxr-x---   3 john.doe  staff   96 Nov  2 10:31 content
drwxr-x---   3 john.doe  staff   96 Nov  2 10:31 manifest

./TaniumServer/content_public_keys/console:
total 8
drwxr-x---  3 john.doe  staff   96 Nov  2 10:31 .
drwxr-x---  5 john.doe  staff  160 Oct 24 18:10 ..
-r--r-----  1 john.doe  staff  586 Oct 24 18:10 console-release.pub

./TaniumServer/content_public_keys/content:
total 8
drwxr-x---  3 john.doe  staff   96 Nov  2 10:31 .
drwxr-x---  5 john.doe  staff  160 Oct 24 18:10 ..
-r--r-----  1 john.doe  staff  586 Oct 24 18:10 content-release.pub

./TaniumServer/content_public_keys/manifest:
total 8
drwxr-x---  3 john.doe  staff   96 Nov  2 10:31 .
drwxr-x---  5 john.doe  staff  160 Oct 24 18:10 ..
-r--r-----  1 john.doe  staff  586 Oct 24 18:10 manifest-release.pub

./TaniumZoneServer:
total 208
drwxr-xr-x   4 john.doe  staff    128 Nov  2 14:18 .
[email protected] 11 john.doe  staff    352 Nov  2 14:24 ..
-rw-r-----   1 john.doe  staff  86016 Nov  2 14:05 pki.db
-rw-r-----   1 john.doe  staff  20480 Nov  2 10:31 zoneserver.db

./config:
total 104
drwxr-xr-x  13 john.doe  staff   416 Nov  2 14:06 .
[email protected] 11 john.doe  staff   352 Nov  2 14:24 ..
-rw-r--r--   1 john.doe  staff    36 Nov  2 14:06 .tanium-user-key-encryption.key
-rw-r--r--   1 john.doe  staff   171 Nov  2 14:05 TaniumServer-TDL-config.txt
-rw-r--r--   1 john.doe  staff   994 Nov  2 14:05 TaniumServer-config.txt
-rw-r--r--   1 john.doe  staff   122 Nov  2 14:05 TaniumZoneServer-config.txt
-r-xr-xr-x   1 john.doe  staff  8041 Nov  2 14:05 appliance_array.json
drwxr-xr-x   3 john.doe  staff    96 Nov  2 14:06 certs
-rw-r--r--   1 john.doe  staff  1299 Nov  2 14:05 fstab
-rw-r--r--   1 john.doe  staff   126 Nov  2 14:05 ifcfg-ens160
-rw-r--r--   1 john.doe  staff    78 Nov  2 14:05 ifcfg-lo
-rw-r--r--   1 john.doe  staff  4701 Nov  2 14:06 slapd.ldif
-rw-r--r--   1 john.doe  staff  4563 Nov  2 14:05 tanos_environment

./config/certs:
total 0
drwxr-xr-x   3 john.doe  staff   96 Nov  2 14:06 .
drwxr-xr-x  13 john.doe  staff  416 Nov  2 14:06 ..
drwxr-xr-x   3 john.doe  staff   96 Nov  2 14:06 root

./config/certs/root:
total 16
drwxr-xr-x  3 john.doe  staff    96 Nov  2 14:06 .
drwxr-xr-x  3 john.doe  staff    96 Nov  2 14:06 ..
-rw-r--r--  1 john.doe  staff  7732 Nov  2 14:06 redhat-uep.pem

./database:
total 7360
drwxr-xr-x   3 john.doe  staff       96 Nov  2 14:05 .
[email protected] 11 john.doe  staff      352 Nov  2 14:24 ..
-rw-r--r--   1 john.doe  staff  3764562 Nov  2 14:06 tsdb.pgdump

./enc_ssl:
total 16
drwxr-xr-x   5 john.doe  staff  160 Nov  2 14:05 .
[email protected] 11 john.doe  staff  352 Nov  2 14:24 ..
-rw-r--r--   1 john.doe  staff   33 Nov  2 14:05 iv.txt
drwxr-xr-x   3 john.doe  staff   96 Nov  2 14:05 pubkeys
-rw-r--r--   1 john.doe  staff  384 Nov  2 14:05 symmetric.enc.24cc7840d640bbcc6a15a756c49428e50323c2d9

./enc_ssl/pubkeys:
total 8
drwxr-xr-x  3 john.doe  staff   96 Nov  2 14:05 .
drwxr-xr-x  5 john.doe  staff  160 Nov  2 14:05 ..
-rw-r--r--  1 john.doe  staff  624 Nov  2 14:05 24cc7840d640bbcc6a15a756c49428e50323c2d9.pub

Beginning with TanOS 1.6.3, a core backup replaces a minimal backup and a Tanium database backup. If you previously scheduled a Tanium database backup through TanOS, the schedule is reused by the core backup after upgrade.

A core backup is recommended in most situations for the Tanium Servers. You should run the core backup on each of your Tanium Servers. The Tanium Module Server core backup only includes the files to restore the appliance configuration; it does not include any module data. If you are using a Standby Tanium Module Server in your environment, the core backup is sufficient. Otherwise, consider a comprehensive backup instead.

You have the option to transfer the backup file to a remote location using SCP, or to copy the backup into the /outgoing directory for manual collection.

Comprehensive backup

Perform a comprehensive backup to produce a comprehensive recovery bundle. A comprehensive recovery bundle includes the same content as a core recovery bundle, in addition to Tanium Server downloads and module data:

Appliance, array and network settings
LDAP database contents (locally managed Tanium accounts)
Critical Tanium Server, Tanium Module Server, and Tanium Zone Server configuration and key material
LDAP and root CA certificates
Tanium Server database contents (primary Tanium Server only)
Tanium Server downloads
Module data

Sample contents of a comprehensive recovery bundle

$ ls -laR
total 40
[email protected] 11 john.doe  staff   352 Nov  2 14:31 .
[email protected] 46 john.doe  staff  1472 Nov  2 14:29 ..
[email protected]  1 john.doe  staff  6148 Nov  2 14:30 .DS_Store
-rw-r--r--   1 john.doe  staff  2238 Nov  2 14:06 README.txt
drwxr-xr-x  12 john.doe  staff   384 Nov  2 14:31 TaniumServer
drwxr-xr-x   4 john.doe  staff   128 Nov  2 14:31 TaniumZoneServer
drwxr-xr-x  13 john.doe  staff   416 Nov  2 14:06 config
drwxr-xr-x   3 john.doe  staff    96 Nov  2 14:06 database
-rwxr-xr-x   1 john.doe  staff  1493 Nov  2 14:06 decrypt-payload.sh
drwxr-xr-x   5 john.doe  staff   160 Nov  2 14:06 enc_ssl
-rw-r--r--   1 john.doe  staff  1090 Nov  2 14:07 manifest.txt

./TaniumServer:
total 288
drwxr-xr-x   12 john.doe  staff    384 Nov  2 14:31 .
[email protected]  11 john.doe  staff    352 Nov  2 14:31 ..
drwxr-x---    4 john.doe  staff    128 Nov  2 10:31 Backup
drwxr-x---  999 john.doe  staff  31968 Nov  2 11:59 Downloads
-r--------    1 john.doe  staff   1111 Oct  9 14:55 SOAPServer.crt
-r--------    1 john.doe  staff   1704 Oct  9 14:55 SOAPServer.key
drwxr-x---    5 john.doe  staff    160 Oct 24 18:10 content_public_keys
-rw-r-----    1 john.doe  staff  90112 Nov  2 14:05 pki.db
-rw-r-----    1 john.doe  staff  20480 Nov  2 10:31 server.db
-rw-r-----    1 john.doe  staff    158 Oct  9 14:55 tanium.pub
-rw-r-----    1 john.doe  staff  20480 Oct  9 14:55 tdownloader.db
-rw-r-----    1 john.doe  staff   1133 Oct  9 15:01 trusted-module-servers.crt

./TaniumServer/Backup:
total 352
drwxr-x---   4 john.doe  staff    128 Nov  2 10:31 .
drwxr-xr-x  12 john.doe  staff    384 Nov  2 14:31 ..
-rw-r-----   1 john.doe  staff  90112 Nov  2 10:31 pki-backup.db
-rw-r-----   1 john.doe  staff  90112 Oct  9 15:12 pki.db

./TaniumServer/Downloads:
total 1815088
drwxr-x---  999 john.doe  staff      31968 Nov  2 11:59 .
drwxr-xr-x   12 john.doe  staff        384 Nov  2 14:31 ..
-rw-r-----    1 john.doe  staff         56 Nov  2 11:50 005a3cd39f819b3392cc39f99e62f7831c1011c95d971f9c23d8cbadc00642a9
-rw-r-----    1 john.doe  staff      15738 Nov  2 11:59 00dbf6a76715a7c3d4005b7ca451d23985a6e060300b24129f7a7933c73a3fb1
-rw-r-----    1 john.doe  staff       4068 Nov  2 11:29 027e5bf6248a58f3f73e6f51a72bd9e0d21cf6de75c6e6cf27ead0b62514fe52
drwxr-x---  281 john.doe  staff       8992 Nov  2 11:57 DownloadAttempts
-rw-r-----    1 john.doe  staff        269 Oct  9 15:08 DownloadProgress_1
-rw-r-----    1 john.doe  staff         76 Oct  9 15:08 DownloadProgress_1.modified
-rw-r-----    1 john.doe  staff        205 Oct  9 15:08 DownloadProgress_10
-rw-r-----    1 john.doe  staff        269 Nov  2 11:48 DownloadProgress_100
-rw-r-----    1 john.doe  staff         85 Nov  2 11:15 DownloadProgress_100.error
-rw-r-----    1 john.doe  staff         76 Nov  2 11:48 DownloadProgress_100.modified
-rw-r-----    1 john.doe  staff        269 Nov  2 11:48 DownloadProgress_101
-rw-r-----    1 john.doe  staff         85 Nov  2 11:15 DownloadProgress_101.error
-rw-r-----    1 john.doe  staff         76 Nov  2 11:48 DownloadProgress_101.modified
-rw-r-----    1 john.doe  staff      13461 Nov  2 11:37 a06ce519c0b087489a32c4d4cac24fbc914ec7ca085b0dca53fd49a222e5cf1d
-rw-r-----    1 john.doe  staff      99192 Nov  2 11:56 a0cfcbfcfbf9d28f07e10c209df01b35e48e65ab77b3985eb30eb74b7112f0e6
-rw-r-----    1 john.doe  staff        262 Nov  2 11:37 a144a623d5d8019c8628d020189a74df67d4a06c5d43bf8b3f9dbbf70f9bf4f7
-rw-r-----    1 john.doe  staff  102047744 Nov  2 11:18 tdl-crls.db
drwxr-x---    2 john.doe  staff         64 Nov  2 11:59 tmp
-rw-r-----    1 john.doe  staff         68 Nov  2 10:31 upload_hosts.txt

./TaniumServer/Downloads/DownloadAttempts:
total 2232
drwxr-x---  281 john.doe  staff   8992 Nov  2 11:57 .
drwxr-x---  999 john.doe  staff  31968 Nov  2 11:59 ..
-rw-r-----    1 john.doe  staff    257 Oct  9 15:08 1.json
-rw-r-----    1 john.doe  staff    266 Oct  9 15:04 10.json
-rw-r-----    1 john.doe  staff    268 Oct  9 15:06 11.json
-rw-r-----    1 john.doe  staff    284 Oct  9 15:06 12.json
-rw-r-----    1 john.doe  staff    284 Oct  9 15:06 13.json
-rw-r-----    1 john.doe  staff    278 Oct  9 15:06 14.json
-rw-r-----    1 john.doe  staff    274 Oct  9 15:06 15.json
-rw-r-----    1 john.doe  staff    276 Oct  9 15:06 16.json
-rw-r-----    1 john.doe  staff    280 Oct  9 15:06 17.json
-rw-r-----    1 john.doe  staff    293 Oct  9 15:09 18.json
-rw-r-----    1 john.doe  staff    268 Oct  9 15:09 19.json
-rw-r-----    1 john.doe  staff    265 Oct  9 15:08 2.json
-rw-r-----    1 john.doe  staff    266 Oct  9 15:08 20.json
-rw-r-----    1 john.doe  staff    276 Oct  9 15:08 21.json
-rw-r-----    1 john.doe  staff    286 Oct  9 15:08 22.json
-rw-r-----    1 john.doe  staff    283 Oct  9 15:08 23.json
-rw-r-----    1 john.doe  staff    274 Oct  9 15:08 24.json
-rw-r-----    1 john.doe  staff    279 Oct  9 15:08 25.json
-rw-r-----    1 john.doe  staff    277 Oct  9 15:08 26.json
-rw-r-----    1 john.doe  staff    268 Oct  9 15:08 27.json
-rw-r-----    1 john.doe  staff    286 Oct  9 15:08 28.json
-rw-r-----    1 john.doe  staff    284 Oct  9 15:08 29.json
-rw-r-----    1 john.doe  staff    271 Oct  9 15:08 3.json
-rw-r-----    1 john.doe  staff    269 Oct  9 15:08 30.json
-rw-r-----    1 john.doe  staff    268 Oct  9 15:08 31.json
-rw-r-----    1 john.doe  staff    258 Oct  9 15:08 32.json
-rw-r-----    1 john.doe  staff    271 Oct  9 15:08 33.json
-rw-r-----    1 john.doe  staff    268 Oct  9 15:08 4.json
-rw-r-----    1 john.doe  staff    266 Oct  9 15:08 43.json
-rw-r-----    1 john.doe  staff    260 Oct  9 15:08 5.json
-rw-r-----    1 john.doe  staff    280 Oct  9 15:08 6.json
-rw-r-----    1 john.doe  staff    276 Oct  9 15:08 7.json
-rw-r-----    1 john.doe  staff    276 Oct  9 15:08 8.json
-rw-r-----    1 john.doe  staff    263 Oct  9 15:08 9.json

./TaniumServer/Downloads/tmp:
total 0
drwxr-x---    2 john.doe  staff     64 Nov  2 11:59 .
drwxr-x---  999 john.doe  staff  31968 Nov  2 11:59 ..

./TaniumServer/content_public_keys:
total 0
drwxr-x---   5 john.doe  staff  160 Oct 24 18:10 .
drwxr-xr-x  12 john.doe  staff  384 Nov  2 14:31 ..
drwxr-x---   3 john.doe  staff   96 Nov  2 10:31 console
drwxr-x---   3 john.doe  staff   96 Nov  2 10:31 content
drwxr-x---   3 john.doe  staff   96 Nov  2 10:31 manifest

./TaniumServer/content_public_keys/console:
total 8
drwxr-x---  3 john.doe  staff   96 Nov  2 10:31 .
drwxr-x---  5 john.doe  staff  160 Oct 24 18:10 ..
-r--r-----  1 john.doe  staff  586 Oct 24 18:10 console-release.pub

./TaniumServer/content_public_keys/content:
total 8
drwxr-x---  3 john.doe  staff   96 Nov  2 10:31 .
drwxr-x---  5 john.doe  staff  160 Oct 24 18:10 ..
-r--r-----  1 john.doe  staff  586 Oct 24 18:10 content-release.pub

./TaniumServer/content_public_keys/manifest:
total 8
drwxr-x---  3 john.doe  staff   96 Nov  2 10:31 .
drwxr-x---  5 john.doe  staff  160 Oct 24 18:10 ..
-r--r-----  1 john.doe  staff  586 Oct 24 18:10 manifest-release.pub

./TaniumZoneServer:
total 208
drwxr-xr-x   4 john.doe  staff    128 Nov  2 14:31 .
[email protected] 11 john.doe  staff    352 Nov  2 14:31 ..
-rw-r-----   1 john.doe  staff  86016 Nov  2 14:06 pki.db
-rw-r-----   1 john.doe  staff  20480 Nov  2 10:31 zoneserver.db

./config:
total 104
drwxr-xr-x  13 john.doe  staff   416 Nov  2 14:06 .
[email protected] 11 john.doe  staff   352 Nov  2 14:31 ..
-rw-r--r--   1 john.doe  staff    36 Nov  2 14:06 .tanium-user-key-encryption.key
-rw-r--r--   1 john.doe  staff   171 Nov  2 14:06 TaniumServer-TDL-config.txt
-rw-r--r--   1 john.doe  staff   994 Nov  2 14:06 TaniumServer-config.txt
-rw-r--r--   1 john.doe  staff   122 Nov  2 14:06 TaniumZoneServer-config.txt
-r-xr-xr-x   1 john.doe  staff  8041 Nov  2 14:06 appliance_array.json
drwxr-xr-x   3 john.doe  staff    96 Nov  2 14:06 certs
-rw-r--r--   1 john.doe  staff  1299 Nov  2 14:06 fstab
-rw-r--r--   1 john.doe  staff   126 Nov  2 14:06 ifcfg-ens160
-rw-r--r--   1 john.doe  staff    78 Nov  2 14:06 ifcfg-lo
-rw-r--r--   1 john.doe  staff  4701 Nov  2 14:06 slapd.ldif
-rw-r--r--   1 john.doe  staff  4563 Nov  2 14:06 tanos_environment

./config/certs:
total 0
drwxr-xr-x   3 john.doe  staff   96 Nov  2 14:06 .
drwxr-xr-x  13 john.doe  staff  416 Nov  2 14:06 ..
drwxr-xr-x   3 john.doe  staff   96 Nov  2 14:06 root

./config/certs/root:
total 16
drwxr-xr-x  3 john.doe  staff    96 Nov  2 14:06 .
drwxr-xr-x  3 john.doe  staff    96 Nov  2 14:06 ..
-rw-r--r--  1 john.doe  staff  7732 Nov  2 14:06 redhat-uep.pem

./database:
total 7360
drwxr-xr-x   3 john.doe  staff       96 Nov  2 14:06 .
[email protected] 11 john.doe  staff      352 Nov  2 14:31 ..
-rw-r--r--   1 john.doe  staff  3764519 Nov  2 14:06 tsdb.pgdump

./enc_ssl:
total 16
drwxr-xr-x   5 john.doe  staff  160 Nov  2 14:06 .
[email protected] 11 john.doe  staff  352 Nov  2 14:31 ..
-rw-r--r--   1 john.doe  staff   33 Nov  2 14:06 iv.txt
drwxr-xr-x   3 john.doe  staff   96 Nov  2 14:06 pubkeys
-rw-r--r--   1 john.doe  staff  384 Nov  2 14:06 symmetric.enc.24cc7840d640bbcc6a15a756c49428e50323c2d9

./enc_ssl/pubkeys:
total 8
drwxr-xr-x  3 john.doe  staff   96 Nov  2 14:06 .
drwxr-xr-x  5 john.doe  staff  160 Nov  2 14:06 ..
-rw-r--r--  1 john.doe  staff  624 Nov  2 14:06 24cc7840d640bbcc6a15a756c49428e50323c2d9.pub

Beginning with TanOS 1.6.3, a comprehensive backup replaces a full backup. If you previously scheduled a full backup through TanOS, the schedule is reused by the comprehensive backup after upgrade. You should re-evaluate your need for a comprehensive backup; for Tanium Servers, the core backup might meet your needs. The comprehensive backup is more commonly used if you have a single Tanium Module Server in your environment.

You can transfer the backup file to a remote location using SCP, or to drop the backup into the /outgoing directory for manual collection.

A comprehensive backup stops services on the Tanium Module Server to capture the module data. During this time, users can still access Tanium and ask questions, but the module workbenches are unavailable until the backup completes.

Backup exclusions

Generally, any TanOS configuration that is specific to that appliance. This includes:
- Auth key file (SSH keys)
- FQDN, IP, routes
TanOS local user accounts and their account policies (such as password policy)
Scheduled jobs (such as backups)

TanOS backup recommendations

The following section describes standard guidance for backups. You can reference these as standard guidelines. Your backup strategy might differ based on the specific needs of your organization. For additional information and procedures backup procedures, see Maintaining the Tanium Appliance.

Define a disaster recovery plan

Tanium recommends that you define a disaster recovery (DR) plan early during deployment. The restoration and backup options that fit your disaster recovery plan vary depending on your specific needs and requirements.

To properly define a DR plan, define specific restore time objectives (RTO) and restore point objectives (RPO).

RPO: The expected point of recovery. For example, an RPO of 1 month means recovery point within 30 days of the time of failure.
RTO: What is the amount of downtime that is acceptable to recover the system.

Your frequency for backup and recovery might vary depending on your RPO and RTO.

General recommendations

Tanium recommends performing a partition sync on a physical Tanium Appliance or snapshot on a cloud-based Tanium Appliance or virtual Tanium Appliance before any major changes, including:

TanOS upgrades
Tanium Server upgrades
Tanium Module upgrades

Use the automated backup features available in TanOS to automate many of the backup options. For detailed steps, see Configure and run automatic backups.

Backing up a cloud-based Tanium Appliance

For cloud-based Tanium Appliance deployments, Tanium recommends that you back up the Tanium Appliance image file. You can use these image files to restore the Tanium Appliance to the specific time the backup occurred. This is the preferred method to restore the entire appliance after a major failure. Tanium recommends that you back up the image file once a day. You should store as many copies as you deem necessary and in accordance with your company’s RPO.

There are many third party tools on the market that you can use to back up the image file. Contact your virtual infrastructure team to determine if your organization already uses one of these solutions. Tanium does not allow third party backup agents on the appliance; however, open-vm-tools is preinstalled to provide a method for applications to quiesce the file system before taking the snapshots that are used to save the image to persistent storage.

If you are not able to perform image-based backups and you do not use a Tanium cluster, Tanium recommends that you take a snapshot of the virtual image every two weeks, and monthly comprehensive backups for all Tanium Servers.

For environments with Tanium deployed in a Tanium cluster, Tanium recommends monthly snapshots and quarterly comprehensive backups. You can restore the Tanium Server from a secondary Tanium Server in a Tanium cluster with minimal downtime, and the comprehensive backup is less critical in this scenario. It is only necessary to perform the comprehensive backup on the secondary Tanium Server of the cluster to reduce downtime.

Backing up a virtual Tanium Appliance

For virtual Tanium Appliance deployments, Tanium recommends that you back up the virtual Tanium Appliance image file. You can use these image files to restore the Tanium Appliance to the specific time the backup occurred; this is the preferred method to restore the entire appliance after a major failure. Tanium recommends that you back up the image once a day. You should store as many copies as you deem necessary and in accordance with your company’s RPO.

There are a number of third party tools on the market that you can use to back up the image file. Contact your virtual infrastructure team to determine if your organization already uses one of these solutions. Tanium does not allow third party backup agents on the appliance; however, open-vm-tools is preinstalled to provide a method for applications to quiesce the file system before taking the snapshots that are used to save the image to persistent storage.

If you cannot perform image-based backups and you do not use a Tanium cluster, Tanium recommends a daily core backup for the Tanium Server and a weekly comprehensive backup for your Tanium Module Server. For additional protection, you can optionally enable the alternate partition and run a partition sync as needed.

In a Tanium cluster, you should schedule a core backup of both Tanium Servers to protect your deployment. If you have a single Tanium Module Server, schedule a comprehensive backup of that server. If you have a Standby Module Server that is actively syncing, a core backup of the Tanium Module Server is sufficient.

Backing up a physical Tanium Appliance

For physical Tanium Appliances that do not use Tanium Servers in a cluster, Tanium recommends weekly partition syncs, a daily core backup for the Tanium Server, and a weekly comprehensive backup for the Tanium Module Server.

In a Tanium cluster, Tanium recommends that you schedule a core backup of both Tanium Servers to protect your deployment. If you have a single Tanium Module Server, schedule a comprehensive backup of that server. If you have a Standby Module Server that is actively syncing, a core backup of the Tanium Module Server is sufficient.

Backup automation

As a best practice, use the automated backup options that are available in TanOS. For detailed steps, see Configure and run automatic backups.

To fully automate the transfer of backup files to an off-box location, you must set up an SCP destination that the appliance can access.

Testing a planned failover

As part of a planned failover exercise, you might want to failover to your secondary appliance and then fail back.

Before you begin any planned failover activities, check that database replication is healthy by running a health check (3 -5) on both appliances.

In the following scenario, TS1 is a TanOS appliance with the primary database and TS2 is a TanOS appliance with a replicated database from TS1.

Follow the planned failover steps:

On TS1, stop the Tanium Server service (2-1-4-2
).
On TS2, initialize database failover (2-B-A).
On TS1, reinitialize replication (2-B-B).

Follow the planned failback steps:

On TS2, stop the Tanium Server service (2-1-4-2).
On TS1, initialize database failover (2-B-A).
On TS2, reinitialize replication (2-B-B).

TanOS restore options

The following section details specific options for Tanium Appliance restoration. Review each option sequentially to determine the correct procedures to follow for your restoration.

These steps are not designed to be run without the assistance of Tanium Support. Contact Tanium Support before you run any steps in this section. For more information, see Support for Tanium Appliances.

Restore TanOS from a partition sync (physical Tanium Appliance or virtual Tanium Appliance only)

The quickest way to restore to a known good restore point is to use the partition sync feature on TanOS.

Prerequisites

A previous partition sync that is within your RPO
The password that was used on the previous partition sync (if the TanOS password was changed after partition sync)
Access to a user with the tanadmin role

Notes

Your partition syncs for each Tanium Server and the Module Server must be initiated within 30 minutes of each other (to minimize configuration drift between the Tanium Server and the Tanium Module Server).

Steps

Sign in to the TanOS console as a user with the tanadmin role. Make sure the appliance is booted to the primary partition in normal mounting mode.
Initiate a partition swap (A-X-3). TanOS automatically reboots.
Boot into the TanOS Active partition. You are now in inverted mounting mode.
Perform a partition sync (B-1-P). This syncs your secondary partition (which is active) to your primary partition (inactive).

Changes on the primary partition are overwritten.

Initiate a partition swap (A-X-3). TanOS automatically reboots.
Repeat these steps for all Tanium Servers and Tanium Module Servers in the cluster.
- Start with all Tanium Servers followed by the Tanium Module Servers.

Restore TanOS using a VM image or snapshot (cloud-based Tanium Appliance or virtual Tanium Appliance only)

These steps are only available for cloud-based Tanium Appliances or virtual Tanium Appliances as these steps require an available snapshot (or image) for restoration.

Prerequisites

A known good image or snapshot for appliance restore.
The restore point must have Tanium Module Server and Tanium Server images that are at the same time.

Steps

Turn off all appliances (Tanium Server, Tanium Module Server).
Restore the appliances to the image or snapshot state.
Start the Tanium Servers first, followed by the Tanium Module Server.
Perform your standard checkout steps, including the following:
1. Import/reimport/upgrade a module (upgrade is okay in this state instead of a reimport).
2. Make sure modules can load without errors.
3. Make sure module service accounts are present.
4. Make sure plugin schedules are set to running and not disabled.

Recover a failed member of a redundant TS cluster

If you have a Tanium cluster set up with multiple Tanium Servers, you can perform the following steps to recover the cluster.

Prerequisites

One member of the cluster must be in good working order.
Replication and communication must be healthy within the cluster prior to failure.

(Primary server failure only) Initiate the Database server failover

If the primary Tanium Server is down, you must perform the following steps to restore Tanium functionality:

Sign in to the TanOS console of the appliance with the secondary database as a user with the tanadmin role.
Enter 3 to go to the Tanium Support menu.
Enter 3 to go to the Database Operations menu.
Enter F to go to the Database Server Failover menu.
Follow the prompts to initiate the failover.

This procedure effectively promotes this Tanium Server to primary. Deploy a new secondary server by following the steps in Restore a secondary Tanium Server.

Restore a secondary Tanium Server

If your secondary Tanium Server is no longer available and your environment is operational but in a reduced redundancy state, perform the following steps to restore a secondary Tanium Server to the cluster:

Set up a new appliance. See Getting started.
Conduct initial configuration:
- For a physical Tanium Appliance, see Completing the initial setup (physical Tanium Appliance).
- For a virtual Tanium Appliance, see Completing the initial setup (virtual Tanium Appliance).
- For a cloud-based Tanium Appliance, see Completing the initial setup (cloud-based Tanium Appliance).
Use the same FQDN and IP as the backup to make restoration easier.
Add the newly configured appliance to the array. See Add members to the array.
Assign the appliance the Tanium Server role and install new roles. See Assign roles and Install Tanium roles.

Make sure that you match the platform version of the primary server.
In a web browser, sign in to the Tanium Console on the new appliance and reimport all modules to ensure they match the same version as the primary Tanium Server. Be careful not to accidentally upgrade modules on the secondary Tanium Server.

After each module import, verify that the module loads and that service accounts and other settings are restored.
Manually reconfigure TanOS settings and backup schedules as necessary.

If you are using an LDAPS or StartTLS configuration, the configuration and stored root CA certificates are automatically copied to the new secondary Tanium Server when you add it to the array and install the role.
Verify the deployment on the new appliance. See Tanium Core Platform Deployment Guide for Windows: Verifying the deployment.

Recover a failed Module Server with standby Module Server

Perform the following procedure to recover a failed Module Server with a standby (inactive) Module Server.

Prerequisites

The most recent TMS sync must be successful and meet your RPO.

Steps

Promote the standby Module Server to active. See Promote the standby Module Server.
Make sure plugin schedules are in place and perform standard checkout steps:
1. Import/reimport/upgrade a module. It is okay to upgrade modules in this state.
2. Make sure modules can load without errors.
3. Make sure package files are cached.
4. Make sure packages, sensors, users, computer groups, and other objects are restored.
Your environment is now fully operational.
Deploy a new standby Module Server:
1. Deploy a new appliance:
  - For a physical Tanium Appliance, see Completing the initial setup (physical Tanium Appliance).
  - For a virtual Tanium Appliance, see Completing the initial setup (virtual Tanium Appliance).
  - For a cloud-based Tanium Appliance, see Completing the initial setup (cloud-based Tanium Appliance).
2. Add the appliance to the array. See Add members to the array.
3. Assign the appliance the Tanium Module Server role and install new roles. See Assign roles and Install Tanium roles.

Full restoration to a new appliance or file-level restoration to an existing appliance

You can perform a full restoration of a Tanium Core Platform server from a core or comprehensive recovery bundle, or in some cases you can perform a file-level restoration for limited impact. However, the necessary steps for either of these restorations types vary based on your environment, and these procedures require assistance from Tanium Support. For help with either of these restoration types, contact Tanium Support.