Managing bandwidth throttling

Bandwidth throttling overview

You can configure throttles to limit the bandwidth and the number of concurrent connections that Tanium as a Service (TaaS) the Tanium Server or Tanium Zone Server uses to send data to Tanium Clients. In deployments where numerous Tanium Clients connect with TaaS the server at the same time to download sensors and packages, spikes in bandwidth usage might occur. Throttles prevent the spikes from degrading network performance by ensuring that TaaS the server never exceeds a specific bandwidth across your entire network or in specific subnets when sending Tanium data. To enforce the limits, TaaS the server delays sending data that would exceed the maximum bandwidth and rejects connections beyond the allowed maximum number.

Bandwidth throttles control only the rate at which TaaS the Tanium Server or Zone Server sends data to Tanium Clients, not the rate at which the Tanium Clients send data to TaaS the server. The throttles do not affect data exchanges between any other Tanium components.

For the user role permissions to see and manage bandwidth throttle configurations, see Manage bandwidth throttling.

Before you begin

When you configure throttles, strike a balance between providing TaaS the Tanium Server or Zone Server enough resources (bandwidth and concurrent connections) to complete tasks in a reasonable time frame and mitigating the impact of those tasks on your network. Setting limits too low might prevent TaaS the server from sending all the sensors and packages that endpoints need in time to respond to questions and perform actions. Setting limits too high might allow spikes in Tanium traffic to hinder other tasks that the endpoints must perform. Therefore, work with your network administrator and Tanium Technical Account Manager (TAM) to determine the following aspects of your network:

  • Bandwidth trends: Evaluate bandwidth trends for data that TaaS the Tanium Server sends to endpoints. The trends will enable you to gauge how much the traffic affects your network and determine the maximum resources that the server requires for sending the data. You can configure separate throttles for all data that TaaS the server sends and for sensor or package data. Note that the throttles for all data must accommodate every type of outbound data (such as registration information), not just sensor and package data. The best practice is leave the all-data throttle unconfigured so that no limit applies to it. If configuring the all-data throttle is necessary, set it to at least 150 megabits per second (Mbps) above the sum of the sensor and package throttles. For example, if you set the sensors bandwidth throttle to 200 Mbps and the packages throttle to 400 Mbps, set the all-data throttle to 750 Mbps or higher to accommodate all additional data types. Work with your TAM to determine the throttles required for all data types.
  • Site throttles: Determine whether you need site throttles: subnet-specific throttles that are more restrictive than the throttles for the rest of your network. For example, you might want to set a lower bandwidth limit for Tanium traffic in sites that are dedicated to high priority tasks or that experience more non-Tanium traffic. Note that more restrictive throttles override less restrictive ones when multiple throttles apply to the same server-to-endpoint connections. For example, if you set a site-specific throttle to 1 Mbps and the global (network-wide) throttle to 5 Mbps, the server applies the 1 Mbps throttle to the site.
  • Zone Server throttles: In a deployment with Tanium Zone Servers, determine whether they require throttles that differ from the Tanium Servers. For example, the connections between a Zone Server and Tanium Clients in external networks might support and need more or less bandwidth than the connections between the Tanium Server and Tanium Clients in internal networks.
  • Overlapping IP address ranges: Determine whether the sites that require separate throttles have overlapping IP address ranges. Only the throttle for the site with the smallest IP address range applies to an endpoint that has an address within the ranges of multiple sites. For example, the endpoints in subnet 192.168.2.0/26 are a small subset of the endpoints in subnet 192.168.2.0/24. Therefore, the 192.168.2.0/26 site throttle would override the 192.168.2.0/24 site throttle for an endpoint that is in both subnets, such as IP address 192.168.2.1.

Configure global throttles

Configure bandwidth and connection throttles for the data that TaaS the Tanium Server or Zone Server sends to all the endpoints in your network. Repeat these steps for each data type that requires a distinct throttle: all data combined (packages, sensors, and all other types), just package files, or just sensors. By default, global throttles apply to both Tanium Servers and Zone Servers, but you can configure Zone Server-specific throttles if necessary.

  1. From the Main menu, select Administration > Configuration > Tanium Server and click Bandwidth Throttling.
  2. In the Global Throttles section, click Edit beside the data type that you want to throttle.
  3. Enter the maximum bandwidth in Mbps. For a new TaaS instance Tanium Server installation, the default is 0 (no limit) for all data, 45 Mbps for packages, and 45 Mbps for sensors. Existing settings are preserved after upgrades.
  4. Enter the maximum number of concurrent TaasTanium Server-to-endpoint connections. For a new TaaS instance Tanium Server installation, the default is 0 (no limit) for all data, 300 for packages, and 10 for sensors. Existing settings are preserved after upgrades.
  5. Save your changes.
  6. (Zone Server only) For each Zone Server that requires global throttles that differ from those you configured for the Tanium Server, create a JSON file named server-throttles.json. The file must be in the Zone Server installation folder. (For the steps to move the file to the Tanium Appliance, consult your TAM.) The file contents must match the following format. Note that the bandwidth and connection limit values are numbers (not enclosed in quotation marks) and the bandwidth units are bytes per second (Bps). If you want to calculate the equivalent bits per second (bps) rate, multiply the values by a factor of eight. For example, 12.5 MBps equals 100 Mbps.

    // server-throttles.json
    { "data": [{
       "bandwidth_bytes_limit" : <maximum bandwidth for all data>,
       "connection_limit" : <maximum number of concurrent connections for all data>,
       "download_bandwidth_bytes_limit" : <maximum bandwidth for packages>,
       "download_connection_limit" : <maximum number of concurrent connections for packages>,
       "sensor_bandwidth_bytes_limit" : <maximum bandwidth for sensors>,
       "sensor_connection_limit" : <maximum number of concurrent connections for sensors>
    }]}

    When this file is present, the Zone Server does not inherit any global or site throttle settings from the Tanium Server. Any settings that you omit from the file have a default value of 0 (no limit). You do not have to restart the Zone Server service to apply your changes after you add, edit, or delete the file.

Configure site throttles

Configure bandwidth throttles for the data that TaaS the Tanium Server or Zone Server sends to specific Tanium Client subnets. By default, site throttles apply to both Tanium Servers and Zone Servers, but you can configure Zone Server-specific throttles if necessary.

Base throttles on local or NAT-translated IP addresses

When defining sites for bandwidth throttling, you can specify local or NAT-translated IP addresses, but not both. By default, Tanium Servers and Zone Servers treat the IP addresses as NAT-translated. If you need to change this setting:

  1. From the Main menu, select Administration > Management > Global Settings.
  2. Select site_throttles_use_local_ip and click Edit.
  3. Set the value to 0 (NAT IP addresses) or 1 (local IP addresses).
  4. Ensure the setting Affects the Server (Tanium Server or Zone Server) and save your changes.

Add sites

Add a site for each group of Tanium Client subnets that require the same bandwidth throttles for data received from TaaS the Tanium Server.

For subnets that receive data from Zone Servers, you configure the sites when you Add site throttles.

  1. From the Main menu, select Administration > Configuration > Tanium Server and click Bandwidth Throttling.
  2. In the Site Throttles section, click Add Site.

  3. Enter a Site Name to identify the site.

  4. Enter one or more Subnets in CIDR format (such as 192.168.2.0/24 or 2001:db8::/32). Enter one subnet per line.

  5. Select whether to apply throttles for the site to the Total bandwidth shared across all subnets in bundle or to the Individual bandwidth of each subnet in bundle.

  6. Save your changes. The Tanium Console then displays each subnet that you added to the site.

Add site throttles

Configure site-specific bandwidth throttles that apply to all data combined (packages, sensors, and all other types), just package files, or just sensors.

  1. From the Main menu, select Administration > Configuration > Tanium Server and click Bandwidth Throttling.
  2. Scroll down to the Site Throttles section, which has a <site_name> subsection for each site that you added.
  3. For each data type that you want to throttle, click Add in the <site_name> subsection, enter the maximum bandwidth in Mbps (default is 0, which specifies no limit), and save your changes.
  4. (Zone Server only) For each Zone Server that requires site throttles that differ from those you configured for the Tanium Server, create a JSON file named site-throttles.json. The file must be in the Zone Server installation folder. (For the steps to move the file to the Tanium Appliance, consult your TAM.) The file contents must match the following format. Note that the bandwidth and connection limit values are numbers (not enclosed in quotation marks) and the bandwidth units are bytes per second (Bps). If you want to calculate the equivalent bits per second (bps) rate, multiply the values by a factor of eight. For example, 12.5 MBps equals 100 Mbps. This example specifies two sites: one with three subnets and one with two subnets. The all_subnets_flag controls whether the throttles apply individually to each subnet (0) or apply to the total bandwidth that is shared across all the subnets (1).

    // site-throttles.json
    { "data": [
       {
          "all_subnets_flag" : [0 | 1],
          "bandwidth_bytes_limit" : <maximum bandwidth for all data>,
          "download_bandwidth_bytes_limit" : <maximum bandwidth for packages>,
          "sensor_bandwidth_bytes_limit" : <maximum bandwidth for sensors>,
          "subnets" : [
             {
               "range" : "<subnet CIDR>"
             },
             {
               "range" : "<subnet CIDR>"
             },
             {
               "range" : "<subnet CIDR>"
             }
          ]
       },
       {
          "all_subnets_flag" : [0 | 1],
          "bandwidth_bytes_limit" : <maximum bandwidth for all data>,
          "download_bandwidth_bytes_limit" : <maximum bandwidth for packages>,
          "sensor_bandwidth_bytes_limit" : <maximum bandwidth for sensors>,
          "subnets" : [
             {
               "range" : "<subnet CIDR>"
             },
             {
               "range" : "<subnet CIDR>"
             }
          ]
       }
    ]}

    When this file is present, the Zone Server does not inherit any global or site throttle settings from the Tanium Server. Any settings that you omit from the file have a default value of 0 (no limit). You do not have to restart the Zone Server service to apply your changes after you add, edit, or delete the file.

Verify throttle delays

To see the current delays (Queue delay values) that TaaS the Tanium Server applies to enforce the throttles on the data it sends to endpoints:

  1. From the Main menu, select Administration > Configuration > Tanium Server.
  2. Click Bandwidth Throttling and scroll to the type of throttles for which you want to see the delays.

For example, if you set the bandwidth limit for the Global Throttle for All Data to 400 Mbps and TaaS the Tanium Server starts sending 400 megabits of data, the Global Throttle for All Data section initially displays a Queue delay of 1,000 milliseconds (ms). After the download completes, the Queue delay drops to 0 ms until TaaS the Tanium Server sends more data.

The Tanium Console displays queue delays only for the Tanium Server. For the queue delays associated with Zone Servers, consult your TAM.

The Tanium Console uses the following icons to indicate the severity level of a Queue delay. The severity levels indicate the likelihood that the delay will prevent TaaS the Tanium Server from sending all the sensors and packages that endpoints need in time to respond to questions and perform actions.

  • 0 to 9,999 ms: Little or no risk of disrupting Tanium functions.

  • 10,000 to 44,999 ms: Moderate risk of disrupting Tanium functions.

  • 45,000 ms or more: High risk of disrupting Tanium functions.

To see the delays associated with specific subnets within a site, hover over a subnet in the corresponding <site_name> subsection.

Figure  1:  Queue delays associated with bandwidth throttles