Configuring an Elasticsearch destination

With Elasticsearch, you can search, analyze, and get actionable insights in real time from almost any type of structured and unstructured data source. With Connect, Tanium can write data directly to Elasticsearch.

Specify general connection information

  1. On the Connect home page, click Create Connection > Create.
  2. Enter a name and description for your connection.
  3. Enable the connection to run on a schedule.
    Select Enable. You can set up the schedule when you configure the rest of the connection. If the schedule is not enabled, the connection only runs when you manually run it.
  4. (Optional) Set the logging level.
    By default, the logging is set to Information. Set the log level to Trace or Debug if you are debugging the connection. To reduce the amount of logging, you can set the log level to Warning, Error, or Fatal.

Configure the connection source

The connection source determines what data you are sending to the destination. This data is usually information from Tanium, such as a saved question, question log, system status, or event. The settings vary depending on which source you choose.

If you want to use nested JSON for your Elasticsearch data, turn off the Flatten setting in the Advanced Settings for the source. If you leave the data flattened, a single JSON item with all of the rows is returned.

Configure Elasticsearch destination

  1. Name the destination.
    You can either indicate a unique name to save the configuration information as a new destination, or you can select an existing Elasticsearch destination from the list.

  2. Define a URL to access the Elasticsearch API.
    The URL is in the following format:
    <protocol>://<elasticSearchHost>:<elasticSearchPort>/<index>/<type>/_bulk
    where:

    protocol

    Specifies the protocol for accessing the Elasticsearch server. (http or https)

    elasticSearchHost

    Specifies the fully qualified domain name (FQDN) or IP of the Elasticsearch server.

    elasticSearchPort

    Specifies the port on which the Elasticsearch server is listening. (default: 9200)

    index

    Specifies the unique identifier for this API request. The common format for this identifier is: {name}-{0:yyyy.MM.dd}. This date format creates a unique index each day.

    type

    Specifies the data label that identifies the data source type. For example, you might use tanium for Tanium, and syslog for syslog data.


  3. Define the Batch Size, which determines the number of messages to batch before sending data to Elasticsearch.
  4. If you have a proxy configured for your Tanium Module Server, select Use Tanium Module Server Proxy Setting. When selected, the proxy makes the outbound connection to the specified Elasticsearch instance provided in the URL.

Configure filters

(Optional) You can use filters to modify the data that you are getting from your connection source before it is sent to the destination.

For more information about the types of filters you can configure, see Reference: Filtering options.

Format data for Elasticsearch

Choose the columns that you want to use as an index in Elasticsearch. If a saved question is selected as a source, you can expand Advanced Settings and select Nested JSON to return the results as an array of objects instead of an array of simple value types. If you customize the columns, leave the Value Type as Unmodified to get the expected object output for Nested JSON.

You can change the Destination name of each column and Value Type to force the column to be a String, Numeric, or DateTime value. If you choose Numeric for the value, you can specify a default value that is used if the data cannot be coerced into a numeric value. You can specify any negative or positive number. If you choose DateTime for the value, specify the Date/Time format that you want to use for the column.

Schedule the connection

Connections can run at a highly configurable time interval, such as multiple times per hour, day, week, or month.

Update the schedule: 

  • Use the Generate Cron tab to build a schedule based on some common time intervals. This tab generates a Cron expression.
  • To view or edit the Cron expression directly, click the Edit Cron Expression tab.

Save and verify connection

  1. Click Create Connection > Create. When the connection gets created, your new connection displays in the list on the Connections page.
  2. To view details about when the connection is running, click the name of the connection. On the resulting connection details page, click the Runs tab.
  3. To view individual run logs, click the link in the Status column in the Runs table.

Last updated: 9/18/2018 4:07 PM | Feedback