Indexing file systems
Tanium Index 2.3.3.0003
Use Tanium™ Index to index the local file systems on Tanium Client endpoints that are running Windows, Linux, and Mac OS X operating systems. Index is optimized to minimize endpoint resource utilization and work with journaling file systems, when available. The solution indexes local file systems, computes file hashes, and gathers file attributes and magic numbers. This information is recorded in an SQLite database for detection and reporting of threat indicators for files at rest.
Index creates and maintains an inventory of the file system on an individual endpoint with the following steps:
The file system inventory is saved in the SQLite database on the endpoint.
On Windows endpoints, Index uses the Master File Table (MFT) for the initial inventory. Index uses the NTFS journal for Windows endpoints by default to do a full, fast, and efficient inventory pass on all fixed drives on the endpoint by drive letter.
On Linux endpoints, Index uses real-time event recorder data from Tanium™ Trace to collect file events. If Trace is not available on the endpoint, a platform-independent method is used to index the file system on the endpoint.
Mac OS X
On Mac OS X endpoints, a platform-independent method is used to index the file system on the endpoint.
Any new file system changes are captured in the database.
On Windows systems, after the initial indexing is complete, changes are detected within a few seconds of when the change occurs if the drive is in journaled mode. If the drive is not journaled, Index operates in platform-independent mode, and changes are detected during the next indexing pass.
Index starts an asynchronous thread that checks the NTFS journal for changes, and updates the inventory in the database. If a file is modified, any existing saved hashes for that file are removed. When a file creation or modification is detected, the file is indexed to include the file name, file size, file creation time, file modification time, and directory name.
For files that have one or more NTFS hard links, Index records all hard links by the associated file reference number. This ensures that even if a hard link is removed, the remaining hard links are still in the database.
After initial indexing, Index does not detect changes made to only the attributes of a file, such as creation or modification timestamps. If the contents of a file are modified, Index records the new file modification time stamp, but does not update the file creation time stamp.
If Tanium Trace packages are deployed to the endpoint, Index queries the file event information that has been collected in the Trace database.
If Trace is not available, Index uses the platform-independent indexing method. With the platform-independent indexing, changes take longer to pick up because Index gets file changes by traversing the directory tree. Instead of tracking only changed files, the entire index gets rebuilt before each hashing pass.
Mac OS X
With the platform-independent indexing method, changes take longer to pick up because Index gets file changes by traversing the directory tree. Instead of tracking only changed files, the entire index gets rebuilt before each hashing pass.
After the initial inventory of the file system is complete, Index computes and stores the hashes of files in the database. The file hashes are not immediately computed when a file is indexed. The Index hashing thread sleeps for the configured rescan interval. This interval is one hour by default. When the thread wakes up, it calculates hash values for files in the database that do not have hashes.
Index supports the computation of three different hash types: MD5, SHA-1, or SHA-256. By default, only MD5 hashes are calculated. Index can be configured to compute a mix of hash types. The hash columns in the database are cleared whenever a change in the file is detected. You can disable calculation of hashes if necessary.
The magic number is the first 4 bytes of the file. You can use the magic number to identify the file type. Magic numbers are recorded during the hashing pass for files that do not have a magic number entry.
- Windows Workstation: Windows XP with SP3 and later
- Windows Server: Windows Server 2003 with SP2 and later
- Linux: See Tanium Client Deployment Guide: Host system requirements (with Index 2.0.0 and later)
- Mac OS X: Mac OS X 10.8 and later (with Index 18.104.22.168 and later)
For Index to start, a minimum of 1 GB of free space must be available on the drive where Tanium Client is installed.
How much space the Index installation uses varies depending on how much space is used on the local disks that are being indexed. The actual space that is required for the Index database is proportional to the number of files and directories on the local disks and what hashes are configured. For a rough estimate, the Index database uses approximately 1 MB of space for each 1 GB of drive space that is used.
For more information about calculating the amount of space that is required for the Index database, ask your TAM.
Index monitors the CPU usage on the endpoint and throttles if needed. Indexing pauses if the calculated CPU usage exceeds the configured CPU usage limit value during the file system inventory and computation of hashes. Pausing indexing ensures that the overall CPU utilization averages out to the defined CPU usage limit. However, the CPU usage might briefly be higher than the limit.
The CPU usage is set via the CPUUsageLimit parameter in the Index configuration file. We suggest setting it to 3-5%. When the limit is reached or exceeded, the Index process pauses using a dynamic and adaptive algorithm based on the process run time and configured CPU usage limit value.
- The endpoints must have Tanium Client installed. For more information, see Tanium Client Deployment Guide.
- Install the Tanium Index solution. For more information, see Install Index. If you are upgrading, see Upgrading Incident Response.
- (Linux endpoints) For live file event monitoring with Trace, the endpoint packages for Trace must have file event recording enabled. Check for the following sensor results:
- Run the Tanium Trace Status sensor and verify that it returns No issues found.
- Run the Tanium Trace Endpoint Filters sensor and verify that no file events are listed.
- Exempt the following process from antivirus or other host-based security solutions: <Tanium Client>\Tools\EPI\TaniumEndpointIndex.exe. For more information about AV exclusions for Tanium, see Tanium Support Knowledge Base: Security Software Exceptions (login required).
Deploy the latest Tanium endpoint Index tools to the appropriate endpoints with a scheduled action. You must target the endpoints by operating system. One way to target by operating system is to create OS-specific computer groups.
The actions that deploy Index to the endpoints are disabled by default.
- From the Main menu, click Actions > Scheduled Actions.
- Select the appropriate tool deployment action and click Edit.
- Deploy Distribute Tanium Endpoint Index Tools
- Deploy Distribute Tanium Endpoint Index Tools for Linux
- Deploy Distribute Tanium Endpoint Index Tools for Mac
- Specify the scheduling details and target systems for the endpoint package distribution.
Selecting a Reissue interval ensures that endpoints that come online later get the Index tools.
- Choose an action group of endpoints for the package.
- Click Show Preview to Continue. Review the list of targeted endpoints and adjust the action group if necessary.
- Click Save Action.
The action runs at the specified time or interval to distribute the Distribute Tanium Endpoint Index Tools, Distribute Tanium Endpoint Index Tools For Linux, and Distribute Tanium Endpoint Index Tools For Mac packages to the targeted endpoints.
The tools are deployed by default to the <Tanium Client>\Tools\EPI directory. An SQLite 3 database is used to store file indexes and the associated file hashes in the following location: <Tanium Client>\Tools\EPI\EndpointIndex.db.
Verify that Index is installed on endpoints and confirm that the Index tools are up to date.
The version of Index that is on the endpoint.
Specifies a number of Linux, Mac, or Windows endpoints that have a version of Index deployed on the endpoint that does not match the version of the solution that is imported on the server. You need to deploy the Index package to those endpoints.
Customize Index configuration settings with the Distribute Tanium Endpoint Index Config, Distribute Tanium Endpoint Index Config For Linux, or Distribute Tanium Endpoint Index Config For Mac packages. The default packages contain a sample configuration file to use as a temple to customize the Index settings to your environment.
If the config.ini file is already installed on an endpoint, the Index tools packages and actions cannot overwrite the modified config.ini file on the endpoint. To update an existing config.ini file on the endpoint, use the Distribute Tanium Index Config package.
- From the Main menu, go to Content > Packages.
- Select the appropriate package and click Edit.
- Add your config.ini file.
- To download the sample_config.ini, click Download .
- Update the file to the settings that you want and save it as config.ini.
The following table provides a description of the settings in the config.ini files for Index and the operating systems that they apply to:
Setting Windows Mac Linux
Indexing pauses if the calculated CPU usage exceeds the configured CPU usage limit value during the file system inventory and computation of hashes.
A percentage of CPU use that alerts a CPU use violation.
The duration in seconds for a CPU use violation to be tolerated before triggering a violation.
The number of reported CPU use violations to allow before stopping index. Setting this value to 0 disables the automatic stopping of indexing.
Captures a full stack trace when index detects a resource use violation.
specifies which hash types to calculate and store for each file hashed. Valid values are MD5, SHA1, and SHA256.
Limits hashing to only files whose size is less than specified. This is important because hashing is resource intensive for large files, for example, VMDK and other virtualization resources.
The number of files to index before checking if throttling is necessary. Throttling ensures that the overall CPU utilization averages out to the defined CPU usage limit.
The number of updates to the hash digest to perform before checking if throttling is necessary.
When the hashing phase is complete, this is the length of time that Index monitors for file changes before going to the next indexing phase.
List the permissions that are associated with files when an index is performed.
* * *
Enables real time file change notifications.
The level of logging to apply to index operations.
Exclude local (and otherwise indexable) file systems from indexing based on file system type. Provide a comma-separated list of file system types.
Sets the frequency of reading pending file change events from the Event Recorder.
Sets the maximum number of file change events to read from the Event Recorder in each batch.
Specifies a non-standard Event Recorder installation location.
Create exclusions to keep specific files and paths out of file system indexes.
* * *
Create exclusions from hashing to exclude specific files and paths from having hash values calculated.
* * *
* = The setting is not enabled by default.
- Click Add to upload the customized config.ini file.
- Click Save.
- Deploy the Distribute Tanium Endpoint Index Config, Distribute Tanium Endpoint Index Config For Linux, or Distribute Tanium Endpoint Index Config For Mac packages.
- Click Actions > Scheduled Actions.
- Select the package that you want to deploy and click Edit.
- Edit the deployment details and target the package distribution to a specific action group.
By default, the config.ini file is located in the <Tanium Client>\Tools\EPI directory.
To ensure that updates to the modified config.ini file are preserved when you upgrade Index, see Upgrade Tanium Index .
To start indexing on the endpoints that have Index tools installed, use the Deploy Start Indexing, Deploy Start Indexing For Linux, and Deploy Start Indexing For Mac saved actions. To ensure that indexing gets restarted when a computer restarts, configure the saved action as a scheduled action. For example, you might schedule Deploy Start Indexing to run every hour.
For more information about these actions, see:
- Tanium Knowledge Base Index Reference: Start indexing
- Tanium Knowledge Base Index Reference: Start indexing For Linux
- Tanium Knowledge Base Index Reference: Start indexing For Mac
To check indexing status, use the Index Status sensor. For more information about the status values, see Tanium Knowledge Base Index Reference: Index Status.
Use the Index Query File sensors to get details about files that have been indexed.
The Index Query File Details sensors return Created and Last Modified time stamps. The time stamps in the results make the strings that are returned for each file unique. To reduce the overall number of strings, use the following workflow:
- Start with one of the following sensors that are less likely to return as many unique strings:
- Index Query File Path Using Name
- Index Query File Path and Hash
- Index Query File Exists
- Index Query File Hash Recently Changed
- Index Query File Count
Index Query File Permissions*
- After getting results from the sensors above, you can drill down to get more details with the following sensors:
- Index Query File Details
- Index Query File Details Using Name
- Index Query File Details by Last Modified
- Index Query File Details Using Name Sort By Largest
- Index Query File Permissions*
* = To use the Index Query File Permissions sensor, the ScanFilePermissions setting in the Index config.ini must be enabled and set to true.
For more information about these sensors, see Tanium Knowledge Base Index Reference: Sensors.
You can provide a blacklist of hashes and compare that list with the hashes that are computed by Tanium Index. You can use MD5, SHA1, or SHA256 hashes.
- Edit the package.
- In the Tanium Console, go to Content > Packages.
- Select the Distribute Index Query Blacklist, Distribute Index Query Blacklist For Linux, or Distribute Index Query Blacklist For Mac package.
- Click Edit.
- Update the blacklist.txt file.
The file contains a list of hashes that are separated by commas or carriage returns. If the hashes are separated by commas, group the hashes of the same type together.
The blacklist has been successfully tested with over 100,000 entries, but start with a smaller number of hashes and update the blacklist on a regular basis.
- To download the current file, click Download .
- Remove the file that is currently in the package .
- Edit the blacklist.txt file.
- Click Add to upload the updated blacklist.txt file.
- Click Save.
- In the Tanium Console, use an operating system-based question to locate computers on which to deploy the Package. Drill down to the endpoints and click Deploy Action. Choose the Distribute Index Query Blacklist, Distribute Index Query Blacklist For Linux, or Distribute Index Query Blacklist For Mac package.
- Perform comparison of deployed blacklist with hashes computed by Index.
Use the Get Index Query Find Blacklist Matches sensor. This sensor returns a list of the file paths and hashes that are listed in the blacklist.
Index not running
By default, the configuration deployment packages contain a sample configuration file: sample_config.ini. If you did not replace this file with a customized config.ini file, the Index Status sensor returns: Missing Config File. Verify that you have replaced the sample_config.ini file with a customized config.ini file. See Customize Index endpoint settings.
Files and directory paths reported by Index are different compared to other methods
You might notice a difference in the files and directory paths that are reported by Index versus other methods. Windows uses hard links, symbolic links, and junctions for some of the files that the user sees. Enumerating the files with the Master File Table (MFT) shows the source path of the first hard link of a file, but not all of the hard links. As a result, scanning the MFT can yield different directory paths for files than a typical directory traversal.
Links in the MFT can cause problems with finding the full path of file. If you search for a file that seems to be "missing" from the System32 directory, you might find the file in a different location, such as the C:\Windows\SysWow64 directory. Another example, the C:\Users\all users directory, is symbolically linked to the C:\ProgramData directory. Index follows the link and records the files in the database under C:\ProgramData. The hashes of these files are correct and match the linked files that are in the directories that are visible to users.
The differences caused by links in the MFT are rarely an issue for forensic analysis. Indicators of Compromise (IOC) rarely have a full path as an indicator item, and instead use a file name and MD5 hash. The blacklists of files that you can get from the government also include MD5 hashes.
Beginning with Index version 1.7.0, hard links in Windows are now tracked. For more information, see Hard links not recorded.
If you see that Index records only the first hard link for a file, and not the other hard link peers, you can verify which machines have the configuration to record all hard links. Use Interact to ask Get Index Status from all machines. If you see the message "Delete Index Database To Enable Hard Link Tracking", this means that these endpoints have a version of Index that can track hard links, but Index needs to reindex the filesystem to get all of the information needed to track the hard links in the filesystem. To enable hard link tracking, deploy the Delete Tanium Endpoint Index database package and initiate a reindex.
Performing reindexing message
If Index is started after not running for a while, either because it was stopped or the endpoint has been offline, you might see an entry in the TaniumEndpointIndex.log file that is similar to the following message:
[2015-10-27 13:28:39]  [Information] [MTFScanner] LAST_JOURNAL_ID_SEEN-C not found in DB. Performing reindexing…
This message indicates that when Index restarted, the NTFS Journal no longer had the last Update Sequence Number (USN), so indexing restarted.
To prevent reindexing from occurring, use the Deploy Start Indexing scheduled action to restart Index every hour if indexing is disabled. Using this scheduled action:
- Catches new endpoints.
- Catches endpoints that are coming online after a restart.
- Starts indexing on these new and restarted endpoints.
Missing hash or magic number for file
Some files might be in the Index database with no hashes or magic number. This situation can happen for the following reasons:
- The file is inventoried, but the initial hashing pass is not complete.
- The file is changed, but the RescanInterval timer has not initiated the file to be rehashed.
- The file is locked, so Index cannot get read lock to hash it.
- The file is larger than the configured MaxFileSizeToHashMB value.
- The file was excluded from hashing with the ExcludeFromHashing regular expression.
The levels for the logging.loggers.root.level property in the config.ini file are in the following list. The levels are listed from least to most verbose:
- none (turns off logging)
- information (default)
- trace (includes the most messages)
If you set the level to debug or trace, expect verbose output in the log file. Most of the information in these levels is meant for debugging by Tanium technical teams.
Log file rotation
Log files are capped at 10 MB. When the file reaches 10 MB, the file gets moved to TaniumEndpointIndex.log.timestamp.gz. Index keeps up to ten log files, removing older log files. The log files are in the <Tanium Client>\Tools\EPI directory.
Dump (.dmp) files
TaniumEndpointIndex_[0-9].dmp log files are created if the Index process crashes. Files rotate, with TaniumEndpointIndex_0.dmp always being the most recent.
For details about the parameters for each Index sensor and package, see Tanium Knowledge Base: Index Reference.
Last updated: 11/28/2018 9:27 AM | Feedback