Moogsoft Docs

Search and Indexing

Moogsoft AIOps uses Elasticsearch to provide search and data indexing functions.

You can control the Elasticsearch service using the following service script:

/etc/init.d/elasticsearch [start|restart|stop]

All Elasticsearch logs are stored in following location:

/var/log/elasticsearch/

Perform an Index

T here are two tools you can use to index Alerts and Situations: the Indexer Moolet and the moog_indexer utility.

Indexer Moolet

The Indexer listens for new Alerts and Situations on the Message Bus and indexes them. AIOps indexes Alerts and Situations as soon as they are are created or changed so you can search for them immediately.

You can configure the Indexer in moog_farmd.conf using the following parameters:

enable_private_teams

Set to ' True' if you use private teams mode in AIOps and you want to configure the Indexer to only index Alerts and Situations visible to the private team.

If disabled, Indexer will index all Alerts and Situations present in AIOps.

Type : Boolen
Default : False

full_scan_batch_size

The maximum number of Alerts or Situations the Indexer scans in each batch as it indexes. This is useful because it is not possible to load all Alerts to the memory in one go.

By default Indexer scans through batches of 1000 Alerts or Situations.

Type : Integer
Default : 1000

full_scan_wait

The number of seconds the Indexer waits between batches. One advantage of this is to free up the CPU and memory being used to index each batch.

It is set t o 0 by default so the Indexer will not wait between batches.

Type: Integer
Default: 0

full_scan_at

Determines the exact time when Indexer runs a full scan. This allows you to ensure the accuracy of search data once per day by performing a full re-index. If left empty, the Indexer does not perform a full scan.

Type : Time (HH:mm:ss)
Default : "02:12:35"

full_scan_at_startup

If enabled, then Indexer performs a full scan when it sta rts . This is useful if you are not using the scheduled scan and only restart moogfarmd once a week.

Type: Boolean
Default: false

historic_scan_frequency

Determines how frequently the Indexer performs a full scan of both active and historic databases. By default, Indexer scans both databases every three days.

Type: Integer
Default: 3

For example, refer to the default Indexer configuration:

# Set to false to disable private teams indexing.
enable_private_teams: false,

# Maximal full scan batch size
full_scan_batch_size: 1000,

# How many seconds to wait between batches (0 not to wait)
full_scan_wait: 0,

# When to run the full scan (HH:mm:ss) leave empty to disable full scan (HH:mm:ss)
full_scan_at: "02:12:35",

# Do we want to run full scan when the moolet starts?
full_scan_at_startup: false

# Scan the historic data once every how many full scans
historic_scan_frequency: 3

moog_indexer

You can run the indexer utility alongside the Indexer Moolet. Before you can run the indexer utility, you must start moogfarmd with a running Indexer Moolet.

The moog_indexer command-line utility accepts the following options:

Argument Input Description
-h,--help  
- Displays the help text with arguments that can be used with the utility.
-f, --full - Scans both the active and historic data. Use this argument if you want data from both databases to be indexed.
-i,--in <arg> 
Integer

Schedule full index to run in a set amount of time (in hours). This can be a decimal. For example, 0.1 = 6 minutes.

-l,--loglevel <arg> 
INFO|WARN|ALL

Specify the log level to choose the amount of debug output

-n,--now   
-

Schedules a full index to run immediately.

-r,--report  
-

Request report from on the last performed full scan index. This report will show the status of previous runs within the lifetime of the moogfarmd process and any runs still in progress. If moogfarmd is restarted, the -r argument will not return any data.

Note

Please note : If you use Private Teams mode, meaning one or more Roles do NOT have the all_data permission set, then you must run both the initial 'full index' and the 'incremental index crontab' moog_indexer commands with the -p argument. If not, users in one Team will be able to see search results for other Teams.

Tune your MySQL database to ensure indexing runs as quickly as possible. See either the Percona or MySQL websites for information on tuning and optimization.

An output example is shown below:

[root@myhost home]# moog_indexer -r
Got report:
	05/10/17 13:43:06 - Starting full scan
	05/10/17 13:43:06 - Scanning for alerts
	05/10/17 13:43:07 - Scanned: [177] alerts
	05/10/17 13:43:07 - Scanning for situations
	05/10/17 13:43:07 - Scanned: [44] situations
	05/10/17 13:43:07 - Full scan complete
	05/10/17 13:43:22 - Starting full scan
	05/10/17 13:43:22 - Scanning for alerts
	05/10/17 13:43:22 - Scanned: [204] alerts
	05/10/17 13:43:22 - Scanning for situations
	05/10/17 13:43:23 - Scanned: [55] situations
	05/10/17 13:43:23 - Full scan complete

Warning

Before you upgrade to AIOps V6.2.1 or later from a version older than 6.2.1, remove or disable the crontab jobs for the old indexer utility.

Elasticsearch Details

Elasticsearch runs on port 9200 by default.

To make Elasticsearch available externally and listen on the external host IP address, run the following command:

$MOOGSOFT_HOME/bin/utils/moog_init_search.sh -r

The script updates the Elasticsearch configuration and restarts the service.