Configure Search and Indexing
Moogsoft Onprem uses Opensearch to provide search and data indexing functions.
You can control the Opensearch service using the following service script:
service opensearch [start|restart|stop]
All Opensearch logs are stored in the following location:
/var/log/opensearch/
Index Alerts and Situations
Two tools are used to index alerts and Situations: the Indexer Moolet and the Moog Indexer utility.
Indexer Moolet
The Indexer listens for new alerts and Situations on the Message Bus and indexes them. Moogsoft Onprem indexes alerts and Situations as soon as they are are created or modified so that they are immediately searchable.
You can configure the Indexer in $MOOGSOFT_HOME/config/moolets/indexer.conf using the following parameters:
enable_private_teams
Set to true if you limit team permissions based upon services, Situations, or alerts assigned to the team. The the indexer applies team permissions to the indexes.
If disabled, the Indexer will index all alerts and Situations present in Moogsoft Onprem.
Type: Boolean
Default: False
full_scan_batch_size
The maximum number of alerts or Situations the Indexer scans in each batch. This is useful because it is not possible to load all alerts to the memory at once.
By default the Indexer scans through batches of one thousand alerts or Situations.
Type: Integer
Default: 1000
full_scan_wait
The number of seconds the Indexer waits between batches. This frees up the CPU and memory used to index each batch.
It is set to zero by default so the Indexer will not wait between batches.
Type: Integer
Default: 0
full_scan_at
Determines the exact time when Indexer runs a full scan. This allows you to ensure the accuracy of search data once per day by performing a full reindex. If left empty, the Indexer does not perform a full scan.
Type: Time (HH:mm:ss)
Default: "02:12:35"
full_scan_at_startup
If enabled, the Indexer performs a full scan when it starts. This is useful if you are not using the scheduled scan and only restart Moogfarmd once a week.
Type: Boolean
Default: false
historic_scan_frequency
Determines how frequently the Indexer performs a full scan of both active and historic databases. By default, the Indexer scans both databases every three days.
Type: Integer
Default: 3
By default the Indexer is configured as follows:
# Set to false to disable private teams indexing. enable_private_teams: false, # Maximal full scan batch size full_scan_batch_size: 1000, # How many seconds to wait between batches (0 not to wait) full_scan_wait: 0, # When to run the full scan (HH:mm:ss) leave empty to disable full scan (HH:mm:ss) full_scan_at: "02:12:35", # Do we want to run full scan when the moolet starts? full_scan_at_startup: false # Scan the historic data once every how many full scans historic_scan_frequency: 3
Moog Indexer
Before you can run the indexer utility, you must start Moogfarmd with a running Indexer Moolet. The moog_indexer accepts the following options:
Argument | Input | Description |
|---|---|---|
-h,--help | - | Displays the help text with arguments that can be used with the utility. |
| - | Scans both the active and historic data. Use this argument if you want data from both databases to be indexed. |
-i,--in <arg> | Integer | Schedule full index to run in a set amount of time (in hours). This can be a decimal. For example, 0.1 = 6 minutes. |
-l,--loglevel <arg> | WARN|INFO|DEBUG|TRACE | Specify the log level to choose the amount of debug output. Defaults to INFO. |
-n,--now | - | Schedules a full index to run immediately. |
-r,--report | - | Request report from on the last performed full scan index. This report will show the status of previous runs within the lifetime of the moogfarmd process and any runs still in progress. If moogfarmd is restarted, the -r argument will not return any data. |
Note
If you use Private Teams mode, meaning one or more Roles do NOT have the all_data permission set, then you must run both the initial 'full index' and the 'incremental index crontab' moog_indexer commands with the -p argument. If not, users in one Team will be able to see search results for other Teams.
Tune your MySQL database to ensure indexing runs as quickly as possible. See either the Percona or MySQL websites for information on tuning and optimization.
An output example is shown below:
[root@myhost home]# moog_indexer -r
Got report:
05/10/17 13:43:06 - Starting full scan
05/10/17 13:43:06 - Scanning for alerts
05/10/17 13:43:07 - Scanned: [177] alerts
05/10/17 13:43:07 - Scanning for situations
05/10/17 13:43:07 - Scanned: [44] situations
05/10/17 13:43:07 - Full scan complete
05/10/17 13:43:22 - Starting full scan
05/10/17 13:43:22 - Scanning for alerts
05/10/17 13:43:22 - Scanned: [204] alerts
05/10/17 13:43:22 - Scanning for situations
05/10/17 13:43:23 - Scanned: [55] situations
05/10/17 13:43:23 - Full scan completeSearch Details
Opensearch runs on port 9200 by default.
To make Opensearch available externally and listen on the external host IP address, run the following command:
$MOOGSOFT_HOME/bin/utils/moog_init_search.sh -r
The script updates the Opensearch configuration and restarts the service.