Configure Historic Data Retention

Moogsoft Enterprise employs two databases, an active database and a historic database to enhance performance of the UI and extend data retention capabilities.

Note

If you are upgrading from Moogsoft Enterprise v. 6.4 or earlier, manually split the database if you want to benefit from using a separate historic database. See Historic Database Benefits for further details.

See Historic Database Benefits for information on the performance and scalability advantages of separating the active and historic databases.

You can use various closing strategies to help maintain your active database at an optimal size:

  • Manually close alerts.

  • Programmatically close alerts.

  • Use the Auto Close feature.

The historic database can grow without affecting performance. When it is time to retire data from the historic database, you can archive selected Situations and alerts.

Control historic data retention

Historic data retention requires the Housekeeper Moolet to work. Verify you have configured the Housekeeper Moolet and that this Moolet is enabled within Moogfarmd. The Housekeeper periodically identifies eligible closed alerts and Situations in the active database and moves them into the historic database.

You can control historic data retention using the database split configurer at $MOOGSOFT_HOME/bin/utils/moog_db_split_configurer.

See the Historic Data Utility Command Reference for a full list of available arguments.

Access historic data

By default, the database split configurer creates a database called historic_moogdb within the same MySQL instance as the active database: moogdb.

The historic database contains alert and Situation related tables that have the same structure as their equivalents in the active database.

You can access this historic data in the UI in read-only format:

  • Search results (when "include close" is selected)

  • Direct URL (for Situation Room, Alert Timeline etc.)

  • Situation Room (including Situation Alert View and Situation Timeline)

  • Similar Situations (historical closed Situations will feature in the Similar Situations list for a Situation)

  • PRC feedback can be set for closed alerts from the historic database (as accessed from the Alerts Tab of a Situation Room)

You can also access historic data from various Graze API endpoints and MoogDb V2 methods. See Graze API EndPoint Reference for details of individual Graze API endpoints or MoogDb V2 API Method Reference for details of individual MoogDb v2 methods.

The active database retains event and snapshot data for two weeks by default. At the end of the retention period, the Housekeeper Moolet removes the event and snapshot data from the active database. You can only access this event and snapshot data from the historic database after this point. You can change the default retention period at $MOOGSOFT_HOME/bin/utils/moog_db_split_configurer.

Moogsoft Enterprise rejects Graze endpoints and MoogDb v2 methods that attempt to modify historic alerts and Situations.

Database split examples

To split the database at 2am every day, with a grace period of 1 hour and an alerts_batch_size and a sigs_batch_size of 1000, run this command:

moog_db_split_configurer -e -r 02:00 -g 1 -a 1000 -s 1000

During the splitting process you can see entries such as the following in the Moogfarmd log:

WARN : [0:House][20180315 15:58:44.207 +0000] [CSplitterService.java]:143 +|Data Splitter started|+
WARN : [0:House][20180315 15:58:44.503 +0000] [CSplitterTask.java]:205 +|Splitter will copy [17] alerts and will move [1983] alerts and [500] situations|+
WARN : [0:House][20180315 15:58:44.706 +0000] [CSplitterTask.java]:205 +|Splitter will copy [152] alerts and will move [1848] alerts and [0] situations|+
WARN : [0:House][20180315 15:58:46.434 +0000] [CSplitterTask.java]:205 +|Splitter will copy [78] alerts and will move [917] alerts and [0] situations|+
WARN : [0:House][20180315 15:58:47.280 +0000] [CSplitterTask.java]:201 +|Nothing more to split|+
WARN : [0:House][20180315 15:58:47.282 +0000] [CSplitterService.java]:145 +|Data Splitter completed|+

If there is no eligible data to move to the historic database, the following entry is logged:

WARN : [0:House][20180315 15:12:22.547 +0000] [CSplitterService.java]:143 +|Data Splitter started|+
WARN : [0:House][20180315 15:12:22.666 +0000] [CSplitterTask.java]:201 +|Nothing more to split|+
WARN : [0:House][20180315 15:12:22.667 +0000] [CSplitterService.java]:145 +|Data Splitter completed|+

Example Split Scenario

The following table illustrates how the Housekeeper Moolet splits an example set of Situations and alerts.

Pre-Split Active Database

Post-Split Active Database

Post-Split Historic Database

Situation 1 (closed) with member alerts:

  • Alert 1 (closed)

  • Alert 2 (closed)

  • Alert 3 (closed)

Situation 2 (closed) with member alerts:

  • Alert 4 (closed)

  • Alert 5 (closed)

  • Alert 6 (closed)

Situation 3 (open) with member alerts:

  • Alert 5 (closed)

  • Alert 6 (closed)

  • Alert 7 (closed)

  • Alert 8 (open)

Situation 4 (open) with member alerts:

  • Alert 8 (open)

  • Alert 9 (open)

Loose alerts:

  • Alert 10 (closed)

  • Alert 11 (open)

Split occurs

Situation 3 (open) with member alerts:

  • Alert 5 (closed)

  • Alert 6 (closed)

  • Alert 7 (closed)

  • Alert 8 (open)

Situation 4 (Open) with member alerts:

  • Alert 8 (open)

  • Alert 9 (open)

Loose alerts:

  • Alert 11 (open)

Situation 1 (closed) with member alerts:

  • Alert 1 (closed)

  • Alert 2 (closed)

  • Alert 3 (closed)

Situation 2 (Closed) with member alerts:

  • Alert 4 (closed)

  • Alert 5 (closed)

  • Alert 6 (closed)

Loose alerts:

  • Alert 10 (closed)

Other alerts:

  • Alert 7 (closed)

Notes on the above:

  • The process copies closed alerts 5 and 6 to the historic database because they are related to open Situation 3. In the historic database they retain their relationship to closed Situation 2, but not open Situation 3 until it is closed and the Housekeeper Moolet moves it to the historic database.

  • The process copies closed Alert 7 to the historic database, but within that database the relationship to open Situation 3 is removed, until Situation 3 is closed and the Housekeeper Moolet moves it to the historic database.