Moogsoft AIOps employs two databases, an active database and a historic database to enhance performance of the UI and extend data retention capabilities.
If you are upgrading from Moogsoft AIOps v. 6.4 or earlier, manually split the database if you want to benefit from using a separate historic database. See Enable the Historic Database for further details.
See Historic Database Benefits for information on the performance and scalability advantages of separating the active and historic databases.
You can use various closing strategies to help maintain your active database at an optimal size:
- Manually close alerts.
- Programmatically close alerts.
- Use the Auto Close feature.
The historic database can grow without affecting performance. When it is time to retire data from the historic database, you can archive selected Situations and alerts.
Control Historic Data Retention
Historic data retention requires the Housekeeper Moolet to work. Verify you have configured the Housekeeper Moolet and that it is enabled within Moogfarmd. The Housekeeper periodically identifies eligible closed alerts and Situations in the active database and moves them into the historic database.
You can control historic data retention using the database split enabler at $MOOGSOFT_HOME/bin/utils/moog_db_split_enabler
.
See the Historic Data Utility Command Reference for a full list of available arguments.
Access Historic Data
By default, the database split enabler creates a database called historic_moogdb within the same MySQL instance as the active database: moogdb.
The historic database contains alert and Situation related tables that have the same structure as their equivalents in the active database.
You can access this historic data in the UI in read-only format:
- Search results (when "include close" is selected)
- Direct URL (for Situation Room, Alert Timeline etc.)
- Situation Room (including Situation Alert View and Situation Timeline)
- Similar Situations (historical closed Situations will feature in the Similar Situations list for a Situation)
- PRC feedback can be set for closed alerts from the historic database (as accessed from the Alerts Tab of a Situation Room)
Historic data is also accessible from various Graze endpoints and their equivalent MoogDb methods:
- getAlertIds
- getAlertDetails
- getSituationAlertIds
- getSituationDetails
- getSituationHosts
- getSituationProcesses
- getSituationServices
- getSituationActions
Moogsoft AIOps rejects Graze endpoints and MoogDb methods that attempt to modify historic alerts and Situations.
Disable Historic Data Retention
When you disable the historic database, the data already in the historic database remains there. The Housekeeper Moolet does not move the data back to the active database. The old historic database is only accessible via SQL queries.
To disable the retention of historic data in a separate database, run the following command:
moog_db_split_enabler -d
Enable the Historic Database
If you are upgrading from Moogsoft AIOps v. 6.4 or earlier you need to manually split the data into an active database and a separate historic database.
Before you split the database, ensure you have met the following requirements:
- The Housekeeper Moolet is configured and running within Moogfarmd.
- You have the username and password for a MySQL user with schema creation privileges.
To split historic data into a separate database, run this command:
moog_db_split_enabler -e
Alert and Situation data that meet the default criteria are moved. If you do not specify alternative criteria, all closed alerts and Situations that have not been updated within the past hour are moved into a historic database.
See the Historic Data Utility Command Reference for a full list of default and available arguments. Any changes to the default settings are picked up and applied immediately to a Moogfarmd with Housekeeper Moolet running.
If the process encounters closed alerts that are eligible to be moved but still have a relationship with an open Situation, it copies the alerts to the historic database instead of moving them. Later, after they're no longer related to an open Situation, the Housekeeper Moolet moves them to the historic database.
If your system contains a large amount of closed data, the first run may take a long time and require additional CPU and memory use for the Moogfarmd process.
When the splitting process is active, you can monitor its progress in the Moogfarmd log by running this command:
tail -f /var/log/moogsoft/moogfarmd.log|grep Splitter
After the database is split, UI based filters do not return closed alerts or Situations that have been moved to the historic database.
Database Split Examples
To split the database with a grace_period and a run_interval of 60 seconds and an alerts_batch_size and a sigs_batch_size of 1000, run this command:
moog_db_split_enabler -e -g 60 -r 60 -a 1000 -s 1000
During the splitting process you can see entries such as the following in the Moogfarmd log:
WARN : [0:House][20180315 15:58:44.207 +0000] [CSplitterService.java]:143 +|Data Splitter started|+ WARN : [0:House][20180315 15:58:44.503 +0000] [CSplitterTask.java]:205 +|Splitter will copy [17] alerts and will move [1983] alerts and [500] situations|+ WARN : [0:House][20180315 15:58:44.706 +0000] [CSplitterTask.java]:205 +|Splitter will copy [152] alerts and will move [1848] alerts and [0] situations|+ WARN : [0:House][20180315 15:58:46.434 +0000] [CSplitterTask.java]:205 +|Splitter will copy [78] alerts and will move [917] alerts and [0] situations|+ WARN : [0:House][20180315 15:58:47.280 +0000] [CSplitterTask.java]:201 +|Nothing more to split|+ WARN : [0:House][20180315 15:58:47.282 +0000] [CSplitterService.java]:145 +|Data Splitter completed|+
If there is no eligible data to move to the historic database, the following entry is logged:
WARN : [0:House][20180315 15:12:22.547 +0000] [CSplitterService.java]:143 +|Data Splitter started|+ WARN : [0:House][20180315 15:12:22.666 +0000] [CSplitterTask.java]:201 +|Nothing more to split|+ WARN : [0:House][20180315 15:12:22.667 +0000] [CSplitterService.java]:145 +|Data Splitter completed|+
Example Split Scenario
The following table illustrates how the Housekeeper Moolet splits an example set of Situations and alerts.
Pre-Split Active Database | Post-Split Active Database | Post-Split Historic Database | |
---|---|---|---|
Situation 1 (closed) with member alerts:
Situation 2 (closed) with member alerts:
Situation 3 (open) with member alerts:
Situation 4 (open) with member alerts:
Loose alerts:
| →Split occurs | Situation 3 (open) with member alerts:
Situation 4 (Open) with member alerts:
Loose alerts:
| Situation 1 (closed) with member alerts:
Situation 2 (Closed) with member alerts:
Loose alerts:
Other alerts:
|
Notes on the above:
- The process copies closed alerts 5 and 6 to the historic database because they are related to open Situation 3. In the historic database they retain their relationship to closed Situation 2, but not open Situation 3 until it is closed and the Housekeeper Moolet moves it to the historic database.
- The process copies closed Alert 7 to the historic database, but within that database the relationship to open Situation 3 is removed, until Situation 3 is closed and the Housekeeper Moolet moves it to the historic database.
43 Comments
James Scott
I think call it either historic database or historic data store. People are probably more likely to search for "historic database" IMO.
James Scott
Style thing - we need to agree on a style for version + number. In release notes we have V6 and v6.4.0 etc. In supported environments we do v. 5 etc. We probably need to pick one and add it to the style guide.
James Scott
database
James Scott
Not sure all customers will know what sigs and sig_alerts are. Might be worth doing "Situations (sigs) and Situation alerts (sig_alerts).
James Scott
Could link to Graze API doc and MoogdbV2 doc from here.
James Scott
Passive + future. I probably wrote this
Suggest changing to: "Moogsoft AIOps rejects Graze endpoints and moogdb methods that attempt to modify historic alerts and Situations"
James Scott
As previously mentioned
James Scott
As above.
James Scott
remove
James Scott
remove
James Scott
I think it's worth putting a comma here.
James Scott
I don't like Neil's arrow. Shall we just put a → in there instead?
Also might be better to have "Split occurs" or similar instead of SPLIT
Neil Billett
I agree with James. I don't like the name Data Store in general. moogdb and historic_moogdb are "databases" within the same MySQL instance.
Neil Billett
I'd probably change this to "of a split active/historic database" - I think the work segmentation implies something else.
Neil Billett
Maybe this would be better as "If you regularly close alerts and Situations (either manually, programmatically or via Auto Close)"
Neil Billett
Not quite true as ultimately the MySQL instance will still be getting bigger and bigger. Maybe better worded as "The historic database can scale with far less impact on system performance"
Neil Billett
Is it worth repeating this here? Sort of implied in previous paragraphs.
Neil Billett
This utility can enable/configure/disable - worth mentioning?
Neil Billett
Maybe change to "the moog_db_split_enabler utility" just in case its not obvious as we're in a new paragraph.
Neil Billett
I'd be tempted to go with: "The historic database contains Alert and Situation related tables that have the same structure as their equivalents in the active database."
Neil Billett
replace with "UI in the following"
Neil Billett
This deserves a warning box or at least an important prefix.
Neil Billett
We don't really enable the utility. We enable splitting using the utility - not sure how best to word that - I'll leave it to you
Neil Billett
Ditto my last comment.
Neil Billett
Ditto my last comment.
Neil Billett
I see we're calling the splitter itself (which is a task within the Housekeeper moolet) the "utility" which is a bit confusing. Maybe I've had this conversation before with someone - I'm getting deja-vu. Anyhow I'd probably be happier referring to the enabler as the "utility" and the thing that does the splitting as maybe just the "Housekeeper" or maybe the "Splitter". I'll leave it up to you.
Neil Billett
Think this sentence is better as: "The following table illustrates how an example set of Situations and Alerts will be be split"
Charles Smith
+1
Charles Smith
Suggest:
Moogsoft employs two databases, an active database and a historic database to enhance performance of the UI and extend data retention capabilities.
Charles Smith
via – not a good english tech docs word
Suggest
You can use various closing strategies to help maintain your active database at an optimal size:
Charles Smith
+1 for Neil's revision.
Charles Smith
by cows. How about
Historic data retention requires the Housekeeper Moolet to work. Verify you have configured the Housekeeper Moolet and that it is enabled within moogfarmd.
Charles Smith
+1 I think OK to edit out at this point.
Charles Smith
one word for : enable/confiugre/disable?
control?
You can control historic data retention using the Database Split Enabler at <path>
Charles Smith
+1 for Neils second revision
Charles Smith
If it is dangerous, maybe we make this the last option on the page. Can we think of a reason why a customer would want to do this? Are we only documenting it so people understand the impact & won't do it?
How about instead of a box, we move this information before you run the command. Change the intro a little bit:
You can disable the retention of historic data in a separate database. When you disable the historic database, the data already in the historic database remains there. The Housekeeper Moolet does not move the data back to the active database.
Neil Billett is the data truly inaccessibble -or only accesible via SQL queries/accessing the database directly?
I realize it's more words, but it's kind of a way to reiterate the impact to make people think.
Charles Smith
How about we don't use the future passive:
The following table illustrates how the Housekeeper Moolet splits an example set of Situations and Alerts.
Charles Smith
hard stop at end of sentence
Charles Smith
no _ in the English name
Charles Smith
I don't think this is actually a "code" reference here. suggest moogfarmd w/o monospace type
Jill Osborne
Not quite true. It would be better to write "The historic database can scale with far less impact on system performance".
Jill Osborne
test
Jill Osborne
Not quite true. It would be more accurate to write "The historic database can scale with far less impact on system performance".