Skip to end of metadata
Go to start of metadata

Moogsoft AIOps employs two databases, an active database and a historic database to enhance performance of the UI and extend data retention capabilities.

If you are upgrading from Moogsoft AIOps v. 6.4 or earlier, manually split the database if you want to benefit from using a separate historic database. See Enable the Historic Database for further details.

See Historic Database Benefits for information on the performance and scalability advantages of separating the active and historic databases.

You can use various closing strategies to help maintain your active database at an optimal size:

  • Manually close alerts.
  • Programmatically close alerts.
  • Use the Auto Close feature.

The historic database can grow without affecting performanceWhen it is time to retire data from the historic database, you can archive selected Situations and alerts.

Control Historic Data Retention

Historic data retention requires the Housekeeper Moolet to work. Verify you have configured the Housekeeper Moolet and that it is enabled within Moogfarmd. The Housekeeper periodically identifies eligible closed alerts and Situations in the active database and moves them into the historic database. 

You can control historic data retention using the database split enabler at $MOOGSOFT_HOME/bin/utils/moog_db_split_enabler.

See the Historic Data Utility Command Reference for a full list of available arguments.

Access Historic Data

By default, the database split enabler creates a database called historic_moogdb within the same MySQL instance as the active database: moogdb.

The historic database contains alert and Situation related tables that have the same structure as their equivalents in the active database.

You can access this historic data in the UI in read-only format:

  • Search results (when "include close" is selected)
  • Direct URL (for Situation Room, Alert Timeline etc.)
  • Situation Room (including Situation Alert View and Situation Timeline)
  • Similar Situations (historical closed Situations will feature in the Similar Situations list for a Situation)
  • PRC feedback can be set for closed alerts from the historic database (as accessed from the Alerts Tab of a Situation Room)

Historic data is also accessible from various Graze endpoints and their equivalent MoogDb methods:

  • getAlertIds
  • getAlertDetails
  • getSituationAlertIds
  • getSituationDetails
  • getSituationHosts
  • getSituationProcesses
  • getSituationServices
  • getSituationActions

Moogsoft AIOps rejects Graze endpoints and MoogDb methods that attempt to modify historic alerts and Situations.

Disable Historic Data Retention

When you disable the historic database, the data already in the historic database remains there. The Housekeeper Moolet does not move the data back to the active database. The old historic database is only accessible via SQL queries.

To disable the retention of historic data in a separate database, run the following command:

moog_db_split_enabler -d

Enable the Historic Database

If you are upgrading from Moogsoft AIOps v. 6.4 or earlier you need to manually split the data into an active database and a separate historic database.

Before you split the database, ensure you have met the following requirements:

  • The Housekeeper Moolet is configured and running within Moogfarmd.
  • You have the username and password for a MySQL user with schema creation privileges.

To split historic data into a separate database, run this command:

moog_db_split_enabler -e

Alert and Situation data that meet the default criteria are moved. If you do not specify alternative criteria, all closed alerts and Situations that have not been updated within the past hour are moved into a historic database.

See the Historic Data Utility Command Reference for a full list of default and available arguments. Any changes to the default settings are picked up and applied immediately to a Moogfarmd with Housekeeper Moolet running.

If the process encounters closed alerts that are eligible to be moved but still have a relationship with an open Situation, it copies the alerts to the historic database instead of moving them. Later, after they're no longer related to an open Situation, the Housekeeper Moolet moves them to the historic database.

If your system contains a large amount of closed data, the first run may take a long time and require additional CPU and memory use for the Moogfarmd process.

When the splitting process is active, you can monitor its progress in the Moogfarmd log by running this command:

tail -f /var/log/moogsoft/moogfarmd.log|grep Splitter

After the database is split, UI based filters do not return closed alerts or Situations that have been moved to the historic database.

Database Split Examples

To split the database with a grace_period and a run_interval of 60 seconds and an alerts_batch_size and a sigs_batch_size of 1000, run this command:

moog_db_split_enabler -e -g 60 -r 60 -a 1000 -s 1000

During the splitting process you can see entries such as the following in the Moogfarmd log:

WARN : [0:House][20180315 15:58:44.207 +0000] [CSplitterService.java]:143 +|Data Splitter started|+
WARN : [0:House][20180315 15:58:44.503 +0000] [CSplitterTask.java]:205 +|Splitter will copy [17] alerts and will move [1983] alerts and [500] situations|+
WARN : [0:House][20180315 15:58:44.706 +0000] [CSplitterTask.java]:205 +|Splitter will copy [152] alerts and will move [1848] alerts and [0] situations|+
WARN : [0:House][20180315 15:58:46.434 +0000] [CSplitterTask.java]:205 +|Splitter will copy [78] alerts and will move [917] alerts and [0] situations|+
WARN : [0:House][20180315 15:58:47.280 +0000] [CSplitterTask.java]:201 +|Nothing more to split|+
WARN : [0:House][20180315 15:58:47.282 +0000] [CSplitterService.java]:145 +|Data Splitter completed|+

If there is no eligible data to move to the historic database, the following entry is logged:

WARN : [0:House][20180315 15:12:22.547 +0000] [CSplitterService.java]:143 +|Data Splitter started|+
WARN : [0:House][20180315 15:12:22.666 +0000] [CSplitterTask.java]:201 +|Nothing more to split|+
WARN : [0:House][20180315 15:12:22.667 +0000] [CSplitterService.java]:145 +|Data Splitter completed|+

Example Split Scenario

The following table illustrates how the Housekeeper Moolet splits an example set of Situations and alerts.

Pre-Split Active Database
Post-Split Active DatabasePost-Split Historic Database

Situation 1 (closed) with member alerts:

  • Alert 1 (closed)
  • Alert 2 (closed)
  • Alert 3 (closed)

Situation 2 (closed) with member alerts:

  • Alert 4 (closed)
  • Alert 5 (closed)
  • Alert 6 (closed)

Situation 3 (open) with member alerts:

  • Alert 5 (closed)
  • Alert 6 (closed)
  • Alert 7 (closed)
  • Alert 8 (open)

Situation 4 (open) with member alerts:

  • Alert 8 (open)
  • Alert 9 (open)

Loose alerts:

  • Alert 10 (closed)
  • Alert 11 (open)





Split occurs


Situation 3 (open) with member alerts:

  • Alert 5 (closed)
  • Alert 6 (closed)
  • Alert 7 (closed)
  • Alert 8 (open)

Situation 4 (Open) with member alerts:

  • Alert 8 (open)
  • Alert 9 (open)

Loose alerts:

  • Alert 11 (open)


Situation 1 (closed) with member alerts:
  • Alert 1 (closed)
  • Alert 2 (closed)
  • Alert 3 (closed)

Situation 2 (Closed) with member alerts:

  • Alert 4 (closed)
  • Alert 5 (closed)
  • Alert 6 (closed)

Loose alerts:

  • Alert 10 (closed)

Other alerts:

  • Alert 7 (closed)


Notes on the above:

  • The process copies closed alerts 5 and 6 to the historic database because they are related to open Situation 3. In the historic database they retain their relationship to closed Situation 2, but not open Situation 3 until it is closed and the Housekeeper Moolet moves it to the historic database.
  • The process copies closed Alert 7 to the historic database, but within that database the relationship to open Situation 3 is removed, until Situation 3 is closed and the Housekeeper Moolet moves it to the historic database.


43 Comments

  1. user-1ea8e

    I think call it either historic database or historic data store. People are probably more likely to search for "historic database" IMO.

  2. user-1ea8e

    Style thing - we need to agree on a style for version + number. In release notes we have V6 and v6.4.0 etc. In supported environments we do v. 5 etc. We probably need to pick one and add it to the style guide.

  3. user-1ea8e

  4. user-1ea8e

    Not sure all customers will know what sigs and sig_alerts are. Might be worth doing "Situations (sigs) and Situation alerts (sig_alerts).

  5. user-1ea8e

    Could link to Graze API doc and MoogdbV2 doc from here.

  6. user-1ea8e

    Passive + future. I probably wrote this (smile) Suggest changing to: "Moogsoft AIOps rejects Graze endpoints and moogdb methods that attempt to modify historic alerts and Situations"

  7. user-1ea8e

    As previously mentioned

  8. user-1ea8e

  9. user-1ea8e

  10. user-1ea8e

  11. user-1ea8e

    I think it's worth putting a comma here.

  12. user-1ea8e

    I don't like Neil's arrow. Shall we just put a → in there instead? 


    Also might be better to have "Split occurs" or similar instead of SPLIT

  13. I agree with James. I don't like the name Data Store in general. moogdb and historic_moogdb are "databases" within the same MySQL instance.

  14. I'd probably change this to "of a split active/historic database" - I think the work segmentation implies something else.

  15. Maybe this would be better as "If you regularly close alerts and Situations (either manually, programmatically or via Auto Close)"

  16. Not quite true as ultimately the MySQL instance will still be getting bigger and bigger. Maybe better worded as "The historic database can scale with far less impact on system performance"

  17. Is it worth repeating this here? Sort of implied in previous paragraphs.

  18. This utility can enable/configure/disable - worth mentioning?

  19. Maybe change to "the moog_db_split_enabler utility" just in case its not obvious as we're in a new paragraph.

  20. I'd be tempted to go with: "The historic database contains Alert and Situation related tables that have the same structure as their equivalents in the active database."

  21. replace with "UI in the following"

  22. This deserves a warning box or at least an important prefix.

  23. We don't really enable the utility. We enable splitting using the utility - not sure how best to word that - I'll leave it to you (smile)

  24. Ditto my last comment.

  25. Ditto my last comment.

  26. I see we're calling the splitter itself (which is a task within the Housekeeper moolet) the "utility" which is a bit confusing. Maybe I've had this conversation before with someone - I'm getting deja-vu. Anyhow I'd probably be happier referring to the enabler as the "utility" and the thing that does the splitting as maybe just the "Housekeeper" or maybe the "Splitter". I'll leave it up to you.

  27. Think this sentence is better as: "The following table illustrates how an example set of Situations and Alerts will be be split"

  28. Suggest:

    Moogsoft employs two databases, an active database and a historic database to enhance performance of the UI and extend data retention capabilities.

  29. via – not a good english tech docs word 

    Suggest

    You can use various closing strategies to help maintain your active database at an optimal size:

    • Manually close alerts
    • Programmatically close alerts
    • Use the Auto CLose feature
  30. +1 for Neil's revision.

  31. by cows. How about

    Historic data retention requires the Housekeeper Moolet to work. Verify you have configured the Housekeeper Moolet and that it is enabled within moogfarmd.

  32. +1 I think OK to edit out at this point.

  33. one word for : enable/confiugre/disable?

    control? 

    You can control historic data retention using the Database Split Enabler at <path>

  34. +1 for Neils second revision


  35. If it is dangerous, maybe we make this the last option on the page. Can we think of a reason why a customer would want to do this? Are we only documenting it so people understand the impact & won't do it?

    How about instead of a box, we move this information before you run the command. Change the intro a little bit:

    You can disable the retention of historic data in a separate database. When you disable the historic database, the data already in the historic database remains there. The Housekeeper Moolet does not move the data back to the active database.

    Neil Billett is the data truly inaccessibble -or only accesible via SQL queries/accessing the database directly?

    I realize it's more words, but it's kind of a way to reiterate the impact to make people think. 

  36. How about we don't use the future passive:

    The following table illustrates how the Housekeeper Moolet splits an example set of Situations and Alerts.

  37. hard stop at end of sentence

  38. no _ in the English name

  39. I don't think this is actually a "code" reference here. suggest moogfarmd w/o monospace type

  40. Not quite true. It would be better to write "The historic database can scale with far less impact on system performance".

  41. Not quite true. It would be more accurate to write "The historic database can scale with far less impact on system performance".