Historic Database Benefits

Prior to the release of Moogsoft AIOps 6.4, the single-database architecture limited the amount of data you could retain while running a performant system. Systems with tens of millions of alerts could be slow to display the left navigation filters, especially alert filters. Sluggish performance could also affect the rendering of filter data or a refresh after a filter modification. The threshold for acceptable performance hovered around 15M alerts. In very large systems this typically represented two months of data. Aggressive archive management was the only solution to improve performance.

With Moogsoft AIOps 6.5, separate active and historic databases are created as part of the installation process. The Housekeeper Moolet periodically migrates closed Situation and alert data from the active to the historic database.

The segmentation of open Situations and alerts from closed ones yields a number of performance and scalability improvements. The extent of the benefits depends on your system size and the size of your database.

If you are upgrading from Moogsoft AIOps 6.4 or an earlier release, manually split the database if you want to benefit from using a separate historic database. See Configure Historic Data Retention for further details.

Performance Improvement Metrics

Moogsoft engineering used the following baselines for testing performance differences between split-database and single-database systems:

  • Single Linux server with 64 x 2.3GHz cores

  • 128 GB RAM

  • 13 months of data from a very large system (100M events,14M alerts and 1M Situations)

  • 3% open alerts and Situations.

Splitting the database improved all filtering and loading times. The bar chart below displays the loading times before and after the database split on 13 months of data. Note: most customers could not have operated in production for 13 months with such performance.

29959903.png

The percentage improvements to loading times for different areas of Moogsoft AIOps on the 13 months of data were as follows:

29959904.png

Other improvements included:

  • 12% reduction in the time to run a full search re-index

  • 12% reduction in the time taken for the Alert Builder to process 1M raw events from a Socket LAM to 600,000 events and 4,000 alerts in the database

  • the Cookbook Sigaliser algorithms improved performance.

In this test environment example, the open:closed ratio for alerts and Situations was 3:97. The database split only impacts closed alerts and situations, so you might see different results if your system has a higher proportion of open alerts and Situations.

Next Steps

If you are upgrading to Moogsoft AIOps v. 6.5 and your system has performance issues related to the amount of data in your database, you can learn more about the Configure Historic Data Retention feature and evaluate its benefits:

  • Read the documentation: Configure Historic Data Retention

  • Follow the instructions to implement a database split in a non-production environment so you can test the performance improvements and understand any potential impacts to your workflow.

  • If you had previously implemented aggressive archiving, relax it incrementally and observe the system and user performance over time.

Be aware of the following impacts when you implement database splitting:

  • General system performance depends on the database splitting settings. Aggressive splitting can lead to an increased CPU load because both MySQL and moogfarmd are consuming CPU. You can run the Housekeeper Moolet in its own moogfarmd on a separate host to potentially mitigate any performance impacts.

  • Retaining more data means that your database will increase in size. Implement monitors to check your database growth.