Skip to main content

Monitor Component Performance

Moogsoft Onprem features the ability to ingest large amounts of event data from various sources, process the data using configurable logic, and display the data to multiple concurrent users. This document outlines the various system components and how their interactions can impact system performance. It includes performance tuning suggestions where applicable.

To learn about opportunities to plan your implementation for increased performance capabilities, see Scale Your Moogsoft Onprem Implementation.Scale Your Moogsoft Onprem Implementation

For information on monitoring your system performance and handling performance issues, see Monitor and Troubleshoot Moogsoft Onprem.

System Component Summary

Moogsoft Onprem comprises several components which have tuning and configuration options available:

  • Integrations and LAMs that listen or poll for data, parse and encode them into discrete events, and then pass the events to the Message Bus.

  • The Message Bus (RabbitMQ) that receives published messages from integrations and LAMs. It publishes messages destined for data processing (Moogfarmd) and the web application server.

  • The system datastore (MySQL) that handles transactional data from other parts of the system: integrations and LAMs, data processing, and the web application server.

  • The data processing component (Moogfarmd), an application that consumes messages from the Message Bus. It processes event data in a series of servlet-like modules called Moolets. Moogfarmd reads and writes to the database and publishes messages to the bus. See Monitor Moogfarmd Health Logs for more details.

  • The web application server (Apache Tomcat) that reads and writes to the bus and the database.

The diagram below shows the general data flow of the components:

v9.png

Other components include:

  • A proxy (Ngnix) for the web application server and for integrations. See the Ngnix docs for more information.

  • The search engine (OpenSearch) for the UI that indexes documents from the indexer Moolet in the data processing series. It returns search results to Apache Tomcat. See the OpenSearch documentation for more information.

Integration Performance

Event data enters the system via integrations and LAMs. Integrations and LAMs running on a large system can normally process up to 10,000 events per secondand publish them to the Message Bus at an equivalent rate. Integrations can buffer events under event storm conditions. The following factors affect the capacity to process events:

  • CPU clock speed and number of available cores.

  • Threads setting.

  • Complexity of the LAMbot logic.

  • Whether you have enabled "guaranteed delivery settings". For example, rest_response_mode for the REST LAM.

  • Log level. For example, DEBUG is the slowest.

You can specify a value for the number of threads in the LAM's configuration file to control the number of active threads for the integration. To tune the socket LAM, for example, edit socket_lam.conf. Increasing the number of threads can improve ingestion performance. However it will also result in higher CPU usage and may cause internal buffering. Buffering increases memory usage until the buffer is cleared.

Message Bus Performance

RabbitMQ is very lightweight and, in all known cases, has been able to process the incoming event rate from Integrations. Refer to the RabbitMQ documentation for its performance tuning options.

Database Performance

Manage and tune your MySQL instance as you would any other database system in your enterprise. In addition to the standard tuning options in the MySQL documentation, consider the following recommendations for settings in /etc/my.cnf :

  • On servers with >= 16 GB RAM that run MySQL and Moogsoft Onprem applications ,set innodb-buffer-pool-size to 50% of system RAM.

  • On servers where only MySQL is running, set innodb-buffer-pool-size to 80% of system RAM.

  • If innodb-buffer-pool-size > 8 GB, increase the innodb-buffer-pool-instances to divide the buffer buffer pool into 1 G (GB) chunks to the maximum supported value of 64 G. For example, if your buffer pool size is 64 GB:

    innodb-buffer-pool-size=64G
innodb_buffer_pool_instances=64

JVM Performance

Integrations and LAMs, Moogfarmd, and Tomcat are all Java processes so you can tune the memory allocation pool settings for the JVM to optimize performance. This -Xmx setting defines the maximum allowed Java heap size of the process. The default memory allocation for a Java process is one quarter of the server's RAM.

For LAMs, integrations and Moogfarmd, you can add the -Xmx argument to the line in $MOOGSOFT_HOME/bin/<lam name> or $MOOGSOFT_HOME/bin/moog_farmd where the JVM is launched.

For example, to set the maximum Java heap size for the Moogfarmd process to 16 GB, add "-Xmx16g" to the java_vm command line in $MOOGSOFT_HOME/bin/moog_farmd as follows:

#Run app
$java_vm -server -Xmx16g -DprocName=$proc_name -DMOOGSOFT_HOME=$MOOGSOFT_HOME -classpath $java_classpath $java_main_class "$@" &

For Tomcat, the default setting is 2GB. If you need to change it you can edit the service script /etc/init.d/apache-tomcat.

In this section: