Configure the Message Bus
The Moogsoft Messaging System (MooMS) is the Message Bus component of Moogsoft Onprem and shares event data. This is subscribed to by the various Moolets.
The Message Bus is a publish-subscribe message brokering system implemented with RabbitMQ which uses AMQP, an open standard for message-orientated middleware, over TCP.
Message Handling
The Message Bus handles the data it receives (e.g. raw event data, new alerts, Situation activity etc) by placing it in queues, which are lines of messages waiting to be handled.
Moogsoft Onprem does not enforce any size or time limits on queues, so the maximum number of messages in a queue is limited by the available RAM and disk space on the server. It also depends on the size of the alerts and Situations being generated. The size limit is 128kb for Alerts and 64kb for Situations.
Once the maximum number of messages has been reached, the broker drops messages from the front of the queue to make room for new messages. By default, Moogsoft Onprem applications use exclusive transient queues. For example, if Moogsoft Onprem or the broker shuts down or dies then the queue and all of its messages are lost. Durable queues can be enabled using the message_persistence
setting in $MOOGSOFT_HOME/config/system.conf
.
For more information see the RabbitMQ docs on queues and queue length.
Default Configuration
Moogsoft Onprem is installed with a single RabbitMQ broker by default, running on the same machine as the other components (LAMs, Moogfarmd, the Moolets, etc).
The out-of-the-box configuration in $MOOGSOFT_HOME/config/system.conf
is as follows:
port: 5672 zone: <none> username: moogsoft password: m00gs0ft
The username and password must match the Message Bus broker configuration. If commented out, a default "guest" user will be used (guest: guest).
Zones
You can use zones, or virtual hosts, to share a single RabbitMQ broker cluster among multiple instances of Moogsoft Onprem.
If multiple instances of Moogsoft Onprem share a single RabbitMQ broker then each instance uses a different zone name to prevent message interference. In RabbitMQ, a zone is called a virtual host (vhost
). Clients connecting to one vhost
cannot see messages sent to a different vhost
.
The default deployment does not use zones. If you specify a zone name, you must also configure a vhost
with the same name in the RabbitMQ broker.
By default, Moogsoft Onprem clients connect to the vhost
specified during moog-init.sh
setup with the moogsoft
username and the password.
For distributed installations using multiple RabbitMQ brokers, this must be configured. A zone (vhost
) name is required by the moog-init.sh
setup script. See Message System Deployment.
Message Persistence
You can control and minimize message loss during a shutdown or failure using the following settings in $MOOGSOFT_HOME/config/system.conf
. See the section below for the configurable properties. For other properties, see System Configuration
connections_per_producer_pool
The number of connections to use for each message sender pool. For example, if a message sender pool has 20 channels and this property is set to 2, the channels are split across both connections so that each has 10 channels. To configure this property, you must manually add it to the mooms
section.
Type | Integer |
Required | No |
Default | 2 |
zone
Name of the zone.
Type | String |
Required | No |
Default | N/A |
brokers
Hostname and port number of the RabbitMQ broker.
Type | Array |
Required | No |
Default |
|
username
Username of the RabbitMQ user. This needs to match the RabbitMQ broker configuration. If commented out, it uses the default "guest" user.
Type | String |
Required | No |
Default |
|
password
Password for the RabbitMQ user. You can choose to either have a password or an encrypted password, you cannot use both.
Type | String |
Required | Yes. If you are not using |
Default |
|
encrypted_password
Encrypted password for the RabbitMQ user. You can choose to either have a password or an encrypted password, you cannot use both. See Moog Encryptor if you want to encrypt your password.
Type | String |
Required | Yes. If you are not using |
Default | N/A |
threads
Number of threads a process can create in order to consume the messages from the Message Bus. If not specified, the thread limit = (Number of processors x 2) + 1. Altering this limit affects the performance of Moogsoft Onprem processes such as Moogfarmd and Moogpoller.
If your logs indicate an issue in creating threads, Moogsoft advises that you increase the ulimit, the maximum number of file descriptors each process can use, for the Moogsoft Onprem user. You can set this limit in /etc/security/limits.conf
.
Type | Integer |
Required | No |
Default |
|
message_persistance
Controls whether RabbitMQ persists important messages. Message queues are durable by default and data is replicated between nodes in High Availability mode. Setting this value to false
means that replicated data is not stored to disk.
Type | Boolean |
Required | No |
Default |
|
message_prefetch
Controls how many messages a process can take from the Message Bus and store in memory as a buffer for processing. This configuration allows processes to regulate message consumption which can ease backlog and memory consumption issues. The higher the number, the more messages held in the process's memory. Set to 0 for unlimited processing. To achieve high availability of messages and ensure messages are processed, the value of this should be higher than 0.
Type | Integer |
Required | No |
Default |
|
max_retries
Maximum number of attempts to resend a message that failed to send. Moogsoft Onprem only attempts a retry when there is a network outage or if cache_on_failure
is enabled.
You can use this in conjunction with the retry_interval
property. For example, a combination of 100 maximum retries and 200 milliseconds for retry interval leads to a total of 20 seconds. The combined default value for these properties was chosen to handle the typical time for a broker failover in a clustered environment.
Type | Integer |
Required | No |
Default |
|
retry_interval
Maximum length of time to wait in milliseconds between each attempt to retry and send a message that failed to send.
You can use this in conjunction with the max_retries
property. The combined value for these properties was chosen to handle the typical time for broker failover in a clustered environment.
Type | Integer |
Required | No |
Default |
|
cache_on_failure
Controls whether Moogsoft Onprem caches the message internally and resends it if there is an initial retry failure. The system attempts to resend any cached messages in the order they were cached until the time-to-live value, defined by the cache_ttl
property, is reached.
Type | Boolean |
Required | No |
Default |
|
cache_ttl
Length of time in seconds that Moogsoft Onprem keeps cached messages in the cache list before discarding them. If a message is not successfully resent within this timeframe it is still discarded.
This defaults to 900 seconds (15 minutes). Increasing this value has a direct impact on sender process memory.
Type | Integer |
Required | No |
Default |
|
confirmation_timeout
Length of time in milliseconds to wait for the Message Bus to confirm that a broker has received a message. Moogsoft does not advise changing this value.
Type | Integer |
Required | No |
Default |
|