Configure the Message Bus
The Moogsoft Messaging System (MooMS) is the Message Bus component of Moogsoft AIOps and shares event data. This is subscribed to by the various Moolets.
The Message Bus is a publish-subscribe message brokering system implemented with RabbitMQ which uses AMQP, an open standard for message-orientated middleware, over TCP.
Message Handling
The Message Bus handles the data it receives (e.g. raw event data, new alerts, Situation activity etc) by placing it in queues, which are lines of messages waiting to be handled.
Moogsoft AIOps does not enforce any size or time limits on queues, so the maximum number of messages in a queue is limited by the available RAM and disk space on the server. It also depends on the size of the alerts and Situations being generated. The size limit is 128kb for Alerts and 64kb for Situations.
Once the maximum number of messages has been reached, the broker drops messages from the front of
the queue to make room for new messages. By default, Moogsoft AIOps
applications use exclusive transient queues. For example, if Moogsoft
AIOps or the broker shuts down or dies then the queue and all of its messages are lost.
Durable queues can be enabled using the message_persistence
setting in
$MOOGSOFT_HOME/config/system.conf
.
For more information see the RabbitMQ docs on queues and queue length.
Default Configuration
Moogsoft AIOps is installed with a single RabbitMQ broker by default, running on the same machine as the other components (LAMs, Moogfarmd, the Moolets, etc).
The out-of-the-box configuration in $MOOGSOFT_HOME/config/system.conf
is as follows:
port: 5672 zone: <none> username: moogsoft password: m00gs0ft
The username and password must match the Message Bus broker configuration. If commented out, a default "guest" user will be used (guest: guest).
Zones
You can use zones, or virtual hosts, to share a single RabbitMQ broker cluster among multiple instances of Moogsoft AIOps.
If multiple instances of Moogsoft AIOps share a single RabbitMQ
broker then each instance uses a different zone name to prevent message interference. In RabbitMQ,
a zone is called a virtual host (vhost
). Clients connecting to one vhost
cannot see messages sent to a different vhost
.
The default deployment does not use zones. If you specify a zone name, you must also configure a
vhost
with the same name in the RabbitMQ broker.
By default, Moogsoft AIOps clients connect to the vhost
specified during moog-init.sh
setup with the
moogsoft
username and the password.
For distributed installations using multiple RabbitMQ brokers, this must be configured. A zone
(vhost
) name is required by the moog-init.sh
setup script. See Message System Deployment.
Message Persistence
You can control and minimize message loss during a shutdown or failure using the following
settings in $MOOGSOFT_HOME/config/system.conf
. See the section below for
the configurable properties. For other properties, see System
Configuration
connections_per_producer_pool
The number of connections to use for each message sender pool. For example, if a message sender
pool has 20 channels and this property is set to 2, the channels are split across both
connections so that each has 10 channels. To configure this property, you must manually add it
to the mooms
section.
Type |
Integer |
Required |
No |
Default |
2 |
zone
Name of the zone.
Type |
String |
Required |
No |
Default |
N/A |
brokers
Hostname and port number of the RabbitMQ broker.
Type |
Array |
Required |
No |
Default |
|
username
Username of the RabbitMQ user. This needs to match the RabbitMQ broker configuration. If commented out, it uses the default "guest" user.
Type |
String |
Required |
No |
Default |
|
password
Password for the RabbitMQ user. You can choose to either have a password or an encrypted password, you cannot use both.
Type |
String |
Required |
Yes. If you are not using |
Default |
|
encrypted_password
Encrypted password for the RabbitMQ user. You can choose to either have a password or an encrypted password, you cannot use both. See Moog Encryptor if you want to encrypt your password.
Type |
String |
Required |
Yes. If you are not using |
Default |
N/A |
threads
Number of threads a process can create in order to consume the messages from the Message Bus. If not specified, the thread limit = (Number of processors x 2) + 1. Altering this limit affects the performance of Moogsoft AIOps processes such as Moogfarmd and Moogpoller.
If your logs indicate an issue in creating threads, Moogsoft
advises that you increase the ulimit, the maximum number of file descriptors each process can
use, for the Moogsoft AIOps user. You can set this limit in /etc/security/limits.conf
.
Type |
Integer |
Required |
No |
Default |
|
message_persistance
Controls whether RabbitMQ persists important messages. Message queues are durable by default
and data is replicated between nodes in High Availability mode. Setting this value to false
means that replicated data is not stored to disk.
Type |
Boolean |
Required |
No |
Default |
|
message_prefetch
Controls how many messages a process can take from the Message Bus and store in memory as a buffer for processing. This configuration allows processes to regulate message consumption which can ease backlog and memory consumption issues. The higher the number, the more messages held in the process's memory. Set to 0 for unlimited processing. To achieve high availability of messages and ensure messages are processed, the value of this should be higher than 0.
Type |
Integer |
Required |
No |
Default |
|
max_retries
Maximum number of attempts to resend a message that failed to send. Moogsoft AIOps only attempts a retry when there is a network outage or
if cache_on_failure
is enabled.
You can use this in conjunction with the retry_interval
property. For
example, a combination of 100 maximum retries and 200 milliseconds for retry interval leads to a
total of 20 seconds. The combined default value for these properties was chosen to handle the
typical time for a broker failover in a clustered environment.
Type |
Integer |
Required |
No |
Default |
|
retry_interval
Maximum length of time to wait in milliseconds between each attempt to retry and send a message that failed to send.
You can use this in conjunction with the max_retries
property. The
combined value for these properties was chosen to handle the typical time for broker failover in
a clustered environment.
Type |
Integer |
Required |
No |
Default |
|
cache_on_failure
Controls whether Moogsoft AIOps caches the message internally and
resends it if there is an initial retry failure. The system attempts to resend any cached
messages in the order they were cached until the time-to-live value, defined by the cache_ttl
property, is reached.
Type |
Boolean |
Required |
No |
Default |
|
cache_ttl
Length of time in seconds that Moogsoft AIOps keeps cached messages in the cache list before discarding them. If a message is not successfully resent within this timeframe it is still discarded.
This defaults to 900 seconds (15 minutes). Increasing this value has a direct impact on sender process memory.
Type |
Integer |
Required |
No |
Default |
|
confirmation_timeout
Length of time in milliseconds to wait for the Message Bus to confirm that a broker has received a message. Moogsoft does not advise changing this value.
Type |
Integer |
Required |
No |
Default |
|