Configure the Message Bus

The Moogsoft Messaging System (MooMS) is the Message Bus component of Moogsoft AIOps and shares event data. This is subscribed to by the various Moolets.

The Message Bus is a publish-subscribe message brokering system implemented with RabbitMQ which uses AMQP, an open standard for message-orientated middleware, over TCP.

Message Handling

The Message Bus handles the data it receives (e.g. raw event data, new alerts, Situation activity etc) by placing it in queues, which are lines of messages waiting to be handled.

Moogsoft AIOps does not enforce any size or time limits on queues, so the maximum number of messages in a queue is limited by the available RAM and disk space on the server. It also depends on the size of the alerts and Situations being generated. The size limit is 128kb for Alerts and 64kb for Situations.

Once the maximum number of messages has been reached, the broker drops messages from the front of the queue to make room for new messages. By default, Moogsoft AIOps applications use exclusive transient queues. For example, if Moogsoft AIOps or the broker shuts down or dies then the queue and all of its messages are lost. Durable queues can be enabled using the message_persistence setting in $MOOGSOFT_HOME/config/system.conf.

For more information see the RabbitMQ docs on queues and queue length.

Default Configuration

Moogsoft AIOps is installed with a single RabbitMQ broker by default, running on the same machine as the other components (LAMs, Moogfarmd, the Moolets, etc).

The out-of-the-box configuration in $MOOGSOFT_HOME/config/system.conf is as follows:

port: 5672
zone: <none>
username: moogsoft
password: m00gs0ft

The username and password must match the Message Bus broker configuration. If commented out, a default "guest" user will be used (guest: guest).

Zones

You can use zones, or virtual hosts, to share a single RabbitMQ broker cluster among multiple instances of Moogsoft AIOps.

If multiple instances of Moogsoft AIOps share a single RabbitMQ broker then each instance uses a different zone name to prevent message interference. In RabbitMQ, a zone is called a virtual host (vhost). Clients connecting to one vhost cannot see messages sent to a different vhost.

The default deployment does not use zones. If you specify a zone name, you must also configure a vhost with the same name in the RabbitMQ broker.

By default, Moogsoft AIOps clients connect to the vhost specified during moog-init.sh setup with the moogsoft username and the password.

For distributed installations using multiple RabbitMQ brokers, this must be configured. A zone (vhost) name is required by the moog-init.sh setup script. See Message System Deployment.

Message Persistence

You can control and minimize message loss during a shutdown or failure using the following settings in $MOOGSOFT_HOME/config/system.conf. See the section below for the configurable properties. For other properties, see System Configuration

connections_per_producer_pool

The number of connections to use for each message sender pool. For example, if a message sender pool has 20 channels and this property is set to 2, the channels are split across both connections so that each has 10 channels. To configure this property, you must manually add it to the mooms section.

Type

Integer

Required

No

Default

2

zone

Name of the zone.

Type

String

Required

No

Default

N/A

brokers

Hostname and port number of the RabbitMQ broker.

Type

Array

Required

No

Default

"host" : "localhost", "port" : 5672

username

Username of the RabbitMQ user. This needs to match the RabbitMQ broker configuration. If commented out, it uses the default "guest" user.

Type

String

Required

No

Default

guest

password

Password for the RabbitMQ user. You can choose to either have a password or an encrypted password, you cannot use both.

Type

String

Required

Yes. If you are not using encrypted password.

Default

guest

encrypted_password

Encrypted password for the RabbitMQ user. You can choose to either have a password or an encrypted password, you cannot use both. See Moog Encryptor if you want to encrypt your password.

Type

String

Required

Yes. If you are not using password.

Default

N/A

threads

Number of threads a process can create in order to consume the messages from the Message Bus. If not specified, the thread limit = (Number of processors x 2) + 1. Altering this limit affects the performance of Moogsoft AIOps processes such as Moogfarmd and Moogpoller.

If your logs indicate an issue in creating threads, Moogsoft advises that you increase the ulimit, the maximum number of file descriptors each process can use, for the Moogsoft AIOps user. You can set this limit in /etc/security/limits.conf.

Type

Integer

Required

No

Default

10

message_persistance

Controls whether RabbitMQ persists important messages. Message queues are durable by default and data is replicated between nodes in High Availability mode. Setting this value to false means that replicated data is not stored to disk.

Type

Boolean

Required

No

Default

true

message_prefetch

Controls how many messages a process can take from the Message Bus and store in memory as a buffer for processing. This configuration allows processes to regulate message consumption which can ease backlog and memory consumption issues. The higher the number, the more messages held in the process's memory. Set to 0 for unlimited processing. To achieve high availability of messages and ensure messages are processed, the value of this should be higher than 0.

Type

Integer

Required

No

Default

0

max_retries

Maximum number of attempts to resend a message that failed to send. Moogsoft AIOps only attempts a retry when there is a network outage or if cache_on_failure is enabled.

You can use this in conjunction with the retry_interval property. For example, a combination of 100 maximum retries and 200 milliseconds for retry interval leads to a total of 20 seconds. The combined default value for these properties was chosen to handle the typical time for a broker failover in a clustered environment.

Type

Integer

Required

No

Default

100

retry_interval

Maximum length of time to wait in milliseconds between each attempt to retry and send a message that failed to send.

You can use this in conjunction with the max_retries property. The combined value for these properties was chosen to handle the typical time for broker failover in a clustered environment.

Type

Integer

Required

No

Default

200

cache_on_failure

Controls whether Moogsoft AIOps caches the message internally and resends it if there is an initial retry failure. The system attempts to resend any cached messages in the order they were cached until the time-to-live value, defined by the cache_ttl property, is reached.

Type

Boolean

Required

No

Default

false

cache_ttl

Length of time in seconds that Moogsoft AIOps keeps cached messages in the cache list before discarding them. If a message is not successfully resent within this timeframe it is still discarded.

This defaults to 900 seconds (15 minutes). Increasing this value has a direct impact on sender process memory.

Type

Integer

Required

No

Default

900

confirmation_timeout

Length of time in milliseconds to wait for the Message Bus to confirm that a broker has received a message. Moogsoft does not advise changing this value.

Type

Integer

Required

No

Default

2000