Moogsoft Docs

Set Up the Redundancy Server Role

In Moogsoft AIOps HA architecture, both RabbitMQ and ElasticSearch run as three-node clusters. The three-node clusters prevent issues with ambiguous data state, such as a "split-brain".

RabbitMQ is the Message Bus used by Moogsoft AIOps. Elasticsearch delivers the search functionality.

The three nodes are distributed across the two Core roles and the redundancy server.

HA architecture

In our distributed HA installation, the RabbitMQ and Elasticsearch components are installed on servers 6, 7 and 8.

  • Server 6: RabbitMQ Node 1 (part of Core 1)

  • Server 7: RabbitMQ Node 2 (part of Core 2)

  • Server 8: RabbitMQ Node 3 (part of Redundancy Server)

  • Server 6: Elasticsearch Node 1 (part of Core 1)

  • Server 7: Elasticsearch Node 2 (part of Core 2)

  • Server 8: Elasticsearch Node 3 (part of Redundancy Server)

Fully distributed installation

See Distributed HA Installation for a reference diagram and steps to achieve a fully distributed installation.

Minimally distributed installation

For a minimally distributed installation follow the same instructions below, replacing server 8 with the relevant value for your architecture.

Install Redundancy server

  1. Install the Moogsoft AIOps components on the Redundancy server.

    On server 8 install the following Moogsoft AIOps components:

    VERSION=7.3.1; yum -y install moogsoft-common-${VERSION} \
        moogsoft-mooms-${VERSION} \
        moogsoft-search-${VERSION} \
  2. Initialize RabbitMQ Cluster Node 3 on the Redundancy server and join the cluster.

    1. On server 8 initialise RabbitMQ. Set a zone name: -pz <ZONE>
    2. Copy and replace the /var/lib/rabbitmq/.erlang.cookie file from server 6 to the same location on this server.

    3. Restart the rabbit-mq server service:

      systemctl restart rabbitmq-server
    4. Run the following commands to form the cluster:

      rabbitmqctl stop_app
      rabbitmqctl join_cluster rabbit@<server 6 hostname>
      rabbitmqctl start_app
    5. Apply the HA mirrored queues policy:

      rabbitmqctl set_policy -p <ZONE> ha-all ".+\.HA" '{"ha-mode":"all"}'


      Replace <ZONE> with the zone name you used earlier.

    6. Verify the cluster status and queue policy. For example:

      rabbitmqctl cluster_status
      Cluster status of node rabbit@ldev02 ...
      [root@ldev02 rabbitmq]# rabbitmqctl -p MOOG list_policies
      Listing policies for vhost "MOOG" ...
      MOOG    ha-all  .+\.HA  all {"ha-mode":"all"}   0
  3. Initialise, configure and start Elasticsearch Cluster Node 3 on the Redundancy server.

    1. On server 8 initialize Elasticsearch:
    2. Edit the /etc/elasticsearch/elasticsearch.yml file, replacing its contents with the following: aiops <server 8 hostname>
      http.port: 9200 [ "<server 6 hostname>","<server 7 hostname>","<server 8 hostname>" ]
      discovery.zen.minimum_master_nodes: 1
      gateway.recover_after_nodes: 1
      node.master: true
    3. Restart Elasticsearch on all three servers:

      systemctl restart elasticsearch
  4. Verify that the Elasticsearch nodes are working correctly. For example:

    curl -X GET "localhost:9200/_cat/health?v&pretty"