Moogsoft Docs

Set Up the Core Role for HA

In Moogsoft AIOps HA architecture, Core 1 and Core 2 run in an active / passive HA pair.

HA architecture

In our distributed HA installation, the Core components are installed on servers 6, 7, and 8:

HA_core.png

Server 6: Core 1 (Moogfarmd), Elastic Node 1, RabbitMQ Node 1.

Server 7: Core 2 (Moogfarmd), Elastic Node 2, RabbitMQ Node 2.

Server 8: Elastic Node 3, RabbitMQ Node 3.

Fully distributed installation

See Distributed HA Installation for a reference diagram and steps to achieve a fully distributed installation.

Minimally distributed installation

For a minimally distributed installation follow the instructions below, replacing server 6, server 7 and server 8 with the relevant values for your architecture.

Install Core 1

  1. Initialize RabbitMQ Cluster Node 1 on the Core Primary Server.

    On server 6 initialize RabbitMQ. Set the zone:

    moog_init_mooms.sh -pz <ZONE>
  2. Initialize, configure and start Elasticsearch Cluster Node 1 on the Core Primary Server.

    1. Initialize Elasticsearch on server 6:

      moog_init_search.sh
    2. Edit the /etc/elasticsearch/elasticsearch.yml file, replacing its contents with the following.

      Substitute the values as appropriate, for example <server 6 hostname> :

      cluster.name: aiops
      node.name: <server 6 hostname>
      network.host: 0.0.0.0
      http.port: 9200
      discovery.zen.ping.unicast.hosts: [ "<server 6 hostname>","<server 7 hostname>","<server 8 hostname>" ]
      discovery.zen.minimum_master_nodes: 1
      gateway.recover_after_nodes: 1
      node.master: true
  3. Configure system.conf on the Core Primary server.

    1. On server 6 edit the $MOOGSOFT_HOME/config/system.conf file and set the mooms.zone and mooms.brokers properties as follows:

      "zone" : "<ZONE>",
      "brokers" : [ { "host" : "<server 6 hostname>", "port" : 5672 },
         { "host" : "<server 7 hostname>", "port" : 5672 },
         { "host" : "<server 8 hostname>", "port" : 5672 }
      ],
    2. In the same file, edit the search.nodes property as follows:

      "nodes" : [ { "host" : "<server 6 hostname>", "port" : 9200 },
         { "host" : "<server 7 hostname>", "port" : 9200 },
         { "host" : "<server 8 hostname>", "port" : 9200 }
      ]

      Note

      The brokers object must list all hostnames that form the RabbitMQ cluster and nodes must list the hostnames that form the Elasticsearch cluster.

    3. In the same file, set the ha section as follows:

      "ha": { "cluster": "PRIMARY" },
    4. Restart Elasticsearch:

      systemctl restart elasticsearch
  4. Install and configure HA Proxy on the Core Primary server for connection to Percona XtraDB Cluster.

    On server 6 install, configure and start HAProxy.

Install Core 2

  1. Install Moogsoft AIOps components on the Core Secondary server.

    On server 7 install the following Moogsoft AIOps components:

    yum -y install moogsoft-common-7.3* \
        moogsoft-mooms-7.3* \
        moogsoft-search-7.3* \
        moogsoft-server-7.3* \
        moogsoft-utils-7.3* \
        moogsoft-integrations-7.3* 
  2. Initialize RabbitMQ Cluster Node 2 on the Core Secondary server and create the cluster.

    1. On server 7 initialize RabbitMQ. Set a zone name:

      moog_init_moos.sh -pz <ZONE>
    2. Copy and replace the /var/lib/rabbitmq/.erlang.cookie file from server 6 to the same location on this server.

    3. Restart the rabbitmq-server service:

      systemctl restart rabbitmq-server
    4. Run the following commands to create the cluster:

      rabbitmqctl stop_app
      rabbitmqctl join_cluster rabbit@<server 6 hostname>
      rabbitmqctl start_app
    5. Apply HA mirrored queues policy:

      rabbitmqctl set_policy -p <ZONE> ha-all ".+\.HA" '{"ha-mode":"all"}'

      Note

      Replace <ZONE> with the zone name you used earlier.

    6. Verify the cluster status and queue policy. For example:

      rabbitmqctl cluster_status
      Cluster status of node rabbit@ldev02 ...
      [{nodes,[{disc,[rabbit@ldev01,rabbit@ldev02]}]},
       {running_nodes,[rabbit@ldev01,rabbit@ldev02]},
       {cluster_name,<<"rabbit@ldev02">>},
       {partitions,[]},
       {alarms,[{rabbit@ldev01,[]},{rabbit@ldev02,[]}]}]
      [root@ldev02 rabbitmq]# rabbitmqctl -p <ZONE> list_policies
      Listing policies for vhost "MOOG" ...
      <ZONE>    ha-all  .+\.HA  all {"ha-mode":"all"}   0
      
  3. Initialize, configure and start Elasticsearch Cluster Node 2 on the Core Secondary server.

    1. On server 7 initialize Elasticsearch:

      moog_init_search.sh
    2. Edit the /etc/elasticsearch/elasticsearch.yml file, replacing its contents with the following:

      cluster.name: aiops
      node.name: <server 7 hostname>
      network.host: 0.0.0.0
      http.port: 9200
      discovery.zen.ping.unicast.hosts: [ "<server 6 hostname>","<server 7 hostname>","<server 8 hostname>" ]
      discovery.zen.minimum_master_nodes: 1
      gateway.recover_after_nodes: 1
      node.master: true
  4. Configure system.conf on the Core Secondary server.

    1. On server 7 edit the $MOOGSOFT_HOME/config/system.conf and set the mooms.zone and mooms.brokers properties as follows:

      "zone" : "<ZONE>",
      "brokers" : [ { "host" : "<server 6 hostname>", "port" : 5672 },
         { "host" : "<server 7 hostname>", "port" : 5672 },
         { "host" : "<server 8 hostname>", "port" : 5672 }
      ],
    2. In the same file, edit the search.nodes property as follows:

      "nodes" : [ { "host" : "<server 6 hostname>", "port" : 9200 },
         { "host" : "<server 7 hostname>", "port" : 9200 },      
         { "host" : "<server 8 hostname>", "port" : 9200 }
      ]

      Note

      The brokers object must list all hostnames that form the RabbitMQ cluster and nodes must list the hostnames that form the Elasticsearch cluster.

    3. In the same file, set the ha section as follows:

      "ha": { "cluster": "SECONDARY" }
    4. Restart Elasticsearch:

      systemctl restart elasticsearch

    On server 7 edit the $MOOGSOFT_HOME/config/system.conf and set the mooms.zone and mooms.brokers properties as follows:

    and the search.nodes property as follows:

  5. Install and configure HA Proxy on the Core Secondary server for connection to Percona XtraDB Cluster.

    On server 7 install, configure and start HAProxy.

Enable failover

  1. Stop the Moogfarmd service on both Primary, server 6 and Secondary, server 7:

    systemctl stop moogfarmd
  2. On server 6 and server 7 edit the $MOOGSOFT_HOME/config/system.conf file and set the failover.automatic_failover:

    "automatic_failover" : true,
  3. Start the Moogfarmd service on both Primary, server 6 and Secondary, server 7:

    systemctl start moogfarmd