Single Site High Availability
This topic guides you through the most basic high availability setup for Moogsoft AIOps where you install two clusters, one primary and one secondary. It covers the configuration for the core Moogsoft AIOps components as follows:
-
How to designate multiple hosts for the Message Bus and search components.
-
How to configure system failover options for the UI servlets, Moogfarmd, and LAMs.
-
How to configure the cluster for the overall system, Moogfarmd, and LAMs.
These instructions don't include guidance for setting up load balancing or other infrastructure topics. For HA for third-party components: RabbitMQ, Elasticsearch, and MySQL, see High Availability for Third Party Component Dependencies.
Note
For the sake of brevity and clarity the sample configurations in this document do not include the default comments delivered in the file.
Before you begin
Before you start your highly available deployment of Moogsoft AIOps:
-
Familiarize yourself with the single-server deployment process: Install Moogsoft AIOps and Upgrade Moogsoft AIOps.
-
Read the High Availability Overview and review the HA Reference Architecture.
-
Set up Moogsoft AIOps on three hosts following the standard installation instructions. The example here uses the
host1
andhost2
andhost3
as example machine names. On the third host, only initialize search and the Message Bus.Tip
If you are using virtual hosts to perform your HA setup, you can set up Moogsoft AIOps and perform the HA configurations on the first host. Then clone your first host to create the second host and third host. Then you only need to change the configurations that differ on the other hosts.
-
Verify that that the hosts can access the required ports on the other hosts in the group. See HA Reference Architecture.
-
Set up master > standby replication for your database. See High Availability for Third Party Component Dependencies.
-
Set up HA for RabbitMQ and Elasticsearch on all three hosts. See High Availability for Third Party Component Dependencies.
Configure system-level HA settings
Edit the system-wide settings to configure HA in the following file: MOOGSOFT_HOME/config/system.conf
. For details on
system.conf, see System
Configuration Reference.
-
Edit the Message Bus (
mooms
) brokers object to include all three hosts. The setting is the same on both host1 and host2. For example:"mooms": { "zone" : "MY_ZONE", "brokers" : [ { "host" : "host1", "port" : 5672 }, { "host" : "host2", "port" : 5672 }, { "host" : "host3", "port" : 5672 }, ], }
-
Edit the database (
mysql
) object to target one of the following:-
The master database. This is OK for testing HA, but requires a manual failover to the standby database server.
-
A VIP or a load balancer for the MySQL database.
The setting is the same on both host1 and host2. For example, if you target the master database on host1:
"mysql": { "host" : "host1", "moogdb_database_name" : "moogdb", "referencedb_database_name" : "moog_reference", "username" : "ermintrude", "password" : "m00", "port" : 3306 }
-
-
Edit the search object to include the host and port configuration for all Elasticsearch hosts. The setting is the same for both host1 and host2. For example:
"search": { "connection_timeout" : 1000, "request_timeout" : 10000, "limit" : 1000, "nodes" : [ { "host" : "host1", "port" : 9200 }, { "host" : "host2", "port" : 9200 }, { "host" : "host3", "port" : 9200 } ] }
-
The failover settings are the same for both host1 and host2. Configure the failover object as follows:
-
Set
persist_state
to true. -
Optionally set
auto_increment
to true if you want Hazelcast to attempt to automatically increment the port number to find an available port in the case that 5701 is not available. -
Add both hosts where you are running Moogfarmd and LAMs to the hosts lists. For example host1 and host2.
-
Disable the
man_center
UI for Hazelcast. -
Set
cluster_per_group
to true. -
Set
automatic_failover
to true. Otherwise, you must use the HA Control utility,ha_cntl
, to manually trigger failover in the event of a component failure. -
In most cases, the default settings for Moogfarmd failover work fine. For noisy environments, you can increase the margin value to 30. For more information on individual settings, see In most cases, the default settings for Moogfarmd failover work fine. For noisy environments, you can increase the margin value to 30. For more information on individual settings, see System Configuration Reference.
For example:
"failover": { "persist_state" : true, "hazelcast": { "network_port" : 5701, "auto_increment" : true, "hosts" : ["host1", "host2"], "man_center": { "enabled" : false, "host" : "localhost", "port" : 8091 }, "cluster_per_group" : true }, "keepalive_interval" : 5, "margin" : 10, "failover_timeout" : 10, "automatic_failover" : true, "heartbeat_failover_after" : 2 },
-
-
Configure the cluster name. This setting differs from host to host. For example on host1:
"ha": { "cluster" : "host1" }
Enable HA for servlets
Edit the servlets settings to configure HA in the following file: MOOGSOFT_HOME/config/servlets.conf
.
Uncomment the HA object and edit the HA properties. This vary from cluster to
cluster. In order to run the UI in an active /active configuration, use
different values for the group name. Additionally, provide descriptive
instance names. Both instances are active, so set start_as_passove
to false. For example, on host1:
ha: { #cluster : "MOO", instance : "servlets-host1", group : "UI_host1", start_as_passive : false }
You don't need to edit the cluster name, the servlets inherit it from system.conf
.
Enable HA for Moogfarmd
Edit the Moogfarmd settings in the following file: $MOOGSOFT_HOME/config/moogfarmd.conf
. The
configurations vary from cluster to cluster based upon your instance names.
Uncomment the HA object and edit the HA properties. For example on host1:
ha: { #cluster : "MOO", group : "moog_farmd", instance : "moogfarmd_host1", default_leader : true, start_as_passive : false }
Tip
Use the host name in your cluster name to help you identify where your processes are running.
Enable HA for LAMs
Each LAM has its own configuration file under MOOGSOFT_HOME/config/
. This example references rest_lam.conf
.
These instructions apply to the the default active / passive mode. Uncomment the HA object and edit the HA properties. This will vary from cluster to cluster based upon your instance names. Moogsoft AIOps automatically manages the active and passive role for the LAMs in a single process group. For example on host1:
ha: { group : "rest_lam", instance : "rest_lam_host1", duplicate_source : false },
Restart components
After you have finished your configurations, restart the Moogsoft AIOps on the hosts running the full stack. See Control Moogsoft AIOps Processes. You can use the HA Control command line utility to check the status of your clusters. See HA Control Utility Command Reference.