Single Site High Availability

This topic guides you through the most basic high availability setup for Moogsoft AIOps where you install two clusters, one primary and one secondary. It covers the configuration for the core Moogsoft AIOps components as follows:

  • How to designate multiple hosts for the Message Bus and search components.

  • How to configure system failover options for the UI servlets, Moogfarmd, and LAMs.

  • How to configure the cluster for the overall system, Moogfarmd, and LAMs.

These instructions don't include guidance for setting up load balancing or other infrastructure topics. For HA for third-party components: RabbitMQ, Elasticsearch, and MySQL, see High Availability for Third Party Component Dependencies.

Note

For the sake of brevity and clarity the sample configurations in this document do not include the default comments delivered in the file.

Before you begin

Before you start your highly available deployment of Moogsoft AIOps:

Configure system-level HA settings

Edit the system-wide settings to configure HA in the following file: MOOGSOFT_HOME/config/system.conf. For details on system.conf, see System Configuration Reference.

  1. Edit the Message Bus (mooms) brokers object to include all three hosts. The setting is the same on both host1 and host2. For example:

    "mooms":
    {
       "zone" : "MY_ZONE",
       "brokers" : [
           {
               "host" : "host1",
               "port" : 5672
           },
           {
               "host" : "host2",
               "port" : 5672 
           },
           {
               "host" : "host3",
               "port" : 5672 
           },
       ],
    }
  2. Edit the database (mysql) object to target one of the following:

    • The master database. This is OK for testing HA, but requires a manual failover to the standby database server.

    • A VIP or a load balancer for the MySQL database.

    The setting is the same on both host1 and host2. For example, if you target the master database on host1:

    "mysql":
    {
        "host"                      : "host1",
        "moogdb_database_name"      : "moogdb",
        "referencedb_database_name" : "moog_reference",
        "username"                  : "ermintrude",
        "password"                  : "m00",
        "port"                      : 3306
    }
  3. Edit the search object to include the host and port configuration for all Elasticsearch hosts. The setting is the same for both host1 and host2. For example:

    "search":
    {
        "connection_timeout"  : 1000,
        "request_timeout" : 10000,
        "limit" : 1000,
        "nodes" : [
        {
            "host" : "host1",
            "port" : 9200
        },
        {
            "host" : "host2",
            "port" : 9200
        },
        {
             "host" : "host3",
             "port" : 9200
        }
        ]
    }
  4. The failover settings are the same for both host1 and host2. Configure the failover object as follows:

    • Set persist_state to true.

    • Optionally set auto_increment to true if you want Hazelcast to attempt to automatically increment the port number to find an available port in the case that 5701 is not available.

    • Add both hosts where you are running Moogfarmd and LAMs to the hosts lists. For example host1 and host2.

    • Disable the man_center UI for Hazelcast.

    • Set cluster_per_group to true.

    • Set automatic_failover to true. Otherwise, you must use the HA Control utility, ha_cntl, to manually trigger failover in the event of a component failure.

    • In most cases, the default settings for Moogfarmd failover work fine. For noisy environments, you can increase the margin value to 30. For more information on individual settings, see In most cases, the default settings for Moogfarmd failover work fine. For noisy environments, you can increase the margin value to 30. For more information on individual settings, see System Configuration Reference.

    For example:

    "failover":
    {
        "persist_state" : true,
        "hazelcast":
        {
            "network_port"         : 5701,
            "auto_increment"       : true,
            "hosts"                : ["host1", "host2"],
            "man_center":
            {
                "enabled"          : false,
                "host"             : "localhost",
                "port"             : 8091
            },
            "cluster_per_group"    : true
        },
        "keepalive_interval"       : 5,
        "margin"                   : 10,
        "failover_timeout"         : 10,
        "automatic_failover"       : true,
        "heartbeat_failover_after" : 2
    },
  5. Configure the cluster name. This setting differs from host to host. For example on host1:

    "ha": 
    {
        "cluster" : "host1"
    }

Enable HA for servlets

Edit the servlets settings to configure HA in the following file: MOOGSOFT_HOME/config/servlets.conf.

Uncomment the HA object and edit the HA properties. This vary from cluster to cluster. In order to run the UI in an active /active configuration, use different values for the group name. Additionally, provide descriptive instance names. Both instances are active, so set start_as_passove to false. For example, on host1:

ha:
{
    #cluster         : "MOO",
    instance         : "servlets-host1",
    group            : "UI_host1",
    start_as_passive : false
}

You don't need to edit the cluster name, the servlets inherit it from system.conf.

Enable HA for Moogfarmd

Edit the Moogfarmd settings in the following file: $MOOGSOFT_HOME/config/moogfarmd.conf. The configurations vary from cluster to cluster based upon your instance names.

Uncomment the HA object and edit the HA properties. For example on host1:

ha:
{
    #cluster         : "MOO",
    group            : "moog_farmd",
    instance         : "moogfarmd_host1",
    default_leader   : true,
    start_as_passive : false
}

Tip

Use the host name in your cluster name to help you identify where your processes are running.

Enable HA for LAMs

Each LAM has its own configuration file under MOOGSOFT_HOME/config/. This example references rest_lam.conf.

These instructions apply to the the default active / passive mode. Uncomment the HA object and edit the HA properties. This will vary from cluster to cluster based upon your instance names. Moogsoft AIOps automatically manages the active and passive role for the LAMs in a single process group. For example on host1:

ha:
{
    group            : "rest_lam",
    instance         : "rest_lam_host1",
    duplicate_source : false
},

Restart components

After you have finished your configurations, restart the Moogsoft AIOps on the hosts running the full stack. See Control Moogsoft AIOps Processes. You can use the HA Control command line utility to check the status of your clusters. See HA Control Utility Command Reference.