Basic HA Installation - Tarball

This topic describes the basic High Availability (HA) installation for Moogsoft Enterprise using tarball. This installation configuration has three servers; two for the primary and secondary clusters, and a redundancy server.

3_-_Basic_HA_Installation.png

This topic describes how to perform the following tasks for the core Moogsoft Enterprise components:

  • Stage the installation files on all servers.

  • Install the Moogsoft Enterprise packages and set the environment variables.

  • Set up the Percona XtraDB database and HA Proxy.

  • Configure the RabbitMQ message broker and Elasticsearch search service.

  • Configure high availability for the Moogsoft Enterprise core processing components.

  • Initialize the user interface (UI).

  • Configure high availability for data ingestion.

Before you begin

Before you start the tarball basic HA installation of Moogsoft Enterprise:

Install files

  1. Log in as the Linux user for the primary, secondary and redundancy servers. If you have not yet done so, create a working directory in the overall installation directory, and move the pre-install files to this directory.

    Refer to the Moogsoft Enterprise - Offline Tarball pre-installation or Moogsoft Enterprise - Online Tarball pre-installation instructions for information on how big the working directory should be.

    For the following example, /opt/moogsoft is the installation directory and /opt/moogsoft/v7312wd is the working directory.

    mkdir /opt/moogsoft/{VERSION} (recommend making the Linux user the owner of /opt/moogsoft)
    cd /opt/moogsoft/{VERSION}
    mv /tmp/preinstall_files{VERSION}.tgz /opt/moogsoft/{VERSION}
    tar xzf preinstall_files{VERSION}.tgz
  2. Install Kernel Asynchronous I/O (AIO) Support for Linux. For example:

    mkdir -p ~/install/libraries/ 
    mv ./libaio-0.3.109-13.el7.x86_64.rpm ~/install/libraries
    cd ~/install/libraries 
    rpm2cpio ./libaio-0.3.109-13.el7.x86_64.rpm | cpio -idmv && \
    rm -f ./libaio-0.3.109-13.el7.x86_64.rpm && \
    rm -f ~/install/libraries/lib64/libaio.so.1 && \
    ln -s ~/install/libraries/lib64/libaio.so.1.0.1 ~/install/libraries/lib64/libaio
    .so.1 && \echo "export LD_LIBRARY_PATH=`pwd`/lib64:\$LD_LIBRARY_PATH" >> ~/.bashrc && \
    source ~/.bashrc
    cd -
    
  3. Install libgfortran. For example:

    mv ./libquadmath-4.8.5-39.el7.x86_64.rpm ./libgfortran-4.8.5-39.el7.x86_64.rpm ~/install/libraries/
    cd -/install/libraries/
    for PACKAGE in libquadmath-4.8.5-39.el7.x86_64.rpm libgfortran-4.8.5-39.el7.x86_64.rpm; do
        rpm2cpio $PACKAGE | cpio -idmv && \
        rm -f $PACKAGE
    done
    echo "export LD_LIBRARY_PATH=$(pwd)/usr/lib64:\$LD_LIBRARY_PATH" >> ~/.bashrc
    source ~/.bashrc
    cd –
  4. Install Percona dependencies on all servers that will house a database node (requires root permissions):

    yum install *.rpm
    

Prepare to install

Before you install Moogsoft Enterprise:

  • Log in as the Linux user on the primary, secondary and redundancy servers and go to the working directory on each server.

  • Make sure that the directories where you will install Moogsoft Enterprise meet the size requirements stated in the Moogsoft Enterprise - Offline Tarball pre-installation or Moogsoft Enterprise - Online Tarball pre-installation instructions. Some files such as the Percona database and Elasticsearch files will be installed into the Linux user's HOME/install directory.

  • Remove existing environment variables such as $MOOGSOFT_HOME from previous installations.

Install database and Moogsoft Enterprise files on the primary server

  1. Run the following commands on the primary, secondary and redundancy servers that will house a database node:

    cp percona-xtrabackup-2.4.14-Linux-x86_64.libgcrypt153.tar.gz ~/install
    cp Percona-XtraDB-Cluster-5.7.26-rel29-31.37.1.Linux.x86_64.ssl102.tar.gz ~/install
    cp socat-1.7.3.2-2.el7.x86_64.rpm ~/install
    
  2. Run the Percona install script on the primary database server. Substitute the IP addresses of your servers and choose the password for the sstuser.

    bash install_percona_nodes_tarball.sh -p -i <primary_ip>,<secondary_ip>,<redundancy_ip> -u sstuser -w <sstpassword>
  3. To verify that the Percona install was successful, run the following command on the primary server. Substitute the IP address of your primary server:

    curl http://<primary_ip>:9198

    If successful, you see the following message:

    Percona XtraDB Cluster Node is synced.
  4. Source the ~/.bashrc file on the primary server:

    source ~/.bashrc
  5. Now that you've installed the Percona database, you can install Moogsoft Enterprise. Go to your working directory on the primary server, and extract the Moogsoft Enterprise distribution archive:

    source ~/.bashrc
    cd /opt/moogsoft/v{VERSION}
    tar -xf moogsoft-aiops-{VERSION}.tgz
  6. Run the installation script in your primary server working directory to install Moogsoft Enterprise:

    bash moogsoft-aiops-install-{VERSION}.sh

    When prompted, enter the working directory to install Moogsoft Enterprise on the primary server. For example, /opt/moogsoft/aiops. The script guides you through the installation process. You can modify the default installation directory displayed for your environment.

  7. Set the $MOOGSOFT_HOME environment variable to point to your primary server installation directory, and add the following directories to the PATH variable set in ~/.bashrc. In this example, /opt/moogsoft/aiops is the $MOOGSOFT_HOME directory.

    echo "export MOOGSOFT_HOME=/opt/moogsoft/aiops" >> ~/.bashrc
    

    Insert the following code after PATH= in ~/.bashrc:

    .:/opt/moogsoft/aiops/bin:/opt/moogsoft/aiops/bin/utils:/opt/moogsoft/aiops/cots/erlang/bin:/opt/moogsoft/aiops/cots/rabbitmq-server/sbin:
  8. Configure the Tool Runner to execute locally:

    sed -i 's/# execute_locally: false,/,execute_locally: true/1' $MOOGSOFT_HOME/config/servlets

Initialize Moogsoft Enterprise on the primary server

Initialize the database on the primary server:

moog_init_db.sh -qIu root

When prompted for a password, enter the password for the root database user instead of the Linux user. If you are installing Percona on this machine for the first time, leave the password blank and press Enter to continue. The script prompts you to accept the End User License Agreement (EULA) and guides you through the initialization process.

Install Moogsoft Enterprise files on the secondary and redundancy servers

  1. Run the Percona install script on the secondary and redundancy servers. Substitute the IP addresses of your servers and use the same password as for the primary server.

    bash install_percona_nodes_tarball.sh -i <primary IP address>,<secondary IP address>,<redundancy IP address> -u sstuser -w <sstpassword>
    
  2. To verify that the Percona install was successful, run the following command on the secondary and redundancy servers. Substitute the IP address of your primary server:

    curl http://<primary IP address>:9198

    If successful, you see the following message:

    Percona XtraDB Cluster Node is synced 

    It takes a moment for the secondary and redundancy servers to sync. If you do not get this message, wait for a few moments and then try again.

  3. Source the ~/.bashrc file on the secondary and redundancy servers:

    source ~/.bashrc
  4. Now that you've installed the Percona database, you can install Moogsoft Enterprise.Go to your working directory in the secondary and redundancy servers, and extract the Moogsoft Enterprise distribution archive:

    source ~/.bashrc
    cd /opt/moogsoft/v{VERSION}
    tar -xf moogsoft-aiops-{VERSION}.tgz
  5. Run the installation script in your working directory in both servers to install Moogsoft AIOps:

    bash moogsoft-aiops-install-{VERSION}.sh

    When prompted, enter the directory to install AIOps. For example, /opt/moogsoft/aiops. The script guides you through the installation process. You can modify the default installation directory displayed for your environment.

  6. Set the $MOOGSOFT_HOME environment variable to point to your installation directory, and add $MOOGSOFT_HOME/bin/utils to the path. For example:

    echo "export MOOGSOFT_HOME=/opt/moogsoft/aiops" >> ~/.bashrc

    Insert the following code after PATH= in ~/.bashrc:

    .:/opt/moogsoft/aiops/bin:/opt/moogsoft/aiops/bin/utils:/opt/moogsoft/aiops/cots/erlang/bin:/opt/moogsoft/aiops/cots/rabbitmq-server/sbin:

    Source the ~/.bashrc file:

    source ~/.bashrc
  7. Configure the Tool Runner to execute locally:

    sed -i 's/# execute_locally: false,/,execute_locally: true/1' $MOOGSOFT_HOME/config/servlets

Install HA Proxy on primary and secondary servers

Install HA Proxy on the primary and secondary servers (root permission required).

  1. Run the following command, using your chosen value for MOOGSOFT_HOME. Substitute the IP addresses of your servers.

    export MOOGSOFT_HOME=/opt/moogsoft/aiops
    $MOOGSOFT_HOME/bin/utils/haproxy_installer.sh -l 3309 -c -i <primary_ip>:3306,<secondary_ip>:3306,<redundancy_ip>:3306
  2. Restart Apache Tomcat and Moogfarmd to start using HA Proxy:

    process_cntl --process_name apache-tomcat restart
    
    process_cntl --process_name moog_farmd restart
    
  3. Run the following script to confirm successful installation:

    $MOOGSOFT_HOME/bin/utils/check_haproxy_connections.sh

    If successful, you see a script output similar to the following example:

    HAProxy Connection Counts
    Frontend:
        0.0.0.0:3309 : 27
    Backend:
        mysql_node_1 172.31.82.211:3306 : 27
        mysql_node_2 172.31.82.133:3306 : 0
        mysql_node_3 172.31.85.42:3306 : 0

Set up RabbitMQ on all servers

Initialize and configure RabbitMQ on all three servers.

Primary, Secondary and Redundancy servers:

  1. Run the following command. Substitute a name for your zone. You must use the same zone name for all servers.

    moog_init_mooms.sh -pz <my_zone>
  2. Stop RabbitMQ on the secondary and redundancy servers:

    process_cntl --process_name rabbitmq stop
  3. The primary server erlang cookie is located in the Linux user's HOME directory: ~/.erlang.cookie. The erlang cookie must be the same for all RabbitMQ nodes.

    Replace the erlang cookie on the secondary and redundancy servers with the erlang cookie from the primary server. You may need to change the file permissions on the secondary and redundancy erlang cookies first to allow those files to be overwritten.

    After copying the cookie, as the Linux user, make the cookies on the secondary and redundancy servers read-only. For example:

    cd ~
    chmod 406 ~/.erlang.cookie
    mv .erlang.cookie .erlang.cookie.orig
    scp moogsoft@<PRIMARY_IP>:.erlang.cookie .
    chmod 400 .erlang.cookie
    

Secondary and Redundancy servers:

  1. Restart RabbitMQ on the secondary and redundancy servers and join those servers to the cluster. Substitute the short hostname of your primary server and the name of your zone.

    The short hostname is the full hostname excluding the DNS domain name. For example, if the hostname is ip-172-31-82-78.ec2.internal, the short hostname is ip-172-31-82-78. To find out the short hostname, run rabbitmqctl cluster_status on the primary server.

    The restart command differs depending on whether you are a root or a non-root user:

    • Root: systemctl restart rabbitmq-server

    • Non-root: process_cntl --process_name rabbitmq restart

    <restart command>
    rabbitmqctl stop_app
    rabbitmqctl join_cluster rabbit@<primary_short_hostname>
    rabbitmqctl start_app
    rabbitmqctl set_policy -p <my_zone> ha-all ".+\.HA" '{"ha-mode":"all"}'
    
  2. Run rabbitmqctl cluster_status to get the cluster status. Example output is as follows:

    Cluster status of node rabbit@secondary ...
    [{nodes,[{disc,[rabbit@primary,rabbit@secondary]}]},
     {running_nodes,[rabbit@primary,rabbit@secondary]},
     {cluster_name,<<"rabbit@secondary">>},
     {partitions,[]},
     {alarms,[{rabbit@primary,[]},{rabbit@secondary,[]}]}]

Set up Elasticsearch on all servers

You must start Elasticsearch with specific memory parameters to support HA. The process_cntl utility starts Elasticsearch.

  1. Run the following on the primary, secondary and redundancy servers to adjust the process_cntl utility:

    cp –p $MOOGSOFT_HOME/bin/utils/process_cntl $MOOGSOFT_HOME/bin/utils/process_cntl.orig
    sed -i 's/-Xms256m/-Xms2g/' $MOOGSOFT_HOME/bin/utils/process_cntl
  2. Initialize and configure Elasticsearch on all three servers:

    moog_init_search.sh
  3. Uncomment and edit the properties in $MOOGSOFT_HOME/cots/elasticsearch/config/elasticsearch.ymlon all three servers as follows

    cluster.name: aiops
    node.name: <server_hostname>
    ...
    network.host: 0.0.0.0
    http.port: 9200
    discovery.zen.ping.unicast.hosts: ["<PRIMARY_HOSTNAME>","<SECONDARY_HOSTNAME>","<REDUNDANCY_HOSTNAME>"]
    discovery.zen.minimum_master_nodes: 2
    gateway.recover_after_nodes: 1
    node.master: true
  4. Restart Elasticsearch on all three servers:

    process_cntl --process_name elasticsearch restart
  5. Get the health status of the cluster by running the following on the primary server:

    curl -X GET "localhost:9200/_cat/health?v&pretty"

    Example cluster health status:

    epoch timestamp cluster status 
    node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1580490422 17:07:02 aiops green 3 3 0 0 0 0 0 0 - 100.0%

    If you've configured all three servers, the node.total will be 3.

Configure Moogsoft Enterprise

Configure Moogsoft Enterprise by editing the Moogfarmd and system configuration files on the primary and secondary servers.

Primary and Secondary servers:

  1. Make a copy of the $MOOGSOFT_HOME/config/system.conf file:

    cp -p $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.orig
  2. Edit $MOOGSOFT_HOME/config/system.conf and set the following properties. Substitute the name of your RabbitMQ zone, the server hostnames, and the cluster names:

    "mooms" :
       {
    ...
    "zone" : "<my_zone>",
    
    "brokers" : [
        {"host" : "<primary_hostname>", "port" : 5672},
        {"host" : "<secondary_hostname>", "port" : 5672},
        {"host" : "<redundancy_hostname>", "port" : 5672}
    ],
    ...
    "cache_on_failure" : true,
    ...
    "search" :
        {
    ...
      "nodes" : [
          {"host" : "<primary_hostname>", "port" : 9200},
          {"host" : "<secondary_hostname>", "port" : 9200},
          {"host" : "<redundancy_hostname>", "port" : 9200}
    ]
    ...
    "failover" :
        {
        "persist_state" : true,
        "hazelcast" :
            {
            "hosts" : ["<primary_hostname>","<secondary_hostname>"],
            "cluster_per_group" : true
            }
        "automatic_failover" : true,
        }
    ...
    "ha": 
        { "cluster": "<cluster_name, primary or secondary>" }
  3. Make a copy of the $MOOGSOFT_HOME/config/moog_farmd.conf file:

    cp -p $MOOGSOFT_HOME/config/moog_farmd.conf $MOOGSOFT_HOME/config/moog_farmd.conf.orig
  4. Uncomment and edit the following properties in $MOOGSOFT_HOME/config/moog_farmd.conf. Note the importance of the initial comma. Delete the cluster line in this section of the file:

    Primary server

    ,
    ha:
    {
        group: "moog_farmd",
        instance: "moog_farmd",
        default_leader: true,
        start_as_passive: false
    }

    Secondary server

    ,
    ha:
    {
        group: "moog_farmd",
        instance: "moog_farmd",
        default_leader: false,
        start_as_passive: false
    }
  5. Start Moogfarmd on the primary and secondary servers:

    process_cntl --process_name moog_farmd start

Initialize the user interface

Run the initialization script moog_init_ui.sh on the primary server. Substitute the name of your RabbitMQ zone and primary hostname.

When asked if you want to change the configuration hostname, say yes and enter the public hostname/domain name used in the URL to access AIOps via browser. If the URL uses a different hostname/domain name (e.g., an alias) the system will reject your login.

Primary server:

moog_init_ui.sh -twfz <MY_ZONE> -c <primary_hostname>:15672 -m <primary_hostname>:5672 -s <primary_hostname>:9200 -d <primary_hostname>:3309 -n

Edit the servlets settings on the primary server in the file $MOOGSOFT_HOME/config/servlets.conf. Note the importance of the initial comma.

,ha :
{    
    instance: "servlets",    
    group: "servlets_primary",    
    start_as_passive: false
}

Restart Apache Tomcat and Moogfarmd on the primary server:

process_cntl --process_name apache-tomcat restart
process_cntl --process_name moog_farmd restart

Run the initialization script moog_init_ui.sh on the secondary server. Substitute the name of your RabbitMQ zone.

When asked if you want to change the configuration hostname, say yes and enter the public hostname/domain name used in the URL to access AIOps via browser.

Secondary server:

moog_init_ui.sh -twfz MY_ZONE -c <SECONDARY_HOSTNAME>:15672 -m <SECONDARY_HOSTNAME>:5672 -s <SECONDARY_HOSTNAME>:9200 -d <SECONDARY_HOSTNAME>:3309 -n

Edit the servlets settings in the secondary server $MOOGSOFT_HOME/config/servlets.conf file. Note the importance of the initial comma.

,ha :
{
    instance: "servlets",     
    group: "servlets_secondary",    
    start_as_passive: false
}

Restart Apache Tomcat and Moogfarmd on the primary server secondary server:

process_cntl --process_name apache-tomcat restart
process_cntl --process_name moog_farmd restart

Enable HA for LAMs

There are two types of HA configuration for LAMs; Active/Active and Active/Passive:

  • Receiving LAMs that listen for events are configured as Active/Active. For example, the REST LAM.

  • Polling LAMs are configured as Active/Passive. For example, the SolarWinds LAM.

Every LAM has its own configuration file under $MOOGSOFT_HOME/config/. This example references rest_lam.conf and solarwinds_lam.conf.

Primary and Secondary servers

Edit the HA properties in the primary and secondary servers' LAM configuration files. Moogsoft Enterprise automatically manages the active and passive role for the LAMs in a single process group:

# Receiving LAM (Active / Active)
# Configuration on Primary
ha:
{
    group            : "rest_lam_primary",
    instance         : "rest_lam",
    duplicate_source : false
},
...
# Configuration on Secondary
ha:
{
    group            : "rest_lam_secondary",
    instance         : "rest_lam",
    duplicate_source : false
},
# Polling LAM (Active / Passive)
# Configuration on Primary
ha:
{
    group                    : "solarwinds_lam",
    instance                 : "solarwinds_lam",
    only_leader_active       : true,
    default_leader           : true,
    accept_conn_when_passive : false,
    duplicate_source         : false
},
...
# Configuration on Secondary
ha:
{
    group                    : "solarwinds_lam",
    instance                 : "solarwinds_lam",
    only_leader_active       : true,
    default_leader           : false,
    accept_conn_when_passive : false,
    duplicate_source         : false
},

Start the LAMs:

process_cntl --process_name restlamd restart 
process_cntl --process_name solarwindslamd restlamd restart 

Run the HA Control command line utility ha_cntl -v to check the status of the LAMS:

Moogsoft AIOps Version{VERSION}
(C) Copyright 2012-2020 Moogsoft, Inc.
All rights reserved.
Executing: ha_cntl
Getting system status
Cluster: [PRIMARY] active
        ...
	Process Group: [rest_lam_primary] Active (no leader - all can be active)
		Instance: [rest_lam] Active
        ...
        Process Group: [solarwinds_lam] Active (only leader should be active)
                Instance: [solarwinds_lam] Active Leader 

Cluster: [SECONDARY] partially active
        ...
	Process Group: [rest_lam_secondary] Passive (no leader - all can be active)
		Instance: [rest_lam] Active
        ...
        Process Group: [solarwinds_lam] Passive (only leader should be active)
                Instance: [solarwinds_lam] Passive

For more information, see the HA Control Utility Command Reference.

Validate the installation