Basic HA Installation - Tarball

This topic describes the basic High Availability (HA) installation for Moogsoft Enterprise using tarball. This installation configuration has three servers; two for the primary and secondary clusters, and a redundancy server.

This topic describes how to perform the following tasks for the core Moogsoft Enterprise components:

Stage the installation files on all servers.
Install the Moogsoft Enterprise packages and set the environment variables.
Set up the Percona XtraDB database and HA Proxy.
Configure the RabbitMQ message broker and Opensearch/Elasticsearch search service.
Configure high availability for the Moogsoft Enterprise core processing components.
Initialize the user interface (UI).
Configure high availability for data ingestion.

Before you begin

Before you start the tarball basic HA installation of Moogsoft Enterprise:

Familiarize yourself with the single-server deployment process: Install Moogsoft Enterprise and Upgrade Moogsoft Enterprise.
Read the High Availability Overview and the HA Reference Architecture.
Verify that the hosts can access the required ports on the other hosts in the group. See HA Reference Architecture for more information.
Complete either the Moogsoft Enterprise - Offline Tarball pre-installation or Moogsoft Enterprise - Online Tarball pre-installation instructions.

Install files

Log in as the Linux user for the primary, secondary and redundancy servers. If you have not yet done so, create a working directory in the overall installation directory, and move the pre-install files to this directory.
Refer to the Moogsoft Enterprise - Offline Tarball pre-installation or Moogsoft Enterprise - Online Tarball pre-installation instructions for information on how big the working directory should be.
For the following example, /opt/moogsoft is the installation directory and /opt/moogsoft/v800wd is the working directory.
```
VERSION=8.2.0
mkdir /opt/moogsoft/${VERSION} #(recommend making the Linux user the owner of /opt/moogsoft)
cd /opt/moogsoft/${VERSION}
mv /tmp/preinstall_files${VERSION}.tgz /opt/moogsoft/${VERSION}
tar xzf preinstall_files${VERSION}.tgz
```

Install Kernel Asynchronous I/O (AIO) Support for Linux. For example:

mkdir -p ~/install/libraries/ 
mv preinstall_files/libaio-0.3.109-13.el7.x86_64.rpm ~/install/libraries
cd ~/install/libraries 
rpm2cpio ./libaio-0.3.109-13.el7.x86_64.rpm | cpio -idmv && \
rm -f ./libaio-0.3.109-13.el7.x86_64.rpm && \
rm -f ~/install/libraries/lib64/libaio.so.1 && \
ln -s ~/install/libraries/lib64/libaio.so.1.0.1 ~/install/libraries/lib64/libaio.so.1 && \
echo "export LD_LIBRARY_PATH=`pwd`/lib64:\$LD_LIBRARY_PATH" >> ~/.bashrc && \
source ~/.bashrc
cd -

Install libgfortran. For example:

mv preinstall_files/libquadmath-4.8.5-39.el7.x86_64.rpm preinstall_files/libgfortran-4.8.5-39.el7.x86_64.rpm ~/install/libraries/
cd ~/install/libraries/
for PACKAGE in libquadmath-4.8.5-39.el7.x86_64.rpm libgfortran-4.8.5-39.el7.x86_64.rpm; do
    rpm2cpio $PACKAGE | cpio -idmv && \
    rm -f $PACKAGE
done
echo "export LD_LIBRARY_PATH=$(pwd)/usr/lib64:\$LD_LIBRARY_PATH" >> ~/.bashrc
source ~/.bashrc
cd –

Install Percona dependencies on all servers that will house a database node (requires root permissions):
```
yum install *.rpm
```

Prepare to install

Before you install Moogsoft Enterprise:

Log in as the Linux user on the primary, secondary and redundancy servers and go to the working directory on each server.
Make sure that the directories where you will install Moogsoft Enterprise meet the size requirements stated in the Moogsoft Enterprise - Offline Tarball pre-installation or Moogsoft Enterprise - Online Tarball pre-installation instructions. Some files such as the Percona database and Opensearch/Elasticsearch files will be installed into the Linux user's HOME/install directory.
Remove existing environment variables such as $MOOGSOFT_HOME from previous installations.

Install database and Moogsoft Enterprise files on the primary server

Run the following commands on the primary, secondary and redundancy servers that will house a database node:

cp percona-xtrabackup-2.4.24-Linux-x86_64.glibc2.12.tar.gz ~/install
cp Percona-XtraDB-Cluster-5.7.36-rel39-55.1.Linux.x86_64.glibc2.12.tar.gz ~/install
cp socat-1.7.3.2-2.el7.x86_64.rpm ~/install

Run the Percona install script on the primary database server. Substitute the IP addresses of your servers and choose the password for the sstuser.
```
bash install_percona_nodes_tarball.sh -p -i <primary_ip>,<secondary_ip>,<redundancy_ip> -u sstuser -w <sstpassword>
```
To verify that the Percona install was successful, run the following command on the primary server. Substitute the IP address of your primary server:
```
curl http://<primary_ip>:9198
```
If successful, you see the following message:
```
Percona XtraDB Cluster Node is synced.
```
Now that you've installed the Percona database, you can install Moogsoft Enterprise. Go to your working directory on the primary server, and extract the Moogsoft Enterprise distribution archive:
```
source ~/.bashrc
cd /opt/moogsoft/${VERSION}
tar -xf moogsoft-aiops-${VERSION}.tgz
```
Run the installation script in your primary server working directory to install Moogsoft Enterprise:
```
bash moogsoft-aiops-install-${VERSION}.sh
```
When prompted, enter the working directory to install Moogsoft Enterprise on the primary server. For example, /opt/moogsoft/aiops. The script guides you through the installation process. You can modify the default installation directory displayed for your environment.
Edit the .bashrc file to set the MOOGSOFT_HOME variable and update the PATH variable to include various directories under MOOGSOFT_HOME that contain binaries and scripts.
For example:
```
export MOOGSOFT_HOME=/opt/moogsoft/aiops
export PATH=$PATH:$MOOGSOFT_HOME/bin:$MOOGSOFT_HOME/bin/utils:/$MOOGSOFT_HOME/cots/erlang/bin:$MOOGSOFT_HOME/cots/rabbitmq-server/sbin
```
And then source the ~/bashrc file:
```
source ~/.bashrc
```

Configure the Tool Runner to execute locally:

sed -i 's/# execute_locally: false,/,execute_locally: true/1' $MOOGSOFT_HOME/config/servlets.conf

Initialize Moogsoft Enterprise on the primary server

Initialize the database on the primary server:

moog_init_db.sh -qIu root

When prompted for a password, enter the password for the root database user instead of the Linux user. If you are installing Percona on this machine for the first time, leave the password blank and press Enter to continue. The script prompts you to accept the End User License Agreement (EULA) and guides you through the initialization process.

Install Moogsoft Enterprise files on the secondary and redundancy servers

Run the Percona install script on the secondary and redundancy servers. Substitute the IP addresses of your servers and use the same password as for the primary server.
```
bash install_percona_nodes_tarball.sh -i <primary IP address>,<secondary IP address>,<redundancy IP address> -u sstuser -w <sstpassword>
```
To verify that the Percona install was successful, run the following command on the secondary and redundancy servers. Substitute the IP address of your primary server:
```
curl http://<primary IP address>:9198
```
If successful, you see the following message:
```
Percona XtraDB Cluster Node is synced
```
It takes a moment for the secondary and redundancy servers to sync. If you do not get this message, wait for a few moments and then try again.
Now that you've installed the Percona database, you can install Moogsoft Enterprise.Go to your working directory in the secondary and redundancy servers, and extract the Moogsoft Enterprise distribution archive:
```
source ~/.bashrc
cd /opt/moogsoft/${VERSION}
tar -xf moogsoft-aiops-${VERSION}.tgz
```
Run the installation script in your working directory in both servers to install Moogsoft AIOps:
```
bash moogsoft-aiops-install-${VERSION}.sh
```
When prompted, enter the directory to install AIOps. For example, /opt/moogsoft/aiops. The script guides you through the installation process. You can modify the default installation directory displayed for your environment.
Edit the .bashrc file to set the MOOGSOFT_HOME variable and update the PATH variable to include various directories under MOOGSOFT_HOME containing binaries and scripts.
For example:
```
export MOOGSOFT_HOME=/opt/moogsoft/aiops
export PATH=$PATH:$MOOGSOFT_HOME/bin:$MOOGSOFT_HOME/bin/utils:/$MOOGSOFT_HOME/cots/erlang/bin:$MOOGSOFT_HOME/cots/rabbitmq-server/sbin
```
Source the ~/.bashrc file:
```
source ~/.bashrc
```

Configure the Tool Runner to execute locally:

sed -i 's/# execute_locally: false,/,execute_locally: true/1' $MOOGSOFT_HOME/config/servlets.conf

Install HA Proxy on primary and secondary servers

Install HA Proxy on the primary and secondary servers (root permission required).

Run the following command, using your chosen value for MOOGSOFT_HOME. Substitute the IP addresses of your servers.

export MOOGSOFT_HOME=/opt/moogsoft/aiops
$MOOGSOFT_HOME/bin/utils/haproxy_installer.sh -l 3309 -c -i <primary_ip>:3306,<secondary_ip>:3306,<redundancy_ip>:3306

Perform the remaining steps as the moogsoft Linux user
Check that you can connect to MySQL to confirm successful installation:
```
mysql -h localhost -P3309 -u root
```

Set up RabbitMQ on all servers

Initialize and configure RabbitMQ on all three servers.

Primary, Secondary and Redundancy servers:

Run the following command. Substitute a name for your zone. You must use the same zone name for all servers.
```
moog_init_mooms.sh -pz <my_zone>
```
Stop RabbitMQ on the secondary and redundancy servers:
```
process_cntl --process_name rabbitmq stop
```
The primary server erlang cookie is located in the Linux user's HOME directory: ~/.erlang.cookie. The erlang cookie must be the same for all RabbitMQ nodes.
Replace the erlang cookie on the secondary and redundancy servers with the erlang cookie from the primary server. You may need to change the file permissions on the secondary and redundancy erlang cookies first to allow those files to be overwritten.
After copying the cookie, as the Linux user, make the cookies on the secondary and redundancy servers read-only. For example:
```
cd ~
chmod 406 ~/.erlang.cookie
mv .erlang.cookie .erlang.cookie.orig
scp moogsoft@<PRIMARY_IP>:.erlang.cookie .
chmod 400 .erlang.cookie
```

Secondary and Redundancy servers:

Restart RabbitMQ on the secondary and redundancy servers and join those servers to the cluster. Substitute the short hostname of your primary server and the name of your zone.
The short hostname is the full hostname excluding the DNS domain name. For example, if the hostname is ip-172-31-82-78.ec2.internal, the short hostname is ip-172-31-82-78. To find out the short hostname, run rabbitmqctl cluster_status on the primary server.
The restart command differs depending on whether you are a root or a non-root user:
- Root: systemctl restart rabbitmq-server
- Non-root: process_cntl --process_name rabbitmq restart
```
<restart command>
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@<primary_short_hostname>
rabbitmqctl start_app
rabbitmqctl set_policy -p <my_zone> ha-all ".+\.HA" '{"ha-mode":"all"}'
```

Run rabbitmqctl cluster_status to get the cluster status. Example output is as follows:

Cluster status of node rabbit@secondary ...
[{nodes,[{disc,[rabbit@primary,rabbit@secondary]}]},
 {running_nodes,[rabbit@primary,rabbit@secondary]},
 {cluster_name,<<"rabbit@secondary">>},
 {partitions,[]},
 {alarms,[{rabbit@primary,[]},{rabbit@secondary,[]}]}]

Set up Opensearch/Elasticsearch on all servers

You must start Opensearch/Elasticsearch with specific memory parameters to support HA. The process_cntl utility starts Opensearch/Elasticsearch.

For ElasticSearch ONLY (v8.1.x and older)

Run the following on the primary, secondary and redundancy servers to adjust the process_cntl utility:

cp –p $MOOGSOFT_HOME/bin/utils/process_cntl $MOOGSOFT_HOME/bin/utils/process_cntl.orig
sed -i 's/-Xms256m/-Xms2g/' $MOOGSOFT_HOME/bin/utils/process_cntl

Initialize and configure Opensearch/Elasticsearch on all three servers:
Pre v8.2.x:
```
moog_init_search.sh
```
v8.2.x and newer:
```
moog_init_search.sh -i
```

Configure Elasticsearch/Opensearch:

Pre v8.2.x:

Uncomment and edit the properties in $MOOGSOFT_HOME/cots/elasticsearch/config/elasticsearch.ymlon all three servers as follows

cluster.name: aiops
node.name: <server_hostname>
...
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: ["<PRIMARY_HOSTNAME>","<SECONDARY_HOSTNAME>","<REDUNDANCY_HOSTNAME>"]
discovery.zen.minimum_master_nodes: 2
gateway.recover_after_nodes: 1
node.master: true

v8.2.x and newer:

Edit the following properties in $MOOGSOFT_HOME/cots/opensearch/config/opensearch.yml on all three servers as follows, changing the node names etc as appropriate:

cluster.name: moog-opensearch-cluster
node.name: node1
network.host: 0.0.0.0
node.master: true 
discovery.seed_hosts: ["node1.example.com",
                      "node2.example.com",
                      "node3.example.com"]
cluster.initial_master_nodes: ["node1","node2","node3"]
plugins.security.allow_default_init_securityindex: true
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.restapi.roles_enabled: ["all_access"]
plugins.security.ssl.http.enabled: false
plugins.security.authcz.admin_dn:
  - 'CN=ADMIN,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
# Give value according to SSL certificate.
plugins.security.nodes_dn:
    - 'CN=node1.example.com,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
    - 'CN=node2.example.com,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
    - 'CN=node2.example.com,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'

Restart Opensearch/Elasticsearch on all three servers:

# For deployments < v8.2.x
$MOOGSOFT_HOME/bin/utils/process_cntl --process_name elasticsearch restart
# For deployments >= v8.2.x
$MOOGSOFT_HOME/bin/utils/process_cntl --process_name opensearch restart

Get the health status of the cluster by running the following on the primary server:
```
curl -X GET "localhost:9200/_cat/health?v&pretty"
```
- If authentication has been enabled for Opensearch/Elasticsearch (Opensearch is secure by default), the curl command will need to be provided with user credentials as follows: curl -u <username>:<password> -X ...
Example cluster health status:
```
epoch timestamp cluster status 
node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1580490422 17:07:02 aiops green 3 3 0 0 0 0 0 0 - 100.0%
```
If you've configured all three servers, the node.total will be 3.

Configure Moogsoft Enterprise

Configure Moogsoft Enterprise by editing the Moogfarmd and system configuration files on the primary and secondary servers.

Primary and Secondary servers:

Make a copy of the $MOOGSOFT_HOME/config/system.conf file:

cp -p $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.orig

Edit $MOOGSOFT_HOME/config/system.conf and set the following properties. Substitute the name of your RabbitMQ zone, the server hostnames, and the cluster names:

"mooms" :
   {
...
"zone" : "<my_zone>",

"brokers" : [
    {"host" : "<primary_hostname>", "port" : 5672},
    {"host" : "<secondary_hostname>", "port" : 5672},
    {"host" : "<redundancy_hostname>", "port" : 5672}
],
...
"cache_on_failure" : true,
...
"search" :
    {
...
  "nodes" : [
      {"host" : "<primary_hostname>", "port" : 9200},
      {"host" : "<secondary_hostname>", "port" : 9200},
      {"host" : "<redundancy_hostname>", "port" : 9200}
]
...
"failover" :
    {
    "persist_state" : true,
    "hazelcast" :
        {
        "hosts" : ["<primary_hostname>","<secondary_hostname>"],
        "cluster_per_group" : true
        }
    "automatic_failover" : true,
    }
...
"ha": 
    { "cluster": "<cluster_name, primary or secondary>" }

Make a copy of the $MOOGSOFT_HOME/config/moog_farmd.conf file:

cp -p $MOOGSOFT_HOME/config/moog_farmd.conf $MOOGSOFT_HOME/config/moog_farmd.conf.orig

Uncomment and edit the following properties in $MOOGSOFT_HOME/config/moog_farmd.conf. Note the importance of the initial comma. Delete the cluster line in this section of the file:

Primary server

,
ha:
{
    group: "moog_farmd",
    instance: "moog_farmd",
    default_leader: true,
    start_as_passive: false
}

Secondary server

,
ha:
{
    group: "moog_farmd",
    instance: "moog_farmd",
    default_leader: true,
    start_as_passive: false
}

Start Moogfarmd on the primary and secondary servers:
```
process_cntl --process_name moog_farmd start
```

Initialize the user interface

Run the initialization script moog_init_ui.sh on the primary server. Substitute the name of your RabbitMQ zone and primary hostname.

When asked if you want to change the configuration hostname, say yes and enter the public hostname/domain name used in the URL to access AIOps via browser. If the URL uses a different hostname/domain name (e.g., an alias) the system will reject your login.

Primary server:

moog_init_ui.sh -twfz <MY_ZONE> -c <primary_hostname>:15672 -m <primary_hostname>:5672 -s <primary_hostname>:9200 -d <primary_hostname>:3309 -n

Edit the servlets settings on the primary server in the file $MOOGSOFT_HOME/config/servlets.conf. Note the importance of the initial comma.

,ha :
{    
    instance: "servlets",    
    group: "servlets_primary",    
    start_as_passive: false
}

Restart Apache Tomcat and Moogfarmd on the primary server:

process_cntl --process_name apache-tomcat restart
process_cntl --process_name moog_farmd restart

Run the initialization script moog_init_ui.sh on the secondary server. Substitute the name of your RabbitMQ zone.

When asked if you want to change the configuration hostname, say yes and enter the public hostname/domain name used in the URL to access AIOps via browser.

Secondary server:

moog_init_ui.sh -twfz MY_ZONE -c <SECONDARY_HOSTNAME>:15672 -m <SECONDARY_HOSTNAME>:5672 -s <SECONDARY_HOSTNAME>:9200 -d <SECONDARY_HOSTNAME>:3309 -n

Edit the servlets settings in the secondary server $MOOGSOFT_HOME/config/servlets.conf file. Note the importance of the initial comma.

,ha :
{
    instance: "servlets",     
    group: "servlets_secondary",    
    start_as_passive: false
}

Restart Apache Tomcat and Moogfarmd on the primary server secondary server:

process_cntl --process_name apache-tomcat restart
process_cntl --process_name moog_farmd restart

Enable HA for LAMs

There are two types of HA configuration for LAMs; Active/Active and Active/Passive:

Receiving LAMs that listen for events are configured as Active/Active. For example, the REST LAM.
Polling LAMs are configured as Active/Passive. For example, the SolarWinds LAM.

Every LAM has its own configuration file under $MOOGSOFT_HOME/config/. This example references rest_lam.conf and solarwinds_lam.conf.

Primary and Secondary servers

Edit the HA properties in the primary and secondary servers' LAM configuration files. Moogsoft Enterprise automatically manages the active and passive role for the LAMs in a single process group:

# Receiving LAM (Active / Active)
# Configuration on Primary
ha:
{
    group            : "rest_lam_primary",
    instance         : "rest_lam",
    duplicate_source : false
},
...
# Configuration on Secondary
ha:
{
    group            : "rest_lam_secondary",
    instance         : "rest_lam",
    duplicate_source : false
},

# Polling LAM (Active / Passive)
# Configuration on Primary
ha:
{
    group                    : "solarwinds_lam",
    instance                 : "solarwinds_lam",
    only_leader_active       : true,
    default_leader           : true,
    accept_conn_when_passive : false,
    duplicate_source         : false
},
...
# Configuration on Secondary
ha:
{
    group                    : "solarwinds_lam",
    instance                 : "solarwinds_lam",
    only_leader_active       : true,
    default_leader           : false,
    accept_conn_when_passive : false,
    duplicate_source         : false
},

Start the LAMs:

process_cntl --process_name restlamd restart 
process_cntl --process_name solarwindslamd restlamd restart

Run the HA Control command line utility ha_cntl -v to check the status of the LAMS:

Moogsoft AIOps Version{VERSION}
(C) Copyright 2012-2020 Moogsoft, Inc.
All rights reserved.
Executing: ha_cntl
Getting system status
Cluster: [PRIMARY] active
        ...
	Process Group: [rest_lam_primary] Active (no leader - all can be active)
		Instance: [rest_lam] Active
        ...
        Process Group: [solarwinds_lam] Active (only leader should be active)
                Instance: [solarwinds_lam] Active Leader 

Cluster: [SECONDARY] partially active
        ...
	Process Group: [rest_lam_secondary] Passive (no leader - all can be active)
		Instance: [rest_lam] Active
        ...
        Process Group: [solarwinds_lam] Passive (only leader should be active)
                Instance: [solarwinds_lam] Passive

For more information, see the HA Control Utility Command Reference.

Validate the installation

In this section: