Basic HA Installation - Tarball

Run the following commands on the primary, secondary and redundancy servers that will house a database node:

VERSION=9.1.0
mkdir -p ~/install
cd /opt/moogsoft/${VERSION}/preinstall_files
cp percona-xtrabackup*.tar.gz ~/install
cp Percona-XtraDB-Cluster*.tar.gz ~/install

Run the Percona install script on the primary database server. Substitute the IP addresses of your servers and choose the password for the sstuser.

bash install_percona_nodes_tarball.sh -p -i <primary_ip>,<secondary_ip>,<redundancy_ip>

To verify that the Percona install was successful, run the following command on the primary server. Substitute the IP address of your primary server:

curl http://<primary_ip>:9198

If successful, you see the following message : (it can take a few minutes for the node to be synced):

Percona XtraDB Cluster Node is synced.

Now that you've installed the Percona database, you can install Moogsoft Onprem. Go to your working directory on the primary server, and extract the Moogsoft Onprem distribution archive:

tar -xf moogsoft-enterprise-${VERSION}.tgz

Run the installation script in your primary server working directory to install Moogsoft Onprem:

bash moogsoft-enterprise-install-${VERSION}.sh

When prompted, enter the working directory to install Moogsoft Onprem on the primary server. For example, /opt/moogsoft/aiops. The script guides you through the installation process. You can modify the default installation directory displayed for your environment.

Edit the .bashrc file to set the MOOGSOFT_HOME variable and update the PATH variable to include various directories under MOOGSOFT_HOME that contain binaries and scripts.

For example:

export MOOGSOFT_HOME=/opt/moogsoft/aiops
export PATH=$PATH:$MOOGSOFT_HOME/bin:$MOOGSOFT_HOME/bin/utils:/$MOOGSOFT_HOME/cots/erlang/bin:$MOOGSOFT_HOME/cots/rabbitmq-server/sbin

And then source the ~/bashrc file:

source ~/.bashrc

Configure the Tool Runner to execute locally:

sed -i 's/# execute_locally: false,/,execute_locally: true/1' $MOOGSOFT_HOME/config/servlets.conf

Run the Percona install script on the secondary and redundancy servers. Substitute the IP addresses of your servers and use the same password as for the primary server.

bash install_percona_nodes_tarball.sh -i <primary IP address>,<secondary IP address>,<redundancy IP address>

To verify that the Percona install was successful, run the following command on the secondary and redundancy servers. Substitute the IP address of your primary server:

curl http://<primary IP address>:9198

If successful, you see the following message:

Percona XtraDB Cluster Node is synced

It takes a moment for the secondary and redundancy servers to sync. If you do not get this message, wait for a few moments and then try again.

Now that you've installed the Percona database, you can install Moogsoft Onprem.Go to your working directory in the secondary and redundancy servers, and extract the Moogsoft Onprem distribution archive:

source ~/.bashrc
cd /opt/moogsoft/${VERSION}
tar -xf moogsoft-aiops-${VERSION}.tgz

Run the installation script in your working directory in both servers to install Moogsoft AIOps:

bash moogsoft-aiops-install-${VERSION}.sh

When prompted, enter the directory to install AIOps. For example, /opt/moogsoft/aiops. The script guides you through the installation process. You can modify the default installation directory displayed for your environment.

Edit the .bashrc file to set the MOOGSOFT_HOME variable and update the PATH variable to include various directories under MOOGSOFT_HOME containing binaries and scripts.

For example:

export MOOGSOFT_HOME=/opt/moogsoft/aiops
export PATH=$PATH:$MOOGSOFT_HOME/bin:$MOOGSOFT_HOME/bin/utils:/$MOOGSOFT_HOME/cots/erlang/bin:$MOOGSOFT_HOME/cots/rabbitmq-server/sbin

Source the ~/.bashrc file:

source ~/.bashrc

Configure the Tool Runner to execute locally:

sed -i 's/# execute_locally: false,/,execute_locally: true/1' $MOOGSOFT_HOME/config/servlets.conf

Run the following command, using your chosen value for MOOGSOFT_HOME. Substitute the IP addresses of your servers.

export MOOGSOFT_HOME=/opt/moogsoft/aiops
$MOOGSOFT_HOME/bin/utils/haproxy_installer.sh -l 3309 -c -i <primary_ip>:3306,<secondary_ip>:3306,<redundancy_ip>:3306

Perform the remaining steps as the moogsoft Linux user

Check that you can connect to MySQL to confirm successful installation:

mysql -h localhost -P3309 -u root

Run the following command. Substitute a name for your zone. You must use the same zone name for all servers.

moog_init_mooms.sh -pz <my_zone>

Stop RabbitMQ on the secondary and redundancy servers:

process_cntl --process_name rabbitmq stop

The primary server erlang cookie is located in the Linux user's HOME directory: ~/.erlang.cookie. The erlang cookie must be the same for all RabbitMQ nodes.

Replace the erlang cookie on the secondary and redundancy servers with the erlang cookie from the primary server. You may need to change the file permissions on the secondary and redundancy erlang cookies first to allow those files to be overwritten.

After copying the cookie, as the Linux user, make the cookies on the secondary and redundancy servers read-only. For example:

cd ~
chmod 406 ~/.erlang.cookie
mv .erlang.cookie .erlang.cookie.orig
scp moogsoft@<PRIMARY_IP>:.erlang.cookie .
chmod 400 .erlang.cookie

Restart RabbitMQ on the secondary and redundancy servers and join those servers to the cluster. Substitute the short hostname of your primary server and the name of your zone.

The short hostname is the full hostname excluding the DNS domain name. For example, if the hostname is ip-172-31-82-78.ec2.internal, the short hostname is ip-172-31-82-78. To find out the short hostname, run rabbitmqctl cluster_status on the primary server.

The restart command differs depending on whether you are a root or a non-root user:

<restart command>
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@<primary_short_hostname>
rabbitmqctl start_app
rabbitmqctl set_policy -p <my_zone> ha-all ".+\.HA" '{"ha-mode":"all"}'

Run rabbitmqctl cluster_status to get the cluster status. Example output is as follows with extra less important output marked as '...' :

Cluster status of node rabbit@secondary ...
...
Basics

Cluster name rabbit@secondary

Disk Nodes

rabbit@primary
rabbit@secondary

Running Nodes

rabbit@primary
rabbit@secondary

Versions
...

Make a copy of the $MOOGSOFT_HOME/config/system.conf file:

cp -p $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.orig

Edit $MOOGSOFT_HOME/config/system.conf and set the following properties. Substitute the name of your RabbitMQ zone, the server hostnames, and the cluster names:

"mooms" :
   {
...
"zone" : "<my_zone>",

"brokers" : [
    {"host" : "<primary_hostname>", "port" : 5672},
    {"host" : "<secondary_hostname>", "port" : 5672},
    {"host" : "<redundancy_hostname>", "port" : 5672}
],
...
"cache_on_failure" : true,
...
"search" :
    {
...
  "nodes" : [
      {"host" : "<primary_hostname>", "port" : 9200},
      {"host" : "<secondary_hostname>", "port" : 9200},
      {"host" : "<redundancy_hostname>", "port" : 9200}
]
...
"failover" :
    {
    "persist_state" : true,
    "hazelcast" :
        {
        "hosts" : ["<primary_hostname>","<secondary_hostname>"],
        "cluster_per_group" : true
        }
    "automatic_failover" : true,
    }
...
"ha": 
    { "cluster": "<cluster_name, primary or secondary>" }

Make a copy of the $MOOGSOFT_HOME/config/moog_farmd.conf file:

cp -p $MOOGSOFT_HOME/config/moog_farmd.conf $MOOGSOFT_HOME/config/moog_farmd.conf.orig

Uncomment and edit the following properties in $MOOGSOFT_HOME/config/moog_farmd.conf. Note the importance of the initial comma. Delete the cluster line in this section of the file:

Primary server

,
ha:
{
    group: "moog_farmd",
    instance: "moog_farmd",
    default_leader: true,
    start_as_passive: false
}

Secondary server

,
ha:
{
    group: "moog_farmd",
    instance: "moog_farmd",
    default_leader: true,
    start_as_passive: false
}

Start Moogfarmd on the primary and secondary servers:

process_cntl --process_name moog_farmd start