Upgrade Moogsoft Onprem Tarball v9.0.x or v9.1.x to v9.2.0.1

This topic describes how to patch a Tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.0.0 or 9.0.0.x or 9.0.1.x, 9.1.0.x, or 9.1.1.x

Important

This process requires the presence of a third Moogsoft Onprem server to act a redundancy server as per : Fully Distributed HA Installation

If this server is not already provisioned and running, it must be provisioned before starting this process, and a clean installation of Moogsoft Onprem v9.2.0 performed on it, with just the RabbitMQ and OpenSearch processes left running

In the process below, Server 1 and Server 2 are existing Moogsoft Onprem servers running MoogFarmD, Apache-Tomcat, RabbitMQ etc

In the process below, Server 3 is the Redundancy Server running RabbitMQ and possibly OpenSearch too

One-time multi-server upgrade step
Update all Percona nodes to the latest version using the instructions here: Percona Cluster 8.4 Tarball Minor Version Upgrade

On Server 1 or Server 2

IMPORTANT: Make a note of the existing RabbitMQ Events.HA queue name by running this command:

export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
VHOST=$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k mooms.zone)
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl -p "${VHOST}" list_queues | grep Events.HA | awk '{print $1}' | head -n 1

This queue name is needed later in the upgrade process

On Server 3

Stop RabbitMQ

$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;

On Server 1
1. Remove the Server 3 RabbitMQ node from the cluster:
```
export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl forget_cluster_node rabbit@<SERVER_3_IP_OR_HOSTNAME>
```
2. Comment out any reference to the Server 1 RabbitMQ broker from $MOOGSOFT_HOME/config/system.conf
  The entry under mooms -> brokers should be commented out
3. Comment out any reference to the Server 3 RabbitMQ broker from $MOOGSOFT_HOME/config/system.conf
  The entry under mooms -> brokers should be commented out
4. Stop MoogFarmD to pause event processing
```
$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;
```
5. Restart all Moogsoft Onprem services (excluding MoogFarmD)
```
$MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat restart;
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart;
$MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart
```
On Server 2
1. Comment out any reference to the Server 1 and the Server 3 RabbitMQ brokers from $MOOGSOFT_HOME/config/system.conf
  The entries under mooms -> brokers should be commented out
2. Stop MoogFarmD to pause event processing
```
$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;
```
3. Restart all Moogsoft Onprem services (excluding MoogFarmD)
```
$MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat restart;
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart;
$MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart
```
On Server 3 (Redundancy Server)
1. Stop all services
```
$MOOGSOFT_HOME/bin/utils/process_cntl opensearch stop;
```
2. Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade
```
rm -rf $MOOGSOFT_HOME/var/lib/rabbitmq/mnesia/*
```
3. Upgrade the packages on the server
  Upgrade Moogsoft Onprem Tarball to v9.2.0.1
  This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.0.0 or v9.0.0.x or v9.0.1.x, v9.1.0.x, or 9.1.1
  Ensure the patch tarball is available to each server being patched:
  Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):
  curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.1.tgz"
  Copy the tarball to all servers where the product is installed.
  Optional GPG key validation of the tarball to validate the installation file.
  Follow the same instructions for GPG validation on this page: Moogsoft Onprem Online Tarball Pre-installation
  Gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):
  export OLD_DIST_PATH=$(readlink -f $MOOGSOFT_HOME/bin|awk -F '/' 'sub(FS $NF,x)'); export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';')); export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';')); export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf); export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
  Place the downloaded tarball file into an empty directory called patch_directory9.2.0.1 and execute the following commands to start the patch installation:
  cd patch_directory9.2.0.1; tar -xf moogsoft-enterprise-9.2.0.1.tgz; bash moogsoft-enterprise-install-9.2.0.1.sh;
  There are changes in controller.conf, system.conf, and security.conf in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:
  diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0.1/config | grep -i 'differ'
  Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.2.0.1 version of these files (if appropriate/needed).
  FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 or v9.0.0.2 OR EARLIER ONLY
  A fix has been made for SAML authentication around WantAssertionsSigned handling which can impact SAML logins for v9.0.1. Signing is now always required, even when WantAssertionsSigned is disabled. WantAssertionsSigned now requires explicit signing of the assertions, not the response.
  A way to disable this new check (if needed) is to change the WantAssertionsSigned setting in the SP Metadata file from true to false as per the example below (change the file name and path as needed below).
  sed -i 's/WantAssertionsSigned="true"/WantAssertionsSigned="false"/g' ${MOOGSOFT_HOME}/config/keycloak.my_sp_metadata.xml
  FOR ALL VERSIONS
  Upgrade OpenSearch. This step will remove the existing copy of OpenSearch and upgrade it to the latest one (single node deployment):
  $MOOGSOFT_HOME/bin/utils/process_cntl opensearch stop; $MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i
  This step will overwrite the opensearch_user password in $MOOGSOFT_HOME/config/system.conf If OpenSearch needs to be clustered, it can be done after all the nodes are upgraded fully.
  Verify the remaining processes are working properly and update the crontab:
  $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status; $MOOGSOFT_HOME/bin/utils/process_cntl nginx status; $MOOGSOFT_HOME/bin/utils/process_cntl opensearch status; (crontab -l | sed -e 's/^\#\+$.*events_analyser.*$/\1/') | crontab - (crontab -l | sed -e 's/^\#\+$.*keepalive.*$/\1/') | crontab -
4. Create the RabbitMQ vhost and Events.HA quorum queue on that node (to allow events to queue here during the upgrade)
```
VHOST=<your_vhost/zone_name>;
$MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -pz ${VHOST};
QUEUE_NAME="<YOUR_EVENTS_HA_QUEUE_NAME>";
chmod u+x ${MOOGSOFT_HOME}/cots/rabbitmq-server/plugins/rabbitmq_management-*/priv/www/cli/rabbitmqadmin;
${MOOGSOFT_HOME}/cots/rabbitmq-server/plugins/rabbitmq_management-*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft --vhost=${VHOST} declare queue name=${QUEUE_NAME} durable=true arguments='{"x-queue-type":"quorum"}';
${MOOGSOFT_HOME}/cots/rabbitmq-server/plugins/rabbitmq_management-*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft --vhost=${VHOST} declare exchange name=MOOMS.GENERAL.TOPIC_EXCHANGE type=topic durable=true
${MOOGSOFT_HOME}/cots/rabbitmq-server/plugins/rabbitmq_management-*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft --vhost=${VHOST} declare binding source="MOOMS.GENERAL.TOPIC_EXCHANGE" destination_type="queue" destination="${QUEUE_NAME}" routing_key="events"
```

On Server 2

Confirm that the RabbitMQ HA policy is still present on the Events queue:

export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl list_queues name durable policy -p <YOUR_VHOST> | grep Events.HA

This command should return a value. If it does not, the policy should be reapplied with this command:

${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl set_policy -p <YOUR_ZONE> ha-all ".+.HA" '{"ha-mode":"all"}'

Create a RabbitMQ shovel from the existing Events.HA queue to the Server 3 Events.HA queue to buffer events during the upgrade. Update the commands below with accurate variable values for the hostnames, VHosts, and credentials.

SERVER_2_RABBITMQ_CREDS="moogsoft:m00gs0ft";
SERVER_2_RABBITMQ_VHOST="<YOUR_VHOST_NAME>";
SERVER_2_RABBITMQ_HOST="127.0.0.1";
QUEUE="<YOUR_EVENTS_HA_QUEUE_NAME>";
SERVER_3_RABBITMQ_CREDS="moogsoft:m00gs0ft";
SERVER_3_RABBITMQ_VHOST="<YOUR_VHOST_NAME>";
SERVER_3_RABBITMQ_HOST="YOUR_SERVER_3_IP_OR_HOSTNAME";
export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-plugins enable rabbitmq_shovel;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-plugins enable rabbitmq_shovel_management;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl -p "${SERVER_2_RABBITMQ_VHOST}" set_parameter shovel my_shovel "$(cat <<EOF
{
   "src-uri": "amqp://${SERVER_2_RABBITMQ_CREDS}@${SERVER_2_RABBITMQ_HOST}/${SERVER_2_RABBITMQ_VHOST}",
   "src-queue": "${QUEUE}",
   "dest-uri": "amqp://${SERVER_3_RABBITMQ_CREDS}@${SERVER_3_RABBITMQ_HOST}/${SERVER_3_RABBITMQ_VHOST}",
   "add-forward-headers": false,
   "dest-exchange": "MOOMS.GENERAL.TOPIC_EXCHANGE",
   "ack-mode": "on-confirm",
   "delete-after": "never"
}
EOF
)"

Confirm that the Shovel was created successfully by running the command below.
The "state" must show "running" in this output. If the Shovel is not running at this stage, events may be lost depending on the overall process being carried out.
```
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl -p <YOUR_VHOST> shovel_status
```
Assuming that events are still being ingested via LAMs and UI integrations, the count for the Events.HA queue one the SERVER 3 node should now be increasing. You can check this by using this CLI command on Server 3:
```
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl -p <YOUR_VHOST> list_queues
```

On Server 1

Confirm that the RabbitMQ HA policy is still present on the Events queue:

export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl list_queues name durable policy -p <YOUR_VHOST> | grep Events.HA

This command should return a value. If it does not, the policy should be reapplied with this command:

${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl set_policy -p <YOUR_VHOST> ha-all ".+.HA" '{"ha-mode":"all"}'

Stop RabbitMQ

$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;

On Server 2

Remove the Server 1 RabbitMQ node from the cluster:

export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl forget_cluster_node rabbit@<SERVER_1_IP_OR_HOSTNAME>

On Server 1
1. Stop all services (update the commands below as needed)
```
$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;
$MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop;
kill -9 $(ps -ef | grep java | grep lam | awk '{print $2}') 2>/dev/null;
(crontab -l | sed -e 's/^$.*events_analyser.*$$/#\1/') | crontab -
(crontab -l | sed -e 's/^$.*keepalive.*$$/#\1/') | crontab -
ps -ef | grep java | egrep 'events_analyser|keepalive' | awk '{print $2}' | xargs kill 2>/dev/null;
kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null;
```
2. Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade
```
rm -rf $MOOGSOFT_HOME/var/lib/rabbitmq/mnesia/*
```
3. Upgrade the packages on the server
  Upgrade Moogsoft Onprem Tarball to v9.2.0.1
  This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.0.0 or v9.0.0.x or v9.0.1.x, v9.1.0.x, or 9.1.1
  Ensure the patch tarball is available to each server being patched:
  Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):
  curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.1.tgz"
  Copy the tarball to all servers where the product is installed.
  Optional GPG key validation of the tarball to validate the installation file.
  Follow the same instructions for GPG validation on this page: Moogsoft Onprem Online Tarball Pre-installation
  Gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):
  export OLD_DIST_PATH=$(readlink -f $MOOGSOFT_HOME/bin|awk -F '/' 'sub(FS $NF,x)'); export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';')); export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';')); export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf); export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
  Place the downloaded tarball file into an empty directory called patch_directory9.2.0.1 and execute the following commands to start the patch installation:
  cd patch_directory9.2.0.1; tar -xf moogsoft-enterprise-9.2.0.1.tgz; bash moogsoft-enterprise-install-9.2.0.1.sh;
  There are changes in controller.conf, system.conf, and security.conf in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:
  diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0.1/config | grep -i 'differ'
  Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.2.0.1 version of these files (if appropriate/needed).
  FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 or v9.0.0.2 OR EARLIER ONLY
  A fix has been made for SAML authentication around WantAssertionsSigned handling which can impact SAML logins for v9.0.1. Signing is now always required, even when WantAssertionsSigned is disabled. WantAssertionsSigned now requires explicit signing of the assertions, not the response.
  A way to disable this new check (if needed) is to change the WantAssertionsSigned setting in the SP Metadata file from true to false as per the example below (change the file name and path as needed below).
  sed -i 's/WantAssertionsSigned="true"/WantAssertionsSigned="false"/g' ${MOOGSOFT_HOME}/config/keycloak.my_sp_metadata.xml
  FOR ALL VERSIONS
  Upgrade OpenSearch. This step will remove the existing copy of OpenSearch and upgrade it to the latest one (single node deployment):
  $MOOGSOFT_HOME/bin/utils/process_cntl opensearch stop; $MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i
  This step will overwrite the opensearch_user password in $MOOGSOFT_HOME/config/system.conf If OpenSearch needs to be clustered, it can be done after all the nodes are upgraded fully.
  Verify the remaining processes are working properly and update the crontab:
  $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status; $MOOGSOFT_HOME/bin/utils/process_cntl nginx status; $MOOGSOFT_HOME/bin/utils/process_cntl opensearch status; (crontab -l | sed -e 's/^\#\+$.*events_analyser.*$/\1/') | crontab - (crontab -l | sed -e 's/^\#\+$.*keepalive.*$/\1/') | crontab -
4. Upgrade the database schema
  Upgrade the database schema
  Upgrade the database schema and refresh all stored procedures (provide the 'ermintrude' DB user password when prompted):
```
$MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 9.2.0 -u ermintrude
```
5. If the Enrichment Database (Create the Enrichment API Data Store) is being used, ensure the schema is updated using the schema update command on that page.
6. Cluster RabbitMQ with the Node 3 RabbitMQ
  Join the RabbitMQ v4 cluster (Tarball)
  The primary erlang cookie is located at $HOME/.erlang.cookie on whichever RabbitMQ node has been upgraded first.
  The erlang cookie must be the same for all RabbitMQ nodes. Replace the erlang cookie on all other RabbitMQ nodes with primary erlang cookie
  Make the cookies on non-primary nodes read-only:
```
chmod 400 $HOME/.erlang.cookie
```
  You may need to change the file permissions on the existing erlang cookies first to allow those files to be overwritten. For example:
```
chmod 600 $HOME/.erlang.cookie
```
  Restart RabbitMQ on nodes where the erlang cookie has been replaced
```
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart
```
  To join a RabbitMQ node to a cluster, run these commands on the servers needing to join the cluster, but first substitute the short hostname of the RabbitMQ server you wish to cluster with. The short hostname is the full hostname excluding the DNS domain name. For example, if the hostname is ip-172-31-82-78.ec2.internal, the short hostname is ip-172-31-82-78. To find out the short hostname, run rabbitmqctl cluster_status on the appropriate server.
```
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart

export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl stop_app
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl join_cluster rabbit@<NODE_3_SHORTNAME>
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl start_app
```
  Run rabbitmqctl cluster_status to get the cluster status. Example output is as follows:
```
Cluster status of node rabbit@ip-172-31-93-201 Basics Cluster name rabbit@ip-172-31-93-201.ec2.internal Disk Nodes rabbit@ip-172-31-93-201
rabbit@ip-172-31-85-42
rabbit@ip-172-31-93-201 Running Nodes rabbit@ip-172-31-93-201
rabbit@ip-172-31-85-42
rabbit@ip-172-31-93-201 Versions
...
```
7. Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 3
  The entries under mooms -> brokers should be added or uncommented as needed
8. Upgrade Apache-Tomcat
  Upgrade Apache-Tomcat and Nginx (Tarball)
  Warning
  This process will completely remove and re-deploy Apache-Tomcat. If the Apache-Tomcat logs need to be kept, a copy should be made before proceeding.
  Deploy the new version of Apache Tomcat with the latest webapps, and Nginx:
  IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the $MOOGSOFT_HOME/bin/utils/process_cntl script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat process.
  Run these commands in the same terminal as the one where the variables were saved previously.
```
$MOOGSOFT_HOME/bin/utils/process_cntl nginx stop;
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula
cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/cots/nginx/ssl/;
cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/cots/nginx/ssl/;
sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx restart;
```
9. Perform final checks
  Final upgrade steps (Tarball)
  Validate the patch:
  $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh
  If there are any errors from the above validators, please contact Moogsoft Support.
  Re-start non-UI integrations or LAMs
  Important
  Re-install the latest Add-ons pack: Install Moogsoft Add-ons
  The OpenSearch Cluster now needs to be recreated: Opensearch Clustering Guide - Tarball
  Restart any event feeds if they were stopped
10. Start the LAMs (update the commands below as needed)
```
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart
$MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart
```
On Server 2
1. Stop Tomcat and LAMs (update the commands below as needed)
```
$MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop;
kill -9 $(ps -ef | grep java | grep lam | awk '{print $2}') 2>/dev/null;
(crontab -l | sed -e 's/^$.*events_analyser.*$$/#\1/') | crontab -
(crontab -l | sed -e 's/^$.*keepalive.*$$/#\1/') | crontab -
ps -ef | grep java | egrep 'events_analyser|keepalive' | awk '{print $2}' | xargs kill 2>/dev/null;
kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null;
```
2. Wait for the old Events.HA RabbitMQ queue to drain via the shovel to the new nodes. The command below needs to return a value of 0 before proceeding:
```
export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
VHOST=$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k mooms.zone)
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl -p "${VHOST}" list_queues name messages | grep Events.HA
```
3. Stop RabbitMQ
```
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;
```
4. Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade
```
rm -rf $MOOGSOFT_HOME/var/lib/rabbitmq/mnesia/*
```
5. Upgrade the packages on the server
  Upgrade Moogsoft Onprem Tarball to v9.2.0.1
  This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.0.0 or v9.0.0.x or v9.0.1.x, v9.1.0.x, or 9.1.1
  Ensure the patch tarball is available to each server being patched:
  Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):
  curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.1.tgz"
  Copy the tarball to all servers where the product is installed.
  Optional GPG key validation of the tarball to validate the installation file.
  Follow the same instructions for GPG validation on this page: Moogsoft Onprem Online Tarball Pre-installation
  Gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):
  export OLD_DIST_PATH=$(readlink -f $MOOGSOFT_HOME/bin|awk -F '/' 'sub(FS $NF,x)'); export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';')); export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';')); export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf); export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
  Place the downloaded tarball file into an empty directory called patch_directory9.2.0.1 and execute the following commands to start the patch installation:
  cd patch_directory9.2.0.1; tar -xf moogsoft-enterprise-9.2.0.1.tgz; bash moogsoft-enterprise-install-9.2.0.1.sh;
  There are changes in controller.conf, system.conf, and security.conf in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:
  diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0.1/config | grep -i 'differ'
  Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.2.0.1 version of these files (if appropriate/needed).
  FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 or v9.0.0.2 OR EARLIER ONLY
  A fix has been made for SAML authentication around WantAssertionsSigned handling which can impact SAML logins for v9.0.1. Signing is now always required, even when WantAssertionsSigned is disabled. WantAssertionsSigned now requires explicit signing of the assertions, not the response.
  A way to disable this new check (if needed) is to change the WantAssertionsSigned setting in the SP Metadata file from true to false as per the example below (change the file name and path as needed below).
  sed -i 's/WantAssertionsSigned="true"/WantAssertionsSigned="false"/g' ${MOOGSOFT_HOME}/config/keycloak.my_sp_metadata.xml
  FOR ALL VERSIONS
  Upgrade OpenSearch. This step will remove the existing copy of OpenSearch and upgrade it to the latest one (single node deployment):
  $MOOGSOFT_HOME/bin/utils/process_cntl opensearch stop; $MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i
  This step will overwrite the opensearch_user password in $MOOGSOFT_HOME/config/system.conf If OpenSearch needs to be clustered, it can be done after all the nodes are upgraded fully.
  Verify the remaining processes are working properly and update the crontab:
  $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status; $MOOGSOFT_HOME/bin/utils/process_cntl nginx status; $MOOGSOFT_HOME/bin/utils/process_cntl opensearch status; (crontab -l | sed -e 's/^\#\+$.*events_analyser.*$/\1/') | crontab - (crontab -l | sed -e 's/^\#\+$.*keepalive.*$/\1/') | crontab -
6. Cluster RabbitMQ with Server 3 RabbitMQ
  Join the RabbitMQ v4 cluster (Tarball)
  The primary erlang cookie is located at $HOME/.erlang.cookie on whichever RabbitMQ node has been upgraded first.
  The erlang cookie must be the same for all RabbitMQ nodes. Replace the erlang cookie on all other RabbitMQ nodes with primary erlang cookie
  Make the cookies on non-primary nodes read-only:
```
chmod 400 $HOME/.erlang.cookie
```
  You may need to change the file permissions on the existing erlang cookies first to allow those files to be overwritten. For example:
```
chmod 600 $HOME/.erlang.cookie
```
  Restart RabbitMQ on nodes where the erlang cookie has been replaced
```
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart
```
  To join a RabbitMQ node to a cluster, run these commands on the servers needing to join the cluster, but first substitute the short hostname of the RabbitMQ server you wish to cluster with. The short hostname is the full hostname excluding the DNS domain name. For example, if the hostname is ip-172-31-82-78.ec2.internal, the short hostname is ip-172-31-82-78. To find out the short hostname, run rabbitmqctl cluster_status on the appropriate server.
```
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart

export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl stop_app
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl join_cluster rabbit@<NODE_3_SHORTNAME>
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl start_app
```
  Run rabbitmqctl cluster_status to get the cluster status. Example output is as follows:
```
Cluster status of node rabbit@ip-172-31-93-201 Basics Cluster name rabbit@ip-172-31-93-201.ec2.internal Disk Nodes rabbit@ip-172-31-93-201
rabbit@ip-172-31-85-42
rabbit@ip-172-31-93-201 Running Nodes rabbit@ip-172-31-93-201
rabbit@ip-172-31-85-42
rabbit@ip-172-31-93-201 Versions
...
```
7. Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 2, and Server 3
  The entries under mooms -> brokers should be added or uncommented as needed
8. Upgrade Apache-Tomcat
  Upgrade Apache-Tomcat and Nginx (Tarball)
  Warning
  This process will completely remove and re-deploy Apache-Tomcat. If the Apache-Tomcat logs need to be kept, a copy should be made before proceeding.
  Deploy the new version of Apache Tomcat with the latest webapps, and Nginx:
  IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the $MOOGSOFT_HOME/bin/utils/process_cntl script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat process.
  Run these commands in the same terminal as the one where the variables were saved previously.
  $MOOGSOFT_HOME/bin/utils/process_cntl nginx stop; $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/cots/nginx/ssl/; cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/cots/nginx/ssl/; sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf; sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf; $MOOGSOFT_HOME/bin/utils/process_cntl nginx restart;
9. Perform final checks
  Final upgrade steps (Tarball)
  Validate the patch:
  $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh
  If there are any errors from the above validators, please contact Moogsoft Support.
  Re-start non-UI integrations or LAMs
  Important
  Re-install the latest Add-ons pack: Install Moogsoft Add-ons
  The OpenSearch Cluster now needs to be recreated: Opensearch Clustering Guide - Tarball
  Restart any event feeds if they were stopped
10. Start the LAMs (update the commands below as needed)
```
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart
$MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart
```

On Server 1

Ensure all RabbitMQ quorum queues are updated to have the right number of replicas:

export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues grow rabbit@<SERVER1_HOSTNAME_OR_IP> all;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues grow rabbit@<SERVER2_HOSTNAME_OR_IP> all;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues grow rabbit@<SERVER3_HOSTNAME_OR_IP> all;

Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 2, and Server 3
The entries under mooms -> brokers should be added or uncommented as needed

Start MoogFarmD

$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd restart

Restart Apache-Tomcat

$MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat restart

On Server 2

Start MoogFarmD

$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd restart

Upgrade the Moogsoft Bridge server(s) :
Upgrade Moogsoft Onprem Bridge Tarball to v9.2.0.1
This topic describes how to patch a Tarball-based distribution of Moogsoft Onprem Bridge to v9.2.0.1 from v9.0.0 or 9.0.0.x or 9.0.1.x, 9.1.0.x, or 9.1.1.x
Upgrade Moogsoft Onprem Bridge Tarball to v9.2.0.1
Important
While a Moogsoft Onprem Bridge environment can run with one or two RabbitMQ nodes, as of Moogsoft v9.2.0 / RabbitMQ v4, the new Quorum queues require at least three nodes to provide full protection to events in the queues in the case of a cluster node failure.
For example, three RabbitMQ v4 nodes will protect against a single node failing, and five RabbitMQ v4 nodes will protect against two nodes failing etc
So while the Moogsoft Onprem Bridge can run against a single RabbitMQ node, events will potentially be lost during this upgrade process. Events will be saved if there are two nodes, but there will be no protection on the queue should one of the nodes go down in the future. Hence it is recommended to provision a third RabbitMQ node for the Bridge to communicate with.
Documentation for creating a three node setup is available here: Fully Distributed HA Installation
Ensure that the main Moogsoft Onprem servers have been upgraded before performing the Bridge upgrades
For Bridge deployments with a single RabbitMQ node
Warning
Events may be lost during the upgrade process in this deployment type - it is recommended to upgrade to a three node RabbitMQ cluster to protect against event loss
Stop all services (ensure that all LAMs are stopped - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^$.*moogsoft_bridge.*$$/#\1/') | crontab - kill -9 $(ps -ef | grep java | grep moogsoft_bridge | awk '{print $2}') 2>/dev/null $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop
Upgrade the packages on the server using the instructions in the section below : ???
Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the queue upgrade
rm -rf $MOOGSOFT_HOME/var/lib/rabbitmq/mnesia/*
Re-create the RabbitMQ vhost
$MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -pz ${YOUR_VHOST};
Restart all services (ensure that all LAMs are started - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^\#\+$.*moogsoft_bridge.*$/\1/') | crontab - $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam start $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam start
For Bridge deployments with two RabbitMQ nodes
Warning
Having only two nodes will not provide protection against event loss in RabbitMQ v4 - if one node fails, the cluster will fail. It is recommended to upgrade to a three node RabbitMQ cluster which can be done after this upgrade if needed.
On Server 1
Stop all services (ensure that all LAMs are stopped - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^$.*moogsoft_bridge.*$$/#\1/') | crontab - kill -9 $(ps -ef | grep java | grep moogsoft_bridge | awk '{print $2}') 2>/dev/null $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop
Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade
rm -rf $MOOGSOFT_HOME/var/lib/rabbitmq/mnesia/*
Upgrade the packages on the server using the instructions in the section below : ???
Create the RabbitMQ vhost
$MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -pz ${YOUR_VHOST};
Restart all services (ensure that all LAMs are started - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^\#\+$.*moogsoft_bridge.*$/\1/') | crontab - $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart
On Server 2
Stop all services (ensure that all LAMs are stopped - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^$.*moogsoft_bridge.*$$/#\1/') | crontab - kill -9 $(ps -ef | grep java | grep moogsoft_bridge | awk '{print $2}') 2>/dev/null $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop
Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade
rm -rf $MOOGSOFT_HOME/var/lib/rabbitmq/mnesia/*
Upgrade the packages on the server using the instructions in the section below : ???
Cluster the Rabbit Nodes together
Join the RabbitMQ v4 cluster (Tarball)
The primary erlang cookie is located at $HOME/.erlang.cookie on whichever RabbitMQ node has been upgraded first.
The erlang cookie must be the same for all RabbitMQ nodes. Replace the erlang cookie on all other RabbitMQ nodes with primary erlang cookie
Make the cookies on non-primary nodes read-only:
chmod 400 $HOME/.erlang.cookie
You may need to change the file permissions on the existing erlang cookies first to allow those files to be overwritten. For example:
chmod 600 $HOME/.erlang.cookie
Restart RabbitMQ on nodes where the erlang cookie has been replaced
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart
To join a RabbitMQ node to a cluster, run these commands on the servers needing to join the cluster, but first substitute the short hostname of the RabbitMQ server you wish to cluster with. The short hostname is the full hostname excluding the DNS domain name. For example, if the hostname is ip-172-31-82-78.ec2.internal, the short hostname is ip-172-31-82-78. To find out the short hostname, run rabbitmqctl cluster_status on the appropriate server.
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH; ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl stop_app ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl join_cluster rabbit@<NODE_3_SHORTNAME> ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl start_app
Run rabbitmqctl cluster_status to get the cluster status. Example output is as follows:
Cluster status of node rabbit@ip-172-31-93-201 Basics Cluster name rabbit@ip-172-31-93-201.ec2.internal Disk Nodes rabbit@ip-172-31-93-201 rabbit@ip-172-31-85-42 rabbit@ip-172-31-93-201 Running Nodes rabbit@ip-172-31-93-201 rabbit@ip-172-31-85-42 rabbit@ip-172-31-93-201 Versions ...
Restart all services (ensure that all LAMs are started - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^\#\+$.*moogsoft_bridge.*$/\1/') | crontab - $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart
For Bridge deployments with more than two RabbitMQ nodes
For deployments with three or more RabbitMQ nodes, the same process as for two nodes can be followed - upgrade one server fully to start, then select the next server, upgrade the applications, join it to the first RabbitMQ node cluster, before restarting all services, and repeating for the remaining servers
Upgrade the Moogsoft bridge packages to v9.2.0.1
This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.0.0 or v9.0.0.x or v9.0.1.x, v9.1.0.x, or 9.1.1
Important
Enabling the "latency performance" RHEL profile is strongly recommended. This profile allows RabbitMQ to operate much more efficiently so that throughput is increased and smoothed out.
For more information on performance profiles, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/getting-started-with-tuned_monitoring-and-managing-system-status-and-performance
Enable the profile by running the following command as root:
tuned-adm profile latency-performance
This setting will survive machine restarts and only needs to be set once.
‌
Warning
For deployments upgrading from v9.0.0 or v9.0.0.1
The upgrade path from v9.0.0/v9.0.0.1 to v9.0.1 onwards (any pre v9.0.0.2 release going to any post v9.0.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters. All rabbit nodes will need to be stopped before their binaries are upgraded. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.
For deployments upgrading from v9.0.0.2
The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.
The following command must be run on all RabbitMQ server nodes before the following steps are performed:
export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH; ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all
Ensure the patch tarball is available to each server being patched:
Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):
curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.1.tgz"
Copy the tarball to all servers where the product is installed.
Optional GPG key validation of the tarball to validate the installation file.
For servers without internet access (if server has internet access go to the next step):
Download the key from this site:
https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
Copy the key to the server onto which the tarball will be installed (it will be an .asc file).
Import the key:
gpg --import 0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
For servers with internet access, run the following command:
curl https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 | gpg —-import
Download the patch tarball and matching '.sig' file from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:
https://speedy.moogsoft.com/v9/installer/
Copy the patch tgz and matching '.sig' file into the same folder.
Copy the following code into a bash terminal and run it to perform the validation:
gpg --verify moogsoft-enterprise-9.2.0.1.tgz.sig moogsoft-enterprise-9.2.0.1.tgz
Confirm that the command reports:
Good signature from "Moogsoft <security@moogsoft.com>"
Place the downloaded tarball file into an empty directory called patch_directory9.2.0.1 and execute the following commands to start the patch installation:
cd patch_directory9.2.0.1; tar -xf moogsoft-enterprise-9.2.0.1.tgz; bash moogsoft-enterprise-install-9.2.0.1.sh;
There are changes in controller.conf, system.conf, and security.conf in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:
diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0.1/config | grep -i 'differ'
Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.2.0.1 version of these files (if appropriate/needed).

‌

In this section:

Upgrade Moogsoft Onprem Tarball v9.0.x or v9.1.x to v9.2.0.1

Important

Warning

Important

Upgrade Apache-Tomcat and Nginx (Tarball)

Warning

Final upgrade steps (Tarball)

Important

Important

Warning

Warning

Important

Warning

Search results