Patch Moogsoft Onprem Tarball v9.2.0 to v9.2.0.1

This topic describes how to patch a Tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.2.0

Important

This process requires the presence of a third Moogsoft Onprem server to act a redundancy server as per : Fully Distributed HA Installation

In the process below, Server 1 and Server 2 are existing Moogsoft Onprem servers running MoogFarmD, Apache-Tomcat, RabbitMQ etc

In the process below, Server 3 is the Redundancy Server running RabbitMQ and possibly OpenSearch too

Important

If any of the RabbitMQ commands (rabbitmqctl / rabbit-upgrade etc) fail with any errors mentionging escript/erl/erlang, run this command to add erlang to the PATH in the shell, then retry the command:

[[ ":$PATH:" != *"$MOOGSOFT_HOME/cots/erlang/bin/"* ]] && export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;

OPTIONAL (but recommended) : On any server - perform cluster health checks

Confirm the RabbitMQ cluster is healthy:
```
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl cluster_status
```
This output should show three 'Running nodes' in the cluster

Confirm the Percona cluster is healthy:

moog_mysql_client -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

The above command should return 3 for a 3-node cluster etc.

Now check all nodes are synced:

for CLUSTER_NODE in $(moog_mysql_client -sNe "SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME='wsrep_incoming_addresses'" 2>/dev/null | tr ',' '\n' | cut -d: -f1); do
  echo -n "${CLUSTER_NODE}: "; curl -s http://${CLUSTER_NODE}:9198
done

This command show show 'Percona XtraDB Cluster Node is synced' for every node

Confirm the OpenSearch cluster is healthy:

curl -XGET 'https://localhost:9200/_cluster/health?pretty' -u moog_opensearchuser:$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k search.password)

This should report: "number_of_nodes" : 3

One-time multi-server upgrade step
Update all Percona nodes to the latest version using the instructions here: Percona Cluster 8.4 Tarball Minor Version Upgrade
One-time multi-server upgrade step
Upgrade RabbitMQ to v4.2.0 (Tarball)
Important
RabbitMQ v4.3.4 can only be upgraded to from v4.2.0 onwards, so any clusters running RabbitMQ running v4.1.x need to be upgraded using this process
For the purposes of this document, Server3 is the redundancy node, and Server1/Server2 are application nodes also running RabbitMQ
1. Check the version of RabbitMQ (this command can be run on any node):
```
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl version
```
  If this command returns v4.1.x, proceed with these instructions. If the command returns v4.2.x, you can skip this process and continue the rest of the upgrade
2. Enable all feature flags for RabbitMQ v4.1.x (this command only needs to be run once, and on any node):
```
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all
```
3. Starting with the redundancy RabbitMQ node (Server3):
  Drain the current node:
  ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade drain
  Stop the RabbitMQ process:
  $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;
  Upgrade RabbitMQ on the current node:
  Online deployments:
  ${MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -f https://github.com/rabbitmq/rabbitmq-server/releases/download/v4.2.0/rabbitmq-server-generic-unix-4.2.0.tar.xz
  Offline deployments:
  Download the package on a server with internet access:
  curl -L -O https://github.com/rabbitmq/rabbitmq-server/releases/download/v4.2.0/rabbitmq-server-generic-unix-4.2.0.tar.xz
  Copy the package to the RabbitMQ node in question, and upgrade it:
  ${MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -f rabbitmq-server-generic-unix-4.2.0.tar.xz
  Restart RabbitMQ:
  $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart;
  Revive the node:
  ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade revive
  Confirm that all three 'Running nodes' are still reporting as present and running:
  ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl cluster_status
4. Repeat the previous step on Server1
5. Repeat the previous step on Server2
6. After all three nodes are upgraded to RabbitMQ v4.2.0, enable all new feature flags (this command only needs to be run once, against any node):
```
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all
```
On Server 1
1. Stop MoogFarmD to pause event processing (events will buffer in RabbitMQ)
```
$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;
```
On Server 2
1. Stop MoogFarmD to pause event processing (events will buffer in RabbitMQ)
```
$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;
```
On Server 3 (Redundancy Server)
1. Stop all services
```
$MOOGSOFT_HOME/bin/utils/process_cntl opensearch stop;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade drain
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;
```
2. Upgrade the packages on the server
  Upgrade Moogsoft Onprem Tarball to v9.2.0.1
  This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.2.0
  Ensure the patch tarball is available to each server being patched:
  Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):
  curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.1.tgz"
  Copy the tarball to all servers where the product is installed.
  Optional GPG key validation of the tarball to validate the installation file.
  Follow the same instructions for GPG validation on this page: Moogsoft Onprem Online Tarball Pre-installation
  Record the DIST path for file comparison later (do not close the terminal these commands are run in, otherwise they will be lost):
  export OLD_DIST_PATH=$(readlink -f ${MOOGSOFT_HOME}/bin|awk -F '/' 'sub(FS $NF,x)');
  On servers running NGinX only, gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):
  export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';')); export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';')); export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf); export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
  Place the downloaded tarball file into an empty directory called patch_directory9.2.0.1 and execute the following commands to start the patch installation:
  cd patch_directory9.2.0.1; tar -xf moogsoft-enterprise-9.2.0.1.tgz; bash moogsoft-enterprise-install-9.2.0.1.sh;
  There may be changes in config files in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:
  diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0.1/config | grep -i 'differ'
  Update files in ${MOOGSOFT_HOME}/config with any changes introduced in the v9.2.0.1 version of these files (if appropriate/needed).
  Uncomment entries in crontab:
  (crontab -l | sed -e 's/^\#\+$.*events_analyser.*$/\1/') | crontab - (crontab -l | sed -e 's/^\#\+$.*keepalive.*$/\1/') | crontab -
3. Restart the RabbitMQ service and rejoin the cluster:
```
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade revive
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues rebalance quorum
```
4. Confirm this RabbitMQ node has rejoined the existing cluster:
```
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl cluster_status
```
  Three nodes should be listed
On Server 1
1. Stop all services (update the commands below as needed)
```
$MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade drain
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;
kill -9 $(ps -ef | grep java | grep lam | awk '{print $2}') 2>/dev/null;
(crontab -l | sed -e 's/^$.*events_analyser.*$$/#\1/') | crontab -
(crontab -l | sed -e 's/^$.*keepalive.*$$/#\1/') | crontab -
ps -ef | grep java | egrep 'events_analyser|keepalive' | awk '{print $2}' | xargs kill 2>/dev/null;
kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null;
```
2. Upgrade the packages on the server
  Upgrade Moogsoft Onprem Tarball to v9.2.0.1
  This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.2.0
  Ensure the patch tarball is available to each server being patched:
  Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):
  curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.1.tgz"
  Copy the tarball to all servers where the product is installed.
  Optional GPG key validation of the tarball to validate the installation file.
  Follow the same instructions for GPG validation on this page: Moogsoft Onprem Online Tarball Pre-installation
  Record the DIST path for file comparison later (do not close the terminal these commands are run in, otherwise they will be lost):
  export OLD_DIST_PATH=$(readlink -f ${MOOGSOFT_HOME}/bin|awk -F '/' 'sub(FS $NF,x)');
  On servers running NGinX only, gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):
  export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';')); export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';')); export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf); export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
  Place the downloaded tarball file into an empty directory called patch_directory9.2.0.1 and execute the following commands to start the patch installation:
  cd patch_directory9.2.0.1; tar -xf moogsoft-enterprise-9.2.0.1.tgz; bash moogsoft-enterprise-install-9.2.0.1.sh;
  There may be changes in config files in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:
  diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0.1/config | grep -i 'differ'
  Update files in ${MOOGSOFT_HOME}/config with any changes introduced in the v9.2.0.1 version of these files (if appropriate/needed).
  Uncomment entries in crontab:
  (crontab -l | sed -e 's/^\#\+$.*events_analyser.*$/\1/') | crontab - (crontab -l | sed -e 's/^\#\+$.*keepalive.*$/\1/') | crontab -
3. If the Enrichment Database (Create the Enrichment API Data Store) is being used, ensure the schema is updated using the schema update command on that page.
4. Restart the RabbitMQ service and rejoin the cluster:
```
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade revive
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues rebalance quorum
```
5. Confirm this RabbitMQ node has rejoined the existing cluster:
```
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl cluster_status
```
  Three nodes should be listed
6. Upgrade Apache-Tomcat
  Upgrade Apache-Tomcat and Nginx (Tarball)
  Warning
  This process will completely remove and re-deploy Apache-Tomcat. If the Apache-Tomcat logs need to be kept, a copy should be made before proceeding.
  Deploy the new version of Apache Tomcat with the latest webapps, and Nginx:
  IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the $MOOGSOFT_HOME/bin/utils/process_cntl script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat process.
  Run these commands in the same terminal as the one where the variables were saved previously.
```
$MOOGSOFT_HOME/bin/utils/process_cntl nginx stop;
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula
cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/cots/nginx/ssl/;
cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/cots/nginx/ssl/;
sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx restart;
```
7. Perform final checks
  Final upgrade steps (Tarball)
  Validate the patch:
  $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh
  If there are any errors from the above validators, please contact Moogsoft Support.
  Re-start non-UI integrations or LAMs
  Important
  Re-install the latest Add-ons pack: Install Moogsoft Add-ons
8. Start the LAMs (update the commands below as needed)
```
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart
$MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart
```

On Server 2

Stop Tomcat and LAMs and RabbitMQ (update the commands below as needed)

$MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop;
kill -9 $(ps -ef | grep java | grep lam | awk '{print $2}') 2>/dev/null;
(crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
(crontab -l | sed -e 's/^\(.*keepalive.*\)$/#\1/') | crontab -
ps -ef | grep java | egrep 'events_analyser|keepalive' | awk '{print $2}' | xargs kill 2>/dev/null;
kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade drain
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;

Upgrade the packages on the server
Upgrade Moogsoft Onprem Tarball to v9.2.0.1
This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.2.0
1. Ensure the patch tarball is available to each server being patched:
  Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):
  curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.1.tgz"
  Copy the tarball to all servers where the product is installed.
2. Optional GPG key validation of the tarball to validate the installation file.
  Follow the same instructions for GPG validation on this page: Moogsoft Onprem Online Tarball Pre-installation
3. Record the DIST path for file comparison later (do not close the terminal these commands are run in, otherwise they will be lost):
```
export OLD_DIST_PATH=$(readlink -f ${MOOGSOFT_HOME}/bin|awk -F '/' 'sub(FS $NF,x)');
```
4. On servers running NGinX only, gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):
```
export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
```
5. Place the downloaded tarball file into an empty directory called patch_directory9.2.0.1 and execute the following commands to start the patch installation:
```
cd patch_directory9.2.0.1;
tar -xf moogsoft-enterprise-9.2.0.1.tgz;
bash moogsoft-enterprise-install-9.2.0.1.sh;
```
6. There may be changes in config files in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:
```
diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0.1/config | grep -i 'differ'
```
  Update files in ${MOOGSOFT_HOME}/config with any changes introduced in the v9.2.0.1 version of these files (if appropriate/needed).
7. Uncomment entries in crontab:
```
(crontab -l | sed -e 's/^\#\+$.*events_analyser.*$/\1/') | crontab -
(crontab -l | sed -e 's/^\#\+$.*keepalive.*$/\1/') | crontab -
```

Restart the RabbitMQ service and rejoin the cluster:

$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade revive
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues rebalance quorum

Confirm this RabbitMQ node has rejoined the existing cluster:
```
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl cluster_status
```
Three nodes should be listed
Upgrade Apache-Tomcat
Upgrade Apache-Tomcat and Nginx (Tarball)
Warning
This process will completely remove and re-deploy Apache-Tomcat. If the Apache-Tomcat logs need to be kept, a copy should be made before proceeding.
Deploy the new version of Apache Tomcat with the latest webapps, and Nginx:
IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the $MOOGSOFT_HOME/bin/utils/process_cntl script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat process.
Run these commands in the same terminal as the one where the variables were saved previously.
```
$MOOGSOFT_HOME/bin/utils/process_cntl nginx stop;
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula
cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/cots/nginx/ssl/;
cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/cots/nginx/ssl/;
sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx restart;
```
Perform final checks
Final upgrade steps (Tarball)
1. Validate the patch:
```
$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh
```
  If there are any errors from the above validators, please contact Moogsoft Support.
2. Re-start non-UI integrations or LAMs
3. Important
  Re-install the latest Add-ons pack: Install Moogsoft Add-ons

Start the LAMs (update the commands below as needed)

$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart
$MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart

Enable all RabbitMQ feature flags:

${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all

The OpenSearch Cluster now needs to be recreated: Opensearch Clustering Guide - Tarball

OPTIONAL (but recommended) : On any server - perform cluster health checks

Confirm the RabbitMQ cluster is healthy:
```
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl cluster_status
```
This output should show three 'Running nodes' in the cluster

Confirm the Percona cluster is healthy:

moog_mysql_client -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

The above command should return 3 for a 3-node cluster etc.

Now check all nodes are synced:

for CLUSTER_NODE in $(moog_mysql_client -sNe "SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME='wsrep_incoming_addresses'" 2>/dev/null | tr ',' '\n' | cut -d: -f1); do
  echo -n "${CLUSTER_NODE}: "; curl -s http://${CLUSTER_NODE}:9198
done

This command show show 'Percona XtraDB Cluster Node is synced' for every node

Confirm the OpenSearch cluster is healthy:

curl -XGET 'https://localhost:9200/_cluster/health?pretty' -u moog_opensearchuser:$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k search.password)

This should report: "number_of_nodes" : 3

On Server 1

Start MoogFarmD

$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd restart

On Server 2

Start MoogFarmD

$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd restart

Restart any event feeds if they were stopped
Upgrade the Moogsoft Bridge server(s) :
Upgrade Moogsoft Onprem Bridge Tarball to v9.2.0.1
Patch Moogsoft Onprem Bridge Tarball to v9.2.0.1
This topic describes how to patch a Tarball-based distribution of Moogsoft Onprem Bridge to v9.2.0.1 from v9.2.0
Important
While a Moogsoft Onprem Bridge environment can run with one or two RabbitMQ nodes, as of Moogsoft v9.2.0 / RabbitMQ v4, the new Quorum queues require at least three nodes to provide full protection to events in the queues in the case of a cluster node failure.
For example, three RabbitMQ v4 nodes will protect against a single node failing, and five RabbitMQ v4 nodes will protect against two nodes failing etc
So while the Moogsoft Onprem Bridge can run against a single RabbitMQ node, events will potentially be lost during this upgrade process. Events will be saved if there are two nodes, but there will be no protection on the queue should one of the nodes go down in the future. Hence it is recommended to provision a third RabbitMQ node for the Bridge to communicate with.
Documentation for creating a three node setup is available here: Fully Distributed HA Installation
Ensure that the main Moogsoft Onprem servers have been upgraded before performing the Bridge upgrades
For Bridge deployments with a single RabbitMQ node
Warning
Events may be lost during the upgrade process in this deployment type - it is recommended to upgrade to a three node RabbitMQ cluster to protect against event loss
Stop all services (ensure that all LAMs are stopped - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^$.*moogsoft_bridge.*$$/#\1/') | crontab - kill -9 $(ps -ef | grep java | grep moogsoft_bridge | awk '{print $2}') 2>/dev/null $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam stop ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade drain $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop
Upgrade the packages on the server using the instructions in the section below :
Upgrade Moogsoft Onprem Bridge packages (Tarball) to v9.2.0.1
This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.2.0
Important
Enabling the "latency performance" RHEL profile is strongly recommended. This profile allows RabbitMQ to operate much more efficiently so that throughput is increased and smoothed out.
For more information on performance profiles, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/getting-started-with-tuned_monitoring-and-managing-system-status-and-performance
Enable the profile by running the following command as root:
tuned-adm profile latency-performance
This setting will survive machine restarts and only needs to be set once.
‌
Warning
For deployments upgrading from v9.0.0.2
The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.
The following command must be run on all RabbitMQ server nodes before the following steps are performed:
export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH; ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all
Ensure the patch tarball is available to each server being patched:
Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):
curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.1.tgz"
Copy the tarball to all servers where the product is installed.
Optional GPG key validation of the tarball to validate the installation file.
For servers without internet access (if server has internet access go to the next step):
Download the key from this site:
https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
Copy the key to the server onto which the tarball will be installed (it will be an .asc file).
Import the key:
gpg --import 0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
For servers with internet access, run the following command:
curl https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 | gpg —-import
Download the patch tarball and matching '.sig' file from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:
https://speedy.moogsoft.com/v9/installer/
Copy the patch tgz and matching '.sig' file into the same folder.
Copy the following code into a bash terminal and run it to perform the validation:
gpg --verify moogsoft-enterprise-9.2.0.1.tgz.sig moogsoft-enterprise-9.2.0.1.tgz
Confirm that the command reports:
Good signature from "Moogsoft <security@moogsoft.com>"
Place the downloaded tarball file into an empty directory called patch_directory9.2.0.1 and execute the following commands to start the patch installation:
cd patch_directory9.2.0.1; tar -xf moogsoft-enterprise-9.2.0.1.tgz; bash moogsoft-enterprise-install-9.2.0.1.sh;
There may be changes in configuration files in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:
diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0.1/config | grep -i 'differ'
Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.2.0.1 version of these files (if appropriate/needed).
Restart the RabbitMQ service and rejoin the cluster:
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart; ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade revive ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues rebalance quorum
Confirm this RabbitMQ node has rejoined the existing cluster:
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl cluster_status
Two nodes should be listed
Restart all services (ensure that all LAMs are started - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^\#\+$.*moogsoft_bridge.*$/\1/') | crontab - $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam start $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam start
For Bridge deployments with two RabbitMQ nodes
Warning
Having only two nodes will not provide protection against event loss in RabbitMQ v4 - if one node fails, the cluster will fail. It is recommended to upgrade to a three node RabbitMQ cluster which can be done after this upgrade if needed.
On Server 1
Stop all services (ensure that all LAMs are stopped - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^$.*moogsoft_bridge.*$$/#\1/') | crontab - kill -9 $(ps -ef | grep java | grep moogsoft_bridge | awk '{print $2}') 2>/dev/null $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam stop ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade drain $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop
Upgrade the packages on the server using the instructions in the section below
Upgrade Moogsoft Onprem Bridge packages (Tarball) to v9.2.0.1
This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.2.0
Important
Enabling the "latency performance" RHEL profile is strongly recommended. This profile allows RabbitMQ to operate much more efficiently so that throughput is increased and smoothed out.
For more information on performance profiles, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/getting-started-with-tuned_monitoring-and-managing-system-status-and-performance
Enable the profile by running the following command as root:
tuned-adm profile latency-performance
This setting will survive machine restarts and only needs to be set once.
‌
Warning
For deployments upgrading from v9.0.0.2
The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.
The following command must be run on all RabbitMQ server nodes before the following steps are performed:
export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH; ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all
Ensure the patch tarball is available to each server being patched:
Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):
curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.1.tgz"
Copy the tarball to all servers where the product is installed.
Optional GPG key validation of the tarball to validate the installation file.
For servers without internet access (if server has internet access go to the next step):
Download the key from this site:
https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
Copy the key to the server onto which the tarball will be installed (it will be an .asc file).
Import the key:
gpg --import 0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
For servers with internet access, run the following command:
curl https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 | gpg —-import
Download the patch tarball and matching '.sig' file from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:
https://speedy.moogsoft.com/v9/installer/
Copy the patch tgz and matching '.sig' file into the same folder.
Copy the following code into a bash terminal and run it to perform the validation:
gpg --verify moogsoft-enterprise-9.2.0.1.tgz.sig moogsoft-enterprise-9.2.0.1.tgz
Confirm that the command reports:
Good signature from "Moogsoft <security@moogsoft.com>"
Place the downloaded tarball file into an empty directory called patch_directory9.2.0.1 and execute the following commands to start the patch installation:
cd patch_directory9.2.0.1; tar -xf moogsoft-enterprise-9.2.0.1.tgz; bash moogsoft-enterprise-install-9.2.0.1.sh;
There may be changes in configuration files in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:
diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0.1/config | grep -i 'differ'
Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.2.0.1 version of these files (if appropriate/needed).
Restart the RabbitMQ service and rejoin the cluster:
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart; ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade revive ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues rebalance quorum
Confirm this RabbitMQ node has rejoined the existing cluster:
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl cluster_status
Two nodes should be listed
Restart all services (ensure that all LAMs are started - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^\#\+$.*moogsoft_bridge.*$/\1/') | crontab - $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart
On Server 2
Stop all services (ensure that all LAMs are stopped - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^$.*moogsoft_bridge.*$$/#\1/') | crontab - kill -9 $(ps -ef | grep java | grep moogsoft_bridge | awk '{print $2}') 2>/dev/null $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam stop ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade drain $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop
Upgrade the packages on the server using the instructions in the section below
Upgrade Moogsoft Onprem Bridge packages (Tarball) to v9.2.0.1
This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.2.0
Important
Enabling the "latency performance" RHEL profile is strongly recommended. This profile allows RabbitMQ to operate much more efficiently so that throughput is increased and smoothed out.
For more information on performance profiles, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/getting-started-with-tuned_monitoring-and-managing-system-status-and-performance
Enable the profile by running the following command as root:
tuned-adm profile latency-performance
This setting will survive machine restarts and only needs to be set once.
‌
Warning
For deployments upgrading from v9.0.0.2
The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.
The following command must be run on all RabbitMQ server nodes before the following steps are performed:
export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH; ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all
Ensure the patch tarball is available to each server being patched:
Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):
curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.1.tgz"
Copy the tarball to all servers where the product is installed.
Optional GPG key validation of the tarball to validate the installation file.
For servers without internet access (if server has internet access go to the next step):
Download the key from this site:
https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
Copy the key to the server onto which the tarball will be installed (it will be an .asc file).
Import the key:
gpg --import 0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
For servers with internet access, run the following command:
curl https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 | gpg —-import
Download the patch tarball and matching '.sig' file from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:
https://speedy.moogsoft.com/v9/installer/
Copy the patch tgz and matching '.sig' file into the same folder.
Copy the following code into a bash terminal and run it to perform the validation:
gpg --verify moogsoft-enterprise-9.2.0.1.tgz.sig moogsoft-enterprise-9.2.0.1.tgz
Confirm that the command reports:
Good signature from "Moogsoft <security@moogsoft.com>"
Place the downloaded tarball file into an empty directory called patch_directory9.2.0.1 and execute the following commands to start the patch installation:
cd patch_directory9.2.0.1; tar -xf moogsoft-enterprise-9.2.0.1.tgz; bash moogsoft-enterprise-install-9.2.0.1.sh;
There may be changes in configuration files in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:
diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0.1/config | grep -i 'differ'
Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.2.0.1 version of these files (if appropriate/needed).
Restart the RabbitMQ service and rejoin the cluster:
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart; ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-upgrade revive ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues rebalance quorum
Confirm this RabbitMQ node has rejoined the existing cluster:
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl cluster_status
Two nodes should be listed
Restart all services (ensure that all LAMs are started - Rest LAM and Socket LAM are included below as examples but these should be replaced with the LAMs running in your deployment)
(crontab -l | sed -e 's/^\#\+$.*moogsoft_bridge.*$/\1/') | crontab - $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart
For Bridge deployments with more than two RabbitMQ nodes
For deployments with three or more RabbitMQ nodes, the same process as for two nodes can be followed - upgrade one server fully to start, then select the next server, upgrade the applications, join it to the first RabbitMQ node cluster, before restarting all services, and repeating for the remaining servers

‌

In this section:

Patch Moogsoft Onprem Tarball v9.2.0 to v9.2.0.1

Important

Important

Important

Warning

Important

Warning

Important

Important

Warning

Important

Warning

Warning

Important

Warning

Important

Warning

Search results