Skip to main content

Upgrade Moogsoft Onprem Tarball to v9.2.0

This topic describes how to patch a Tarball-based distribution of Moogsoft Onprem to v9.2.0 from v9.0.0 or 9.0.0.x or 9.0.1.x, 9.1.0.x, or 9.1.1.x

Important

This process requires the presence of a third Moogsoft Onprem server to act a redundancy server as per : Fully Distributed HA Installation

If this server is not already provisioned and running, it must be provisioned before starting this process, and a clean installation of Moogsoft Onprem v9.2.0 performed on it, with just the RabbitMQ and OpenSearch processes left running

In the process below, Server 1 and Server 2 are existing Moogsoft Onprem servers running MoogFarmD, Apache-Tomcat, RabbitMQ etc

In the process below, Server 3 is the Redundancy Server running RabbitMQ and possibly OpenSearch too

  1. On Server 1 or Server 2

    1. IMPORTANT: Make a note of the existing RabbitMQ Events.HA queue name by running this command:

      export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
VHOST=$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k mooms.zone)
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl -p "${VHOST}" list_queues | grep Events.HA | awk '{print $1}' | head -n 1

      This queue name is needed later in the upgrade process

  2. On Server 3

    1. Stop RabbitMQ

      $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;

  3. On Server 1

    1. Remove the Server 3 RabbitMQ node from the cluster:

      export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl forget_cluster_node rabbit@<SERVER_3_IP_OR_HOSTNAME>

    2. Comment out any reference to the Server 3 RabbitMQ broker from $MOOGSOFT_HOME/config/system.conf

      The entry under mooms -> brokers should be commented out

    3. Stop MoogFarmD to pause event processing

      $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;

    4. Restart all Moogsoft Onprem services (excluding MoogFarmD)

      $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat restart;
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart;
$MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart

  4. On Server 2

    1. Comment out any reference to the Server 1 and the Server 3 RabbitMQ brokers from $MOOGSOFT_HOME/config/system.conf

      The entries under mooms -> brokers should be commented out

    2. Stop MoogFarmD to pause event processing

      $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;

    3. Restart all Moogsoft Onprem services (excluding MoogFarmD)

      $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat restart;
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart;
$MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart

  5. On Server 3 (Redundancy Server)

    1. Stop all services

      $MOOGSOFT_HOME/bin/utils/process_cntl opensearch stop;

    2. Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade

      rm -rf $MOOGSOFT_HOME/var/lib/rabbitmq/mnesia/*

    3. Upgrade the packages on the server

      Upgrade Moogsoft Onprem Tarball to v9.2.0

      This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0 from v9.0.0 or v9.0.0.x or v9.0.1.x, v9.1.0.x, or 9.1.1

      Warning

      Moogsoft Onprem v9.1.1 and newer requires the underlying Percona XtraDB cluster to be v8.4 as part of the upgrade process (see Percona Cluster 8.4 Tarball Minor Version Upgrade)

      An issue with Percona XtraDB Cluster 8.4 has been identified (https://perconadev.atlassian.net/browse/PXC-4658) and this appears to be related to binary logging and async replication – which is recommended for a DR (disaster recovery) architecture. The recommendations are:

      • If you are not running a DR architecture using binary logging and async replication then it is safe to upgrade to v9.1.1

      • If you are running a DR architecture and can delay the upgrade to v9.1.1, do not upgrade until Percona have resolved the issue, Moogsoft has verified this fix, and this document is updated accordingly

      • If you are running a DR architecture and cannot delay the upgrade to v9.1.1, ignore the Percona Database upgrade step ( Step 5 ) in this process. The existing version of Percona PXC (8.0) is supported and compatible with v9.1.1

      If you have any questions please contact your usual Moogsoft technical contact and they will discuss the best route forward.

      Important

      Enabling the "latency performance" RHEL profile is strongly recommended. This profile allows RabbitMQ to operate much more efficiently so that throughput is increased and smoothed out.

      For more information on performance profiles, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/getting-started-with-tuned_monitoring-and-managing-system-status-and-performance

      Enable the profile by running the following command as root:

      tuned-adm profile latency-performance

      This setting will survive machine restarts and only needs to be set once.

      Warning

      • For deployments upgrading from v9.0.0 or v9.0.0.1

        The upgrade path from v9.0.0/v9.0.0.1 to v9.0.1 onwards (any pre v9.0.0.2 release going to any post v9.0.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters. All rabbit nodes will need to be stopped before their binaries are upgraded. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.

      • For deployments upgrading from v9.0.0.2

        The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.

        The following command must be run on all RabbitMQ server nodes before the following steps are performed:

        export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all

      1. Ensure the patch tarball is available to each server being patched:

        • Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):

          curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.tgz"

          Copy the tarball to all servers where the product is installed.

      2. Gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):

        export OLD_DIST_PATH=$(readlink -f $MOOGSOFT_HOME/bin|awk -F '/' 'sub(FS $NF,x)');
export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);

      3. FOR ALL VERSIONS

        IMPORTANT: Please ensure you have read the Warning at the top of this page regarding this step

        Update Percona to the latest version using the instructions here: Percona Cluster 8.4 Tarball Minor Version Upgrade

      4. Place the downloaded tarball file into an empty directory called patch_directory9.2.0 and execute the following commands to start the patch installation:

        cd patch_directory9.2.0;
tar -xf moogsoft-enterprise-9.2.0.tgz;
bash moogsoft-enterprise-install-9.2.0.sh;

      5. There are changes in controller.conf, system.conf, and security.conf in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:

        diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0/config | grep -i 'differ'

        Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.2.0 version of these files (if appropriate/needed).

      6. FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 or v9.0.0.2 OR EARLIER ONLY

        A fix has been made for SAML authentication around WantAssertionsSigned handling which can impact SAML logins for v9.0.1. Signing is now always required, even when WantAssertionsSigned is disabled. WantAssertionsSigned now requires explicit signing of the assertions, not the response.

        A way to disable this new check (if needed) is to change the WantAssertionsSigned setting in the SP Metadata file from true to false as per the example below (change the file name and path as needed below).

        sed -i 's/WantAssertionsSigned="true"/WantAssertionsSigned="false"/g' ${MOOGSOFT_HOME}/config/keycloak.my_sp_metadata.xml

      7. FOR ALL VERSIONS

        Upgrade OpenSearch. This step will remove the existing copy of OpenSearch and upgrade it to the latest one (single node deployment):

        $MOOGSOFT_HOME/bin/utils/process_cntl opensearch stop;
$MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i

        This step will overwrite the opensearch_user password in $MOOGSOFT_HOME/config/system.conf If OpenSearch needs to be clustered, it can be done after all the nodes are upgraded fully.

      8. Verify the remaining processes are working properly and update the crontab:

        $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx status;
$MOOGSOFT_HOME/bin/utils/process_cntl opensearch status;
(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
(crontab -l | sed -e 's/^\#\+\(.*keepalive.*\)/\1/') | crontab -

    4. Create the RabbitMQ vhost and Events.HA quorum queue on that node (to allow events to queue here during the upgrade)

      VHOST=<your_vhost/zone_name>;
$MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -pz ${VHOST};
QUEUE_NAME="<YOUR_EVENTS_HA_QUEUE_NAME>";
chmod u+x ${MOOGSOFT_HOME}/cots/rabbitmq-server/plugins/rabbitmq_management-*/priv/www/cli/rabbitmqadmin;
${MOOGSOFT_HOME}/cots/rabbitmq-server/plugins/rabbitmq_management-*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft --vhost=${VHOST} declare queue name=${QUEUE_NAME} durable=true arguments='{"x-queue-type":"quorum"}';
${MOOGSOFT_HOME}/cots/rabbitmq-server/plugins/rabbitmq_management-*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft --vhost=${VHOST} declare exchange name=MOOMS.GENERAL.TOPIC_EXCHANGE type=topic durable=true
${MOOGSOFT_HOME}/cots/rabbitmq-server/plugins/rabbitmq_management-*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft --vhost=${VHOST} declare binding source="MOOMS.GENERAL.TOPIC_EXCHANGE" destination_type="queue" destination="${QUEUE_NAME}" routing_key="events"

  6. On Server 2

    1. Confirm that the RabbitMQ HA policy is still present on the Events queue: 

      export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl list_queues name durable policy -p <YOUR_VHOST> | grep Events.HA

      This command should return a value. If it does not, the policy should be reapplied with this command: 

      ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl set_policy -p <YOUR_ZONE> ha-all ".+.HA" '{"ha-mode":"all"}'

    2. Create a RabbitMQ shovel from the existing Events.HA queue to the Server 3 Events.HA queue to buffer events during the upgrade. Update the commands below with accurate variable values for the hostnames, VHosts, and credentials.

      SERVER_2_RABBITMQ_CREDS="moogsoft:m00gs0ft";
SERVER_2_RABBITMQ_VHOST="<YOUR_VHOST_NAME>";
SERVER_2_RABBITMQ_HOST="127.0.0.1";
QUEUE="<YOUR_EVENTS_HA_QUEUE_NAME>";
SERVER_3_RABBITMQ_CREDS="moogsoft:m00gs0ft";
SERVER_3_RABBITMQ_VHOST="<YOUR_VHOST_NAME>";
SERVER_3_RABBITMQ_HOST="YOUR_SERVER_3_IP_OR_HOSTNAME";
export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-plugins enable rabbitmq_shovel;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-plugins enable rabbitmq_shovel_management;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl -p "${SERVER_2_RABBITMQ_VHOST}" set_parameter shovel my_shovel "$(cat <<EOF
{
   "src-uri": "amqp://${SERVER_2_RABBITMQ_CREDS}@${SERVER_2_RABBITMQ_HOST}/${SERVER_2_RABBITMQ_VHOST}",
   "src-queue": "${QUEUE}",
   "dest-uri": "amqp://${SERVER_3_RABBITMQ_CREDS}@${SERVER_3_RABBITMQ_HOST}/${SERVER_3_RABBITMQ_VHOST}",
   "add-forward-headers": false,
   "dest-exchange": "MOOMS.GENERAL.TOPIC_EXCHANGE",
   "ack-mode": "on-confirm",
   "delete-after": "never"
}
EOF
)"

    3. Confirm that the Shovel was created successfully by running the command below.

      The "state" must show "running" in this output. If the Shovel is not running at this stage, events may be lost depending on the overall process being carried out.

      ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl -p <YOUR_VHOST> shovel_status

    4. Assuming that events are still being ingested via LAMs and UI integrations, the count for the Events.HA queue one the SERVER 3 node should now be increasing. You can check this by using this CLI command on Server 3:

      ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl -p <YOUR_VHOST> list_queues

  7. On Server 1

    1. Confirm that the RabbitMQ HA policy is still present on the Events queue: 

      export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl list_queues name durable policy -p <YOUR_VHOST> | grep Events.HA

      This command should return a value. If it does not, the policy should be reapplied with this command: 

      ${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl set_policy -p <YOUR_VHOST> ha-all ".+.HA" '{"ha-mode":"all"}'

    2. Stop RabbitMQ

      $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;

  8. On Server 2

    1. Remove the Server 1 RabbitMQ node from the cluster: 

      export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl forget_cluster_node rabbit@<SERVER_1_IP_OR_HOSTNAME>

  9. On Server 1

    1. Stop all services (update the commands below as needed) 

      $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;
$MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop;
kill -9 $(ps -ef | grep java | grep lam | awk '{print $2}') 2>/dev/null;
(crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
(crontab -l | sed -e 's/^\(.*keepalive.*\)$/#\1/') | crontab -
ps -ef | grep java | egrep 'events_analyser|keepalive' | awk '{print $2}' | xargs kill 2>/dev/null;

    2. Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade 

      rm -rf $MOOGSOFT_HOME/var/lib/rabbitmq/mnesia/*

    3. Upgrade the packages on the server

      Upgrade Moogsoft Onprem Tarball to v9.2.0

      This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0 from v9.0.0 or v9.0.0.x or v9.0.1.x, v9.1.0.x, or 9.1.1

      Warning

      Moogsoft Onprem v9.1.1 and newer requires the underlying Percona XtraDB cluster to be v8.4 as part of the upgrade process (see Percona Cluster 8.4 Tarball Minor Version Upgrade)

      An issue with Percona XtraDB Cluster 8.4 has been identified (https://perconadev.atlassian.net/browse/PXC-4658) and this appears to be related to binary logging and async replication – which is recommended for a DR (disaster recovery) architecture. The recommendations are:

      • If you are not running a DR architecture using binary logging and async replication then it is safe to upgrade to v9.1.1

      • If you are running a DR architecture and can delay the upgrade to v9.1.1, do not upgrade until Percona have resolved the issue, Moogsoft has verified this fix, and this document is updated accordingly

      • If you are running a DR architecture and cannot delay the upgrade to v9.1.1, ignore the Percona Database upgrade step ( Step 5 ) in this process. The existing version of Percona PXC (8.0) is supported and compatible with v9.1.1

      If you have any questions please contact your usual Moogsoft technical contact and they will discuss the best route forward.

      Important

      Enabling the "latency performance" RHEL profile is strongly recommended. This profile allows RabbitMQ to operate much more efficiently so that throughput is increased and smoothed out.

      For more information on performance profiles, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/getting-started-with-tuned_monitoring-and-managing-system-status-and-performance

      Enable the profile by running the following command as root:

      tuned-adm profile latency-performance

      This setting will survive machine restarts and only needs to be set once.

      Warning

      • For deployments upgrading from v9.0.0 or v9.0.0.1

        The upgrade path from v9.0.0/v9.0.0.1 to v9.0.1 onwards (any pre v9.0.0.2 release going to any post v9.0.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters. All rabbit nodes will need to be stopped before their binaries are upgraded. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.

      • For deployments upgrading from v9.0.0.2

        The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.

        The following command must be run on all RabbitMQ server nodes before the following steps are performed:

        export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all

      1. Ensure the patch tarball is available to each server being patched:

        • Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):

          curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.tgz"

          Copy the tarball to all servers where the product is installed.

      2. Gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):

        export OLD_DIST_PATH=$(readlink -f $MOOGSOFT_HOME/bin|awk -F '/' 'sub(FS $NF,x)');
export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);

      3. FOR ALL VERSIONS

        IMPORTANT: Please ensure you have read the Warning at the top of this page regarding this step

        Update Percona to the latest version using the instructions here: Percona Cluster 8.4 Tarball Minor Version Upgrade

      4. Place the downloaded tarball file into an empty directory called patch_directory9.2.0 and execute the following commands to start the patch installation:

        cd patch_directory9.2.0;
tar -xf moogsoft-enterprise-9.2.0.tgz;
bash moogsoft-enterprise-install-9.2.0.sh;

      5. There are changes in controller.conf, system.conf, and security.conf in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:

        diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0/config | grep -i 'differ'

        Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.2.0 version of these files (if appropriate/needed).

      6. FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 or v9.0.0.2 OR EARLIER ONLY

        A fix has been made for SAML authentication around WantAssertionsSigned handling which can impact SAML logins for v9.0.1. Signing is now always required, even when WantAssertionsSigned is disabled. WantAssertionsSigned now requires explicit signing of the assertions, not the response.

        A way to disable this new check (if needed) is to change the WantAssertionsSigned setting in the SP Metadata file from true to false as per the example below (change the file name and path as needed below).

        sed -i 's/WantAssertionsSigned="true"/WantAssertionsSigned="false"/g' ${MOOGSOFT_HOME}/config/keycloak.my_sp_metadata.xml

      7. FOR ALL VERSIONS

        Upgrade OpenSearch. This step will remove the existing copy of OpenSearch and upgrade it to the latest one (single node deployment):

        $MOOGSOFT_HOME/bin/utils/process_cntl opensearch stop;
$MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i

        This step will overwrite the opensearch_user password in $MOOGSOFT_HOME/config/system.conf If OpenSearch needs to be clustered, it can be done after all the nodes are upgraded fully.

      8. Verify the remaining processes are working properly and update the crontab:

        $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx status;
$MOOGSOFT_HOME/bin/utils/process_cntl opensearch status;
(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
(crontab -l | sed -e 's/^\#\+\(.*keepalive.*\)/\1/') | crontab -

    4. Upgrade the database schema

      Upgrade the database schema

      Upgrade the database schema and refresh all stored procedures (provide the 'ermintrude' DB user password when prompted):

      $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 9.2.0 -u ermintrude

    5. Cluster RabbitMQ with the Node 3 RabbitMQ

      Join the RabbitMQ v4 cluster (Tarball)

      The primary erlang cookie is located at $HOME/.erlang.cookie on whichever RabbitMQ node has been upgraded first.

      The erlang cookie must be the same for all RabbitMQ nodes. Replace the erlang cookie on all other RabbitMQ nodes with primary erlang cookie

      Make the cookies on non-primary nodes read-only:

      chmod 400 $HOME/.erlang.cookie

      You may need to change the file permissions on the existing erlang cookies first to allow those files to be overwritten. For example:

      chmod 600 $HOME/.erlang.cookie

      Restart RabbitMQ on nodes where the erlang cookie has been replaced

      $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart

      To join a RabbitMQ node to a cluster, run these commands on the servers needing to join the cluster, but first substitute the short hostname of the RabbitMQ server you wish to cluster with. The short hostname is the full hostname excluding the DNS domain name. For example, if the hostname is ip-172-31-82-78.ec2.internal, the short hostname is ip-172-31-82-78. To find out the short hostname, run rabbitmqctl cluster_status on the appropriate server.

      $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart

export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl stop_app
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl join_cluster rabbit@<NODE_3_SHORTNAME>
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl start_app

      Run rabbitmqctl cluster_status to get the cluster status. Example output is as follows:

      Cluster status of node rabbit@ip-172-31-93-201 Basics Cluster name rabbit@ip-172-31-93-201.ec2.internal Disk Nodes rabbit@ip-172-31-93-201
rabbit@ip-172-31-85-42
rabbit@ip-172-31-93-201 Running Nodes rabbit@ip-172-31-93-201
rabbit@ip-172-31-85-42
rabbit@ip-172-31-93-201 Versions
...

    6. Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 3

      The entries under mooms -&gt; brokers should be added or uncommented as needed

    7. Upgrade Apache-Tomcat

      Upgrade Apache-Tomcat (Tarball)

      Warning

      This process will completely remove and re-deploy Apache-Tomcat. If the Apache-Tomcat logs need to be kept, a copy should be made before proceeding.

      Deploy the new version of Apache Tomcat and the latest webapps:

      IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the $MOOGSOFT_HOME/bin/utils/process_cntl script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat process.

      Run these commands in the same terminal as the one where the variables were saved previously. Then restart nginx:

      $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula
cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/cots/nginx/ssl/;
cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/cots/nginx/ssl/;
sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx restart;

    8. Perform final checks

      Final upgrade steps (Tarball)

      1. Validate the patch: 

        $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

        If there are any errors from the above validators, please contact Moogsoft Support.

      2. Re-start non-UI integrations or LAMs

      3. Re-install the latest Add-ons pack: Install Moogsoft Add-ons

      4. The OpenSearch Cluster now needs to be recreated (if it has not been already): Opensearch Clustering Guide - Tarball

      5. Restart any event feeds if they were stopped

    9. Start the LAMs (update the commands below as needed)

      $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart
$MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart

  10. On Server 2

    1. Stop Tomcat and LAMs (update the commands below as needed) 

      $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop;
kill -9 $(ps -ef | grep java | grep lam | awk '{print $2}') 2>/dev/null;
(crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
(crontab -l | sed -e 's/^\(.*keepalive.*\)$/#\1/') | crontab -
ps -ef | grep java | egrep 'events_analyser|keepalive' | awk '{print $2}' | xargs kill 2>/dev/null;

    2. Wait for the old Events.HA RabbitMQ queue to drain via the shovel to the new nodes. The command below needs to return a value of 0 before proceeding:

      export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
VHOST=$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k mooms.zone)
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl -p "${VHOST}" list_queues name messages | grep Events.HA

    3. Stop RabbitMQ 

      $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;

    4. Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade 

      rm -rf $MOOGSOFT_HOME/var/lib/rabbitmq/mnesia/*

    5. Upgrade the packages on the server

      Upgrade Moogsoft Onprem Tarball to v9.2.0

      This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.2.0 from v9.0.0 or v9.0.0.x or v9.0.1.x, v9.1.0.x, or 9.1.1

      Warning

      Moogsoft Onprem v9.1.1 and newer requires the underlying Percona XtraDB cluster to be v8.4 as part of the upgrade process (see Percona Cluster 8.4 Tarball Minor Version Upgrade)

      An issue with Percona XtraDB Cluster 8.4 has been identified (https://perconadev.atlassian.net/browse/PXC-4658) and this appears to be related to binary logging and async replication – which is recommended for a DR (disaster recovery) architecture. The recommendations are:

      • If you are not running a DR architecture using binary logging and async replication then it is safe to upgrade to v9.1.1

      • If you are running a DR architecture and can delay the upgrade to v9.1.1, do not upgrade until Percona have resolved the issue, Moogsoft has verified this fix, and this document is updated accordingly

      • If you are running a DR architecture and cannot delay the upgrade to v9.1.1, ignore the Percona Database upgrade step ( Step 5 ) in this process. The existing version of Percona PXC (8.0) is supported and compatible with v9.1.1

      If you have any questions please contact your usual Moogsoft technical contact and they will discuss the best route forward.

      Important

      Enabling the "latency performance" RHEL profile is strongly recommended. This profile allows RabbitMQ to operate much more efficiently so that throughput is increased and smoothed out.

      For more information on performance profiles, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/getting-started-with-tuned_monitoring-and-managing-system-status-and-performance

      Enable the profile by running the following command as root:

      tuned-adm profile latency-performance

      This setting will survive machine restarts and only needs to be set once.

      Warning

      • For deployments upgrading from v9.0.0 or v9.0.0.1

        The upgrade path from v9.0.0/v9.0.0.1 to v9.0.1 onwards (any pre v9.0.0.2 release going to any post v9.0.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters. All rabbit nodes will need to be stopped before their binaries are upgraded. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.

      • For deployments upgrading from v9.0.0.2

        The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.

        The following command must be run on all RabbitMQ server nodes before the following steps are performed:

        export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all

      1. Ensure the patch tarball is available to each server being patched:

        • Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):

          curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.2.0.tgz"

          Copy the tarball to all servers where the product is installed.

      2. Gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):

        export OLD_DIST_PATH=$(readlink -f $MOOGSOFT_HOME/bin|awk -F '/' 'sub(FS $NF,x)');
export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);

      3. FOR ALL VERSIONS

        IMPORTANT: Please ensure you have read the Warning at the top of this page regarding this step

        Update Percona to the latest version using the instructions here: Percona Cluster 8.4 Tarball Minor Version Upgrade

      4. Place the downloaded tarball file into an empty directory called patch_directory9.2.0 and execute the following commands to start the patch installation:

        cd patch_directory9.2.0;
tar -xf moogsoft-enterprise-9.2.0.tgz;
bash moogsoft-enterprise-install-9.2.0.sh;

      5. There are changes in controller.conf, system.conf, and security.conf in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:

        diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.2.0/config | grep -i 'differ'

        Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.2.0 version of these files (if appropriate/needed).

      6. FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 or v9.0.0.2 OR EARLIER ONLY

        A fix has been made for SAML authentication around WantAssertionsSigned handling which can impact SAML logins for v9.0.1. Signing is now always required, even when WantAssertionsSigned is disabled. WantAssertionsSigned now requires explicit signing of the assertions, not the response.

        A way to disable this new check (if needed) is to change the WantAssertionsSigned setting in the SP Metadata file from true to false as per the example below (change the file name and path as needed below).

        sed -i 's/WantAssertionsSigned="true"/WantAssertionsSigned="false"/g' ${MOOGSOFT_HOME}/config/keycloak.my_sp_metadata.xml

      7. FOR ALL VERSIONS

        Upgrade OpenSearch. This step will remove the existing copy of OpenSearch and upgrade it to the latest one (single node deployment):

        $MOOGSOFT_HOME/bin/utils/process_cntl opensearch stop;
$MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i

        This step will overwrite the opensearch_user password in $MOOGSOFT_HOME/config/system.conf If OpenSearch needs to be clustered, it can be done after all the nodes are upgraded fully.

      8. Verify the remaining processes are working properly and update the crontab:

        $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx status;
$MOOGSOFT_HOME/bin/utils/process_cntl opensearch status;
(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
(crontab -l | sed -e 's/^\#\+\(.*keepalive.*\)/\1/') | crontab -

    6. Cluster RabbitMQ with Server 3 RabbitMQ

      Join the RabbitMQ v4 cluster (Tarball)

      The primary erlang cookie is located at $HOME/.erlang.cookie on whichever RabbitMQ node has been upgraded first.

      The erlang cookie must be the same for all RabbitMQ nodes. Replace the erlang cookie on all other RabbitMQ nodes with primary erlang cookie

      Make the cookies on non-primary nodes read-only:

      chmod 400 $HOME/.erlang.cookie

      You may need to change the file permissions on the existing erlang cookies first to allow those files to be overwritten. For example:

      chmod 600 $HOME/.erlang.cookie

      Restart RabbitMQ on nodes where the erlang cookie has been replaced

      $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart

      To join a RabbitMQ node to a cluster, run these commands on the servers needing to join the cluster, but first substitute the short hostname of the RabbitMQ server you wish to cluster with. The short hostname is the full hostname excluding the DNS domain name. For example, if the hostname is ip-172-31-82-78.ec2.internal, the short hostname is ip-172-31-82-78. To find out the short hostname, run rabbitmqctl cluster_status on the appropriate server.

      $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart

export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl stop_app
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl join_cluster rabbit@<NODE_3_SHORTNAME>
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl start_app

      Run rabbitmqctl cluster_status to get the cluster status. Example output is as follows:

      Cluster status of node rabbit@ip-172-31-93-201 Basics Cluster name rabbit@ip-172-31-93-201.ec2.internal Disk Nodes rabbit@ip-172-31-93-201
rabbit@ip-172-31-85-42
rabbit@ip-172-31-93-201 Running Nodes rabbit@ip-172-31-93-201
rabbit@ip-172-31-85-42
rabbit@ip-172-31-93-201 Versions
...

    7. Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 2, and Server 3

      The entries under mooms -> brokers should be added or uncommented as needed

    8. Upgrade Apache-Tomcat

      Upgrade Apache-Tomcat (Tarball)

      Warning

      This process will completely remove and re-deploy Apache-Tomcat. If the Apache-Tomcat logs need to be kept, a copy should be made before proceeding.

      Deploy the new version of Apache Tomcat and the latest webapps:

      IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the $MOOGSOFT_HOME/bin/utils/process_cntl script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat process.

      Run these commands in the same terminal as the one where the variables were saved previously. Then restart nginx:

      $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula
cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/cots/nginx/ssl/;
cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/cots/nginx/ssl/;
sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx restart;

    9. Perform final checks

      Final upgrade steps (Tarball)

      1. Validate the patch: 

        $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

        If there are any errors from the above validators, please contact Moogsoft Support.

      2. Re-start non-UI integrations or LAMs

      3. Re-install the latest Add-ons pack: Install Moogsoft Add-ons

      4. The OpenSearch Cluster now needs to be recreated (if it has not been already): Opensearch Clustering Guide - Tarball

      5. Restart any event feeds if they were stopped

    10. Start the LAMs (update the commands below as needed)

      $MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart
$MOOGSOFT_HOME/bin/utils/process_cntl socket_lam restart

  11. On Server 1

    1. Ensure all RabbitMQ quorum queues are updated to have the right number of replicas:

      export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues grow rabbit@<SERVER1_HOSTNAME_OR_IP> all;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues grow rabbit@<SERVER2_HOSTNAME_OR_IP> all;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmq-queues grow rabbit@<SERVER3_HOSTNAME_OR_IP> all;

    2. Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 2, and Server 3

      The entries under mooms -> brokers should be added or uncommented as needed

    3. Start MoogFarmD

      $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd restart

    4. Restart Apache-Tomcat

      $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat restart

  12. On Server 2

    1. Start MoogFarmD

      $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd restart

  13. Upgrade the Moogsoft Bridge server(s) :

‌

‌

‌

In this section: