Skip to main content

Patch Moogsoft Onprem Tarball for v9.1.x.x

This topic describes how to patch a tarball-based distribution of Moogsoft Onprem to v9.1.0 from v9.0.0 or v9.0.0.x or v9.0.1.x

Important

Enabling the "latency performance" RHEL profile is strongly recommended. This profile allows RabbitMQ to operate much more efficiently so that throughput is increased and smoothed out.

For more information on performance profiles, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/getting-started-with-tuned_monitoring-and-managing-system-status-and-performance

Enable the profile by running the following command as root:

tuned-adm profile latency-performance

This setting will survive machine restarts and only needs to be set once.

Warning

  • For deployments upgrading from v9.0.0 or v9.0.0.1

    The upgrade path from v9.0.0/v9.0.0.1 to v9.0.1 onwards (any pre v9.0.0.2 release going to any post v9.0.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters. All rabbit nodes will need to be stopped before their binaries are upgraded. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.

  • For deployments upgrading from v9.0.0.2

    The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.

    The following command must be run on all RabbitMQ server nodes before the following steps are performed:

    export PATH=$MOOGSOFT_HOME/cots/erlang/bin/:$PATH;
${MOOGSOFT_HOME}/cots/rabbitmq-server/sbin/rabbitmqctl enable_feature_flag all

  • For users with UI Integrations containing a Rule with a name containing the word 'password'

    A known issue will affect the UI integrations page after upgrade to 9.1.0 . A workaround is to change the name of this rule so it doesn't equal or contain the word 'password'. Alternatively, a hotfix is available. Please contact Moogsoft Support in this case.

  1. Ensure the patch tarball is available to each server being patched:

    • Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):

      curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/v9/installer/moogsoft-enterprise-9.1.0.tgz"

      Copy the tarball to all servers where the product is installed.

  2. Stop all processes on each host:

    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;
$MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop;
kill -9 $(ps -ef | grep java | grep lam | awk '{print $2}') 2>/dev/null;
(crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
(crontab -l | sed -e 's/^\(.*keepalive.*\)$/#\1/') | crontab -
ps -ef | grep java | egrep 'events_analyser|keepalive' | awk '{print $2}' | xargs kill 2>/dev/null;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx stop;

  3. Gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):

    export OLD_DIST_PATH=$(readlink -f $MOOGSOFT_HOME/bin|awk -F '/' 'sub(FS $NF,x)');
export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);

  4. FOR ALL VERSIONS

    Update Percona to the latest version using the instructions here: Percona Cluster 8.0 Tarball Minor Version Upgrade

  5. FOR ALL VERSIONS

    RabbitMQ will be upgraded as part of this process and the nodes need to be stopped one at a time while they are each being upgraded.

    Use the following command to stop RabbitMQ on the current server:

    $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop

  6. Place the downloaded tarball file into an empty directory called patch_directory9.1.0 and execute the following commands to start the patch installation:

    cd patch_directory9.1.0;
tar -xf moogsoft-enterprise-9.1.0.tgz;
bash moogsoft-enterprise-install-9.1.0.sh;

  7. FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 OR EARLIER ONLY

    • Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted. On each node perform the following steps.

      The mnesia directory needs to be deleted using the following command:

      rm -rf $MOOGSOFT_HOME/var/lib/rabbitmq/mnesia/*

      The RabbitMQ node now needs to be re-initialised:

      $MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -pz <YOUR_ZONE_NAME>

      Then, re-initialize the other nodes in the same way using the same command.

      Now create a cluster of all the RabbitMQ nodes: https://www.rabbitmq.com/clustering.html

      Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected

  8. There are changes in controller.conf, system.conf, and security.conf in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:

    diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/9.1.0/config | grep -i 'differ'

    Update files in $MOOGSOFT_HOME/config with any changes introduced in the 9.1.0 version of these files (if appropriate/needed).

  9. FOR ALL VERSIONS

    Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted:

    $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq start

    Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected

  10. FOR ALL VERSIONS

    Refresh all stored procedures (provide the 'ermintrude' DB user password when prompted):

    $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 9.1.0 -u ermintrude

  11. FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 or v9.0.0.2 OR EARLIER ONLY

    A fix has been made for SAML authentication around WantAssertionsSigned handling which can impact SAML logins for v9.0.1. Signing is now always required, even when WantAssertionsSigned is disabled. WantAssertionsSigned now requires explicit signing of the assertions, not the response.

    A way to disable this new check (if needed) is to change the WantAssertionsSigned setting in the SP Metadata file from true to false as per the example below (change the file name and path as needed below).

    sed -i 's/WantAssertionsSigned="true"/WantAssertionsSigned="false"/g' ${MOOGSOFT_HOME}/config/keycloak.my_sp_metadata.xml

  12. FOR ALL VERSIONS

    Upgrade OpenSearch. This step will remove the existing copy of OpenSearch and upgrade it to the latest one (single node deployment):

    $MOOGSOFT_HOME/bin/utils/process_cntl opensearch stop;
$MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i

    This step will overwrite the opensearch_user password in $MOOGSOFT_HOME/config/system.conf If OpenSearch needs to be clustered, it can be done after all the nodes are upgraded fully.

  13. FOR ALL VERSIONS

    Re-deploy Apache Tomcat and reconfigure Nginx. IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the $MOOGSOFT_HOME/bin/utils/process_cntl script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat process. Run these commands in the same terminal as the one where the variables were saved previously. Then restart nginx:

    VERSION=9.1.0;
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula
cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/cots/nginx/ssl/;
cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/cots/nginx/ssl/;
sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/cots/nginx/config/conf.d/moog-ssl.conf;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx restart;

  14. Verify the remaining processes are working properly and update the crontab:

    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status;
$MOOGSOFT_HOME/bin/utils/process_cntl nginx status;
$MOOGSOFT_HOME/bin/utils/process_cntl opensearch status;
(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
(crontab -l | sed -e 's/^\#\+\(.*keepalive.*\)/\1/') | crontab -

  15. Validate the patch:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    If there are any errors from the above validators, please contact Moogsoft Support.

  16. Re-start non-UI integrations or LAMs

  17. Re-install the latest Add-ons pack: Install Moogsoft Add-ons

  18. FOR ALL VERSIONS

    The OpenSearch Cluster now needs to be recreated: Opensearch Clustering Guide - Tarball

  19. Restart moogfarmd:

    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd restart;

  20. Wait for MoogFarmd to start - about two minutes, then trigger a re-index of OpenSearch:

    $MOOGSOFT_HOME/bin/utils/moog_indexer -f -n

  21. Restart any event feeds if they were stopped.

  22. Clear the browser cache and log in to the UI.

In this section: