Skip to main content

Patch Moogsoft Enterprise Tarball for v8.2.x

This topic describes how to patch a tarball-based distribution of Moogsoft Enterprise to v8.2.0 from any v8.1.0.x version.

Important

The v8.2.0 release onwards uses Opensearch instead of Elasticsearch as the Search provider. It is possible to stick with Elasticsearch (Simply ignore the steps to stop Elastic and don't run the moog_init_search.sh command at the end) but this configuration will not be supported if Elastic becomes out of support or loses API/REST compatibility in future.

Opensearch has authentication enabled by default. After it is installed, a new user 'moog_opensearchuser' is added to Opensearch and $MOOGSOFT_HOME/config/system.conf is updated to use this user to connect.

IMPORTANT: Opensearch REQUIRES the system variable vm.max_map_count to be at least 262144. If it is lower, the moog_init_search.sh script will not deploy Opensearch. See: https://opensearch.org/docs/latest/opensearch/install/important-settings/

IMPORTANT: Opensearch REQUIRES some specific ulimits to be set for the user running the application. Example settings for /etc/security/limits.conf are as follows (assumes the non-root user is 'my_moog_user'):

my_moog_user	soft nofile	 65536
my_moog_user	hard nofile	 65536
my_moog_user	soft nproc	65535
my_moog_user	hard nproc	65535
  1. Ensure the patch tarball is available to each server being patched:

    • Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):

      curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/installer/moogsoft-enterprise-8.2.0.tgz"

      Copy the tarball to all servers where the product is installed.

  2. Optional GPG key validation of the tarball to validate the installation file.

    • For servers without internet access (if server has internet access go to the next step):

      1. Download the key from this site:

        https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
      2. Copy the key to the server onto which the tarball will be installed (it will be an .asc file).

      3. Import the key:

        gpg --import 0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
    • For servers with internet access, run the following command:

      curl https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 | gpg —-import
      1. Download the patch tarball and matching '.sig' file from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:

        https://speedy.moogsoft.com/installer/
    • Copy the patch tgz and matching '.sig' file into the same folder.

    • Copy the following code into a bash terminal and run it to perform the validation:

      gpg --verify moogsoft-enterprise-8.2.0.tgz.sig moogsoft-enterprise-8.2.0.tgz
    • Confirm that the command reports:

      Good signature from "Moogsoft <security@moogsoft.com>"
  3. FOR ALL VERSIONS

    Stop all processes on each host:

    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;
    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop;
    kill -9 $(ps -ef | grep java | grep lam | awk '{print $2}') 2>/dev/null;
    (crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
    (crontab -l | sed -e 's/^\(.*keepalive.*\)$/#\1/') | crontab -
    ps -ef | grep java | egrep 'events_analyser|keepalive' | awk '{print $2}' | xargs kill 2>/dev/null;
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx stop;
    $MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch stop;
    kill -9 $(ps -ef | grep org.elasticsearch.bootstrap.Elasticsearch | awk '{print $2}')
    $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop;
  4. FOR ALL VERSIONS

    Gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):

    export OLD_DIST_PATH=$(readlink -f $MOOGSOFT_HOME/bin|awk -F '/' 'sub(FS $NF,x)');
    export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
    export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
    export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
    export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
  5. FOR ALL VERSIONS

    Update Percona to the latest version using the instructions here: Percona Cluster 5.7 Tarball Minor Version Upgrade

  6. FOR ALL VERSIONS

    Place the downloaded tarball file into an empty directory called patch_directory8.2.0 and execute the following commands to start the patch installation:

    cd patch_directory8.2.0;
    tar -xf moogsoft-enterprise-8.2.0.tgz;
    bash moogsoft-enterprise-install-8.2.0.sh;
  7. FOR ALL VERSIONS

    Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted.

    $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart

    Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected

  8. FOR ALL VERSIONS

    Refresh all stored procedures (provide the 'root' DB user password when prompted):

    $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 8.2.0 -u root
  9. FOR ALL VERSIONS

    One of the database views needs to be patched:

    $MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "CREATE OR REPLACE VIEW historic_feedback AS SELECT * FROM historic_moogdb.feedback"
  10. FOR ALL VERSIONS

    Re-deploy Apache Tomcat and reconfigure Nginx. IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the $MOOGSOFT_HOME/bin/utils/process_cntl script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat process. Run these commands in the same terminal as the one where the variables were saved previously. Then restart nginx:

    VERSION=8.2.0;
    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula
    cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/dist/${VERSION}/cots/nginx/ssl/;
    cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/dist/${VERSION}/cots/nginx/ssl/;
    sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/dist/${VERSION}/cots/nginx/config/conf.d/moog-ssl.conf;
    sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/dist/${VERSION}/cots/nginx/config/conf.d/moog-ssl.conf;
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx restart;
  11. FOR ALL VERSIONS

    Verify the remaining processes are working properly and update the crontab:

    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status;
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx status;
    (crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
    (crontab -l | sed -e 's/^\#\+\(.*keepalive.*\)/\1/') | crontab -
    
  12. FOR ALL VERSIONS

    Validate the patch:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    If there are any errors from the above validators, please contact Moogsoft Support.

  13. FOR ALL VERSIONS

    Re-start non-UI integrations or LAMs

  14. FOR ALL VERSIONS

    The 'Addons' pack will need to be re-installed.

    The Addons v2.4.0 bundle is included with this release: $MOOGSOFT_HOME/contrib/Moogsoft-Addons-2.4.0.tar.gz

    Alternatively, the latest 'Addons' pack can always be found by following the instructions here:

  15. FOR ALL VERSIONS

    Elasticsearch is no longer the default Search provider for Moogsoft Enterprise. To install a single-node Opensearch deployment and create a user to connect with in one step on the current server, follow the steps below. For a cluster, follow the instructions here instead: Opensearch Clustering Guide

    Ensure Elasticsearch has been stopped:

    kill -9 $(ps -ef | grep org.elasticsearch.bootstrap.Elasticsearch | grep -v grep | awk '{print $2}') 2>/dev/null

    Install Opensearch, change the default admin password, and create a new account to connect with the following command:

    $MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i

    This will deploy Opensearch, change the default admin account password (the new password will be printed out to STDOUT/the terminal, so be sure to make a note of it), and create a 'moog_opensearchuser' in Opensearch and $MOOGSOFT_HOME/config/system.conf is updated to use this user to connect.

    Warning

    If the OS system variable vm.max_map_count is less than 262144, a system administrator or the root user will need to set this value otherwise the Opensearch install will not proceed. This setting is required by Opensearch.

    If the $MOOGSOFT_HOME/config/system.conf file has all the comments removed, the above script will update the wrong fields with the username and password for Opensearch - to fix this, manually correct the fields in that configuration file before proceeding. A fix for this will be released in an upcoming patch.

    If any customisations had been made to the ElasticSearch jvm.options file, the same changes can be made to the $MOOGSOFT_HOME/cots/opensearch/config/jvm.options.d/moog.options file. A restart of OpenSearch is needed for such changes to take effect.

    Warnings such as the following when OpenSearch starts can be safely ignored:

    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/opt/moogsoft/install/dist/8.2.0/cots/opensearch/plugins/opensearch-anomaly-detection/protostuff-runtime-1.7.4.jar) to field java.lang.Throwable.cause
    WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
    WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
  16. FOR ALL VERSIONS

    Restart apache-tomcat:

    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat restart;
  17. FOR ALL VERSIONS

    Restart moogfarmd:

    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd start;

    Wait for MoogFarmD to finish loading by ensuring this line has appeared in the log: Farmd initialization complete

  18. FOR ALL VERSIONS

    As Opensearch is now being used instead of Elasticsearch, it will need to be re-indexed using the following command:

    $MOOGSOFT_HOME/bin/utils/moog_indexer -f -n
  19. FOR ALL VERSIONS

    Restart any event feeds if they were stopped.

  20. FOR ALL VERSIONS

    Clear the browser cache and log in to the UI.