Patch Moogsoft Enterprise Tarball

This topic describes how to patch a tarball-based distribution of Moogsoft Enterprise to v8.1.0 from any v8.0.0.x version.

Warning

  • The upgrade path from v8.0.x to v8.1.0 onwards (any pre v8.1 release going to any post v8.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters due to incompatibilities between the old (v3.7.4) and the new versions of Rabbit preventing a 'rolling upgrade'.

    All rabbit nodes will need to be stopped before their binaries are upgraded, and then they must be brought back up in the right order. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.

    The version of RabbitMQ being upgraded to in this release supports rolling upgrades, so future upgrades should be less disruptive

  1. Ensure the patch tarball is available to each server being patched:

    • Download the patch tarball using the following command (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):

      curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/installer/moogsoft-enterprise-8.1.0.tgz"

      Copy the tarball to all servers where the product is installed.

  2. Optional GPG key validation of the tarball to validate the installation file.

    • For servers without internet access (if server has internet access go to the next step):

      1. Download the key from this site:

        https://keys.openpgp.org/vks/v1/by-fingerprint/887B323F485F122FE923214FD01B4B4B6F379E7E
      2. Copy the key to the server onto which the tarball will be installed (it will be an .asc file).

      3. Import the key:

        gpg --import 887B323F485F122FE923214FD01B4B4B6F379E7E
    • For servers with internet access, run the following command:

      curl https://keys.openpgp.org/vks/v1/by-fingerprint/887B323F485F122FE923214FD01B4B4B6F379E7E | gpg —import
      1. Download the patch tarball and matching '.sig' file from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:

        https://speedy.moogsoft.com/installer/
    • Copy the patch tgz and matching '.sig' file into the same folder.

    • Copy the following code into a bash terminal and run it to perform the validation:

      gpg --verify moogsoft-enterprise-8.1.0.tgz.sig moogsoft-enterprise-8.1.0.tgz
    • Confirm that the command reports:

      Good signature from "Moogsoft <security@moogsoft.com>"
  3. Stop all processes on each host:

    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop;
    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop;
    kill -9 $(ps -ef | grep java | grep lam | awk '{print $2}') 2>/dev/null;
    (crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
    (crontab -l | sed -e 's/^\(.*keepalive.*\)$/#\1/') | crontab -
    ps -ef | grep java | egrep 'events_analyser|keepalive' | awk '{print $2}' | xargs kill 2>/dev/null;
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx stop;
  4. Gather the required variables (do not close the terminal these commands are run in, otherwise they will be lost):

    export OLD_DIST_PATH=$(readlink -f $MOOGSOFT_HOME/bin|awk -F '/' 'sub(FS $NF,x)');
    export CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
    export CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf|awk '{print $2}'|tr -d ';'));
    export CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
    export CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $OLD_DIST_PATH/cots/nginx/config/conf.d/moog-ssl.conf);
  5. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.5 OR EARLIER ONLY

    Update Percona to the latest version using the instructions here: Percona Cluster 5.7 Tarball Minor Version Upgrade

  6. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    RabbitMQ will be upgraded as part of this process and all nodes need to be stopped. The official RabbitMQ upgrade guidance is here: https://www.rabbitmq.com/upgrade.html#full-stop-upgrades

    From that link:

    • While not strictly necessary, it is a good idea to decide ahead of time which 'disc' node will be the upgrader, stop that node last, and start it first.

    • The last 'disc' node to go down must be the first node to be brought online. Otherwise the started node will emit an error message and fail to start up. Unlike an ordinary cluster restart, upgrading nodes will not wait for the last 'disc' node to come back online.

    Use the following command to stop RabbitMQ on each server in order as specified above:

    $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop
  7. Place the downloaded tarball file into an empty directory called patch_directory and execute the following commands to start the patch installation:

    cd patch_directory;
    tar -xf moogsoft-enterprise-8.1.0.tgz;
    bash moogsoft-enterprise-install-8.1.0.sh;
  8. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted in the right order.

    The last node stopped should have been a 'disc' node (NOT a RAM node). This node must be brought up first using the following command:

    $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq start

    Then, restart the other nodes in the same way.

    Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected

  9. There are changes in controller.conf, system.conf, and security.conf in this release (depending on the version being upgraded from). Identify the config files that have changed between the previously installed version and this version. For example:

    diff -rq $OLD_DIST_PATH/config $MOOGSOFT_HOME/dist/8.1.0/config | grep -i 'differ'

    Update files in $MOOGSOFT_HOME/config with any changes introduced in the 8.1.0 version of these files (if appropriate/needed).

  10. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0 ONLY

    Apply schema changes by running below command:

    $MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "REPLACE INTO event_codes VALUES (11, 'API Alert Modification');"
  11. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.1 OR EARLIER ONLY

    Execute following command to enable auto-pruning of the process_logs table:

    ${MOOGSOFT_HOME}/bin/utils/moog_mysql_client -e "UPDATE system_config SET properties = JSON_SET(properties, '$.aged_logs_period', 90) WHERE config_type = 'Splitter';"

    Execute following command to apply a database configuration fix:

    cat $MOOGSOFT_HOME/etc/moog/schema_upgrades/patch_8_0_0_2/8002_moogdb_default_data.sql | $MOOGSOFT_HOME/bin/utils/moog_mysql_client
  12. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.4 OR EARLIER ONLY

    Execute the following command to ensure the 'moolet_informs' permission is present for all roles with the 'super_privileges' permission:

    ${MOOGSOFT_HOME}/bin/utils/moog_mysql_client -e 'update roles set permissions=JSON_ARRAY_INSERT(permissions, "$[0]", "moolet_informs") where permissions like "%super_privileges%" and permissions not like "%moolet_informs%";'
  13. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.3 OR EARLIER ONLY

    Reconfigure the EventsAnalyser to use the new default settings by running the command below.

    IMPORTANT: This will reset the UI-based Events Analyser (Alert Entropy feature) schedule to incremental runs only with a readage of one day and a keepage of three weeks. After the upgrade is complete and different settings are needed they can be changed via the UI, however, these new settings are the recommended values to minimise memory and performance impact on the rest of the deployment.

    $MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "UPDATE system_config SET properties='{\"enabled\":true,\"file_only_config\":false,\"use_preset\":true,\"preset\":\"d\",\"config\":[{\"name\":\"incremental_run\",\"incremental\":true,\"schedule\":{\"month\":\"*\",\"day_of_month\":\"*\",\"day_of_week\":\"*\",\"hour_of_day\":3,\"minute\":0},\"keepage\":\"3w\",\"readage\":\"1d\"}]}' WHERE name = 'EventsAnalyser' AND config_type = 'EventsAnalyser' AND JSON_EXTRACT(properties, '$.file_only_config') = false;"
  14. FOR ALL VERSIONS

    Refresh all stored procedures (provide the 'root' DB user password when prompted):

    $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 8.1.0 -u root
  15. FOR ALL VERSIONS

    Re-deploy Apache Tomcat and reconfigure Nginx. Run these commands in the same terminal as the one where the variables were saved previously. Then restart nginx:

    VERSION=8.1.0;
    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula
    cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/dist/${VERSION}/cots/nginx/ssl/;
    cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/dist/${VERSION}/cots/nginx/ssl/;
    sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/dist/${VERSION}/cots/nginx/config/conf.d/moog-ssl.conf;
    sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/dist/${VERSION}/cots/nginx/config/conf.d/moog-ssl.conf;
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx restart;
  16. Verify the remaining processes are working properly and update the crontab:

    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status;
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx status;
    $MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch status;
    (crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
    (crontab -l | sed -e 's/^\#\+\(.*keepalive.*\)/\1/') | crontab -
    
  17. Validate the patch:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    If there are any errors from the above validators, please contact Moogsoft Support.

  18. Re-start non-UI integrations or LAMs

  19. Re-install the latest Add-ons pack: https://docs.moogsoft.com/Enterprise.8.0.0/moogsoft-add-ons.html

  20. Restart moogfarmd:

    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd start;
  21. Restart any event feeds if they were stopped.

  22. Clear the browser cache and log in to the UI.