Patch Moogsoft Enterprise RPM

This topic describes how to patch an RPM-based distribution of Moogsoft Enterprise to v8.1.0 from any v8.0.0.x version.

Warning

  • The upgrade path from v8.0.x to v8.1.0 onwards (any pre v8.1 release going to any post v8.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters due to incompatibilities between the old (v3.7.4) and the new versions of Rabbit preventing a 'rolling upgrade'.

    All rabbit nodes will need to be stopped before their binaries are upgraded, and then they must be brought back up in the right order. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.

    The version of RabbitMQ being upgraded to in this release supports rolling upgrades, so future upgrades should be less disruptive

  1. Ensure the patch RPMs are available to each server being patched:

    • For internet-connected hosts, ensure there is a repo file under the /etc/yum.repos.d/ directory pointing to the 'speedy esr' yum repo.

      An example file is below:

      [moogsoft-aiops-80]
      name=moogsoft-aiops-80
      baseurl=https://<username>:<password>@speedy.moogsoft.com/repo/aiops/esr/
      enabled=1
      gpgcheck=0
      sslverify=false
    • For offline-hosts:

      1. Download the patch RPMs using the following commands (SPEEDY_USERNAME and SPEEDY_PASSWORD need to be set as environment variables before running this):

        PATCH_VERSION=8.1.0;
        INCIDENT_BUILD=119045;
        INTEGRATIONS_BUILD=119046;
        for PACKAGE in moogsoft-common-${PATCH_VERSION}-${INCIDENT_BUILD} moogsoft-integrations-${PATCH_VERSION}-${INTEGRATIONS_BUILD} moogsoft-mooms-${PATCH_VERSION}-${INCIDENT_BUILD} moogsoft-server-${PATCH_VERSION}-${INCIDENT_BUILD} moogsoft-utils-${PATCH_VERSION}-${INCIDENT_BUILD} moogsoft-db-${PATCH_VERSION}-${INCIDENT_BUILD} moogsoft-integrations-ui-${PATCH_VERSION}-${INTEGRATIONS_BUILD} moogsoft-search-${PATCH_VERSION}-${INCIDENT_BUILD} moogsoft-ui-${PATCH_VERSION}-${INCIDENT_BUILD};
        do
        	curl -L -O "https://${SPEEDY_USERNAME}:${SPEEDY_PASSWORD}@speedy.moogsoft.com/repo/aiops/esr/${PACKAGE}.rpm"
        done
      2. Copy the RPMs to all servers where the product is installed

  2. Optional GPG key validation of the RPMs to validate the installation files.

    • For servers without internet access (if server has internet access go to the next step):

      1. Download the key from this site:

        https://keys.openpgp.org/vks/v1/by-fingerprint/887B323F485F122FE923214FD01B4B4B6F379E7E
      2. Copy the key to the server onto which the RPMs or tarball will be installed (it will be an .asc file)

      3. Import the key:

        gpg --import 887B323F485F122FE923214FD01B4B4B6F379E7E
    • For servers with internet access, run the following command:

      curl https://keys.openpgp.org/vks/v1/by-fingerprint/887B323F485F122FE923214FD01B4B4B6F379E7E | gpg —import
      1. Download the patch RPMs and matching '.sig' files from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:

        https://speedy.moogsoft.com/repo/aiops/esr/x86_64/
    • Copy the patch RPMs and matching '.sig' files into the same folder (the example below assumes /tmp will be used)

      Copy the following code into a bash terminal and run it to perform the validation:

      while read RPM
      do
          echo "Current RPM: $RPM"
          gpg --verify ${RPM}.sig ${RPM} 2>&1
      done < <(find /tmp -name '*.rpm');
    • Confirm that the command reports:

      Good signature from "Moogsoft <security@moogsoft.com>"
  3. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.5 OR EARLIER ONLY

    Update Percona to the latest version using the instructions here: Percona Cluster 5.7 RPM Minor Version Upgrade

  4. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    RabbitMQ will be upgraded as part of this process and all nodes need to be stopped. The official RabbitMQ upgrade guidance is here: https://www.rabbitmq.com/upgrade.html#full-stop-upgrades

    From that link:

    • While not strictly necessary, it is a good idea to decide ahead of time which 'disc' node will be the upgrader, stop that node last, and start it first.

    • The last 'disc' node to go down must be the first node to be brought online. Otherwise the started node will emit an error message and fail to start up. Unlike an ordinary cluster restart, upgrading nodes will not wait for the last 'disc' node to come back online.

    Use the following command to stop RabbitMQ on each server in order as specified above:

    service rabbitmq-server stop
  5. On each host where moogsoft packages are installed, install the patch RPMs:

    • For internet-connected hosts run the following command:

      yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-8.1.0/')
    • For offline hosts, run the following command in the directory containing the patch RPMs:

      yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-8.1.0*.rpm/')
  6. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    Upgrade Erlang to v23.3 (required for RabbitMQ v3.8.16):

    • Online erlang upgrade command:

      yum upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v23.3/erlang-23.3-2.el7.x86_64.rpm
    • Offline RPM tarball upgrade command:

      yum upgrade erlang-23.3
  7. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted in the right order.

    The last node stopped should have been a 'disc' node (NOT a RAM node). This node must be brought up first using the following command:

    service rabbitmq-server start

    Then, restart the other nodes in the same way.

    Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected

    Note

    If RabbitMQ fails to start, check that Erlang has been upgraded to v23.3 as per the previous step

  8. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0 ONLY

    There is a change in “controller.conf”. Identify the config file that has changed between the previously installed version and this version, and copy any relevant customisations to the new controller.conf file. For example:

    1. FInd what the differences are:

      diff -u $MOOGSOFT_HOME/config/controller.conf $MOOGSOFT_HOME/config/controller.conf.rpmsave
    2. Merge the .rpmsave file with the new file by adding new properties/configuration where needed (from the new version of the file into the old version), so the structure matches the new version of the file.

    3. Ensure the correct version of the file is named controller.conf

  9. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0 ONLY

    Apply schema changes by running below command:

    $MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "REPLACE INTO event_codes VALUES (11, 'API Alert Modification');"
  10. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.1 OR EARLIER ONLY

    Execute following command to enable auto-pruning of process_logs table:

    ${MOOGSOFT_HOME}/bin/utils/moog_mysql_client -e "UPDATE system_config SET properties = JSON_SET(properties, '$.aged_logs_period', 90) WHERE config_type = 'Splitter';"

    Execute following command to apply a database configuration fix:

    cat $MOOGSOFT_HOME/etc/moog/schema_upgrades/patch_8_0_0_2/8002_moogdb_default_data.sql | $MOOGSOFT_HOME/bin/utils/moog_mysql_client
  11. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.4 OR EARLIER ONLY

    Execute the following command to ensure the 'moolet_informs' permission is present for all roles with the 'super_privileges' permission:

    ${MOOGSOFT_HOME}/bin/utils/moog_mysql_client -e 'update roles set permissions=JSON_ARRAY_INSERT(permissions, "$[0]", "moolet_informs") where permissions like "%super_privileges%" and permissions not like "%moolet_informs%";'
  12. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.6 OR EARLIER ONLY

    There are optional changes in “security.conf” and "system.conf". Copy any relevant customisations to the new versions of these files. For example (for the security.conf file):

    1. FInd what the differences are:

      diff -u $MOOGSOFT_HOME/config/security.conf $MOOGSOFT_HOME/config/security.conf.rpmsave
    2. Merge the .rpmsave file with the new file by adding new properties/configuration where needed (from the new version of the file into the old version), so the structure matches the new version of the file.

    3. Ensure the correct version of the file is named security.conf

  13. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.3 OR EARLIER ONLY

    Reconfigure the EventsAnalyser to use the new default settings by running the command below.

    IMPORTANT: This will reset the UI-based Events Analyser (Alert Entropy feature) schedule to incremental runs only with a readage of one day and a keepage of three weeks. After the upgrade is complete and different settings are needed they can be changed via the UI, however, these new settings are the recommended values to minimise memory and performance impact on the rest of the deployment.

    $MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "UPDATE system_config SET properties='{\"enabled\":true,\"file_only_config\":false,\"use_preset\":true,\"preset\":\"d\",\"config\":[{\"name\":\"incremental_run\",\"incremental\":true,\"schedule\":{\"month\":\"*\",\"day_of_month\":\"*\",\"day_of_week\":\"*\",\"hour_of_day\":3,\"minute\":0},\"keepage\":\"3w\",\"readage\":\"1d\"}]}' WHERE name = 'EventsAnalyser' AND config_type = 'EventsAnalyser' AND JSON_EXTRACT(properties, '$.file_only_config') = false;"
  14. FOR ALL VERSIONS

    Refresh all stored procedures (provide the 'root' DB user password when prompted):

    $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 8.1.0 -u root
  15. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    Update the NGiNX configuration file to support a more secure list of ciphers by running the following commands in order on any host with NGiNX installed:

    1. sed -i 's/ssl_ciphers.*/ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";/' /etc/nginx/conf.d/moog-ssl.conf
    2. service nginx restart
  16. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    Upgrade apache-tomcat

    1. Remove the existing Apache Tomcat

      rm -rf /etc/init.d/apache-tomcat
      rm -rf $APPSERVER_HOME
      rm -rf /usr/share/apache-tomcat
    2. Deploy the new version of Apache Tomcat:

      $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tf
    3. If you made any changes to the original Apache Tomcat service script, apply the same changes to the new version

  17. Rebuild the apache-tomcat webapps on the server where moogsoft-ui is installed:

    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  18. Validate the patch:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    If there are any errors from the validators, contact Moogsoft Support

  19. Re-install the latest 'Addons' pack: https://docs.moogsoft.com/Enterprise.8.0.0/moogsoft-add-ons.html

  20. Restart moogfarmd and any LAMs e.g:

    service moogfarmd start;
    service restlamd start;
  21. Restart any event feeds if they were stopped.

  22. Clear the browser cache and log in to the UI.