Skip to main content

Patch Moogsoft Enterprise RPM for v8.1.x

This topic describes how to patch an RPM-based distribution of Moogsoft Enterprise to v8.1.0.3 from v8.0.0.x or any previous 8.1.0.x versions.

Warning

  • The upgrade path from v8.0.x to v8.1.0 onwards (any pre v8.1 release going to any post v8.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters due to incompatibilities between the old (v3.7.4) and the new versions of Rabbit preventing a 'rolling upgrade'.

    All rabbit nodes will need to be stopped before their binaries are upgraded, and then they must be brought back up in the right order. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.

    The version of RabbitMQ being upgraded to in this release supports rolling upgrades, so future upgrades should be less disruptive

  1. Ensure the patch RPMs are available to each server being patched:

    • For internet-connected hosts, ensure there is a repo file under the /etc/yum.repos.d/ directory pointing to the 'speedy esr' yum repo.

      An example file is below:

      [moogsoft-aiops-80]
name=moogsoft-aiops-80
baseurl=https://<username>:<password>@speedy.moogsoft.com/repo/aiops/esr/
enabled=1
gpgcheck=0
sslverify=false

    • For offline-hosts:

      1. Download the two offline yum repository files (requires 'speedy' yum credentials):

        https://speedy.moogsoft.com/offline/aiops/2022-01-19-1642573563-MoogsoftBASE7_offline_repo.tar.gz
https://speedy.moogsoft.com/offline/aiops/2022-01-19-1642573563-MoogsoftESR_8.1.0.3_offline_repo.tar.gz

      2. Move the two offline installer bundle files to each server being upgraded as needed

      3. Create two directories to house the repositories. For example:

        sudo mkdir -p /media/localRPM/BASE/
sudo mkdir -p /media/localRPM/ESR/

      4. Extract the two Tarball files into separate directories. For example:

        tar xzf *-MoogsoftBASE7_offline_repo.tar.gz -C /media/localRPM/BASE/
tar xzf *-MoogsoftESR_8.1.0.3_offline_repo.tar.gz -C /media/localRPM/ESR/

      5. Back up the existing /etc/yum.repos.d directory. For example:

        mv /etc/yum.repos.d /etc/yum.repos.d-backup

      6. Create an empty /etc/yum.repos.d directory. For example:

        mkdir /etc/yum.repos.d

      7. Create a local.repo file in the /etc/yum.repos.d/ folder ready to contain the local repository details for example:

        [BASE]
name=MoogCentOS-$releasever - MoogRPM
baseurl=file:///media/localRPM/BASE/RHEL
gpgcheck=0
enabled=1

[ESR]
name=MoogCentOS-$releasever - MoogRPM
baseurl=file:///media/localRPM/ESR/RHEL
gpgcheck=0
enabled=1

      8. Clean the Yum cache:

        yum clean all

  2. Optional GPG key validation of the RPMs to validate the installation files.

    • For servers without internet access (if server has internet access go to the next step):

      1. Download the key from this site:

        https://keys.openpgp.org/vks/v1/by-fingerprint/887B323F485F122FE923214FD01B4B4B6F379E7E

      2. Copy the key to the server onto which the RPMs or tarball will be installed (it will be an .asc file)

      3. Import the key:

        gpg --import 887B323F485F122FE923214FD01B4B4B6F379E7E

    • For servers with internet access, run the following command:

      curl https://keys.openpgp.org/vks/v1/by-fingerprint/887B323F485F122FE923214FD01B4B4B6F379E7E | gpg --import

      1. Download the patch RPMs and matching '.sig' files from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:

        https://speedy.moogsoft.com/repo/aiops/esr/x86_64/

    • Copy the patch RPMs and matching '.sig' files into the same folder (the example below assumes /tmp will be used)

      Copy the following code into a bash terminal and run it to perform the validation:

      while read RPM
do
    echo "Current RPM: $RPM"
    gpg --verify ${RPM}.sig ${RPM} 2>&1
done < <(find /tmp -name '*.rpm');

    • Confirm that the command reports:

      Good signature from "Moogsoft <security@moogsoft.com>"

  4. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.5 OR EARLIER ONLY

    Update Percona to the latest version using the instructions here: Percona Cluster 5.7 RPM Minor Version Upgrade

  5. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    RabbitMQ will be upgraded as part of this process and all nodes need to be stopped. The official RabbitMQ upgrade guidance is here: https://www.rabbitmq.com/upgrade.html#full-stop-upgrades

    From that link:

    • While not strictly necessary, it is a good idea to decide ahead of time which 'disc' node will be the upgrader, stop that node last, and start it first.

    • The last 'disc' node to go down must be the first node to be brought online. Otherwise the started node will emit an error message and fail to start up. Unlike an ordinary cluster restart, upgrading nodes will not wait for the last 'disc' node to come back online.

    Use the following command to stop RabbitMQ on each server in order as specified above:

    service rabbitmq-server stop

  6. FOR ALL VERSIONS

    On each host where moogsoft packages are installed, install the patch RPMs:

    • For internet-connected hosts run the following command:

      yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-8.1.0.3/')

    • For offline hosts, run the following command in the directory containing the patch RPMs:

      yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-8.1.0.3*.rpm/')

  7. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    Upgrade Erlang to v23.3 (required for RabbitMQ v3.8.16):

    • Online RPM erlang upgrade command:

      yum upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v23.3/erlang-23.3-2.el7.x86_64.rpm

    • Offline RPM erlang upgrade command:

      yum upgrade erlang-23.3

  8. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted in the right order.

    The last node stopped should have been a 'disc' node (NOT a RAM node). This node must be brought up first using the following command:

    service rabbitmq-server start

    Then, restart the other nodes in the same way.

    Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected

    Note

    If RabbitMQ fails to start, check that Erlang has been upgraded to v23.3 as per the previous step

  9. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0 ONLY

    There is a change in “controller.conf”. Identify the config file that has changed between the previously installed version and this version, and copy any relevant customisations to the new controller.conf file. For example:

    1. FInd what the differences are:

      diff -u $MOOGSOFT_HOME/config/controller.conf $MOOGSOFT_HOME/config/controller.conf.rpmsave

    2. Merge the .rpmsave file with the new file by adding new properties/configuration where needed (from the new version of the file into the old version), so the structure matches the new version of the file.

    3. Ensure the correct version of the file is named controller.conf

  10. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0 ONLY

    Apply schema changes by running below command:

    $MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "REPLACE INTO event_codes VALUES (11, 'API Alert Modification');"

  11. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.1 OR EARLIER ONLY

    Execute following command to enable auto-pruning of process_logs table:

    ${MOOGSOFT_HOME}/bin/utils/moog_mysql_client -e "UPDATE system_config SET properties = JSON_SET(properties, '$.aged_logs_period', 90) WHERE config_type = 'Splitter';"

    Execute following command to apply a database configuration fix:

    cat $MOOGSOFT_HOME/etc/moog/schema_upgrades/patch_8_0_0_2/8002_moogdb_default_data.sql | $MOOGSOFT_HOME/bin/utils/moog_mysql_client

  12. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.4 OR EARLIER ONLY

    Execute the following command to ensure the 'moolet_informs' permission is present for all roles with the 'super_privileges' permission:

    ${MOOGSOFT_HOME}/bin/utils/moog_mysql_client -e 'update roles set permissions=JSON_ARRAY_INSERT(permissions, "$[0]", "moolet_informs") where permissions like "%super_privileges%" and permissions not like "%moolet_informs%";'

  13. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.6 OR EARLIER ONLY

    There are optional changes in “security.conf” and "system.conf". Copy any relevant customisations to the new versions of these files. For example (for the security.conf file):

    1. FInd what the differences are:

      diff -u $MOOGSOFT_HOME/config/security.conf $MOOGSOFT_HOME/config/security.conf.rpmsave

    2. Merge the .rpmsave file with the new file by adding new properties/configuration where needed (from the new version of the file into the old version), so the structure matches the new version of the file.

    3. Ensure the correct version of the file is named security.conf

  14. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.3 OR EARLIER ONLY

    Reconfigure the EventsAnalyser to use the new default settings by running the command below.

    IMPORTANT: This will reset the UI-based Events Analyser (Alert Entropy feature) schedule to incremental runs only with a readage of one day and a keepage of three weeks. After the upgrade is complete and different settings are needed they can be changed via the UI, however, these new settings are the recommended values to minimise memory and performance impact on the rest of the deployment.

    $MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "UPDATE system_config SET properties='{\"enabled\":true,\"file_only_config\":false,\"use_preset\":true,\"preset\":\"d\",\"config\":[{\"name\":\"incremental_run\",\"incremental\":true,\"schedule\":{\"month\":\"*\",\"day_of_month\":\"*\",\"day_of_week\":\"*\",\"hour_of_day\":3,\"minute\":0},\"keepage\":\"3w\",\"readage\":\"1d\"}]}' WHERE name = 'EventsAnalyser' AND config_type = 'EventsAnalyser' AND JSON_EXTRACT(properties, '$.file_only_config') = false;"

  15. FOR ALL VERSIONS

    Refresh all stored procedures (provide the 'root' DB user password when prompted):

    $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 8.1.0 -u root

  16. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    Update the NGiNX configuration file to support a more secure list of ciphers by running the following commands in order on any host with NGiNX installed:

    1. sed -i 's/ssl_ciphers.*/ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";/' /etc/nginx/conf.d/moog-ssl.conf
    2. service nginx restart

  17. FOR DEPLOYMENTS BEING UPGRADED FROM v8.0.0.7 OR EARLIER ONLY

    Upgrade apache-tomcat. IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the /etc/init.d/apache-tomcat service script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat service.

    1. Remove the existing Apache Tomcat

      rm -rf /etc/init.d/apache-tomcat
rm -rf $APPSERVER_HOME
rm -rf /usr/share/apache-tomcat

    2. Deploy the new version of Apache Tomcat:

      $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tf

    3. If you made any changes to the original Apache Tomcat service script, apply the same changes to the new version

  18. FOR ALL VERSIONS

    Rebuild the apache-tomcat webapps on the server where moogsoft-ui is installed:

    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w

  19. Validate the patch:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    If there are any errors from the validators, contact Moogsoft Support

  20. Re-install the latest 'Addons' pack: https://docs.moogsoft.com/Enterprise.8.0.0/moogsoft-add-ons.html

  21. FOR DEPLOYMENTS BEING UPGRADED FROM v8.1.0.0 OR EARLIER ONLY

    Restart elasticsearch:

    service elasticsearch restart

  22. Restart moogfarmd and any LAMs e.g:

    service moogfarmd start;
service restlamd start;

  23. Restart any event feeds if they were stopped.

  24. Clear the browser cache and log in to the UI.

In this section: