Skip to main content

Patch Moogsoft Onprem RPM for v9.0.1.x

This topic describes how to patch an RPM-based distribution of Moogsoft Onprem to v9.0.1 from v9.0.0 or 9.0.0.x

Warning

  • For deployments upgrading from v9.0.0 or v9.0.0.1

    The upgrade path from v9.0.0/v9.0.0.1 to v9.0.1 onwards (any pre v9.0.0.2 release going to any post v9.0.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters. All rabbit nodes will need to be stopped before their binaries are upgraded. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.

  • For deployments upgrading from v9.0.0.2

    The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.

    The following command must be run on all RabbitMQ server nodes before the following steps are performed:

    rabbitmqctl enable_feature_flag all
  1. Ensure the patch RPMs are available to each server being patched:

    • For internet-connected hosts, ensure there is a repo file under the /etc/yum.repos.d/ directory pointing to the 'speedy esr' yum repo.

      An example file is below:

      [moogsoft-aiops-90]
      name=moogsoft-aiops-90
      baseurl=https://<username>:<password>@speedy.moogsoft.com/v9/repo/
      enabled=1
      gpgcheck=0
      sslverify=false
    • For offline-hosts:

      1. Download the two offline yum repository files (requires 'speedy' yum credentials):

        https://speedy.moogsoft.com/v9/offline/2023-07-11-1689088698-MoogsoftBASE8_offline_repo.tar.gz
        https://speedy.moogsoft.com/v9/offline/2023-07-11-1689088698-MoogsoftESR_9.0.1_offline_repo.tar.gz
      2. Move the two offline installer bundle files to each server being upgraded as needed

      3. Create two directories to house the repositories. For example:

        sudo mkdir -p /media/localRPM/BASE/
        sudo mkdir -p /media/localRPM/ESR/
      4. Extract the two Tarball files into separate directories. For example:

        tar xzf *-MoogsoftBASE8_offline_repo.tar.gz -C /media/localRPM/BASE/
        tar xzf *-MoogsoftESR_9.0.1_offline_repo.tar.gz -C /media/localRPM/ESR/
      5. Back up the existing /etc/yum.repos.d directory. For example:

        mv /etc/yum.repos.d /etc/yum.repos.d-backup
      6. Create an empty /etc/yum.repos.d directory. For example:

        mkdir /etc/yum.repos.d
      7. Create a local.repo file in the /etc/yum.repos.d/ folder ready to contain the local repository details for example:

        [BASE]
        name=MoogCentOS-$releasever - MoogRPM
        baseurl=file:///media/localRPM/BASE/RHEL
        gpgcheck=0
        enabled=1
        
        [ESR]
        name=MoogCentOS-$releasever - MoogRPM
        baseurl=file:///media/localRPM/ESR/RHEL
        gpgcheck=0
        enabled=1
      8. Clean the Yum cache:

        yum clean all 
  2. Optional GPG key validation of the RPMs to validate the installation files.

    • For servers without internet access (if server has internet access go to the next step):

      1. Download the key from this site:

        https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 
      2. Copy the key to the server onto which the RPMs or tarball will be installed (it will be an .asc file)

      3. Import the key:

        gpg --import 0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 
    • For servers with internet access, run the following command:

      curl https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 | gpg --import
      1. Download the patch RPMs and matching '.sig' files from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:

        https://speedy.moogsoft.com/v9/repo/x86_64/
    • Copy the patch RPMs and matching '.sig' files into the same folder (the example below assumes /tmp will be used)

      Copy the following code into a bash terminal and run it to perform the validation:

      while read RPM
      do
          echo "Current RPM: $RPM"
          gpg --verify ${RPM}.sig ${RPM} 2>&1
      done < <(find /tmp -name '*.rpm');
    • Confirm that the command reports:

      Good signature from "Moogsoft <security@moogsoft.com>"
  3. FOR ALL VERSIONS

    Update Percona to the latest version using the instructions here: Percona Cluster 8.0 RPM Minor Version Upgrade

  4. FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 OR EARLIER ONLY

    RabbitMQ will be upgraded as part of this process and all nodes need to be stopped.

    Use the following command to stop RabbitMQ on each server:

    service rabbitmq-server stop
  5. FOR ALL VERSIONS

    On each host where moogsoft packages are installed, install the patch RPMs:

    • For internet-connected hosts run the following command:

      yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-9.0.1/')
    • For offline hosts, run the following command in the directory containing the patch RPMs:

      yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-9.0.1*.rpm/')
  6. FOR ALL VERSIONS

    In the latest release a number of the configuration files are different out of the box. This means after the RPM upgrade, the following configuration files will be replaced with 'rpmsave' versions of those same files.

    • $MOOGSOFT_HOME/config/system.conf

    • $MOOGSOFT_HOME/config/moog_farmd.conf

    Any customisations made to the pre-upgrade versions of these files (*.rpmsave) should be copied into the non-rpmsave versions of the files. Alternatively, the rpmsave versions of the files can be renamed to replace the new file versions. For example:

    cp $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/901cleansystem.conf.bak;
    mv $MOOGSOFT_HOME/config/system.conf.rpmsave $MOOGSOFT_HOME/config/system.conf
  7. FOR ALL VERSIONS

    Important

    Ensure the RabbitMQ feature flags have been enabled before proceeding. See the start of this document for the required command.

    Upgrade Erlang (required for the new version of RabbitMQ):

    • Online RPM erlang upgrade command:

      yum upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v26.0.1/erlang-26.0.1-1.el8.x86_64.rpm
    • Offline RPM erlang upgrade command:

      yum upgrade erlang-26.0.1
  8. RabbitMQ upgrade

    • FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 OR EARLIER ONLY

      Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted. On each node perform the following steps.

      The mnesia directory needs to be deleted using the following command:

      rm -rf /var/lib/rabbitmq/mnesia/*

      The RabbitMQ node now needs to be re-initialised:

      $MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -pz <YOUR_ZONE_NAME>

      Then, re-initialize the other nodes in the same way using the same commands.

      Now create a cluster of all the RabbitMQ nodes:https://www.rabbitmq.com/clustering.html

      Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected

    • FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0.2 ONLY

      Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted:

      service rabbitmq-server restart

      Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected

  9. FOR ALL VERSIONS

    Refresh all stored procedures (provide the 'ermintrude' DB user password when prompted):

    $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 9.0.1 -u ermintrude
  10. FOR ALL VERSIONS

    New security enhancements in v9.0.1 require enabling the Sign Authentication requests sent over POST and Redirect bindings or Sign Response As Required option on the IDP side, if configurable. We kindly request your SAML team to do this for the <PROD/UAT> environment before the upgrade. If you are unsure after communicating with your SAML team whether this option applies to your setup, please contact Moogsoft support.

    If a new IDP is generated after this change, SAML team should provide its metadata file to the team taking care of the upgrade. During the upgrade, the existing IDP file will be replaced with the one provided. In all cases, the SP metadata file will be regenerated and should be shared with SAML team. They may need to import the new SP metadata or configure the relevant fields with the information supplied in the file to complete the trust configuration.

  11. FOR ALL VERSIONS

    Upgrade opensearch. This step will remove the existing copy of OpenSearch and upgrade it to the latest one (single node deployment):

    service opensearch stop;
    $MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i

    Important

    This step will overwrite the opensearch_user password in $MOOGSOFT_HOME/config/system.conf

    If OpenSearch needs to be clustered, it can be done after all the nodes are upgraded fully.

  12. FOR ALL VERSIONS

    Upgrade apache-tomcat on the server where moogsoft-ui is installed: IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the /etc/init.d/apache-tomcat service script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat service.

    1. Remove the existing Apache Tomcat

      rm -rf /etc/init.d/apache-tomcat
      rm -rf ${APPSERVER_HOME}
      rm -rf /usr/share/apache-tomcat
    2. Deploy the new version of Apache Tomcat:

      $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -twf
    3. If you made any changes to the original Apache Tomcat service script, apply the same changes to the new version

  13. FOR ALL VERSIONS

    Update the NginX configuration file to ensure the UI Integrations tab is accessible:

    sed -i 's;location ^~ /integrations {;location ^~ /integrations/ {;' /etc/nginx/conf.d/moog-ssl.conf

    Then reload NginX:

    service nginx reload
  14. Validate the patch:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    If there are any errors from the validators, contact Moogsoft Support

  15. Re-install the latest 'Addons' pack Install Moogsoft Add-ons

  16. Restart moogfarmd and any LAMs e.g:

    service moogfarmd restart;
    service restlamd start;
  17. If an OpenSearch cluster is needed, create the cluster before performing this step.

    Wait for MoogFarmd to start - about two minutes, then trigger a re-index of OpenSearch:

    $MOOGSOFT_HOME/bin/utils/moog_indexer -f -n
  18. Restart any event feeds if they were stopped.

  19. Clear the browser cache and log in to the UI.