Patch Moogsoft Enterprise RPM for v8.2.x

This topic describes how to patch an RPM-based distribution of Moogsoft Enterprise to v8.2.0 from any previous v8.1.0.x release.

Important

The v8.2.0 release onwards uses Opensearch instead of Elasticsearch as the Search provider. It is possible to stick with Elasticsearch (simply ignore the steps to stop Elastic and don't run the moog_init_search.sh command at the end) but this configuration will not be supported if Elastic becomes out of support or loses API/REST compatibility in future.

Opensearch has authentication enabled by default. After it is installed, a new user 'moog_opensearchuser' is added to Opensearch and $MOOGSOFT_HOME/config/system.conf is updated to use this user to connect.

IMPORTANT: Opensearch REQUIRES the system variable vm.max_map_count to be at least 262144. If it is lower, the moog_init_search.sh script will attempt to set it to 262144 if the script is run as root. See: https://opensearch.org/docs/latest/opensearch/install/important-settings/

IMPORTANT: Opensearch REQUIRES some specific ulimits to be set for the user running the application. In most RPM deployments, it will run as the 'moogsoft' user. Example settings for /etc/security/limits.conf are:

moogsoft	soft nofile	 65536
moogsoft	hard nofile	 65536
moogsoft	soft nproc	65535
moogsoft	hard nproc	65535
  1. Ensure the patch RPMs are available to each server being patched:

    • For internet-connected hosts, ensure there is a repo file under the /etc/yum.repos.d/ directory pointing to the 'speedy esr' yum repo.

      An example file is below:

      [moogsoft-aiops-80]
      name=moogsoft-aiops-80
      baseurl=https://<username>:<password>@speedy.moogsoft.com/repo/aiops/esr/
      enabled=1
      gpgcheck=0
      sslverify=false
    • For offline-hosts:

      1. Download the two offline yum repository files (requires 'speedy' yum credentials):

        https://speedy.moogsoft.com/offline/aiops/2022-03-11-1646996777-MoogsoftBASE7_offline_repo.tar.gz
        https://speedy.moogsoft.com/offline/aiops/2022-03-11-1646996777-MoogsoftESR_8.2.0_offline_repo.tar.gz
      2. Move the two offline installer bundle files to each server being upgraded as needed

      3. Create two directories to house the repositories. For example:

        sudo mkdir -p /media/localRPM/BASE/
        sudo mkdir -p /media/localRPM/ESR/
      4. Extract the two Tarball files into separate directories. For example:

        tar xzf *-MoogsoftBASE7_offline_repo.tar.gz -C /media/localRPM/BASE/
        tar xzf *-MoogsoftESR_8.2.0_offline_repo.tar.gz -C /media/localRPM/ESR/
      5. Back up the existing /etc/yum.repos.d directory. For example:

        mv /etc/yum.repos.d /etc/yum.repos.d-backup
      6. Create an empty /etc/yum.repos.d directory. For example:

        mkdir /etc/yum.repos.d
      7. Create a local.repo file in the /etc/yum.repos.d/ folder ready to contain the local repository details for example:

        [BASE]
        name=MoogCentOS-$releasever - MoogRPM
        baseurl=file:///media/localRPM/BASE/RHEL
        gpgcheck=0
        enabled=1
        
        [ESR]
        name=MoogCentOS-$releasever - MoogRPM
        baseurl=file:///media/localRPM/ESR/RHEL
        gpgcheck=0
        enabled=1
      8. Clean the Yum cache:

        yum clean all 
  2. Optional GPG key validation of the RPMs to validate the installation files.

    • For servers without internet access (if server has internet access go to the next step):

      1. Download the key from this site:

        https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
      2. Copy the key to the server onto which the RPMs or tarball will be installed (it will be an .asc file)

      3. Import the key:

        gpg --import 0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
    • For servers with internet access, run the following command:

      curl https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 | gpg --import
      1. Download the patch RPMs and matching '.sig' files from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:

        https://speedy.moogsoft.com/repo/aiops/esr/x86_64/
    • Copy the patch RPMs and matching '.sig' files into the same folder (the example below assumes /tmp will be used)

      Copy the following code into a bash terminal and run it to perform the validation:

      while read RPM
      do
          echo "Current RPM: $RPM"
          gpg --verify ${RPM}.sig ${RPM} 2>&1
      done < <(find /tmp -name '*.rpm');
    • Confirm that the command reports:

      Good signature from "Moogsoft <security@moogsoft.com>"
  3. FOR ALL VERSIONS

    Update Percona to the latest version using the instructions here: Percona Cluster 5.7 RPM Minor Version Upgrade

  4. FOR ALL VERSIONS

    On each host where moogsoft packages are installed, install the patch RPMs:

    • For internet-connected hosts run the following command:

      yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-8.2.0/')
    • For offline hosts, run the following command in the directory containing the patch RPMs:

      yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-8.2.0*.rpm/')
  5. FOR ALL VERSIONS

    Upgrade Erlang to v23.3.4.11 (required for the RabbitMQ v3.9.13):

    • Online erlang upgrade command:

      yum upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v23.3.4.11/erlang-23.3.4.11-1.el7.x86_64.rpm
    • Offline RPM tarball upgrade command:

      yum upgrade erlang-23.3.4.11
  6. FOR ALL VERSIONS

    Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted.

    service rabbitmq-server restart
  7. FOR ALL VERSIONS

    The $MOOGSOFT_HOME/config/system.conf file has been changed in this release so ensure the right version of the file is in place there. There will be a system.conf.rpmsave version of the file present with the previously working configuration which can be renamed to just system.conf to just use the original configuration. The other option is to manually merge in any differences into the main system.conf file.

  8. FOR ALL VERSIONS

    Refresh all stored procedures (provide the 'root' DB user password when prompted):

    $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 8.2.0 -u root
  9. FOR ALL VERSIONS

    Upgrade apache-tomcat. IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the /etc/init.d/apache-tomcat service script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat service.

    1. Remove the existing Apache Tomcat

      rm -rf /etc/init.d/apache-tomcat
      rm -rf $APPSERVER_HOME
      rm -rf /usr/share/apache-tomcat
    2. Deploy the new version of Apache Tomcat:

      $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tf
    3. If you made any changes to the original Apache Tomcat service script, apply the same changes to the new version

  10. FOR ALL VERSIONS

    Rebuild the apache-tomcat webapps on the server where moogsoft-ui is installed:

    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  11. OPTIONAL

    To enable NginX rate limiting feature for login attempts, the following steps must be taken:

    1. Put the limit_req_zone line from $MOOGSOFT_HOME/etc/cots/nginx/nginx.conf into /etc/nginx/nginx.conf in the same place and uncomment it

    2. Put the two 'location' blocks containing 'limit_req' from $MOOGSOFT_HOME/etc/cots/nginx/moog-ssl.conf into /etc/nginx/conf.d/moog-ssl.conf into the same place and uncomment them

    3. Restart nginx using:

      service nginx restart
  12. FOR ALL VERSIONS

    Validate the patch:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    If there are any errors from the validators, contact Moogsoft Support

  13. FOR ALL VERSIONS

    The 'Addons' pack will need to be re-installed.

    The Addons v2.4.0 bundle is included with this release: $MOOGSOFT_HOME/contrib/Moogsoft-Addons-2.4.0.tar.gz

    Alternatively, the latest 'Addons' pack can always be found by following the instructions here: Install Moogsoft Enterprise Add-ons

  14. FOR ALL VERSIONS

    Elasticsearch is no longer the default Search provider for Moogsoft Enterprise. To install Opensearch and create a user to connect with in one step on the current server, follow the steps below.

    Stop ElasticSearch:

    service elasticsearch stop

    Ensure Elasticsearch has been stopped:

    kill -9 $(ps -ef | grep org.elasticsearch.bootstrap.Elasticsearch  | grep -v grep | awk '{print $2}') 2>/dev/null

    Install Opensearch, change the default admin password, and create a new account to connect with:

    $MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i

    This will deploy Opensearch, change the default admin account password (the new password will be printed out to STDOUT/the terminal, so be sure to make a note of it), and create a 'moog_opensearchuser' in Opensearch and $MOOGSOFT_HOME/config/system.conf is updated to use this user to connect.

    Warning

    If the $MOOGSOFT_HOME/config/system.conf file has all the comments removed, the above script will update the wrong fields with the username and password for Opensearch - to fix this, manually correct the fields in that configuration file before proceeding. A fix for this will be released in an upcoming patch.

  15. FOR ALL VERSIONS

    Restart moogfarmd, apache-tomcat, and any LAMs e.g:

    service apache-tomcat restart;
    service moogfarmd start;
    service restlamd start;

    Wait for MoogFarmD to finish loading by ensuring this line has appeared in the log: Farmd initialization complete

  16. FOR ALL VERSIONS

    If Opensearch is now being used instead of Elasticsearch, it will need to be re-indexed using the following command:

    $MOOGSOFT_HOME/bin/utils/moog_indexer -f -n
  17. FOR ALL VERSIONS

    Restart any event feeds if they were stopped.

  18. FOR ALL VERSIONS

    Clear the browser cache and log in to the UI.