Patch Moogsoft Enterprise RPM for v8.2.x

This topic describes how to patch an RPM-based distribution of Moogsoft Enterprise to v8.2.0 from any previous v8.1.0.x release.

Important

The v8.2.0 release onwards uses Opensearch instead of Elasticsearch as the Search provider. It is possible to stick with Elasticsearch (simply ignore the steps to stop Elastic and don't run the moog_init_search.sh command at the end) but this configuration will not be supported if Elastic becomes out of support or loses API/REST compatibility in future.

Opensearch has authentication enabled by default. After it is installed, a new user 'moog_opensearchuser' is added to Opensearch and $MOOGSOFT_HOME/config/system.conf is updated to use this user to connect.

IMPORTANT: Opensearch REQUIRES the system variable vm.max_map_count to be at least 262144. If it is lower, the moog_init_search.sh script will attempt to set it to 262144 if the script is run as root. See: https://opensearch.org/docs/latest/opensearch/install/important-settings/

IMPORTANT: Opensearch REQUIRES some specific ulimits to be set for the user running the application. In most RPM deployments, it will run as the 'moogsoft' user. Example settings for /etc/security/limits.conf are:

moogsoft	soft nofile	 65536
moogsoft	hard nofile	 65536
moogsoft	soft nproc	65535
moogsoft	hard nproc	65535

Ensure the patch RPMs are available to each server being patched:

For internet-connected hosts, ensure there is a repo file under the /etc/yum.repos.d/ directory pointing to the 'speedy esr' yum repo.
An example file is below:
```
[moogsoft-aiops-80]
name=moogsoft-aiops-80
baseurl=https://<username>:<password>@speedy.moogsoft.com/repo/aiops/esr/
enabled=1
gpgcheck=0
sslverify=false
```

For offline-hosts:

Download the two offline yum repository files (requires 'speedy' yum credentials):

https://speedy.moogsoft.com/offline/aiops/2022-03-11-1646996777-MoogsoftBASE7_offline_repo.tar.gz
https://speedy.moogsoft.com/offline/aiops/2022-03-11-1646996777-MoogsoftESR_8.2.0_offline_repo.tar.gz

Move the two offline installer bundle files to each server being upgraded as needed

Create two directories to house the repositories. For example:

sudo mkdir -p /media/localRPM/BASE/
sudo mkdir -p /media/localRPM/ESR/

Extract the two Tarball files into separate directories. For example:

tar xzf *-MoogsoftBASE7_offline_repo.tar.gz -C /media/localRPM/BASE/
tar xzf *-MoogsoftESR_8.2.0_offline_repo.tar.gz -C /media/localRPM/ESR/

Back up the existing /etc/yum.repos.d directory. For example:
```
mv /etc/yum.repos.d /etc/yum.repos.d-backup
```
Create an empty /etc/yum.repos.d directory. For example:
```
mkdir /etc/yum.repos.d
```

Create a local.repo file in the /etc/yum.repos.d/ folder ready to contain the local repository details for example:

[BASE]
name=MoogCentOS-$releasever - MoogRPM
baseurl=file:///media/localRPM/BASE/RHEL
gpgcheck=0
enabled=1

[ESR]
name=MoogCentOS-$releasever - MoogRPM
baseurl=file:///media/localRPM/ESR/RHEL
gpgcheck=0
enabled=1

Clean the Yum cache:
```
yum clean all 
```

Optional GPG key validation of the RPMs to validate the installation files.
- For servers without internet access (if server has internet access go to the next step):
  1. Download the key from this site:
    https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
  2. Copy the key to the server onto which the RPMs or tarball will be installed (it will be an .asc file)
  3. Import the key:
    gpg --import 0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
- For servers with internet access, run the following command:
```
curl https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 | gpg --import
```
  1. Download the patch RPMs and matching '.sig' files from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:
    https://speedy.moogsoft.com/repo/aiops/esr/x86_64/
- Copy the patch RPMs and matching '.sig' files into the same folder (the example below assumes /tmp will be used)
  Copy the following code into a bash terminal and run it to perform the validation:
```
while read RPM
do
    echo "Current RPM: $RPM"
    gpg --verify ${RPM}.sig ${RPM} 2>&1
done < <(find /tmp -name '*.rpm');
```
- Confirm that the command reports:
```
Good signature from "Moogsoft <security@moogsoft.com>"
```
Stop Apache Tomcat
Run the following command as root to stop the default tomcat service on any servers with the moogsoft-ui package installed and where the Apache Tomcat service is running. Change the service name if you are not using the default.
```
service apache-tomcat stop
```
Stop Moogfarmd
Run the following command as root to stop the default Moogfarmd service on any servers with the moogsoft-server package installed and where the Moogfarmd service is running. Change the service name if you are not using the default.
```
service moogfarmd stop
```
Ensure no more Moogfarmd processes are running with the following command. This will force-kill any remaining java processes which have failed to stop cleanly.
```
kill -9 $(ps -ef | grep java | grep farm | awk '{print $2}') 2>/dev/null
```
Stop LAMs and integrations
Run the following commands as root to query/stop the default LAM/integrations services on any servers with the moogsoft-integrations/moogsoft-integrations-ui package installed and where the LAMs/integrations are running.
Check for running LAM and integration processes and stop them using the relevant service scripts:
```
systemctl status | grep lamd
 
service <lam_service_name> stop
```
Run the following command to stop any remaining active LAM and integration processes:
```
kill -9 $(ps -ef | grep java | grep CLamMain | awk '{print $2}') 2>/dev/null
```
FOR ALL VERSIONS
Update Percona to the latest version using the instructions here: Percona Cluster 5.7 RPM Minor Version Upgrade
FOR ALL VERSIONS
On each host where moogsoft packages are installed, install the patch RPMs:
- For internet-connected hosts run the following command:
```
yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-8.2.0/')
```
- For offline hosts, run the following command in the directory containing the patch RPMs:
```
yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-8.2.0*.rpm/')
```
FOR ALL VERSIONS
Upgrade Erlang to v23.3.4.11 (required for the RabbitMQ v3.9.13):
- Online erlang upgrade command:
```
yum upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v23.3.4.11/erlang-23.3.4.11-1.el7.x86_64.rpm
```
- Offline RPM tarball upgrade command:
```
yum upgrade erlang-23.3.4.11
```
FOR ALL VERSIONS
Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted.
```
service rabbitmq-server restart
```
FOR ALL VERSIONS
The $MOOGSOFT_HOME/config/system.conf file has been changed in this release so ensure the right version of the file is in place there. There will be a system.conf.rpmsave version of the file present with the previously working configuration which can be renamed to just system.conf to just use the original configuration. The other option is to manually merge in any differences into the main system.conf file.
FOR ALL VERSIONS
Refresh all stored procedures (provide the 'root' DB user password when prompted):
```
$MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 8.2.0 -u root
```

FOR ALL VERSIONS

One of the database views needs to be patched:

$MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "CREATE OR REPLACE VIEW historic_feedback AS SELECT * FROM historic_moogdb.feedback"

FOR ALL VERSIONS
Upgrade apache-tomcat. IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the /etc/init.d/apache-tomcat service script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat service.
1. Remove the existing Apache Tomcat
```
rm -rf /etc/init.d/apache-tomcat
rm -rf $APPSERVER_HOME
rm -rf /usr/share/apache-tomcat
```
2. Deploy the new version of Apache Tomcat:
```
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tf
```
3. If you made any changes to the original Apache Tomcat service script, apply the same changes to the new version
FOR ALL VERSIONS
Rebuild the apache-tomcat webapps on the server where moogsoft-ui is installed:
```
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
```
OPTIONAL
To enable NginX rate limiting feature for login attempts, the following steps must be taken:
1. Put the limit_req_zone line from $MOOGSOFT_HOME/etc/cots/nginx/nginx.conf into /etc/nginx/nginx.conf in the same place and uncomment it
2. Put the two 'location' blocks containing 'limit_req' from $MOOGSOFT_HOME/etc/cots/nginx/moog-ssl.conf into /etc/nginx/conf.d/moog-ssl.conf into the same place and uncomment them
3. Restart nginx using:
```
service nginx restart
```

FOR ALL VERSIONS

Validate the patch:

$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

If there are any errors from the validators, contact Moogsoft Support

FOR ALL VERSIONS
The 'Addons' pack will need to be re-installed.
The Addons v2.4.0 bundle is included with this release: $MOOGSOFT_HOME/contrib/Moogsoft-Addons-2.4.0.tar.gz
Alternatively, the latest 'Addons' pack can always be found by following the instructions here: Install Moogsoft Enterprise Add-ons
FOR ALL VERSIONS
Elasticsearch is no longer the default Search provider for Moogsoft Enterprise. To install a single-node Opensearch deployment and create a user to connect with in one step on the current server, follow the steps below. For a cluster, follow the instructions here instead: Opensearch Clustering Guide
Stop ElasticSearch:
```
service elasticsearch stop
```
Ensure Elasticsearch has been stopped:
```
kill -9 $(ps -ef | grep org.elasticsearch.bootstrap.Elasticsearch  | grep -v grep | awk '{print $2}') 2>/dev/null
```
Install Opensearch, change the default admin password, and create a new account to connect with:
```
$MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i
```
This will deploy Opensearch, change the default admin account password (the new password will be printed out to STDOUT/the terminal, so be sure to make a note of it), and create a 'moog_opensearchuser' in Opensearch and $MOOGSOFT_HOME/config/system.conf is updated to use this user to connect.
Warning
If the $MOOGSOFT_HOME/config/system.conf file has all the comments removed, the above script will update the wrong fields with the username and password for Opensearch - to fix this, manually correct the fields in that configuration file before proceeding. A fix for this will be released in an upcoming patch.
If any customisations had been made to the ElasticSearch jvm.options file, the same changes can be made to the /etc/opensearch/jvm.options.d/moog.options file. A restart of OpenSearch is needed for such changes to take effect.
Warnings such as the following when OpenSearch starts can be safely ignored:
```
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/opt/moogsoft/install/dist/8.2.0/cots/opensearch/plugins/opensearch-anomaly-detection/protostuff-runtime-1.7.4.jar) to field java.lang.Throwable.cause
WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
```
FOR ALL VERSIONS
Restart moogfarmd, apache-tomcat, and any LAMs e.g:
```
service apache-tomcat restart;
service moogfarmd start;
service restlamd start;
```
Wait for MoogFarmD to finish loading by ensuring this line has appeared in the log: Farmd initialization complete
FOR ALL VERSIONS
If Opensearch is now being used instead of Elasticsearch, it will need to be re-indexed using the following command:
```
$MOOGSOFT_HOME/bin/utils/moog_indexer -f -n
```
FOR ALL VERSIONS
Restart any event feeds if they were stopped.
FOR ALL VERSIONS
Clear the browser cache and log in to the UI.

In this section:

Patch Moogsoft Enterprise RPM for v8.2.x

Important

Warning

Search results