Patch Moogsoft Onprem RPM for v9.0.1.x

This topic describes how to patch an RPM-based distribution of Moogsoft Onprem to v9.1.0 from v9.0.0 or 9.0.0.x

Warning

For deployments upgrading from v9.0.0 or v9.0.0.1
The upgrade path from v9.0.0/v9.0.0.1 to v9.0.1 onwards (any pre v9.0.0.2 release going to any post v9.0.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters. All rabbit nodes will need to be stopped before their binaries are upgraded. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.
For deployments upgrading from v9.0.0.2
The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.
The following command must be run on all RabbitMQ server nodes before the following steps are performed:
```
rabbitmqctl enable_feature_flag all
```

Ensure the patch RPMs are available to each server being patched:

For internet-connected hosts, ensure there is a repo file under the /etc/yum.repos.d/ directory pointing to the 'speedy esr' yum repo.
An example file is below:
```
[moogsoft-aiops-90]
name=moogsoft-aiops-90
baseurl=https://<username>:<password>@speedy.moogsoft.com/v9/repo/
enabled=1
gpgcheck=0
sslverify=false
```

For offline-hosts:

Download the two offline yum repository files (requires 'speedy' yum credentials):

https://speedy.moogsoft.com/v9/offline/2023-07-11-1689088698-MoogsoftBASE8_offline_repo.tar.gz
https://speedy.moogsoft.com/v9/offline/2023-07-11-1689088698-MoogsoftESR_9.1.0_offline_repo.tar.gz

Move the two offline installer bundle files to each server being upgraded as needed

Create two directories to house the repositories. For example:

sudo mkdir -p /media/localRPM/BASE/
sudo mkdir -p /media/localRPM/ESR/

Extract the two Tarball files into separate directories. For example:

tar xzf *-MoogsoftBASE8_offline_repo.tar.gz -C /media/localRPM/BASE/
tar xzf *-MoogsoftESR_9.1.0_offline_repo.tar.gz -C /media/localRPM/ESR/

Back up the existing /etc/yum.repos.d directory. For example:
```
mv /etc/yum.repos.d /etc/yum.repos.d-backup
```
Create an empty /etc/yum.repos.d directory. For example:
```
mkdir /etc/yum.repos.d
```

Create a local.repo file in the /etc/yum.repos.d/ folder ready to contain the local repository details for example:

[BASE]
name=MoogCentOS-$releasever - MoogRPM
baseurl=file:///media/localRPM/BASE/RHEL
gpgcheck=0
enabled=1

[ESR]
name=MoogCentOS-$releasever - MoogRPM
baseurl=file:///media/localRPM/ESR/RHEL
gpgcheck=0
enabled=1

Clean the Yum cache:
```
yum clean all 
```

Optional GPG key validation of the RPMs to validate the installation files.
- For servers without internet access (if server has internet access go to the next step):
  1. Download the key from this site:
    https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
  2. Copy the key to the server onto which the RPMs or tarball will be installed (it will be an .asc file)
  3. Import the key:
    gpg --import 0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2
- For servers with internet access, run the following command:
```
curl https://keys.openpgp.org/vks/v1/by-fingerprint/0A8FD9AB6F1693A1967B3B8CB919E617EC6946C2 | gpg --import
```
  1. Download the patch RPMs and matching '.sig' files from the 'speedy' yum repository using a browser, providing speedy credentials when asked by the browser:
    https://speedy.moogsoft.com/v9/repo/x86_64/
- Copy the patch RPMs and matching '.sig' files into the same folder (the example below assumes /tmp will be used)
  Copy the following code into a bash terminal and run it to perform the validation:
```
while read RPM
do
    echo "Current RPM: $RPM"
    gpg --verify ${RPM}.sig ${RPM} 2>&1
done < <(find /tmp -name '*.rpm');
```
- Confirm that the command reports:
```
Good signature from "Moogsoft <security@moogsoft.com>"
```
Stop Apache Tomcat
Run the following command as root to stop the default tomcat service on any servers with the moogsoft-ui package installed and where the Apache Tomcat service is running. Change the service name if you are not using the default.
```
service apache-tomcat stop
```
Stop Moogfarmd
Run the following command as root to stop the default Moogfarmd service on any servers with the moogsoft-server package installed and where the Moogfarmd service is running. Change the service name if you are not using the default.
```
service moogfarmd stop
```
Ensure no more Moogfarmd processes are running with the following command. This will force-kill any remaining java processes which have failed to stop cleanly.
```
kill -9 $(ps -ef | grep java | grep farm | awk '{print $2}') 2>/dev/null
```
Stop LAMs and integrations
Run the following commands as root to query/stop the default LAM/integrations services on any servers with the moogsoft-integrations/moogsoft-integrations-ui package installed and where the LAMs/integrations are running.
Check for running LAM and integration processes and stop them using the relevant service scripts:
```
systemctl status | grep lamd
 
service <lam_service_name> stop
```
Run the following command to stop any remaining active LAM and integration processes:
```
kill -9 $(ps -ef | grep java | grep CLamMain | awk '{print $2}') 2>/dev/null
```
FOR ALL VERSIONS
Update Percona to the latest version using the instructions here: Percona Cluster 8.0 RPM Minor Version Upgrade
FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 OR EARLIER ONLY
RabbitMQ will be upgraded as part of this process and all nodes need to be stopped.
Use the following command to stop RabbitMQ on each server:
```
service rabbitmq-server stop
```
FOR ALL VERSIONS
On each host where moogsoft packages are installed, install the patch RPMs:
- For internet-connected hosts run the following command:
```
yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-9.0.1/')
```
- For offline hosts, run the following command in the directory containing the patch RPMs:
```
yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-9.0.1*.rpm/')
```
For ALL RPM-based deployments, ensure the Java JDK folder permissions are correct by running the following command as root (or a user with sudo permissions):
```
chmod -R 755 /usr/java /usr/lib/jvm
```
FOR ALL VERSIONS
In the latest release a number of the configuration files are different out of the box. This means after the RPM upgrade, the following configuration files will be replaced with 'rpmsave' versions of those same files.
- $MOOGSOFT_HOME/config/system.conf
- $MOOGSOFT_HOME/config/moog_farmd.conf
Any customisations made to the pre-upgrade versions of these files (*.rpmsave) should be copied into the non-rpmsave versions of the files. Alternatively, the rpmsave versions of the files can be renamed to replace the new file versions. For example:
```
cp $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/901cleansystem.conf.bak;
mv $MOOGSOFT_HOME/config/system.conf.rpmsave $MOOGSOFT_HOME/config/system.conf
```
FOR ALL VERSIONS
Important
Ensure the RabbitMQ feature flags have been enabled before proceeding. See the start of this document for the required command.
Upgrade Erlang (required for the new version of RabbitMQ):
- Online RPM erlang upgrade command:
```
yum upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v26.0.1/erlang-26.0.1-1.el8.x86_64.rpm
```
- Offline RPM erlang upgrade command:
```
yum upgrade erlang-26.0.1
```
RabbitMQ upgrade
- FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 OR EARLIER ONLY
  Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted. On each node perform the following steps.
  The mnesia directory needs to be deleted using the following command:
```
rm -rf /var/lib/rabbitmq/mnesia/*
```
  The RabbitMQ node now needs to be re-initialised:
```
$MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -pz <YOUR_ZONE_NAME>
```
  Then, re-initialize the other nodes in the same way using the same commands.
  Now create a cluster of all the RabbitMQ nodes:https://www.rabbitmq.com/clustering.html
  Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected
- FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0.2 ONLY
  Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted:
```
service rabbitmq-server restart
```
  Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected
FOR ALL VERSIONS
Refresh all stored procedures (provide the 'ermintrude' DB user password when prompted):
```
$MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 9.0.1 -u ermintrude
```
FOR ALL VERSIONS
New security enhancements in v9.0.1 require enabling the Sign Response As Required option on the IDP side, if configurable. We kindly request your SAML team to do this for the <PROD/UAT> environment before the upgrade. If you are unsure after communicating with your SAML team whether this option applies to your setup, please contact Moogsoft support.
If a new IDP is generated after this change, SAML team should provide its metadata file to the team taking care of the upgrade. During the upgrade, the existing IDP file will be replaced with the one provided. In all cases, the SP metadata file will be regenerated and should be shared with SAML team. They may need to import the new SP metadata or configure the relevant fields with the information supplied in the file to complete the trust configuration.
FOR ALL VERSIONS
Upgrade opensearch. This step will remove the existing copy of OpenSearch and upgrade it to the latest one (single node deployment):
```
service opensearch stop;
$MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i
```
Important
This step will overwrite the opensearch_user password in $MOOGSOFT_HOME/config/system.conf
If OpenSearch needs to be clustered, it can be done after all the nodes are upgraded fully.
FOR ALL VERSIONS
Upgrade apache-tomcat on the server where moogsoft-ui is installed: IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the /etc/init.d/apache-tomcat service script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat service.
1. Remove the existing Apache Tomcat
```
rm -rf /etc/init.d/apache-tomcat
rm -rf ${APPSERVER_HOME}
rm -rf /usr/share/apache-tomcat
```
2. Deploy the new version of Apache Tomcat:
```
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -twf
```
3. If you made any changes to the original Apache Tomcat service script, apply the same changes to the new version

FOR ALL VERSIONS

Update the NginX configuration file to ensure the UI Integrations tab is accessible:

sed -i 's;location ^~ /integrations {;location ^~ /integrations/ {;' /etc/nginx/conf.d/moog-ssl.conf

Then reload NginX:

service nginx reload

Validate the patch:

$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

If there are any errors from the validators, contact Moogsoft Support

Re-install the latest 'Addons' pack Install Moogsoft Add-ons

Restart moogfarmd and any LAMs e.g:

service moogfarmd restart;
service restlamd start;

If an OpenSearch cluster is needed, create the cluster before performing this step.
Wait for MoogFarmd to start - about two minutes, then trigger a re-index of OpenSearch:
```
$MOOGSOFT_HOME/bin/utils/moog_indexer -f -n
```
Restart any event feeds if they were stopped.
Clear the browser cache and log in to the UI.

In this section:

Patch Moogsoft Onprem RPM for v9.0.1.x

Warning

Important

Important

Search results