Patch Moogsoft Onprem RPM for v9.1.x.x

This topic describes how to patch an RPM-based distribution of Moogsoft Onprem to v9.1.0 from v9.0.0 or 9.0.0.x or 9.0.1.x

Important

Enabling the "latency performance" RHEL profile is strongly recommended. This profile allows RabbitMQ to operate much more efficiently so that throughput is increased and smoothed out.

For more information on performance profiles, see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/getting-started-with-tuned_monitoring-and-managing-system-status-and-performance

Enable the profile by running the following command as root:

tuned-adm profile latency-performance

This setting will survive machine restarts and only needs to be set once.

Warning

For deployments upgrading from v9.0.0 or v9.0.0.1
The upgrade path from v9.0.0/v9.0.0.1 to v9.0.1 onwards (any pre v9.0.0.2 release going to any post v9.0.1 release) requires a 'full stop' upgrade of any running RabbitMQ clusters. All rabbit nodes will need to be stopped before their binaries are upgraded. This means there will be a window of time during the upgrade where RabbitMQ cannot be used to store events. Further upgrade details are in the relevant step below.
For deployments upgrading from v9.0.0.2
The RabbitMQ upgrade as part of this process requires all feature flags to be enabled.
The following command must be run on all RabbitMQ server nodes before the following steps are performed:
```
rabbitmqctl enable_feature_flag all
```
For users with UI Integrations containing a Rule with a name containing the word 'password'
A known issue will affect the UI integrations page after upgrade to 9.1.0. A workaround is to change the name of this rule so it doesn't equal or contain the word 'password'. Alternatively, a hotfix is available. Please contact Moogsoft Support in this case.

Ensure the patch RPMs are available to each server being patched:

For internet-connected hosts, ensure there is a repo file under the /etc/yum.repos.d/ directory pointing to the 'speedy esr' yum repo.
An example file is below:
```
[moogsoft-aiops-90]
name=moogsoft-aiops-90
baseurl=https://<username>:<password>@speedy.moogsoft.com/v9/repo/
enabled=1
gpgcheck=0
sslverify=false
```

For offline-hosts:

Download the two offline yum repository files (requires 'speedy' yum credentials):

https://speedy.moogsoft.com/v9/offline/2024-04-08-1712581945-MoogsoftBASE8_offline_repo.tar.gz
https://speedy.moogsoft.com/v9/offline/2024-04-08-1712581945-MoogsoftESR_9.1.0_offline_repo.tar.gz

Move the two offline installer bundle files to each server being upgraded as needed

Create two directories to house the repositories. For example:

sudo mkdir -p /media/localRPM/BASE/
sudo mkdir -p /media/localRPM/ESR/

Extract the two Tarball files into separate directories. For example:

tar xzf *-MoogsoftBASE8_offline_repo.tar.gz -C /media/localRPM/BASE/
tar xzf *-MoogsoftESR_9.1.0_offline_repo.tar.gz -C /media/localRPM/ESR/

Back up the existing /etc/yum.repos.d directory. For example:
```
mv /etc/yum.repos.d /etc/yum.repos.d-backup
```
Create an empty /etc/yum.repos.d directory. For example:
```
mkdir /etc/yum.repos.d
```

Create a local.repo file in the /etc/yum.repos.d/ folder ready to contain the local repository details for example:

[BASE]
name=MoogRHE-$releasever - MoogRPM
baseurl=file:///media/localRPM/BASE/RHEL
gpgcheck=0
enabled=1

[ESR]
name=MoogRHEL-$releasever - MoogRPM
baseurl=file:///media/localRPM/ESR/RHEL
gpgcheck=0
enabled=1

Clean the Yum cache:
```
yum clean all 
```

Stop Apache Tomcat
Run the following command as root to stop the default tomcat service on any servers with the moogsoft-ui package installed and where the Apache Tomcat service is running. Change the service name if you are not using the default.
```
service apache-tomcat stop
```
Stop Moogfarmd
Run the following command as root to stop the default Moogfarmd service on any servers with the moogsoft-server package installed and where the Moogfarmd service is running. Change the service name if you are not using the default.
```
service moogfarmd stop
```
Ensure no more Moogfarmd processes are running with the following command. This will force-kill any remaining java processes which have failed to stop cleanly.
```
kill -9 $(ps -ef | grep java | grep farm | awk '{print $2}') 2>/dev/null
```
Stop LAMs and integrations
Run the following commands as root to query/stop the default LAM/integrations services on any servers with the moogsoft-integrations/moogsoft-integrations-ui package installed and where the LAMs/integrations are running.
Check for running LAM and integration processes and stop them using the relevant service scripts:
```
systemctl status | grep lamd
 
service <lam_service_name> stop
```
Run the following command to stop any remaining active LAM and integration processes:
```
kill -9 $(ps -ef | grep java | grep CLamMain | awk '{print $2}') 2>/dev/null
```
FOR ALL VERSIONS
Update Percona to the latest version using the instructions here: Percona Cluster 8.0 RPM Minor Version Upgrade
FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 OR EARLIER ONLY
RabbitMQ will be upgraded as part of this process and all nodes need to be stopped.
Use the following command to stop RabbitMQ on each server:
```
service rabbitmq-server stop
```
FOR ALL VERSIONS
Important
Ensure the RabbitMQ feature flags have been enabled before proceeding. See the start of this document for the required command.
Upgrade Erlang (required for the new version of RabbitMQ):
- Online RPM erlang upgrade command:
```
yum upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v26.2.1/erlang-26.2.1-1.el8.x86_64.rpm
```
- Offline RPM erlang upgrade command:
```
yum upgrade erlang-26.1.1
```

FOR ALL VERSIONS

Update the NginX yum repo file so it contains the mainline repo:

rm -f /etc/yum.repos.d/nginx.repo;
cat <<END > /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/rhel/8/\$basearch/
gpgcheck=0
enabled=1
module_hotfixes=1
[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/rhel/8/\$basearch/
gpgcheck=0
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=1
END

FOR ALL VERSIONS
On each host where moogsoft packages are installed, install the patch RPMs:
- For internet-connected hosts run the following command:
```
yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-9.1.0/')
```
- For offline hosts, run the following command in the directory containing the patch RPMs:
```
yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-9.1.0*.rpm/')
```
- For ALL RPM-based deployments, ensure the Java JDK folder permissions are correct by running the following command as root (or a user with sudo permissions):
```
chmod -R 755 /usr/java /usr/lib/jvm
```
FOR ALL VERSIONS
Upgrade Nginx:
```
yum -y upgrade nginx
```
RabbitMQ upgrade
- FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 OR EARLIER ONLY
  Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted. On each node perform the following steps.
  The mnesia directory needs to be deleted using the following command:
```
rm -rf /var/lib/rabbitmq/mnesia/*
```
  The RabbitMQ node now needs to be re-initialised:
```
$MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -pz <YOUR_ZONE_NAME>
```
  Then, re-initialize the other nodes in the same way using the same commands.
  Now create a cluster of all the RabbitMQ nodes:https://www.rabbitmq.com/clustering.html
  Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected
FOR ALL VERSIONS
Now the RabbitMQ binaries have been upgraded, the RabbitMQ nodes need to be restarted:
```
service rabbitmq-server restart
```
Perform some health checks (for example as documented here: https://www.rabbitmq.com/monitoring.html#health-checks) to ensure the cluster is operating as expected
FOR ALL VERSIONS
Refresh all stored procedures (provide the 'ermintrude' DB user password when prompted):
```
$MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 9.1.0 -u ermintrude
```
FOR DEPLOYMENTS BEING UPGRADED FROM v9.0.0 OR v9.0.0.1 OR v9.0.0.2 OR EARLIER ONLY
New security enhancements in v9.0.1 require enabling the Sign Response As Required option on the IDP side, if configurable. We kindly request your SAML team to do this for the <PROD/UAT> environment before the upgrade. If you are unsure after communicating with your SAML team whether this option applies to your setup, please contact Moogsoft support.
If a new IDP is generated after this change, SAML team should provide its metadata file to the team taking care of the upgrade. During the upgrade, the existing IDP file will be replaced with the one provided. In all cases, the SP metadata file will be regenerated and should be shared with SAML team. They may need to import the new SP metadata or configure the relevant fields with the information supplied in the file to complete the trust configuration.
FOR ALL VERSIONS
Upgrade opensearch. This step will remove the existing copy of OpenSearch and upgrade it to the latest one (single node deployment):
```
service opensearch stop;
$MOOGSOFT_HOME/bin/utils/moog_init_search.sh -i
```
Important
This step will overwrite the opensearch_user password in $MOOGSOFT_HOME/config/system.conf
If OpenSearch needs to be clustered, it can be done after all the nodes are upgraded fully.
FOR ALL VERSIONS
Upgrade apache-tomcat on the server where moogsoft-ui is installed: IMPORTANT: If the Xmx value for apache-tomcat has been changed from the default in the /etc/init.d/apache-tomcat service script, ensure the customised value is replaced after the upgrade, then restart the apache-tomcat service.
1. Remove the existing Apache Tomcat
```
rm -rf /etc/init.d/apache-tomcat
rm -rf ${APPSERVER_HOME}
rm -rf /usr/share/apache-tomcat
```
2. Deploy the new version of Apache Tomcat:
```
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -twf
```
3. If you made any changes to the original Apache Tomcat service script, apply the same changes to the new version

Validate the patch:

$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

If there are any errors from the validators, contact Moogsoft Support

Re-install the latest 'Addons' pack Install Moogsoft Add-ons
FOR ALL VERSIONS
The OpenSearch Cluster now needs to be recreated: Opensearch Clustering Guide - RPM

Restart moogfarmd and any LAMs e.g:

service moogfarmd restart;
service restlamd start;

If an OpenSearch cluster is needed, create the cluster before performing this step.
Wait for MoogFarmd to start - about two minutes, then trigger a re-index of OpenSearch:
```
$MOOGSOFT_HOME/bin/utils/moog_indexer -f -n
```
Restart any event feeds if they were stopped.
Clear the browser cache and log in to the UI.

In this section:

Patch Moogsoft Onprem RPM for v9.1.x.x

Important

Warning

Important

Important

Search results