RPM - Upgrade UI components

Follow these steps to perform an RPM upgrade on the Moogsoft Enterprise UI components to v8.0.x from v7.1.x, v7.2.x, or 7.3.x:

Nginx
Apache Tomcat
UI integrations

These components should always reside on the same server.

Refer to Upgrade Moogsoft Enterprise or general information and upgrade instructions for other components and versions.

Stop services and processes

Run the following command as root to stop the default tomcat service on any servers with the moogsoft-ui package installed and where the Apache Tomcat service is running. Change the service name if you are not using the default.

service apache-tomcat stop

Run the following commands as root to query/stop the default LAM/integrations services on any servers with the moogsoft-integrations/moogsoft-integrations-ui package installed and where the LAMs/integrations are running.

Check for running LAM and integration processes and stop them using the relevant service scripts:

systemctl status | grep lamd
 
service <lam_service_name> stop

Run the following command to stop any remaining active LAM and integration processes:

kill -9 $(ps -ef | grep java | grep _lam | awk '{print $2}') 2>/dev/null

Delete the Elasticsearch indexes

Run this command on the moogsoft-search/Elasticsearch server to remove the old Elasticsearch indexes:

curl -XDELETE 'http://localhost:9200/alerts/' && curl -XDELETE 'http://localhost:9200/situations/'

If the command completes successfully, the following message is displayed:

{"acknowledged":true}{"acknowledged":true}

Modify the Elasticsearch repo

Note

You can skip this section if you are following the 'Offline RPM' upgrade process, as the Elasticsearch package is obtained from the local Yum repository instead.

Run the following command to modify the Elasticsearch Yum repository to point to v6 instead of v5:

sed -i 's/5.x/6.x/g' $(grep 'artifacts.elastic' /etc/yum.repos.d/* | awk -F: '{ print $1 }' | sort -u | head -1)

Upgrade Moogsoft Enterprise

To upgrade Moogsoft Enterprise, run the upgrade command below that corresponds to your chosen upgrade mechanism.

If you have already run this step on the current host as part of this upgrade (for single-host upgrade for example), you can skip this step.

If you are using a remote or offline Yum repository, run the following command on every host where a Moogsoft Enterprise RPM package is installed:
```
yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-8.0.0.1/')
```
If you are using downloaded RPM files on a host, run the following command from the location where the files are installed:
```
yum -y upgrade moogsoft-*8.0.0.1*.rpm
```

Merge the latest configuration file changes

Note

In Moogsoft Enterprise v7.3.x and 8.0.x, the Cookbooks, Tempus, and merge groups (default and custom) are imported into the database by default, enabling you to to access and configure them via the UI and API. The migration occurs once when Moogfarmd is restarted.

A file_only_config=true flag has been added to the 7.3.x and 8.0.x versions of moog_farmd.conf that you can use to prevent the migration from taking place. If this flag is missing or is set to false, Moogfarmd attempts to perform the import when it starts.

Note

If the file_only_config flag is set to true, UI-based Cookbooks will not run.

The following moolets are no longer supported in v8.0.x and should be removed from the moog_farmd.conf file as part of the upgrade:

Sigaliser Classic
Nexus
Speedbird
AlertRootCause

Version specific config file differences:

v7.1.x-v7.2.x
- $MOOGSOFT_HOME/config/system.conf
  - message_persistence is now enabled by default
- $MOOGSOFT_HOME/config/security.conf
  - The 'Google' realm has been deprecated and removed
- $MOOGSOFT_HOME/config/servlets.conf
  - The toolrunner servlet now optionally supports ssh key authentication as well as username/password-based authentication
- $MOOGSOFT_HOME/config/moog_farmd.conf
  - alert_workflows.conf moolet has been added
  - enrichment_workflows.conf moolet has been added
  - event_workflows.conf moolet has been added
  - situations_workflows.conf moolet has been added
v7.2.x-v7.3.x
- $MOOGSOFT_HOME/config/system.conf
  - New integration database property has been added: intdb_database_name
- $MOOGSOFT_HOME/config/moog_farmd.conf
  - The entire sig_resolution block containing merge groups, retention_period etc has been removed but is still supported as long as file_only_config is true
  - alert_root_cause.conf moolet has been removed and is no longer supported
  - nexus.conf moolet has been removed and is no longer supported
  - speedbird.conf moolet has been removed and is no longer supported
  - sigaliser.conf moolet has been removed and is no longer supported
  - cookbook.conf has been removed but is still supported as long as file_only_config is set to true
  - tempus.conf has been removed but is still supported as long as file_only_config is set to true
v7.3.x-v8.0.x
- $MOOGSOFT_HOME/config/system.conf
  - ElasticSearch now supports basic authentication
- $MOOGSOFT_HOME/config/security.conf
  - There is a new global_settings block which allows control of the CSRF protection feature
- $MOOGSOFT_HOME/config/servlets.conf
  - It is now possible to configure the toolrunner to run on a port other than 22 using toolrunnerport property
- $MOOGSOFT_HOME/config/moog_farmd.conf
  - alert_inform_workflows.conf moolet has been added
  - situation_inform_workflows.conf moolet has been added
  - The 'modules' block has been removed as it only contained Topology-related functionality and this has been deprecated in the v8.0.x release - the Topology feature works differently.

Manually merge and compare .rpmsave versions of files with the new versions of those files. Add any new properties to the older versions of the files. You can skip this step if you have already completed this step as part of this upgrade process on the current host.

The config and bot files from the previous version should not be copied on top of (replace) the new version of those files in 7.3.x, as they are not always forwards-compatible, and some config/bot lines need to be added for the new version to work.

To find files that have been changed, moved or deleted, run these commands:

find $MOOGSOFT_HOME -name '*.rpmsave'
find /etc/init.d/ -name '*.rpmsave'

For example, the following command displays the differences in the new version of the system.conf file:

diff -u $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.rpmsave

Follow this process to merge the file differences:

Rename the new versions of the files, without the .rpmsave extension, to end with .bak.
Merge the .rpmsave file with the new .bak file by adding new properties/configuration where needed (from the new version of the file into the old version), so the structure matches the new version of the file.
Rename the .rpmsave file to remove the .rpmsave extension.

Update JVM to use Java 11

Install the latest Java packages using the command below. It's possible that the yum upgrade of the moogsoft packages will have done this already and even possibly installed a newer version of Java than the one below. In this case, yum will report the packages are already installed and are the latest version. Continue with the subsequent steps even if this is the case.

yum -y install java-11-openjdk-11.0.7.10 \
               java-11-openjdk-devel-11.0.7.10 \
               java-11-openjdk-headless-11.0.7.10

On each server with a Moogsoft Enterprise RPM package installed, run the following command to replace the /usr/java/latest symlink so it points at the JDK11 JAVA_HOME directory:

source $MOOGSOFT_HOME/bin/utils/moog_init_functions.sh

If there are non-Moogsoft Enterprise packages on this server that do not support JDK11, you must update those applications to use a different JAVA_HOME symlink (not /usr/java/latest).

You can skip this step if you have already completed this step as part of this upgrade process on the current host.

To confirm this has worked, run the following command:

$JAVA_HOME/bin/java -version

It should return (as a minimum version):

openjdk version "11.0.7" 2020-04-14 LTS

You can also use the 'alternatives' command to point the system 'java' shortcut to the new version:

alternatives --config java

Remove references to the old MySQL connector

Change ownership of Apache Tomcat folders

Apache Tomcat is now run as the 'moogsoft' system user, which requires a change in ownership for the folders previously owned by Apache Tomcat.

Run the following commands to change ownership:

chown -R moogsoft:moogsoft /var/lib/moogsoft
chown -R moogsoft:moogsoft /var/run/apache-tomcat
chown -R moogsoft:moogsoft $MOOGSOFT_HOME/etc/saml

If SAML SSO is in use in the deployment, the IDP metadata file specified in $MOOGSOFT_HOME/config/security.conf needs to be readable by the 'moogsoft' system user. Use an appropriate chmod/chown command to ensure the readability and ownership is correct.

Upgrade Apache Tomcat and Nginx

Moogsoft Enterprise v8.0.x ships with Apache Tomcat version 9.0.22 and has changes to the nginx configuration files

Note

Moogsoft Enterprise v7.3 no longer runs Apache Tomcat as the 'tomcat' UNIX user. When you follow the instructions below, the new version of Apache Tomcat is deployed to run as the 'moogsoft' user instead. As more threads and processes are now used by the moogsoft UNIX system user, you may need to increase ulimits for this user.

Run the following commands in this section on the server with the moogsoft-ui RPM package installed on it.

Stop Apache Tomcat on any servers where it is running:

service apache-tomcat stop;
ps -ef | grep java | grep tomcat | awk '{print $2}' | xargs kill -9 2>/dev/null

Remove the existing Apache Tomcat:

rm -rf /etc/init.d/apache-tomcat;
rm -rf $APPSERVER_HOME
rm -rf /usr/share/apache-tomcat

Back up the Nginx configuration files and any certificates. Copy the files in the following location to another location before continuing: /etc/nginx/. This folder is based on the default Nginx installation location.
Deploy the new version of Apache Tomcat and Nginx using the command below. The script will ask for a hostname. This hostname or IP must be the same as what is used to access the instance via a browser:
```
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tfn
```
If you made any changes to the original Apache Tomcat service script, apply the same changes to the new version.
Update /etc/nginx/conf.d/moog-ssl.conf with the locations of any certificates used and then restart Nginx:
```
service nginx restart
```

Enable the enhanced Content Security Policy (optional)

Moogsoft has provided an optional enhanced Content Security Policy (CSP) as part of this release. CSP is a security standard introduced to prevent Cross Site Scripting (XSS) and other data injection attacks. For more information, see the Mozilla document on Content Security Policy.

The CSP is controlled by Nginx and is disabled by default. To enable it:

Edit the following file:
```
/etc/nginx/conf.d/moog-ui-headers.conf
```
Uncomment the line that starts with add_header Content-Security-Policy and save the file.
Restart Nginx:
```
service nginx reload
```

Note

If you enable the enhanced CSP you must follow the steps below to allow access to external domains. If you want to access the UI with the Safari web browser, you must follow the steps below to configure Moogsoft Enterprise for use with Safari.

Allow access to external domains

If you enable the enhanced CSP, the following features require additional configuration to allow access to external domains:

Situation Room plugins to external domains
Situation client tools to external URLs

To allow access to required external domains:

Edit the following file:
```
/etc/nginx/conf.d/moog-ui-headers.conf
```

Add a frame-src directive to the Content-Security-Policy header for the required domain. For example, run the following command to allow Google domains:

sed -i "s/add_header Content-Security-Policy\(.*\)\" always/add_header Content-Security-Policy\1; frame-src 'self' *.google.com\" always/" /etc/nginx/conf.d/moog-ui-headers.conf

Restart Nginx:
```
service nginx reload
```

Note

Moogsoft Enterprise allows access to Pendo and WalkMe domains by default.

Configure Moogsoft Enterprise for use with Safari

Due to a known issue in the Safari web browser, you must take additional steps if you've enabled the enhanced CSP and you want to access the UI with Safari:

Edit the following file:
```
/etc/nginx/conf.d/moog-ui-headers.conf
```
Add the following websocket URLs to the Content-Security-Policy section of the file. Substitute your hostname for <webhost>:
```
wss://<webhost>/moogpoller/ws
wss://<webhost>/integrations/ws/v1
```
You can update the configuration using a command similar to the following. Substitute your hostname for <webhost>:
```
sed -i.bak "s;connect-src 'self' app;connect-src 'self' wss://<webhost>/moogpoller/ws wss://<webhost>/integrations/ws/v1 app;g" /etc/nginx/conf.d/moog-ui-headers.conf
```
Restart Nginx:
```
service nginx reload
```

To continue with the upgrade, see RPM - Upgrade Core components.

In this section: