Upgrade RPM v7.0.x to RPM v7.2.x

This topic describes the upgrade procedure of RPM deployments from Moogsoft AIOps v7.0.x to Moogsoft AIOps v7.2.0. Refer to Upgrade - RPM Deployments for general information and links to upgrades for other versions.

Before You Begin

Before you begin the upgrade process, complete the following prerequisite steps as appropriate.

Prepare for an Offline Upgrade

To prepare for an offline upgrade, where the Moogsoft AIOps packages reside in a 'local' Yum repository, follow steps 1-8 in section "Prepare the Local yum Repositories" in the Pre-Install Moogsoft AIOps - Offline RPMs guide. Then follow this guide to perform the upgrade. Take into account any step-specific notes regarding offline upgrades.

Note UI Integrations Configuration

If you are using any UI integrations, make a note of their connection and configuration details before you begin the upgrade. You will need to delete and reconfigure them in the post-upgrade steps, due to UI changes in the new version.

Stop Moogfarmd

Run the following command to stop the default Moogfarmd service on any servers with the moogsoft-server package installed and where the Moogfarmd service is running. Change the service name if you are not using the default.

Run this command as root:

service moogfarmd stop

Ensure no more Moogfarmd processes are running with this command (this will force-kill any remaining java processes that have failed to stop cleanly):

kill -9 $(ps -ef | grep java | grep farm | awk '{print $2}') 2>/dev/null

Stop Apache Tomcat

Run the following command to stop the default Tomcat service on any servers with the moogsoft-ui package installed and where the Apache Tomcat service is running. Change the service name if you are not using the default.

Run this command as root:

service apache-tomcat stop

Stop LAMs and Integrations

Run the following command to query/stop the default LAM/integrations services on any servers with the moogsoft-integrations/moogsoft-integrations-ui package installed and where the LAMs/integrations are running.

Run these commands as root.

Check for running LAM and integration processes and stop them using the relevant service scripts:

systemctl status | grep lamd
 
service <lam_service_name> stop

Ensure no more Moogfarmd processes are running with this command (this will force-kill any remaining java processes that have failed to stop cleanly):

kill -9 $(ps -ef | grep java | grep _lam | awk '{print $2}') 2>/dev/null

Disable the Events Analyser

Run the following command to stop the Events Analyser process on any servers with the moogsoft-server package installed and where the Events Analyser is configured to run.

Run the following command as root to comment out the relevant lines in crontab:

(crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -

Run the following command to stop any active Events Analyser processes:

ps -ef | grep java | egrep 'events_analyser' | awk '{print $2}' | xargs kill 2>/dev/null

Back Up the Existing System

To back up the existing system:

  1. Back up $MOOGSOFT_HOME on all servers with a moogsoft RPM package installed.

  2. Take a snapshot (for VMs).

  3. Back up MySQL on the server with the moogsoft-db RPM package installed

Upgrade Moogsoft AIOps

Choose the upgrade command that corresponds to your installation:

  • Using a remote or local/offline Yum repository. Run the following command to perform the upgrade on every host where a Moogsoft RPM package has been installed:

    yum -y upgrade $(rpm -qa --qf '%{NAME}\n' | grep moogsoft | sed 's/$/-7.2.0.2/')
  • Any host where you have downloaded the RPMs to the current folder. Run the following command to perform the upgrade:

    yum -y upgrade moogsoft-*7.2.0*.rpm

Merge the Latest Configuration File Changes

Moogfarmd Configuration Change

There is a significant change to the configuration of Moogfarmd in this release. In clean installations of this release, the Moogfarmd configuration file now references Moolet configuration files located under $MOOGSOFT_HOME/config/moolets/.

However, this release is backwards-compatible with the old Moogfarmd configuration file format, so, you do not need to split up your Moogfarmd configuration as part of this upgrade, but it is recommended.

Regular Merge Process

Manually merge and compare .rpmsave versions of files with the new versions of those files. Add any new properties to the older versions of the files.

To find files that have been changed, moved, or deleted, run these commands:

find $MOOGSOFT_HOME -name '*.rpmsave'
find /etc/init.d/ -name '*.rpmsave'

An example command to see what differences are present in the $MOOGSOFT_HOME/config/system.conf file is shown below:

diff -u $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.rpmsave

An example of the process to merge the differences is shown below:

  1. Rename the new versions of the files, without the .rpmsave extension, to end with .bak.

  2. Merge the .rpmsave file with the new .bak file by adding new properties/config where needed so the structure matches the new version of the file.

  3. Rename the .rpmsave file to delete the .rpmsave extension.

Update JVM to use Java 11

On each server with a moogsoft RPM package installed, you must run the following command to replace the /usr/java/latest symlink so it points at the new JDK11 JAVA_HOME directory:

source $MOOGSOFT_HOME/bin/utils/moog_init_functions.sh

If there are non-AIOps packages on this server which do not support JDK11, those applications must be updated to use a different JAVA_HOME symlink (not /usr/java/latest).

Remove References to the old MySQL Connector

The MySQL connector has been upgraded in this release.

The original connector might have been used by the ‘External Database’ module in the current deployment (configured in $MOOGSOFT_HOME/config/moog_external_db_details.conf). If this is configured on the current deployment, it should be updated to reference the new ‘mariadb' connector here: $MOOGSOFT_HOME/lib/cots/mariadb-java-client-2.4.0.jar

Upgrade the Moogsoft AIOps Database Schema

To upgrade the Moogsoft AIOps database, you need to provide the Auto Upgrader utility with the credentials of a database user with super privileges. For single-host installs where MySQL was installed as part of the Moogsoft AIOps deployment, you can use the default 'root' user.

  1. Run the following command (on the server where the moogsoft-db package is installed) after substituting the <MySQL-SuperUsername> argument:

    bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 7.2.0 -u <MySQL-SuperUsername>
  2. Enter the password for that user. You can provide the password to the utility with the -p flag but this is not recommended in non-test deployments for security reasons.

Drop Deprecated Historic Database Tables

Run the following commands to drop two tables from the historic database that are no longer used. If they are not dropped at this stage, the moog_db_validator utility at the end of this process will report their presence as a delta.

Run these commands on the server with the moogsoft-db package installed:

bash $MOOGSOFT_HOME/bin/utils/moog_mysql_client -i -e "drop table room_post_sigs";
bash $MOOGSOFT_HOME/bin/utils/moog_mysql_client -i -e "drop table room_posts";

Update the RabbitMQ Configuration

Update the RabbitMQ configuration to support autoheal and other processes that improve stability in HA environments. To do this, run the following commands on each server the moogsoft-mooms / RabbitMQ package is installed on. Replace <VHOST> with the name of your RabbitMQ vhost:

cp -f $MOOGSOFT_HOME/etc/cots/rabbitmq/rabbitmq.config /etc/rabbitmq/
bash $MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -z <VHOST> -p

Upgrade Apache Tomcat and the Webapps

Moogsoft AIOps v7.2.0 ships with Apache Tomcat version 9.0.13.0.

Run the commands in this section on the server with the moogsoft-ui RPM package installed on it.

  1. Remove the existing Apache Tomcat:

    rm -rf /etc/init.d/apache-tomcat;
    rm -rf $APPSERVER_HOME
    rm -rf /usr/share/apache-tomcat
  2. Deploy the new version of Apache Tomcat:

    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tfw

    If the webapps do not extract and the connection to MySQL is SSL-enabled, it is likely that the hostname of the self-signed certificates does not match the ones set. See Encrypt Database Communications for steps to resolve the issue.

  3. Restart Nginx:

    service nginx restart

Update and Reconfigure Integrations

This upgrade process replaces the Moobot file $MOOGSOFT_HOME/bots/moobots/RemedyIntegration.js. If you have customised this file, back it up before continuing.

Run the commands in this section on the server with the moogsoft-integrations-ui package installed on it.

To update the integrations:

  1. The LAMs/Integrations no longer use the 'state' files under $MOOGSOFT_HOME/config. Remove these files by running the following commands:

    rm -f $MOOGSOFT_HOME/config/*_lam.state;
    rm -f $MOOGSOFT_HOME/config/{null.state, TestStateManager.state}
  2. Run the following command to extract the old and new Integrations:

    bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN;

Configure the ServiceNow MID Server to use Java 8

If you are using a ServiceNow MID server installed on the same host as Moogsoft AIOps, you must configure it to point to Java 8. The MID server requires Java 8 (update 152 or later). It will not work with Java 9+. To do this:

  1. Install the latest version of Java 8. See the ServiceNow MID server system requirements for more information.

  2. Stop the MID server by running the appropriate command. For example:

    kill -9 $(ps -ef | grep mid_server | grep -v grep | awk '{print $2}')
  3. Configure the wrapper.java.command property to point to the Java 8 binary in the following file:

    /usr/local/servicenow/moog_mid_server/agent/conf/wrapper-override.conf.

    For example:

    wrapper.java.command=/usr/java/jre1.8.0_171-amd64/bin/java

In the next step you will reconfigure the ServiceNow integration which will restart the MID server.

Reconfigure your UI Integrations

Delete and reconfigure all of your UI integrations, using the configuration details you noted prior to the upgrade. This is required due to UI changes in the new version.

Confirm All Services are Running and Start Moogfarmd

To confirm all services are running:

  1. Run the following commands to check all required services are running and start Moogfarmd:

    service apache-tomcat status
    service moogfarmd start
    service nginx status
    service elasticsearch status
  2. Run the following command to re-enable the Events Analyser cronjobs:

    (crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
  3. Run the following command to restart any previously running Integrations:

    service moogstartupd restart

Add the Keepalive Cronjob (Optional)

A new cronjob is installed for clean installs of 7.2.0 which provides the ability for certain services to be restarted automatically should they fail. For RPM deployments this is usually handled by chkconfig or mysql itself for example, but can be useful in certain cases. See Configure Services to Restart for more information.

Command to enable if required:

(crontab -l; echo -e "*\t*\t*\t*\t* $MOOGSOFT_HOME/bin/utils/process_keepalive.sh 2>&1") | crontab -

Verify the Upgrade

You can verify the upgrade manually or using automatic utilities.

Verify the Upgrade Manually

Perform the following basic steps to ensure that the upgrade to Moogsoft AIOps v7.2.0 was successful:

  1. Check that the UI login page displays "Version 7.2.0" at the top.

  2. Check that the UI "Support Information" window correctly indicates the current version as "7.2.0" and shows the correct schema upgrade history.

Verify the Upgrade Using Automatic Utilities

Run the following automatic utilities to ensure that the upgrade to Moogsoft AIOps v7.2.0 was successful:

  1. Confirm that all Moogsoft AIOps files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
  2. Confirm that all Apache Tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
    • If there are some web app differences, you can resolve them using moog_init_ui.sh -w which extracts the web apps with the right files.

  3. Confirm that the database schema has been upgraded successfully using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    If errors such as the following are seen:

    Differences found in 'moogdb' tables:
    57a58
    >  key 'filter_id' ('filter_id'),
    194a196
    >  key 'enrichment_static_mappings_ibfk_1' ('eid'),
    1196a1199
    >  key 'sig_id' ('sig_id'),
    1325a1329
    >  key 'filter_id' ('filter_id'),

    Then run the following commands to fix the index issues:

    mysql -u root -e "use moogdb; alter table alert_filters_access drop key \'filter_id\'"
    mysql -u root -e "use moogdb; alter table situation_filters_access drop key \'filter_id\'"
    mysql -u root -e "use moogdb; alter table enrichment_static_mappings drop key \'enrichment_static_mappings_ibfk_1\'"
    mysql -u root -e "use moogdb; alter table sig_stats_cache drop key \'sig_id\'"
  4. Confirm that all the steps have completed successfully.

    Some schema differences may be valid (e.g. custom_info related). If there are more substantial differences, you should investigate further to verify that all the prerequisite upgrade scripts have been applied in the right order:

    mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
    mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have any issues, refer to Troubleshooting.