Moogsoft Docs

Upgrade RPM v7.0.x to RPM v7.1.x

This topic describes the upgrade procedure of RPM deployments from Moogsoft AIOps v7.0.x to Moogsoft AIOps v7.1.0. Refer to Upgrade Moogsoft AIOps - RPM Deployments for general information and links to upgrades for other versions.

The following steps outline this workflow:

Before You Begin

To prepare for an offline upgrade, where the Moogsoft AIOps packages reside in a 'local' Yum repository, follow steps 1-8 in section "Prepare the Local Yum Repositories" in the Pre-Install Moogsoft AIOps - Offline RPMs guide. Then follow  this guide to perform the upgrade. Take into account any step-specific notes regarding offline upgrades.

Back Up the Existing System

To back up the existing system:

  1. Back up $MOOGSOFT_HOME .
  2. Take a snapshot (for VMs).
  3. Back up MySQL.

Stop Services and Processes

To stop the services and processes:

  1. Stop Moogfarmd:

    service moogfarmd stop

    If Moogfarmd was started in the UI, stop it in the UI.

  2. Ensure that no Moogfarmd processes are running with this command:

    kill -9 $(ps -ef | grep java | grep farm | awk '{print $2}') 2>/dev/null
  3. Stop Apache Tomcat:

    service apache-tomcat stop
  4. Check for running LAM and integration processes and stop them using the relevant service scripts:

    systemctl status | grep lamd
    
    service <lam_service_name> stop
  5. Run the following command to stop any remaining active LAM and integration processes:

    kill -9 $(ps -ef | grep java | grep _lam | awk '{print $2}') 2>/dev/null
  6. Disable the Events Analyser from running during the upgrade process:

    a. Run the following command to comment out the relevant lines in crontab:

    kill -9 $(ps -ef | grep java | grep _lam | awk '{print $2}') 2>/dev/null

    b. Run the following command to stop any active Events Analyser processes:

    ps -ef | grep java | egrep 'events_analyser' | awk '{print $2}' | xargs kill 2>/dev/null

Upgrade Moogsoft AIOps

Choose the upgrade command that corresponds to your installation:

  • Single host from a remote or local Yum repository. Run the following commands to perform the upgrade:

    yum -y upgrade moogsoft-integrations-7.1.0 moogsoft-integrations-ui-7.1.0 moogsoft-common-7.1.0 moogsoft-server-7.1.0 moogsoft-ui-7.1.0 moogsoft-db-7.1.0 moogsoft-search-7.1.0 moogsoft-mooms-7.1.0 moogsoft-utils-7.1.0
  • Single host from the same machine where the RPMs have been downloaded locally and are in the current folder. Run the following command to perform the upgrade:

    yum -y upgrade moogsoft-*7.1.0*.rpm
  • Distributed installations. Select the relevant update commands to run depending on which package(s) are installed on the current host. For example:

    1. HOST1 running Integrations, Integrations UI, common (required), Moogfarmd server, and Moogsoft AIOps UI packages:

      yum -y upgrade moogsoft-integrations-7.1.0 moogsoft-integrations-ui-7.1.0 moogsoft-common-7.1.0 moogsoft-server-7.1.0 moogsoft-ui-7.1.0
    2. HOST2 running common (required), utilities, database, and search packages:

      yum -y upgrade moogsoft-common-7.1.0 moogsoft-utils-7.1.0 moogsoft-db-7.1.0 moogsoft-search-7.1.0
    3. HOST3 running common (required), Message Bus, and utilities packages:

      yum -y upgrade moogsoft-common-7.1.0 moogsoft-mooms-7.1.0 moogsoft-utils-7.1.0

Merge the Latest Config File Changes

Manually merge and compare .rpmsave versions of files with the new versions of those files. Add any new properties to the older versions of the files.

Moogfarmd configuration change in 7.1.0

There is a significant change to the configuration of Moogfarmd in this release. In clean installations of 7.1.0, the moog_farmd.conf file now references Moolet configuration files located under $MOOGSOFT_HOME/config/moolets/ .

However, 7.1.0 is backwards-compatible with the old Moogfarmd configuration file format so, you do not need to split up your Moogfarmd configuration as part of this upgrade, but it is recommended.

Regular merge process

To find files which have been changed, moved, or deleted, run these commands:

find $MOOGSOFT_HOME -name '*.rpmsave'
find /etc/init.d/ -name '*.rpmsave'

An example command to see what differences are present in the $MOOGSOFT_HOME/config/system.conf file is shown below:

diff -u $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.rpmsave

An example of the process to merge the differences is shown below:

  1. Rename the new versions of the files, without the .rpmsave extension, to end with .bak .
  2. Merge the .rpmsave file with the new .bak file by adding new properties/config where needed so the structure matches the new version of the file.
  3. Rename the .rpmsave file to delete the .rpmsave extension.

Upgrade the Moogsoft AIOps Database Schema

To upgrade the Moogsoft AIOps database, you need to provide the Auto Upgrader utility with the credentials of a database user with super privileges. For single-host installs where MySQL was installed as part of the Moogsoft AIOps deployment, you can use the default 'root' user.

  1. Run the following command after substituting the <MySQL-SuperUsername> argument:

    bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 7.1.0 -u <MySQL-SuperUsername>
  2. Enter the password for that user. You can provide the password to the utility with the -p flag but this is not recommended in non-test deployments for security reasons.

Update the RabbitMQ configuration to support autoheal etc

Run the following commands on each server the moogsoft-mooms / RabbitMQ package is installed on to improve stability in HA environments (replace <VHOST> below with your existing RabbitMQ vhost):

cp -f $MOOGSOFT_HOME/etc/cots/rabbitmq/rabbitmq.config /etc/rabbitmq/
bash $MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -z <VHOST> -p

Upgrade the Apache Tomcat Webapps and Configuration Files

The $APPSERVER_HOME/conf/server.xml and $APPSERVER_HOME/conf/context.xml files have been changed in this release. If the existing deployment has customized versions of those files, please make a backup before continuing.

To upgrade using the script provided, run the following command. This is recommended for standard or single-host installations.

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -wf

If Apache Tomcat startup or shutdown failures are reported when running the above command, such as "Web apps are not rebuilding, please check catalina.out", forcibly shut down Apache Tomcat using kill -9 on the Apache Tomcat java PID, and then repeat the above step.

Alternatively, to upgrade manually, follow these steps:

  1. Stop Apache Tomcat:

    service apache-tomcat stop
  2. Check the Apache Tomcat java process is no longer running:

    service apache-tomcat status
  3. Delete the existing extracted web apps:

    rm -rf $APPSERVER_HOME/webapps/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}
  4. Copy in the new web app WAR files and replace the old ones:

    cp -f $MOOGSOFT_HOME/lib/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}.war $APPSERVER_HOME/webapps/
  5. Copy in the new configuration files:

    cat $MOOGSOFT_HOME/etc/cots/tomcat/context.xml | sed "s;/usr/share/moogsoft/lib;$MOOGSOFT_HOME/lib;g" > $APPSERVER_HOME/conf/context.xml
    cp -f $MOOGSOFT_HOME/etc/cots/tomcat/apache-tomcat-ssl.xml $APPSERVER_HOME/conf/server.xml
  6. Restart Apache Tomcat:

    service apache-tomcat start

Update the Integrations

This upgrade process replaces the $MOOGSOFT_HOME/bots/moobots/RemedyIntegration.js Moobot file. Back up this file if you have customised it, before continuing.

To update the integrations:

  1. The LAMs/integrations no longer use the 'state' files under $MOOGSOFT_HOME/config . You can remove these files with the following commands:

    rm -f $MOOGSOFT_HOME/config/*_lam.state;
    rm -f $MOOGSOFT_HOME/config/{null.state, TestStateManager.state}
  2. Run the following command to extract the old and new integrations:

    bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN;
  3. After you finish the upgrade, you must remove and re-add any existing integrations if they need to be edited in the future (via the UI). If they do not need to be edited post-upgrade, they will work as they did pre-upgrade.

Confirm All Services are Running and Start Moogfarmd

To confirm all services are running:

  1. Run the following commands to check all required services are running and start Moogfarmd:

    service apache-tomcat status
    service moogfarmd start
    service nginx status
    service elasticsearch status
  2. Run the following command to re-enable the Events Analyser cronjobs:

    (crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
  3. Run the following command to restart any previously running integrations:

    service moogstartupd restart

OPTIONAL: Add the new 'Keepalive' cronjob

A new cronjob is installed for clean installs of 7.1.0 which provides the ability for certain services to be restarted automatically should they fail. For RPM deployments this is usually handled by chkconfig or mysql itself for example, but can be useful in certain cases.

See Configure Services to Restart for more information.

Command to enable if required:

(crontab -l; echo -e "*\t*\t*\t*\t* $MOOGSOFT_HOME/bin/utils/process_keepalive.sh 2>&1") | crontab -

Verify the Upgrade

You can verify the upgrade manually or using automatic utilities.

Verify Manually

Perform the following basic steps to ensure that the upgrade to Moogsoft AIOps v7.1.0 was successful:

  1. Check that the UI login page displays "Version 7.1.0" at the top.
  2. Check that the UI "Support Information" window correctly indicates the current version as "7.1.0" and shows the correct schema upgrade history.

Verify using Automatic Utilities

Run the following automatic utilities to ensure that the upgrade to Moogsoft AIOps v7.1.0 was successful:

  1. Confirm that all Moogsoft AIOps files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
  2. Confirm that all Apache Tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
  3. Confirm that the database schema has been upgraded successfully using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh
  4. Confirm that all steps are successful. If there are some web app differences, you can resolve them using moog_init_ui.sh -w which extracts the web apps with the right files.
  5. Some schema differences may be valid (e.g. custom_info related). If there are more substantial differences, investigate further to verify that all the pre-requisite upgrade scripts have been applied in the right order:

    mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
    mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have any issues, refer to Troubleshooting .