Upgrade Tarball v7.1.x to Tarball v7.2.x

This topic describes the upgrade procedure of Tarball deployments from Moogsoft AIOps v7.1.x Tarball to Moogsoft AIOps v7.2.x Tarball.

Before You Begin

To perform the upgrade you must be logged in as the same non-root user that installed and runs the existing Moogsoft AIOps v7.1.x software.

Download the Installer

Run the following command to download the installer:

curl -L -O "https://<username>:<password>@speedy.moogsoft.com/installer/moogsoft-aiops-7.2.0.2.tgz"

Note UI Integrations Configuration

If you are using any of the following UI integrations, make a note of their connection and configuration details before you begin the upgrade. You will need to delete and reconfigure them in the post-upgrade steps, due to UI changes in the new version.

  • AWS CloudWatch

  • Cherwell

  • JIRA Service Desk

  • JIRA Software

  • JMS

  • New Relic

  • Remedy

  • ServiceNow

  • SevOne

  • Slack

  • SolarWinds

  • VMware vCenter

  • VMware vSphere

  • vRealize Log Insight

  • WebSphere MQ

  • xMatters

Stop Services and Processes

  1. Stop Moogfarmd:

    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop

  2. Stop Apache Tomcat:

    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop

  3. Stop the integrations and LAMs, using either the Process Control utility or the kill command:

    $MOOGSOFT_HOME/bin/utils/process_cntl <lam_name> stop

    or:

    kill -9 $(ps -ef | grep java | grep _lam | awk '{print $2}') 2>/dev/null

  4. Disable the Events Analyser from running during the upgrade process:

    a. Run the following command to comment out the relevant lines in crontab:

    (crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -

    b. Run the following command to stop any active Events Analyser processes:

    ps -ef | grep java | egrep 'events_analyser|moog_indexer' | awk '{print $2}' | xargs kill 2>/dev/null

  5. Stop MySQL, RabbitMQ, Elasticsearch, Nginx and Apache Tomcat:

    $MOOGSOFT_HOME/bin/utils/process_cntl mysqld stop
$MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch stop
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop
$MOOGSOFT_HOME/bin/utils/process_cntl nginx stop

  6. Obtain the path variables before re-linking the directories. From this step onwards, use the same terminal session to keep the variables:

    CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $MOOGSOFT_HOME/dist/7.1.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1|awk '{print $2}'|tr -d ';'))
CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $MOOGSOFT_HOME/dist/7.1.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1|awk '{print $2}'|tr -d ';'))
CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $MOOGSOFT_HOME/dist/7.1.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1)
CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $MOOGSOFT_HOME/dist/7.1.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1)

Back Up the Existing System

To back up the existing system:

  1. Back up $MOOGSOFT_HOME.

  2. Take a snapshot (for VMs).

  3. Back up MySQL.

Upgrade Moogsoft AIOps

Run the following commands to perform the upgrade:

tar -xf moogsoft-aiops-7.2.0.2.tgz
bash moogsoft-aiops-install-7.2.0.2.sh

Follow the instructions that appear. The upgrade process detects the v7.1.0 installation and performs an upgrade (a side-by-side deployment).

Run the following command to change the default log level for Elasticsearch. By default it is set to INFO but you can change it to WARN to reduce the size of the Elasticsearch logs.

sed -i -e "s;rootLogger.level = info;rootLogger.level = warn;g" $MOOGSOFT_HOME/cots/elasticsearch/config/log4j2.properties

Run the following commands to restart the MySQL, RabbitMQ and Elasticsearch processes:

$MOOGSOFT_HOME/bin/utils/process_cntl mysqld start
$MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch start
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq start

Merge the Latest Configuration File Changes

The top-level $MOOGSOFT_HOME/config and $MOOGSOFT_HOME/bots folders are the master folder locations for configuration and bot files. The new 'default' v7.2.0 versions of these files are stored in $MOOGSOFT_HOME/dist/7.2.0/config/ and $MOOGSOFT_HOME/dist/7.2.0/bots/ respectively.

  1. Identify the config files that have changed between v7.1.0 and v7.2.0. For example:

    diff -rq $MOOGSOFT_HOME/dist/7.1.*/config $MOOGSOFT_HOME/dist/7.2.0/config | grep -i 'differ'

  2. For the files highlighted by the command above, the differences and new properties in the 7.2.0 versions of these files need to be manually merged into the files in $MOOGSOFT_HOME/config . Do not overwrite the files in $MOOGSOFT_HOME/config.

  3. Identify the contrib files that have changed between v7.1.0 and v7.2.0. For example:

    diff -rq $MOOGSOFT_HOME/dist/7.1.*/contrib $MOOGSOFT_HOME/dist/7.2.0/contrib | grep -i 'differ'

  4. For the files highlighted by the command above, the differences and new properties in the 7.2.0 versions of these files need to be manually merged into the files in $MOOGSOFT_HOME/contrib. Do not overwrite the files in $MOOGSOFT_HOME/contrib.

  5. Identify the bot files that have changed between v7.1.0 and v7.2.0. For example:

    diff -rq $MOOGSOFT_HOME/dist/7.1.*/bots $MOOGSOFT_HOME/dist/7.2.0/bots | grep -i 'differ'

  6. For the files highlighted by the command above, the differences and new properties in the 7.2.0 versions of these files need to be manually merged into the files in $MOOGSOFT_HOME/bots. Do not overwrite the files in $MOOGSOFT_HOME/bots.

Upgrade the Moogsoft AIOps Database Schema

To upgrade the Moogsoft AIOps database schema, you must provide the Auto Upgrader utility with the credentials of a database user with super privileges. For single-host installations where MySQL was installed as part of the Moogsoft AIOps deployment, you can use the default 'root' user.

  1. Run the following command after substituting the <MySQL-SuperUsername> argument:

    bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 7.2.0 -u <MySQL-SuperUsername>

  2. Enter the password for that user. You can provide the password to the utility with the -p flag but this is not recommended in non-test deployments for security reasons.

Upgrade Apache Tomcat and Nginx

Enter the following commands to upgrade Apache Tomcat and Nginx:

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula

Migrate the certificates for the previous deployment of Nginx to the new deployment. Use the following commands as an example of how to do this in the general case where SSL terminates in Nginx (default configuration):

cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/dist/7.2.0/cots/nginx/ssl/
cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/dist/7.2.0/cots/nginx/ssl/
sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/dist/7.2.0/cots/nginx/config/conf.d/moog-ssl.conf
sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/dist/7.2.0/cots/nginx/config/conf.d/moog-ssl.conf

Restart Nginx:

$MOOGSOFT_HOME/bin/utils/process_cntl nginx restart

Update and Reconfigure Integrations

This upgrade process replaces the $MOOGSOFT_HOME/bots/moobots/RemedyIntegration.js Moobot file. Back up this file if you have customised it, before continuing.

To update the integrations:

  1. The LAMs/Integrations no longer use the 'state' files under $MOOGSOFT_HOME/config. These files can be removed with the following command:

    rm -f $MOOGSOFT_HOME/config/*_lam.state;
rm -f $MOOGSOFT_HOME/config/{null.state, TestStateManager.state}
rm -f $MOOGSOFT_HOME/config/observe*.conf

  2. Run the following command to extract the old and new Integrations:

    bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN;
for file in $(diff -qr $MOOGSOFT_HOME/dist/7.2.0/ui/integrations/ \
$MOOGSOFT_HOME/dist/7.1.*/ui/integrations/ | egrep -vi 'only in.*7.2.0|differ' \
| awk '{print $NF}'); do cp -rp $MOOGSOFT_HOME/dist/7.1.*/ui/integrations/$file \
$MOOGSOFT_HOME/dist/7.2.0/ui/integrations/; done

Configure the ServiceNow MID Server to use Java 8

If you are using a ServiceNow MID server installed on the same host as Moogsoft AIOps, you must configure it to point to Java 8. The MID server requires Java 8 (update 152 or later). It will not work with Java 9+. To do this:

  1. Install the latest version of Java 8. See the ServiceNow MID server system requirements for more information.

  2. Stop the MID server with the appropriate command. For example:

    kill -9 $(ps -ef | grep mid_server | grep -v grep | awk '{print $2}')

  3. Configure the wrapper.java.command property to point to the Java 8 binary in the following file:

    /usr/local/servicenow/moog_mid_server/agent/conf/wrapper-override.conf.

    For example:

    wrapper.java.command=/usr/java/jre1.8.0_171-amd64/bin/java

In the next step you will reconfigure the ServiceNow integration which will restart the MID server.

Reconfigure your UI Integrations

Delete and reconfigure all of the following UI integrations, using the configuration details you noted prior to the upgrade. This is required due to UI changes in the new version.

  • AWS CloudWatch

  • Cherwell

  • JIRA Service Desk

  • JIRA Software

  • JMS

  • New Relic

  • Remedy

  • ServiceNow

  • SevOne

  • Slack

  • SolarWinds

  • VMware vCenter

  • VMware vSphere

  • vRealize Log Insight

  • WebSphere MQ

  • xMatters

Confirm All Processes are Running and Start Moogfarmd

To confirm all services are running:

  1. Run the following commands to verify that all required processes are running and start Moogfarmd:

    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status
$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd start
$MOOGSOFT_HOME/bin/utils/process_cntl nginx status
$MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch status

  2. Run the following command to re-enable the Event Analyser cronjobs:

    (crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -

  3. Run the following command to restart any previously running UI-based Integrations:

    bash $MOOGSOFT_HOME/bin/utils/startup_cntl;

Add the Keepalive Cronjob (Optional)

A new cronjob is installed for clean installations of 7.2.0 which provides the ability for certain services to be restarted automatically should they fail. For RPM deployments this is usually handled by chkconfig or MySQL itself for example, but can be useful in certain cases. See Configure Services to Restart for more information.

Command to enable if required:

(crontab -l; echo -e "*\t*\t*\t*\t* $MOOGSOFT_HOME/bin/utils/process_keepalive.sh 2>&1") | crontab -

Migrate the Java Keystore

The upgrade includes a new version of the Java Runtime, so you need to migrate any certificates stored in the old Java keystore into the new keystore, which is now the JRE under $MOOGSOFT_HOME/cots/jre/.

The previous JRE is located in $MOOGSOFT_HOME/dist/7.1.0/cots/jre.

If you did not manually add certificates to the old store, you can skip this step.

Verify the Upgrade

You can verify the upgrade manually or using automatic utilities.

Verify the Upgrade Manually

Perform the following basic steps to ensure the upgrade to Moogsoft AIOps v7.2.0 was successful:

  1. Verify that the UI login page displays "Version 7.2.0.2" at the top.

  2. Check that the UI "Support Information" window correctly indicates the current version as "7.2.0.2" and shows the correct schema upgrade history.

Verify the Upgrade Using Automatic Utilities

Run the following automatic utilities to ensure that the upgrade to 7.2.0 was successful:

  1. Confirm that all Moogsoft AIOps files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

  2. Confirm that all Apache Tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh

    1. Verify that all the steps have completed successfully. If there are some web app differences, run the following command to extract the web app with the correct files: 

      $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w

  3. Confirm that the database schema has upgraded successfully using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    Confirm that all the steps are successful. Some schema differences might be valid (e.g. custom_inforelated). If there are more substantial differences, you should investigate further to verify that all the prerequisite upgrade scripts have been applied in the right order:

    mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have any issues, refer to Troubleshooting.

In this section: