Upgrade Tarball v6.5.x to Tarball v7.2.x

This topic describes the upgrade procedure of Tarball deployments from Moogsoft AIOps v6.5.x Tarball to Moogsoft AIOps v7.2.x Tarball.

Before You Begin

To perform the upgrade you must be logged in as the same non-root user that installed and runs the existing Moogsoft AIOps v6.5.x software.

Download the Installer

Run the following command to download the installer:

curl -L -O "https://<username>:<password>@speedy.moogsoft.com/installer/moogsoft-aiops-7.2.0.2.tgz"

Note UI Integrations Configuration

If you are using any UI integrations, make a note of their connection and configuration details before you begin the upgrade. You will need to delete and reconfigure them in the post-upgrade steps, due to UI changes in the new version.

Stop Services and Processes

  1. Stop Moogfarmd:

    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop
  2. Stop Apache Tomcat:

    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop
  3. Stop the integrations and LAMs, using either the Process Control utility or the kill command:

    $MOOGSOFT_HOME/bin/utils/process_cntl <lam_name> stop

    or:

    kill -9 $(ps -ef | grep java | grep _lam | awk '{print $2}') 2>/dev/null
  4. Disable the Events Analyser from running during the upgrade process:

    a. Run the following command to comment out the relevant lines in crontab:

    (crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -

    b. Run the following command to stop any active Events Analyser processes:

    ps -ef | grep java | egrep 'events_analyser|moog_indexer' | awk '{print $2}' | xargs kill 2>/dev/null
  5. Stop MySQL, RabbitMQ, Elasticsearch, Nginx and Apache Tomcat:

    $MOOGSOFT_HOME/bin/utils/process_cntl mysqld stop
    $MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch stop
    $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx stop
  6. Obtain the path variables before re-linking the directories. From this step onwards, use the same terminal session to keep the variables:

    CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $MOOGSOFT_HOME/dist/6.5.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1|awk '{print $2}'|tr -d ';'))
    CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $MOOGSOFT_HOME/dist/6.5.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1|awk '{print $2}'|tr -d ';'))
    CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $MOOGSOFT_HOME/dist/6.5.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1)
    CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $MOOGSOFT_HOME/dist/6.5.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1)

Back Up the Existing System

To back up the existing system:

  1. Back up $MOOGSOFT_HOME.

  2. Take a snapshot (for VMs).

  3. Back up MySQL.

Upgrade Moogsoft AIOps

Run the following commands to perform the upgrade:

tar -xf moogsoft-aiops-7.2.0.2.tgz
bash moogsoft-aiops-install-7.2.0.2.sh

Follow the instructions that appear. The upgrade process detects the v6.5.0 installation and performs an upgrade (a side-by-side deployment).

Migrate Data Directories and Restart MySQL and Elasticsearch

Run the following commands to migrate the data directories for MySQL, RabbitMQ, and Elasticsearch into the new locations:

mkdir -p $MOOGSOFT_HOME/var/lib/{elasticsearch,mysql,rabbitmq}
cp -frp $MOOGSOFT_HOME/dist/6.5.*/cots/elasticsearch/data $MOOGSOFT_HOME/var/lib/elasticsearch/
sed -i "s;#path.data: /path/to/data;path.data: $MOOGSOFT_HOME/var/lib/elasticsearch/data;g" $MOOGSOFT_HOME/cots/elasticsearch/config/elasticsearch.yml
cp -frp $MOOGSOFT_HOME/dist/6.5.*/cots/mysql/* $MOOGSOFT_HOME/var/lib/mysql/
sed -i "s;$MOOGSOFT_HOME/cots/mysql/;$MOOGSOFT_HOME/var/lib/mysql/;" ~/.my.cnf
cp -frp $MOOGSOFT_HOME/dist/6.5.*/cots/rabbitmq-server/var/lib/rabbitmq/mnesia $MOOGSOFT_HOME/var/lib/rabbitmq/

Run the following command to change the default log level for Elasticsearch. By default it is set to INFO but you can change it to WARN to reduce the size of the Elasticsearch logs.

sed -i -e "s;rootLogger.level = info;rootLogger.level = warn;g" $MOOGSOFT_HOME/cots/elasticsearch/config/log4j2.properties

Run the following commands to restart the MySQL, RabbitMQ and Elasticsearch processes:

$MOOGSOFT_HOME/bin/utils/process_cntl mysqld start
$MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch start
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq start

Merge the Latest Configuration File Changes

The top-level $MOOGSOFT_HOME/config and $MOOGSOFT_HOME/bots folders are the master folder locations for configuration and bot files. The new 'default' v7.2.0 versions of these files are stored in $MOOGSOFT_HOME/dist/7.2.0/config/ and $MOOGSOFT_HOME/dist/7.2.0/bots/ respectively.

  1. Identify the config files that have changed between v6.5.0 and v7.2.0. For example:

    diff -rq $MOOGSOFT_HOME/dist/6.5.*/config $MOOGSOFT_HOME/dist/7.2.0/config | grep -i 'differ'
  2. Update the files in $MOOGSOFT_HOME/config with any changes introduced in the v7.2.0 versions of these files.

  3. Identify the contrib files that have changed between v6.5.0 and v7.2.0. For example:

    diff -rq $MOOGSOFT_HOME/dist/6.5.*/contrib $MOOGSOFT_HOME/dist/7.2.0/contrib | grep -i 'differ'
  4. Update the files in $MOOGSOFT_HOME/contrib with any changes introduced in the v7.2.0 versions of these files.

  5. Identify the bot files that have changed between v6.5.0 and v7.2.0. For example:

    diff -rq $MOOGSOFT_HOME/dist/6.5.*/bots $MOOGSOFT_HOME/dist/7.2.0/bots | grep -i 'differ'
  6. Update the files in $MOOGSOFT_HOME/bots with any changes introduced in the v7.2.0 versions of these files.

Upgrade the Moogsoft AIOps Database Schema

To upgrade the Moogsoft AIOps database schema, you must provide the Auto Upgrader utility with the credentials of a database user with super privileges. For single-host installations where MySQL was installed as part of the Moogsoft AIOps deployment, you can use the default 'root' user.

  1. Run the following command after substituting the <MySQL-SuperUsername> argument:

    bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 7.2.0 -u <MySQL-SuperUsername>
  2. Enter the password for that user. You can provide the password to the utility with the -p flag but this is not recommended in non-test deployments for security reasons.

Drop Deprecated Historic Database Tables

Run the following commands to drop two tables from the historic database that are no longer used. If they are not dropped at this stage, the moog_db_validator utility at the end of this process will report their presence as a delta.

bash $MOOGSOFT_HOME/bin/utils/moog_mysql_client -i -e "drop table room_post_sigs";
bash $MOOGSOFT_HOME/bin/utils/moog_mysql_client -i -e "drop table room_posts";

Update the RabbitMQ Configuration

Update the RabbitMQ configuration to support autoheal and other processes that improve stability in HA environments. To do this, run the following commands on each server the moogsoft-mooms / RabbitMQ package is installed on. Replace <VHOST> with the name of your RabbitMQ vhost:

cp -f $MOOGSOFT_HOME/etc/cots/rabbitmq/rabbitmq.config $MOOGSOFT_HOME/cots/rabbitmq-server/etc/rabbitmq/
bash $MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -z <VHOST> -p

Upgrade Apache Tomcat and Nginx

Enter the following commands to upgrade Apache Tomcat and Nginx:

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula

Migrate the certificates for the previous deployment of Nginx to the new deployment. Use the following commands as an example of how to do this in the general case where SSL terminates in Nginx (default configuration):

cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/dist/7.2.0/cots/nginx/ssl/
cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/dist/7.2.0/cots/nginx/ssl/
sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/dist/7.2.0/cots/nginx/config/conf.d/moog-ssl.conf
sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/dist/7.2.0/cots/nginx/config/conf.d/moog-ssl.conf

Restart Nginx:

$MOOGSOFT_HOME/bin/utils/process_cntl nginx restart

Update and Reconfigure Integrations

This upgrade process replaces the $MOOGSOFT_HOME/bots/moobots/RemedyIntegration.js Moobot file. Back up this file if you have customised it, before continuing.

To update the integrations:

  1. The LAMs/Integrations no longer use the 'state' files under $MOOGSOFT_HOME/config. Remove these files with the following command:

    rm -f $MOOGSOFT_HOME/config/*_lam.state;
    rm -f $MOOGSOFT_HOME/config/{null.state, TestStateManager.state}
    rm -f $MOOGSOFT_HOME/config/observe*.conf
  2. Run the following command to extract the old and new Integrations:

    bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN;
    for file in $(diff -qr $MOOGSOFT_HOME/dist/7.2.0/ui/integrations/ \
    $MOOGSOFT_HOME/dist/6.5.*/ui/integrations/ | egrep -vi 'only in.*7.2.0|differ' \
    | awk '{print $NF}'); do cp -rp $MOOGSOFT_HOME/dist/6.5.*/ui/integrations/$file \
    $MOOGSOFT_HOME/dist/7.2.0/ui/integrations/; done

Configure the ServiceNow MID Server to use Java 8

If you are using a ServiceNow MID server installed on the same host as Moogsoft AIOps, you must configure it to point to Java 8. The MID server requires Java 8 (update 152 or later). It will not work with Java 9+. To do this:

  1. Install the latest version of Java 8. See the ServiceNow MID server system requirements for more information.

  2. Stop the MID server by running the appropriate command. For example:

    kill -9 $(ps -ef | grep mid_server | grep -v grep | awk '{print $2}')
  3. Configure the wrapper.java.command property to point to the Java 8 binary in the following file:

    /usr/local/servicenow/moog_mid_server/agent/conf/wrapper-override.conf.

    For example:

    wrapper.java.command=/usr/java/jre1.8.0_171-amd64/bin/java

In the next step you will reconfigure the ServiceNow integration which will restart the MID server.

Reconfigure your UI Integrations

Delete and reconfigure all of your UI integrations, using the configuration details you noted prior to the upgrade. This is required due to UI changes in the new version.

Confirm All Processes are Running and Start Moogfarmd

  1. Run the following commands to verify that all required processes are running and start Moogfarmd:

    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status
    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd start
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx status
    $MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch status
  2. Run the following command to re-enable the Event Analyser cronjobs:

    (crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
  3. Run the following command to restart any previously running UI-based Integrations:

    bash $MOOGSOFT_HOME/bin/utils/startup_cntl;

Add the Keepalive Cronjob (Optional)

A new cronjob is installed for clean installations of 7.2.0 which provides the ability for certain services to be restarted automatically should they fail. For RPM deployments this is usually handled by chkconfig or MySQL itself for example, but can be useful in certain cases. See Configure Services to Restart for more information.

Command to enable if required:

(crontab -l; echo -e "*\t*\t*\t*\t* $MOOGSOFT_HOME/bin/utils/process_keepalive.sh 2>&1") | crontab -

Migrate the Java Keystore

The upgrade includes a new version of the Java Runtime, so you need to migrate any certificates stored in the old Java keystore into the new keystore, which is now the JRE under $MOOGSOFT_HOME/cots/jre/.

The previous JRE is located in $MOOGSOFT_HOME/dist/6.5.0/cots/jre.

If you did not manually add certificates to the old store, you can skip this step.

Verify the Upgrade

You can verify the upgrade manually or using automatic utilities.

Verify the Upgrade Manually

Perform the following basic steps to ensure the upgrade to Moogsoft AIOps v7.2.0 was successful:

  1. Verify that the UI login page displays "Version 7.2.0.2" at the top.

  2. Check that the UI "Support Information" window correctly indicates the current version as "7.2.0.2" and shows the correct schema upgrade history.

Verify the Upgrade Using Automatic Utilities

Run the following automatic utilities to ensure that the upgrade to 7.2.0.2 was successful:

  1. Confirm that all Moogsoft AIOps files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
  2. Confirm that all Apache Tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
  3. Verify that all the steps have completed successfully. If there are some web app differences, run the following command to extract the web app with the correct files:

    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  4. Confirm that the database schema has upgraded successfully using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    Confirm that all the steps are successful. Some schema differences might be valid (e.g. custom_info related). If there are more substantial differences, you should investigate further to verify that all the pre-requisite upgrade scripts have been applied in the right order:

    mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
    mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have any issues, refer to Troubleshooting.