Moogsoft Docs

Upgrade Non-root v7.0.x to Non-root v7.1.x

This topic describes the upgrade procedure of non-root deployments from Moogsoft AIOps v7.0.x non-root to Moogsoft AIOps v7.1.0 non-root.

The following steps outline this workflow:

Before You Begin

To perform the upgrade you must be logged in as the same non-root user that installed and runs the existing Moogsoft AIOps v7.0.x software.

Run the following command to download the installer:

curl -L -O "https://<username>:<password>@speedy.moogsoft.com/installer/moogsoft-aiops-7.1.0.tgz"


Back Up the Existing System

To back up the existing system:

  1. Back up $MOOGSOFT_HOME .
  2. Take a snapshot (for VMs).
  3. Back up MySQL.

Stop Service and Processes

  1. Stop Moogfarmd:

    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop
  2. Stop Apache Tomcat:

    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop
  3. Stop the integrations and LAMs, using either the Process Control utility or the kill command:

    $MOOGSOFT_HOME/bin/utils/process_cntl <lam_name> stop

    or:

    kill -9 $(ps -ef | grep java | grep _lam | awk '{print $2}') 2>/dev/null
  4. Disable the Events Analyser from running during the upgrade process:

    a. Run the following command to comment out the relevant lines in crontab:

    (crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -

    b. Run the following command to stop any active Events Analyser processes:

    ps -ef | grep java | egrep 'events_analyser|moog_indexer' | awk '{print $2}' | xargs kill 2>/dev/null
  5. Stop MySQL, RabbitMQ, Elasticsearch, Nginx and Apache Tomcat:

    $MOOGSOFT_HOME/bin/utils/process_cntl mysqld stop
    $MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch stop
    $MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq stop
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx stop
  6. Obtain the path variables before re-linking the directories. From this step onwards, use the same terminal session to keep the variables:

    CERT_REAL_PATH_PEM=$(readlink -f $(grep -h 'ssl_certificate ' $MOOGSOFT_HOME/dist/7.0.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1|awk '{print $2}'|tr -d ';'))
    CERT_REAL_PATH_KEY=$(readlink -f $(grep -h 'ssl_certificate_key' $MOOGSOFT_HOME/dist/7.0.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1|awk '{print $2}'|tr -d ';'))
    CERT_PATH_PEM=$(grep -h 'ssl_certificate ' $MOOGSOFT_HOME/dist/7.0.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1)
    CERT_PATH_KEY=$(grep -h 'ssl_certificate_key' $MOOGSOFT_HOME/dist/7.0.*/cots/nginx/config/conf.d/moog-ssl.conf|head -1)

Upgrade Moogsoft AIOps

Run the following commands to perform the upgrade:

tar -xf moogsoft-aiops-7.1.0.tgz
bash moogsoft-aiops-install-7.1.0.sh

Follow the instructions that appear. The upgrade process detects the v7.0.0 installation and performs an upgrade (a side-by-side deployment).

Set Elasticsearch log level and restart MySQL, Elasticsearch and RabbitMQ

Run the following command to change the default log level for Elasticsearch. By default it is set to INFO but you can change it to WARN to reduce the size of the Elasticsearch logs.

sed -i -e "s;rootLogger.level = info;rootLogger.level = warn;g" $MOOGSOFT_HOME/cots/elasticsearch/config/log4j2.properties

Run the following commands to restart the MySQL, RabbitMQ and Elasticsearch processes:

$MOOGSOFT_HOME/bin/utils/process_cntl mysqld start
$MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch start
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq start

Merge the Latest Config File Changes

The top-level $MOOGSOFT_HOME/config and $MOOGSOFT_HOME/bots folders are the master folder locations for configuration and bot files. The new 'default' v7.1.0 versions of these files are stored in $MOOGSOFT_HOME/dist/7.1.0/config/ and $MOOGSOFT_HOME/dist/7.1.0/bots/ respectively.

Moogfarmd configuration change in 7.1.0

There is a significant change in the configuration of Moogfarmd in this release. In clean installations of 7.1.0 the Moogfarmd configuration file now references separate Moolet config files located in $MOOGSOFT_HOME/config/moolets/ .

However, 7.1.0 is backwards-compatible with the old-format of the Moogfarmd configuration. Therefore you do not need to split up your Moogfarmd configuration for this upgrade, although we do recommend it.

Regular merge process

  1. Identify any new config files added in v7.1.0. For example:

    diff -rq $MOOGSOFT_HOME/config $MOOGSOFT_HOME/dist/7.1.0/config/ | grep -i 'Only'
  2. Copy the new config files into $MOOGSOFT_HOME/config .
  3. Identify any new contrib files added in v7.1.0. For example:

    diff -rq $MOOGSOFT_HOME/contrib $MOOGSOFT_HOME/dist/7.1.0/contrib/ | grep -i 'Only'
  4. Copy the new contrib files into $MOOGSOFT_HOME/ contrib .
  5. Identify any new bot files added in v7.1.0. For example:

    diff -rq $MOOGSOFT_HOME/bots $MOOGSOFT_HOME/dist/7.1.0/bots/ | grep -i 'Only'
  6. Copy the new bot files into $MOOGSOFT_HOME/bots (including the subdirectory if appropriate).
  7. Identify the config files that have changed between v7.0.0 and v7.1.0. For example:

    diff -rq $MOOGSOFT_HOME/dist/7.0.*/config $MOOGSOFT_HOME/dist/7.1.0/config | grep -i 'differ'
  8. Update the files in $MOOGSOFT_HOME/config with any changes introduced in the v7.1.0 versions of these files.
  9. Identify the contrib files that have changed between v7.0.0 and v7.1.0. For example:

    diff -rq $MOOGSOFT_HOME/dist/7.0.*/contrib $MOOGSOFT_HOME/dist/7.1.0/contrib | grep -i 'differ'
  10. Update the files in $MOOGSOFT_HOME/contrib with any changes introduced in the v7.1.0 versions of these files.
  11. Identify the bot files that have changed between v7.0.0 and v7.1.0. For example:

    diff -rq $MOOGSOFT_HOME/dist/7.0.*/bots $MOOGSOFT_HOME/dist/7.1.0/bots | grep -i 'differ'
  12. Update the files in $MOOGSOFT_HOME/bots with any changes introduced in the v7.1.0 versions of these files.

Upgrade the Moogsoft AIOps Database Schema

To upgrade the Moogsoft AIOps database schema, you must provide the Auto Upgrader utility with the credentials of a database user with super privileges. For single-host installations where MySQL was installed as part of the Moogsoft AIOps deployment, you can use the default 'root' user.

  1. Run the following command after substituting the <MySQL-SuperUsername> argument:

    bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 7.1.0 -u <MySQL-SuperUsername>
  2. Enter the password for that user. You can provide the password to the utility with the -p flag but this is not recommended in non-test deployments for security reasons.

Update the RabbitMQ configuration to support autoheal etc

Run the following commands on each server the moogsoft-mooms / RabbitMQ package is installed on to improve stability in HA environments (replace <VHOST> below with your existing RabbitMQ vhost):

cp -f $MOOGSOFT_HOME/etc/cots/rabbitmq/rabbitmq.config $MOOGSOFT_HOME/cots/rabbitmq-server/etc/rabbitmq/
bash $MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -z <VHOST> -p

Upgrade Apache Tomcat and Nginx

Enter the following commands to upgrade Apache Tomcat and Nginx:

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -qtnfwz $($MOOGSOFT_HOME/bin/utils/moog_config_reader -k mooms.zone) --accept-eula

Migrate the certificates for the previous deployment of Nginx to the new deployment. Use the following commands as an example of how to do this in the general case where SSL terminates in Nginx (default configuration):

cp -f $CERT_REAL_PATH_PEM $MOOGSOFT_HOME/dist/7.1.0/cots/nginx/ssl/
cp -f $CERT_REAL_PATH_KEY $MOOGSOFT_HOME/dist/7.1.0/cots/nginx/ssl/
sed -i "s|.*ssl_certificate .*|${CERT_PATH_PEM}|" $MOOGSOFT_HOME/dist/7.1.0/cots/nginx/config/conf.d/moog-ssl.conf
sed -i "s|.*ssl_certificate_key.*|${CERT_PATH_KEY}|" $MOOGSOFT_HOME/dist/7.1.0/cots/nginx/config/conf.d/moog-ssl.conf

Restart Nginx

$MOOGSOFT_HOME/bin/utils/process_cntl nginx restart

Update the Integrations

This upgrade process replaces the $MOOGSOFT_HOME/bots/moobots/RemedyIntegration.js Moobot file. Back up this file if you have customised it, before continuing.

To update the integrations:

  1. The LAMs/Integrations no longer use the 'state' files under $MOOGSOFT_HOME/config . These files can be removed with the following command:

    rm -f $MOOGSOFT_HOME/config/*_lam.state;
    rm -f $MOOGSOFT_HOME/config/{null.state, TestStateManager.state}
  2. Run the following command to extract the old and new Integrations:

    bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN;
    for file in $(diff -qr $MOOGSOFT_HOME/dist/7.1.0/ui/integrations/ \
    $MOOGSOFT_HOME/dist/7.0.*/ui/integrations/ | egrep -vi 'only in.*7.1.0|differ' \
    | awk '{print $NF}'); do cp -rp $MOOGSOFT_HOME/dist/7.0.*/ui/integrations/$file \
    $MOOGSOFT_HOME/dist/7.1.0/ui/integrations/; done
  3. After you finish the upgrade, you must remove and re-add any existing integrations if they need to be edited in the future (via the UI). If they do not need to be edited post-upgrade, they will work as they did pre-upgrade.

Confirm All Processes are Running and Start Moogfarmd

To confirm all services are running:

  1. Run the following commands to verify that all required processes are running and start Moogfarmd:

    $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat status
    $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd start
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx status
    $MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch status
  2. Run the following command to re-enable the Event Analyser cronjobs:

    (crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
  3. Run the following command to restart any previously running UI-based Integrations:

    bash $MOOGSOFT_HOME/bin/utils/startup_cntl;

OPTIONAL: Add the new 'Keepalive' cronjob

A new cronjob is installed for clean installations of 7.1.0 which provides the ability for certain services to be restarted automatically should they fail. For RPM deployments this is usually handled by chkconfig or MySQL itself for example, but can be useful in certain cases. See Configure Services to Restart for more information.

Command to enable if required:

(crontab -l; echo -e "*\t*\t*\t*\t* $MOOGSOFT_HOME/bin/utils/process_keepalive.sh 2>&1") | crontab -

Migrate the Java Keystore

The upgrade includes a new version of the Java Runtime, so you need to migrate any certificates stored in the old Java keystore into the new keystore, which is now the JRE under $MOOGSOFT_HOME/cots/jre/ .

The previous JRE is located in $MOOGSOFT_HOME/dist/7.0.0/cots/jre .

If you did not manually add certificates to the old store, you can skip this step.

Verify the Upgrade

You can verify the upgrade manually or using automatic utilities.

Verify the Upgrade Manually

Perform the following basic steps to ensure the upgrade to Moogsoft AIOps v7.1.0 was successful:

  1. Verify that the UI login page displays "Version 7.1.0" at the top.
  2. Check that the UI "Support Information" window correctly indicates the current version as  "7.1.0" and shows the correct schema upgrade history.

Verify the Upgrade Using Automatic Utilities

Run the following automatic utilities to ensure that the upgrade to 7.1.0 was successful:

  1. Confirm that all Moogsoft AIOps files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
  2. Confirm that all Apache Tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
  3. Verify that all the steps have completed successfully. If there are some web app differences, run the following command to extract the web app with the correct files:

    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  4. Confirm that the database schema has upgraded successfully using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

    Confirm that all the steps are successful. Some schema differences might be valid (e.g. custom_info related). If there are more substantial differences, you should investigate further to verify that all the pre-requisite upgrade scripts have been applied in the right order:

    mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
    mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have any issues, refer to Troubleshooting .