Moogsoft Docs

Upgrade v6.5.0 or v6.5.0.x to v7.0.1

This topic describes the upgrade procedure of RPM deployments from Moogsoft AIOps v6.5.0 or v6.5.0.x to Moogsoft AIOps v7.0.1. Refer to Upgrade Moogsoft AIOps - RPM Deployments for general information and links to upgrades for other versions.

The following steps outline this workflow:

Before You Begin

Red Hat Enterprise Linux 6/CentOS 6 ( RHEL6) is not supported for Moogsoft AIOps v7.0.1. See Deprecation Notice: RHEL 6 . If your previous installation runs on RHEL6, see the Upgrade - Migrate from RHEL6 to RHEL7 guide before carrying out this migration.

To prepare for an offline upgrade, where the Moogsoft AIOps packages reside in a 'local' yum repository, follow steps 1-8 in section "Prepare the Local yum Repositories" in the Pre-Install AIOps - Offline RPMs guide . Then follow this guide to perform the upgrade. Take into account any step-specific notes regarding Offline upgrades.

Back Up the Existing System

To back up the existing system:

  1. Back up $MOOGSOFT_HOME .
  2. Take a snapshot (for VMs).
  3. Back up MySQL.

Stop Services and Processes

To stop the services and processes:

  1. Stop Moogfarmd:

    service moogfarmd stop

    Alternatively, if the Moogfarmd instance was started via the UI, you should stop it via the UI at this point.

  2. Ensure no more Moogfarmd processes are running with this command:

    kill -9 $(ps -ef | grep java | grep farm | awk '{print $2}') 2>/dev/null
  3. Stop Apache Tomcat:

    service apache-tomcat stop
  4. Stop the LAMs. You can do this via their service scripts:

    service <lam_service_name> stop

    Alternatively, stop their running processes using the command:

    kill <pid>
  5. Ensure all the above processes have been stopped before continuing:

    service <service_name> status
  6. Disable the events_analyser from running during the upgrade process:

    1. Run the following command to comment out the relevant lines in crontab :

      (crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
    2. Run the following command to stop any running events_analyser processes:

      ps -ef | grep java | egrep 'events_analyser' | awk '{print $2}' | xargs kill 2>/dev/null

Upgrade Moogsoft AIOps

Choose the upgrade command that corresponds to your installation:

  • Single host from a remote or local yum repository. Run the following commands to perform the upgrade:

    yum -y upgrade moogsoft-integrations-7.0.1 moogsoft-integrations-ui-7.0.1 moogsoft-common-7.0.1 moogsoft-server-7.0.1 moogsoft-ui-7.0.1 moogsoft-db-7.0.1 moogsoft-search-7.0.1 moogsoft-mooms-7.0.1 moogsoft-utils-7.0.1
  • Single host from the same machine where the RPMs have been downloaded locally and are in the current folder. Run the following commands to perform the upgrade:

    yum -y upgrade moogsoft-*7.0.1*.rpm
  • Distributed installs. Select the relevant update commands to run depending on which package(s) are installed on the current host. For example:

    1. HOST1 running Integrations, Integrations UI, common (required), Moogfarmd server, and Moogsoft AIOps UI packages:

      yum -y upgrade moogsoft-integrations-7.0.1 moogsoft-integrations-ui-7.0.1 moogsoft-common-7.0.1 moogsoft-server-7.0.1 moogsoft-ui-7.0.1
    2. HOST2 running common (required), utilities, database, and search packages:

      yum -y upgrade moogsoft-common-7.0.1 moogsoft-utils-7.0.1 moogsoft-db-7.0.1 moogsoft-search-7.0.1
    3. HOST3 running common (required), Message Bus, and utility packages:

      yum -y upgrade moogsoft-common-7.0.1 moogsoft-mooms-7.0.1 moogsoft-utils-7.0.1

Merge the Latest Config File Changes

Manually merge and compare .rpmsave versions of files with the new versions of those files. Add any new properties to the older versions of the files.

To find files which have been changed, moved, or deleted, run these commands:

find $MOOGSOFT_HOME -name '*.rpmsave'
find /etc/init.d/ -name '*.rpmsave'

An example command to see what differences are present in the $MOOGSOFT_HOME/config/system.conf file is shown below:

diff -u $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.rpmsave

An example of the process to merge the differences is shown below:

  1. Rename the new versions of the files, without the .rpmsave extension, to end with .bak .
  2. Merge the .rpmsave file with the new .bak file by adding new properties/config where needed so the structure matches the new version of the file.
  3. Rename the .rpmsave file to delete the .rpmsave extension.

Optional: Decrease Elasticsearch Log Verbosity

To decrease the log verbosity of Elasticsearch:

  1. Run the following command to change the default log level for Elasticsearch. By default it is set to info but you can change it to warn to reduce the size of the Elasticsearch logs.

    sed -i -e "s;rootLogger.level = info;rootLogger.level = warn;g" /etc/elasticsearch/log4j2.properties
  2. Restart the Elasticsearch service:

    service elasticsearch restart

Upgrade the Moogsoft AIOps Schema

To upgrade the Moogsoft AIOps database, you need to provide the moog_db_auto_upgrader utility with the credentials of a database user with super privileges. For single-host installs where MySQL was installed as part of the Moogsoft AIOps deployment, you can use the default 'root' user.

  1. Run the following command after substituting the <MySQL-SuperUsername> argument:

    bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 7.0.0 -u <MySQL-SuperUsername>
  2. Enter the password for that user. You can provide the password to the utility with the -p flag but this is not recommended in non-test deployments for security reasons.

Upgrade the Apache Tomcat Webapps and Configuration Files

The $APPSERVER_HOME/conf/server.xml and $APPSERVER_HOME/conf/context.xml files have been changed in this release. If the existing deployment has customized versions of those files, please make a backup before continuing.

To upgrade using the script provided, run the following command. This is recommended for standard or single-host installs.

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -wf

If Apache Tomcat startup or shutdown failures are reported when running the above command, such as "Web apps are not rebuilding, please check catalina.out", then you should forcibly shut down Apache Tomcat using kill -9 on the Apache Tomcat java PID, and then repeat the step above.

Alternatively, to upgrade manually, follow these steps:

  1. Stop Apache Tomcat:

    service apache-tomcat stop
  2. Check the Apache Tomcat java process is no longer running:

    service apache-tomcat status
  3. Delete the existing extracted webapps:

    rm -rf $APPSERVER_HOME/webapps/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}
  4. Copy in the new webapp WAR files and replace the old ones:

    cp -f $MOOGSOFT_HOME/lib/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}.war $APPSERVER_HOME/webapps/
  5. Copy in the new configuration files:

    cat $MOOGSOFT_HOME/etc/cots/tomcat/context.xml | sed "s;/usr/share/moogsoft/lib;$MOOGSOFT_HOME/lib;g" > $APPSERVER_HOME/conf/context.xml
    cp -f $MOOGSOFT_HOME/etc/cots/tomcat/apache-tomcat-ssl.xml $APPSERVER_HOME/conf/server.xml
  6. Restart Apache Tomcat:

    service apache-tomcat start

    If customizations had been made to server.xml or context.xml , update the new version of those files and restart the Apache Tomcat service.

Update the Integrations

To update the Integrations:

  1. Run the following command to extract the Integrations:

    bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN
  2. After you finish the upgrade, un-install and re-install any of the following ticketing integrations that were present before this upgrade:

    • HP OMi
    • HP NNMi
    • Zenoss
    • SolarWinds
    • JDBC
    • Zabbix
    • Dynatrace APM
    • VMware vSphere
    • VMware vCenter
    • VMware vRealize Log Insight
    • Email

Confirm All Services are Running and Start Moogfarmd

To confirm all services are running:

  1. Run the following commands to check all required services are running and to start Moogfarmd:

    service apache-tomcat status
    service moogfarmd start
    service nginx status
    service elasticsearch status
  2. Run the following command to re-enable the events_analyser cronjobs:

    (crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
  3. Run the following command to restart any previously running Integrations:

    service moogstartupd restart

Migrate the Java Keystore

The upgrade includes a new version of the Java Runtime, so you need to migrate any certificates stored in the old Java keystore to the new JAVA_HOME keystore.

If you did not manually add certificates to the old store, you can skip this step.

Verify the Upgrade

You can verify the upgrade manually or using automatic utilities.

Verify the Upgrade Manually

Perform the following basic steps to ensure that the upgrade to Moogsoft AIOps v7.0.1 was successful:

  1. Check that the UI login page displays "Version 7.0.1" at the top.
  2. Check that the UI "Support Information" window correctly indicates the current version as "7.0.1" and shows the correct schema upgrade history.

Verify the Upgrade Using Automatic Utilities

Run the following automatic utilities to ensure that the upgrade to Moogsoft AIOps v7.0.1 was successful:

  1. Confirm all Moog files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
  2. Confirm all Apache Tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
  3. Confirm the schema has been upgraded successfully using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh
  4. Confirm that all the steps are successful. If there are some webapp differences, you can resolve them using moog_init_ui.sh -w which extracts the webapps with the right files.
  5. Some schema differences may be valid (e.g. custom_info related). If there are more substantial differences, you should investigate further to verify that all the pre-requisite upgrade scripts have been applied in the right order:

    mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
    mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have any issues, refer to Troubleshooting .