Moogsoft Docs

Upgrade v6.4.0 or V6.4.0.x to v7.0.1

This topic describes the upgrade procedure of RPM deployments from Moogsoft AIOps v6.4.0 or v6.4.0.x to Moogsoft AIOps v7.0.1. Refer to Upgrade Moogsoft AIOps - RPM Deployments for general information and links to upgrades for other versions.

The following steps outline this workflow:

Before You Begin

Red Hat Enterprise Linux 6/CentOS 6 ( RHEL6) is not supported for Moogsoft AIOps v7.0.1. See Deprecation Notice: RHEL 6 . If your previous installation runs on RHEL6, see the Upgrade - Migrate from RHEL6 to RHEL7 guide before carrying out this migration

Moogsoft AIOps no longer requires the moogsoft-eula RPM package. You accept the  End User License Agreement (EULA) when you run one of the moog_init commands during the upgrade process.  You must accept the EULA to continue with the upgrade.

To prepare for an offline upgrade, where the Moogsoft AIOps packages reside in a 'local' yum repository, follow steps 1-8 in section "Prepare the Local yum Repositories" in the Pre-Install AIOps - Offline RPMs guide. Then follow  this guide to perform the upgrade. Take into account any step-specific notes regarding Offline upgrades.

Back Up the Existing System

To back up the exsting system:

  1. Back up $MOOGSOFT_HOME .
  2. Take a snapshot (for VMs).
  3. Back up MySQL.

Stop Services and Processes

  1. Stop moogfarmd:

    service moogfarmd stop

    Alternatively, if the moogfarmd instance was started via the UI, you should stop it via the UI at this point.

  2. Ensure no more farmd processes are running with this command:

    kill -9 $(ps -ef | grep java | grep farm | awk '{print $2}') 2>/dev/null
  3. Stop Apache Tomcat:

    service apache-tomcat stop
  4. Stop the LAMs. You can do this via their service scripts:

    service <lam_service_name> stop

    Alternatively, stop their running processes using the command:

    kill <pid>
  5. Ensure all the above processes have been stopped before continuing:

    service <service_name> status
  6. Disable the events_analyser from running during the upgrade process:

    1. Run the following command to comment out the relevant lines in crontab :

      (crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
    2. Run the following command to stop any running events_analyser processes:

      ps -ef | grep java | egrep 'events_analyser' | awk '{print $2}' | xargs kill 2>/dev/null

Remove the logrotate cronjob

The logrotate utility is no longer used by Moogsoft AIOps and is replaced by Log4j. Run the following command to remove the logrotate cronjob entry:

(crontab -l | grep -v logrotate) | crontab -

Prepare MySQL for the Upgrade

Check if MySQL is configured to run with --gtid-mode=ON using the following command in the MySQL CLI:

show variables like '%gtid%';

Remember the value of the gtid-mode variable because it affects the steps in the section Upgrade MySQL and the Moogsoft AIOps Schema .

Official instructions for upgrading from v5.6.26 to v5.7.22 are here: https://dev.mysql.com/doc/refman/5.7/en/upgrading.html and the steps are summarized below. It is recommended that you perform a backup of the database before continuing.

Stop the MySQL service and change the password as appropriate:

mysql -u root --password='' --execute="SET GLOBAL innodb_fast_shutdown=0"
service mysqld stop

Upgrade Moogsoft AIOps

Note

Please note that there are some new moogsoft packages (moogsoft-integrations, moogsoft-integrations-ui) which are required by Moogsoft AIOps v7.0.1. As a result, you must perform an install of those packages and an upgrade of the other packages at the same time.

This is performed using a 'yum shell' script.

If you are performing an offline upgrade, you must download the new MySQL packages manually before continuing, as specified in Pre-Install AIOps - Offline RPMs .

Choose the upgrade that corresponds to your installation:

  • Single host from a remote repository. Run the following commands to perform the upgrade:

    echo 'upgrade moogsoft-common-7.0.1' > upgrade_to_700.sh
    echo 'upgrade moogsoft-db-7.0.1' >> upgrade_to_700.sh
    echo 'install moogsoft-integrations-7.0.1' >> upgrade_to_700.sh
    echo 'install moogsoft-integrations-ui-7.0.1' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-mooms-7.0.1' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-search-7.0.1' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-server-7.0.1' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-ui-7.0.1' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-utils-7.0.1' >> upgrade_to_700.sh
    echo 'remove moogsoft-lams' >> upgrade_to_700.sh
    echo 'run' >> upgrade_to_700.sh
    cat upgrade_to_700.sh | yum shell -y

    If this is an offline upgrade and the MySQL RPMs are in the current directory, run the following script instead:

    echo 'upgrade moogsoft-common-7.0.1' > upgrade_to_700.sh
    echo 'upgrade moogsoft-db-7.0.1' >> upgrade_to_700.sh
    echo 'install moogsoft-integrations-7.0.1' >> upgrade_to_700.sh
    echo 'install moogsoft-integrations-ui-7.0.1' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-mooms-7.0.1' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-search-7.0.1' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-server-7.0.1' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-ui-7.0.1' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-utils-7.0.1' >> upgrade_to_700.sh
    echo 'remove moogsoft-lams' >> upgrade_to_700.sh
    echo 'upgrade mysql-community-libs-compat-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_700.sh
    echo 'upgrade mysql-community-server-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_700.sh
    echo 'upgrade mysql-community-client-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_700.sh
    echo 'upgrade mysql-community-common-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_700.sh
    echo 'upgrade mysql-community-libs-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_700.sh
    echo 'run' >> upgrade_to_700.sh
    cat upgrade_to_700.sh | yum shell -y
  • Single host from same machine where the RPMs have been downloaded locally and are in the current folder. Run the following commands to perform the upgrade:

    echo 'upgrade moogsoft-common-7.0.1-28214.x86_64.rpm' > upgrade_to_700.sh
    echo 'install moogsoft-integrations-7.0.1-28218.x86_64.rpm' >> upgrade_to_700.sh
    echo 'install moogsoft-integrations-ui-7.0.1-28218.x86_64.rpm' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-db-7.0.1-28214.x86_64.rpm' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-mooms-7.0.1-28214.x86_64.rpm' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-search-7.0.1-28214.x86_64.rpm' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-server-7.0.1-28214.x86_64.rpm' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-ui-7.0.1-28214.x86_64.rpm' >> upgrade_to_700.sh
    echo 'upgrade moogsoft-utils-7.0.1-28214.x86_64.rpm' >> upgrade_to_700.sh
    echo 'remove moogsoft-lams' >> upgrade_to_700.sh
    echo 'run' >> upgrade_to_700.sh
    cat upgrade_to_700.sh | yum shell -y
  • Distributed installs. Select the relevant update commands to run depending on which package(s) are installed on the current host. For example:

    1. HOST1 running common (required), moogfarmd server, Moogsoft AIOps UI, and Moogsoft AIOps Integrations UI packages:

      echo 'upgrade moogsoft-common-7.0.1' > upgrade_to_700.sh
      echo 'upgrade moogsoft-server-7.0.1' >> upgrade_to_700.sh
      echo 'upgrade moogsoft-ui-7.0.1' >> upgrade_to_700.sh
      echo 'install moogsoft-integrations-ui-7.0.1' >> upgrade_to_700.sh
      echo 'run' >> upgrade_to_700.sh
      cat upgrade_to_700.sh | yum shell -y
    2. HOST2 running common (required), database, and search packages:

      echo 'upgrade moogsoft-common-7.0.1' > upgrade_to_700.sh
      echo 'upgrade moogsoft-db-7.0.1' >> upgrade_to_700.sh
      echo 'upgrade moogsoft-search-7.0.1' >> upgrade_to_700.sh
      echo 'run' >> upgrade_to_700.sh
      cat upgrade_to_700.sh | yum shell -y
    3. HOST3 running common (required), Integrations, message bus, and utilities packages:

      echo 'upgrade moogsoft-common-7.0.1' > upgrade_to_700.sh
      echo 'install moogsoft-integrations-7.0.1' >> upgrade_to_700.sh
      echo 'remove moogsoft-lams' >> upgrade_to_700.sh
      echo 'upgrade moogsoft-mooms-7.0.1' >> upgrade_to_700.sh
      echo 'upgrade moogsoft-utils-7.0.1' >> upgrade_to_700.sh
      echo 'run' >> upgrade_to_700.sh
      cat upgrade_to_700.sh | yum shell -y

Merge the Latest Config File Changes

Manually merge and compare .rpmsave versions of files with the new versions of those files. Add any new properties to the older versions of the files.

To find files which have been changed/moved/deleted run these commands:

find $MOOGSOFT_HOME -name '*.rpmsave'
find /etc/init.d/ -name '*.rpmsave'

An example command to see what differences are present in the $MOOGSOFT_HOME/config/system.conf file is shown below:

diff -u $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.rpmsave

An example of the process to merge the differences is shown below:

  1. Rename the new versions of the files without the .rpmsave extension to end with .bak .
  2. Merge the .rpmsave file with the new .bak file by adding new properties/config where needed so the structure matches the new version of the file.
  3. Rename the .rpmsave file to delete the .rpmsave extension.

Optional: Decrease log verbosity of Elasticsearch

Run the following command to change the default log level for Elasticsearch. By default it is set to info but you can change it to warn to reduce the size of the Elasticsearch logs.

sed -i -e "s;rootLogger.level = info;rootLogger.level = warn;g" /etc/elasticsearch/log4j2.properties

Restart Elasticsearch, RabbitMQ, and Nginx

Run the following commands to restart Elasticsearch, RabbitMQ, and Nginx now that they have been upgraded:

service elasticsearch restart
service rabbitmq-server restart
service nginx restart

Upgrade MySQL and the Moogsoft AIOps Schema

To upgrade MySQL and the Moogsoft AIOps Schema:

  1. Update /etc/my.cnf with two new properties using the script below:

    sed -i 's/\(table-open-cache.*\)/\1\nmax_prepared_stmt_count = 1048576\ngroup_concat_max_len = 1048576/' /etc/my.cnf
  2. Restart MySQL:

    service mysqld restart
  3. If the gtid-mode is set to OFF , based on the command run earlier in the upgrade, run the MySQL upgrade utility. Provide the MySQL root password when prompted or just press <return> if no password set. Then restart MySQL to ensure any changes to system tables are saved:

    mysql_upgrade -u root -p
    service mysqld restart

    More information on GTID and mysql_upgrade is here: https://dev.mysql.com/doc/refman/5.7/en/replication-gtids-restrictions.html#replication-gtids-restrictions-mysql_upgrade .

  4. To upgrade the Moogsoft AIOps database, you need to provide the moog_db_auto_upgrader utility with the credentials of a database user with super privileges. For single-host installs where MySQL was installed as part of the Moogsoft AIOps deployment, you can use the default 'root' user.

    Run the following command after substituting the <MySQL-SuperUsername> argument:

    bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 7.0.0 -u <MySQL-SuperUsername>

    Enter the password for that user. You can provide the password to the utility with the -p flag but this is not recommended in non-test deployments for security reasons.

    1. If you see an error such as this:

      WARN : [main][20180710 20:40:11.376 +0000] [CAutoUpgrader.java:446] +|Encountered errors during execution, exiting...|+
      
      The 643_moogdb_upgrade_all.sql upgrade script contains the following at line 70:

      Change the file $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.4.2-6.4.3/643_moogdb_upgrade_all.sql to remove these lines:

      UPDATE system_config
      SET properties = JSON_SET(properties, "$.\"seed_alert_filter\"", "{}")
      WHERE id=4;
      
      UPDATE system_config
      SET properties = JSON_SET(properties, "$.\"seed_alert_filter\"", "{}")
      WHERE id=5;
      
      UPDATE system_config
      SET properties = JSON_SET(properties, "$.\"seed_alert_filter\"", "{}")
      WHERE id=6;

      And replace them with:

      UPDATE system_config
          SET properties = JSON_SET(properties, "$.\"seed_alert_filter\"", "{}")
          WHERE config_type = 'Recipe' AND  properties ->> '$.seed_alert_filter' IS NULL;

      Then re-run the moog_db_auto_upgrader command.

Upgrade the Apache Tomcat Webapps and Configuration Files

The Apache Tomcat service script has been updated in this version and the new script needs to be used.

If any customizations have been made to the /etc/init.d/apache-tomcat script, e.g. an -Xmx change, please make a backup before continuing.

Replace the old service script with the new one using the following command:

cp -f $MOOGSOFT_HOME/etc/cots/tomcat/apache-tomcat.sh /etc/init.d/apache-tomcat 

If any customizations were made in the old script, they should be made to the new version of the script before continuing.

Similarly, the $APPSERVER_HOME/conf/server.xml and $APPSERVER_HOME/conf/context.xml files need to be replaced with new versions. The command below deploys the new ones. Back up the existing ones if customizations have been made to them.

Now, perform the upgrade of the webapps and server.xml/context.xml .

To upgrade using the script provided, run the following command. This is recommended for standard or single-host installs.

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -wf

If Apache Tomcat startup or shutdown failures are reported when running the above command, such as "Web apps are not rebuilding, please check catalina.out", then you should forcibly shut down Apache Tomcat using kill -9 on the Apache Tomcat java PID, and then repeat the step above.

Alternatively, to upgrade manually, follow these steps:

  1. Stop Apache Tomcat:

    service apache-tomcat stop
  2. Check the Apache Tomcat java process is no longer running:

    service apache-tomcat status
  3. Delete the existing extracted webapps:

    rm -rf $APPSERVER_HOME/webapps/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}
  4. Copy in the new webapp WAR files and replace the old ones:

    cp -f $MOOGSOFT_HOME/lib/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}.war $APPSERVER_HOME/webapps/
  5. Restart Apache Tomcat:

    service apache-tomcat start

Update the Integrations

To update the Integrations:

  1. Run the following command needs to be run to extract Integrations:

    bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN
  2. After you finish the upgrade, un-install and re-install any of the following Ticketing Integrations that were present before this upgrade:

    • HP OMi
    • HP NNMi
    • Zenoss
    • Solarwinds
    • JDBC
    • Zabbix
    • Dynatrace APM
    • vSphere
    • vCenter
    • vRealize Log Insight
    • Email

Confirm All Services are Running and Start moogfarmd

To confirm all services are running:

  1. Run the following commands to check all required services are running and to start moogfarmd:

    service apache-tomcat status
    service moogfarmd start
    service nginx status
    service elasticsearch status
  2. Run the following command to re-enable the events_analyser cronjobs:

    (crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
  3. Run the following command to restart any previously running Integrations:

    service moogstartupd restart

Re-index Elasticsearch

Run the following command on the moogsoft-search/Elasticsearch server to remove the old Elasticsearch indexes:

curl -XDELETE 'http://localhost:9200/alerts/' && curl -XDELETE 'http://localhost:9200/situations/'

If the command completes successfully, the following message should be shown on the screen:

{"acknowledged":true}{"acknowledged":true}

Run the following command to re-index the alerts and Situations in moogfarmd. This requires moogfarmd to be running.

$MOOGSOFT_HOME/bin/utils/moog_indexer --now

The re-index occurs in the background and may take a while to complete. This depends on the number of alerts and Situations in your system.

Migrate the Java Keystore

The upgrade includes a new version of the Java Runtime, so you need to migrate any certificates stored in the old Java keystore to the new JAVA_HOME keystore.

If you did not manually add certificates to the old store, you can skip this step.


Verify the Upgrade

You can verify the upgrade manually or using automatic utilities.

Verify the Upgrade Manually

Perform the following basic steps to ensure the upgrade to Moogsoft AIOps v7.0.1 was successful:

  1. Check that the UI login page displays "Version 7.0.1" at the top.
  2. Check that the UI "Support Information" window correctly indicates the current version as "7.0.1" and shows the correct schema upgrade history.

Verify the Upgrade Using Automatic Utilities

Run the following automatic utilities to ensure that the upgrade to Moogsoft AIOps v7.0.1 was successful:

  1. Confirm all Moog files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
  2. Confirm all Apache Tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:

    $MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
  3. Confirm the schema has been upgraded successfully using this utility:

    $MOOGSOFT_HOME/bin/utils/moog_db_validator.sh
  4. Confirm that all the steps are successful. If there are some webapp differences then this can be resolved using moog_init_ui.sh -w which will extract the webapps with the right files.
  5. Some schema differences may be valid (e.g. custom_info related). If there are more substantial differences, you should investigate further to verify that all the pre-requisite upgrade scripts have been applied in the right order:

    mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
    mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have any issues, refer to Troubleshooting .