Moogsoft Docs

6.5.0-116

This topic describes the upgrade procedure from Moogsoft AIOps v6.4.0 or v6.4.0.x to Moogsoft AIOps v6.5.0. Refer to Upgrade Moogsoft AIOps - RPM Deployments for general information and links to upgrades for other versions.

When preparing for your upgrade, consider the following:

Preparations required for an offline upgrade

To prepare for an offline upgrade (where the AIOps packages are shipped in a 'local' yum repository), follow the steps 1-8 in this guide Pre-Install Offline RPMs . Once those preparations are completed, this guide can be used to perform the actual upgrade whilst taking into account any step-specific notes regarding Offline upgrades.

Backup Existing System

1. Backup $MOOGSOFT_HOME.
2. Take a snapshot (for VMs).
3. Backup MySQL.

Stop Services and Processes

1. moog_farmd:

   service moogfarmd stop

2. apache-tomcat:

   service apache-tomcat stop

3. LAMs. Either via their service scripts:

   service <lam_service_name> stop

   Alternatively, stop their running processes using:

   kill <pid>

4. Ensure all the above processes have been stopped before continuing:

   service <service_name> status

5. Disable the events_analyser from running during the upgrade process:
   
   

   1. Run the following command to comment out the relevant lines in crontab:

      (crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -

   2. Run the following command to stop any running events_analyser processes:

      ps -ef | grep java | egrep 'events_analyser|moog_indexer' | awk '{print $2}' | xargs kill 2>/dev/null

Remove the logrotate cronjob

The logrotate utility is no longer used by AIOps and is replaced by Log4j. The following command should be run to remove the logrotate cronjob entry:

(crontab -l | grep -v logrotate) | crontab -

Prepare MySQL for the upgrade

Check if MySQL is configured to run with --gtid-mode=ON using the following command in the MySQL CLI:

show variables like '%gtid%';

Remember what the value is - it will be needed later.

Official instructions for upgrading from 5.6.26 to 5.7.22 are here: https://dev.mysql.com/doc/refman/5.7/en/upgrading.html and we summarise the steps below. We recommend you perform a backup of the database before continuing.

Stop the mysql service (change the password as appropriate):

mysql -u root --password='' --execute="SET GLOBAL innodb_fast_shutdown=0"
service mysqld stop

Upgrade AIOps

If you are performing an offline upgrade, the new MySQL packages must be downloading manually before continuing as specified in Pre-Install Offline RPMs .

• Single host from a remote repository. Run the following commands to perform the upgrade:

  echo 'upgrade moogsoft-common-6.5.0' > upgrade_to_650.sh
  echo 'upgrade moogsoft-db-6.5.0' >> upgrade_to_650.sh
  echo 'install moogsoft-integrations-6.5.0' >> upgrade_to_650.sh
  echo 'install moogsoft-integrations-ui-6.5.0' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-mooms-6.5.0' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-search-6.5.0' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-server-6.5.0' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-ui-6.5.0' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-utils-6.5.0' >> upgrade_to_650.sh
  echo 'remove moogsoft-lams' >> upgrade_to_650.sh
  echo 'run' >> upgrade_to_650.sh
  cat upgrade_to_650.sh | yum shell -y

  If this is an offline upgrade and the MySQL rpms are in the current directory, the following script should be used instead (replace el7 with el6 as appropriate for your architecture):

  echo 'upgrade moogsoft-common-6.5.0' > upgrade_to_650.sh
  echo 'upgrade moogsoft-db-6.5.0' >> upgrade_to_650.sh
  echo 'install moogsoft-integrations-6.5.0' >> upgrade_to_650.sh
  echo 'install moogsoft-integrations-ui-6.5.0' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-mooms-6.5.0' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-search-6.5.0' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-server-6.5.0' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-ui-6.5.0' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-utils-6.5.0' >> upgrade_to_650.sh
  echo 'remove moogsoft-lams' >> upgrade_to_650.sh
  echo 'upgrade mysql-community-libs-compat-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_650.sh
  echo 'upgrade mysql-community-server-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_650.sh
  echo 'upgrade mysql-community-client-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_650.sh
  echo 'upgrade mysql-community-common-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_650.sh
  echo 'upgrade mysql-community-libs-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_650.sh
  echo 'run' >> upgrade_to_650.sh
  cat upgrade_to_650.sh | yum shell -y

• Single host from same machine (assumes the RPMs have been downloaded locally and are in the current folder). Run the following commands to perform the upgrade:

  echo 'upgrade moogsoft-common-6.5.0-116.x86_64.rpm' > upgrade_to_650.sh
  echo 'install moogsoft-integrations-6.5.0-19744.x86_64.rpm' >> upgrade_to_650.sh
  echo 'install moogsoft-integrations-ui-6.5.0-19744.x86_64.rpm' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-db-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-mooms-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-search-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-server-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-ui-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
  echo 'upgrade moogsoft-utils-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
  echo 'remove moogsoft-lams' >> upgrade_to_650.sh
  echo 'run' >> upgrade_to_650.sh
  cat upgrade_to_650.sh | yum shell -y

• Distributed installs: select the relevant update commands to run depending on which package(s) are installed on the current host. For example:
  
  

  1. HOST1 :

     echo 'upgrade moogsoft-common-6.5.0' > upgrade_to_650.sh
     echo 'upgrade moogsoft-server-6.5.0' >> upgrade_to_650.sh
     echo 'upgrade moogsoft-ui-6.5.0' >> upgrade_to_650.sh
     echo 'install moogsoft-integrations-ui-6.5.0' >> upgrade_to_650.sh
     echo 'run' >> upgrade_to_650.sh
     cat upgrade_to_650.sh | yum shell -y

  2. HOST2 :

     echo 'upgrade moogsoft-common-6.5.0' > upgrade_to_650.sh
     echo 'upgrade moogsoft-db-6.5.0' >> upgrade_to_650.sh
     echo 'upgrade moogsoft-search-6.5.0' >> upgrade_to_650.sh
     echo 'run' >> upgrade_to_650.sh
     cat upgrade_to_650.sh | yum shell -y

  3. HOST3 :

     echo 'upgrade moogsoft-common-6.5.0' > upgrade_to_650.sh
     echo 'install moogsoft-integrations-6.5.0' >> upgrade_to_650.sh
     echo 'remove moogsoft-lams' >> upgrade_to_650.sh
     echo 'upgrade moogsoft-mooms-6.5.0' >> upgrade_to_650.sh
     echo 'upgrade moogsoft-utils-6.5.0' >> upgrade_to_650.sh
     echo 'run' >> upgrade_to_650.sh
     cat upgrade_to_650.sh | yum shell -y

Merge Latest Config File Changes

'.rpmsave' versions of files will need to be manually merged/compared with the new versions of those files (any new properties should be added to the older versions of the files).

To find files which have been changed/moved/deleted run these commands:

find $MOOGSOFT_HOME -name '*.rpmsave'
find /etc/init.d/ -name '*.rpmsave'

An example command to see what differences are present in the $MOOGSOFT_HOME/config/system.conf file:

diff -u $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.rpmsave

An example of the process to merge the differences:

1. Rename the new versions of the files (without the .rpmsave extension) to end with '.bak'.
2. Merge the .rpmsave file with the new '.bak' file by adding new properties/config where needed so the structure matches the new version of the file.
3. Rename the .rpmsave file to delete the '.rpmsave' extension.

Restart ElasticSearch, RabbitMQ, and NGINX

The following commands need to be run to restart ElasticSearch, RabbitMQ, and nginx now that they have been upgraded:

service elasticsearch restart
service rabbitmq-server restart
service nginx restart

Upgrade MySQL and the AIOps schema

1. Update /etc/my.cnf with two new properties using the script below:

   sed -i 's/\(table-open-cache.*\)/\1\nmax_prepared_stmt_count = 1048576\ngroup_concat_max_len = 1048576/' /etc/my.cnf

2. Restart mysql:

   service mysqld restart

3. If the gtid-mode was OFF (based on the command run earlier in the upgrade), run the mysql upgrade utility (provide mysql root password when prompted or just press <return> if no password set), then restart mysql to ensure any changes to system tables are saved:

   mysql_upgrade -u root -p
   service mysqld restart

   More information on GTID and mysql_upgrade is here: https://dev.mysql.com/doc/refman/5.7/en/replication-gtids-restrictions.html#replication-gtids-restrictions-mysql_upgrade

   
   

4. Users upgrading from any of the following versions: v6.4.0.1, v6.4.0.2, or v6.4.0.3, must apply the upgrade to v6.4.1 before continuing to upgrade to v6.5.0 using the moog_db_auto_upgrader utility (step 6):

   1. Scripted (suitable for single host upgrade):

      $MOOGSOFT_HOME/bin/utils/moog_init_db.sh -m $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.4.0-6.4.1/641_moogdb_upgrade_all.sql -r $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.4.0-6.4.1/641_moog_reference_upgrade_all.sql

   2. OR Manual (replace 'moogdb' and 'moog_reference' database names as appropriate):

      mysql -h localhost -P 3306 -u root moogdb < $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.4.0-6.4.1/641_moogdb_upgrade_all.sql
      mysql -h localhost -P 3306 -u root moog_reference < $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.4.0-6.4.1/641_moog_reference_upgrade_all.sql

5. Upgrading the AIOps database requires providing the moog_db_auto_upgrader utility with the credentials of a database user with super privileges.

   For single-host installs where mysql was installed as part of the AIOps deployment, the default 'root' user can be used.

   The following command must be run after substituting the <MySQL-SuperUsername> argument. It will then ask for the password for that user. The password can also be provided to the utility with the ' -p' flag but this is not recommended in non-test deployments for security reasons.

   
   

   bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 6.5.0 -u <MySQL-SuperUsername>

Upgrade the Apache-Tomcat Webapps

The apache-tomcat service script has been updated in this version and the new script will need to be used.

If any customisations (e.g. an -Xmx change) have been made to the /etc/init.d/apache-tomcat script, please make a backup before continuing.

Replace the old service script with the new one using the following command:

cp -f $MOOGSOFT_HOME/etc/cots/tomcat/apache-tomcat.sh /etc/init.d/apache-tomcat 

If any customisations were made in the old script , they should be made to the new version of the script before continuing.

Now, perform the upgrade of the webapps:

Scripted (recommended for standard/single-host installs):

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w

• If tomcat startup/shutdown failures are reported (such as: Web apps are not rebuilding, please check catalina.out ) when running the above command then tomcat should be forcibly shutdown (using kill -9 on the apache-tomcat java PID) then the step repeated

Alternative manual steps:

1. Stop apache-tomcat: service apache-tomcat stop
2. Check the apache-tomcat java process is no longer running: service apache-tomcat status
3. Delete the existing extracted webapps: rm -rf /usr/share/apache-tomcat/webapps/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}
4. Copy in the new webapp WAR files (and replace the old ones): cp -f $MOOGSOFT_HOME/lib/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}.war $APPSERVER_HOME/webapps/
5. Restart apache-tomcat: service apache-tomcat start

Integrations

The following command needs to be run to extract Integrations and SolutionPaks:

bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN

Additionally, any of the following Ticketing integrations present before this upgrade will need to be un-installed and re-installed after the upgrade is complete:

• Cherwell
• JIRA Service Desk
• JIRA Software
• Remedy
• ServiceNow

Confirm all services are running and start moogfarmd

Check all required services are running and start moogfarmd by entering these commands:

service apache-tomcat status
service moogfarmd start
service nginx status
service elasticsearch status

Re-enable the events_analyser cronjobs using the following command:

(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -

Any previously running integrations can be restarted using the following command:

service moogstartupd restart

Elasticsearch Re-Index

The following command must be run on the moogsoft-search/Elasticsearch server to remove the old Elasticsearch indexes.

curl -XDELETE 'http://localhost:9200/alerts/' && curl -XDELETE 'http://localhost:9200/situations/'

If the command completes successfully, the following message should be shown on screen:

{"acknowledged":true}{"acknowledged":true}

To re-index the Alerts and Situations in moogfarmd run the following command (requires moogfarmd to be running):

$MOOGSOFT_HOME/bin/utils/moog_indexer --now

The re-index occurs in the background and may take a while to complete. This will depend on the number of Alerts and Situations in your system.

Java Keystore migration

As the Java runtime has been upgraded, any certificates stored in the old Java keystore need to be migrated into the new JAVA_HOME keystore.

If no certificates were manually added into the old store, no action is needed.

Verification

Manual Verification of Upgrade

Carry out the following basic steps to ensure the upgrade to 6.5.0 was successful:

• Ensure the $MOOGSOFT_HOME/etc/VERSION.txt file contains just " 6.5.0 "
• Check that the UI login page displays " Version 6.5.0 " at the top
• Check that the UI "Support Information" window correctly indicates the current version as 6.5.0 and shows the correct schema upgrade history
• Integrations : If any of the Integrations images are not displaying, please see the 'Integrations' section above and run the final optional step/command.

Automated Verification of Upgrade

Confirm all Moog files have been deployed correctly within $MOOGSOFT_HOME using this utility:

$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh

$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"