Page tree
Skip to end of metadata
Go to start of metadata

 

Clean installations of 6.2.1 need to follow the 'Pre-install' pages (children of this page) for your relevant OS distribution before installing the packages.

Backup Existing System

  1. Backup $MOOGSOFT_HOME.
  2. Take a snapshot (for VMs).
  3. Backup MySQL.

Stop Services and Processes

  1. moog_farmd: 

    service moogfarmd stop
  2. apache-tomcat: 

    service apache-tomcat stop
  3. LAMs. Either via their service scripts:

    service <lam_service_name> stop

    Alternatively, stop their running processes using:

    kill <pid>
  4. Ensure all the above processes have been stopped before continuing:

    service <service_name> status
  5. Disable the events_analyser and moog_indexer from running during the upgrade process:

    1. Run the following command to comment out the relevant lines in crontab:

      (crontab -l | sed -e 's/^\(.*moog_indexer.*\)$/#\1/' -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
    2. Run the following command to stop any running moog_indexer or events_analyser processes:

      ps -ef | grep java | egrep 'events_analyser|moog_indexer' | awk '{print $2}' | xargs kill 2>/dev/null

Prepare MySQL for the upgrade

Official instructions for upgrading from 5.6.26 to 5.7.19 are here: https://dev.mysql.com/doc/refman/5.7/en/upgrading.html and we summarise the steps below:

We recommend you perform a backup of the database before continuing.

Add the MySQL 5.7 yum repo package (and remove the 5.6 package first):

For RHEL6/CentOS6:

yum -y remove mysql-community-release
yum -y install http://repo.mysql.com/mysql57-community-release-el6.rpm

For RHEL7/CentOS7:

yum -y remove mysql-community-release
yum -y install http://repo.mysql.com/mysql57-community-release-el7.rpm

Stop the mysql service (change the password as appropriate):

mysql -u root --password='' --execute="SET GLOBAL innodb_fast_shutdown=0"
service mysqld stop

Upgrade AIOps

Upgrade Mechanism

  1. Single host from a remote repository:

    yum upgrade moogsoft-db-6.2.1 moogsoft-lams-6.2.1 moogsoft-mooms-6.2.1 moogsoft-search-6.2.1 moogsoft-server-6.2.1 moogsoft-ui-6.2.1 moogsoft-utils-6.2.1 moogsoft-common-6.2.1
  2. Single host from same machine (assumes the RPMs have been downloaded locally and are in the current folder):

    yum upgrade moogsoft-*6.2.1*.rpm
  3. Distributed installs: select the relevant update commands to run depending on which package(s) are installed on the current host. For example:

    1. HOST1:

       yum upgrade moogsoft-server-6.2.1 moogsoft-ui-6.2.1
    2. HOST2:

       yum upgrade moogsoft-db-6.2.1 moogsoft-search-6.2.1
    3. HOST3:

       yum upgrade moogsoft-lams-6.2.1 moogsoft-utils-6.2.1 moogsoft-mooms-6.2.1

Merge Latest Config File Changes

'.rpmsave' versions of files will need to be manually merged/compared with the new versions of those files (any new properties should be added to the older versions of the files).

The following files have been changed:

  • /usr/share/moogsoft/config/moog_farmd.conf
  • /usr/share/moogsoft/config/system.conf
  • /usr/share/moogsoft/bots/lambots/RestLam.js
  • /usr/share/moogsoft/config/rest_lam.conf

An example of the process to do this:

  1. Rename the new versions of the files (without the .rpmsave extension) to end with '.bak'.
  2. Merge the .rpmsave file with the new '.bak' file by adding new properties/config where needed so the structure matches the new version of the file.
  3. Rename the .rpmsave file to delete the '.rpmsave' extension.

Upgrade MySQL and the AIOps schema

  1. Prepare the system for MySQL 5.7: 

    sed -i 's/myisam-recover\s*=\s*FORCE,BACKUP/myisam-recover-options = FORCE,BACKUP/' /etc/my.cnf;
    sed -i 's/user\s*=\s*mysql/user = mysql\nsql-mode = NO_ENGINE_SUBSTITUTION/' /etc/my.cnf;
    mkdir /var/log/mysql;
    touch /var/log/mysql/mysql-error.log;
    touch /var/log/mysql/mysql-slow.log;
    chown -R mysql:mysql /var/log/mysql;
  2. Restart the mysql service: 

    service mysqld restart
  3. If mysql is NOT configured to run with --gtid-mode=ON (regular mysql installs do not have this enabled, but Percona/Galera may do), run the mysql upgrade utility (provide mysql root password when prompted or just press <return> if no password set): 

    mysql_upgrade -u root -p

    More information on GTID and mysql_upgrade is here: https://dev.mysql.com/doc/refman/5.7/en/replication-gtids-restrictions.html#replication-gtids-restrictions-mysql_upgrade


  4. Restart mysql to ensure any changes to system tables are saved: 

    service mysqld restart
  5. Apply the AIOps schema changes:

    • Scripted (suitable for single host upgrade): 

      $MOOGSOFT_HOME/bin/utils/moog_init_db.sh -m $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.2.0_ESR6-6.2.1/621_moogdb_upgrade_all.sql -r $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.2.0_ESR6-6.2.1/621_moog_reference_upgrade_all.sql
    • OR Manual (replace 'moogdb' and 'moog_reference' database names as appropriate): 

      mysql -h localhost -P 3306 -u root moogdb < $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.2.0_ESR6-6.2.1/621_moogdb_upgrade_all.sql
      mysql -h localhost -P 3306 -u root moog_reference < $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.2.0_ESR6-6.2.1/621_moog_reference_upgrade_all.sql

Upgrade the Apache-Tomcat Webapps

Stop apache-tomcat:

service apache-tomcat stop

Check the tomcat process has stopped before continuing. If the service/systemctl command hangs or reports that the process is still running, you will need to kill the apache-tomcat java process manually.

Scripted (recommended for standard/single-host installs): 

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  • If tomcat startup/shutdown failures are reported (such as: Web apps are not rebuilding, please check catalina.out) when running the above command then tomcat should be forcibly shutdown (using kill -9 on the apache-tomcat java PID) then the step repeated

Alternative manual steps:

  1. Stop apache-tomcat: service apache-tomcat stop
  2. Check the apache-tomcat java process is no longer running: service apache-tomcat status
  3. Delete the existing extracted webapps: rm -rf /usr/share/apache-tomcat/webapps/{moogpoller,moogsvr,toolrunner,graze,events}
  4. Copy in the new webapp WAR files (and replace the old ones): cp -f $MOOGSOFT_HOME/lib/{moogpoller,moogsvr,toolrunner,graze,events}.war $APPSERVER_HOME/webapps/
  5. Restart apache-tomcat: service apache-tomcat start

Integrations

The following command needs to be run to extract Integrations and SolutionPaks:

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -x


OPTIONAL
: If the images for some Integrations are not visible post-upgrade, the following script can be run to replace the broken image links in the database:

bash $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.1.0_ESR5-6.2.0_ESR6/fix_integration_tmp_image_paths.sh

Confirm all services are running and start moogfarmd

Check all required services are running and start moogfarmd by entering these commands:

service apache-tomcat status
service moogfarmd start
service nginx status
service elasticsearch status

Re-enable the events_analyser cronjobs using the following command:

(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -

Please note that this command does NOT re-enable the crontab lines for the moog_indexer because of the new mechanism by which Indexing works in 6.2.1 (please see the Release Notes for more information).
Any previously running integrations can be restarted using the following command:

service moogstartupd restart

Verification

Manual Verification of Upgrade

Carry out the following basic steps to ensure the upgrade to 6.2.1 was successful:

  • Ensure the $MOOGSOFT_HOME/etc/VERSION.txt file contains just "6.2.1"
  • Check that the last entry in the moogdb.schema_upgrades table in MySQL is the 6.2.1 upgrade entry
  • Check that the UI login page displays "Version 6.2.1" at the top
  • Check that the UI "Support Information" window correctly indicates the current version as 6.2.1 and shows the correct schema upgrade history
  • Integrations: If any of the Integrations images are not displaying, please see the 'Integrations' section above and run the final optional step/command.

Automated Verification of Upgrade

Confirm all Moog files have been deployed correctly within $MOOGSOFT_HOME using this utility:


$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Confirm all apache-tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:


$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh

Confirm that all the steps are successful. If there are some webapp differences then this can be resolved using moog_init_ui.sh -w which will extract the webapps with the right files.

If there are only hostname-related differences in the apache-tomcat server.xml file then these can be safely ignored.

Confirm the schema has been upgraded successfully:


$MOOGSOFT_HOME/bin/utils/mysql_install_validator.sh

Confirm that all the steps are successful. For the current release (6.2.1) we are aware that a number of differences around the keyword 'DEFAULT' will be highlighted by the mysql_install_validator.sh in upgrade scenarios. These can be ignored. A future edge release will fix these differences.

If there are some schema differences, they may be valid (e.g.: custom_info related) but if there are more substantial differences then it would be worth investigating further to check if all the pre-requisite upgrade scripts have been applied in the right order:


mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have an issues please refer to Troubleshooting.

  • No labels