Page tree
Skip to end of metadata
Go to start of metadata

Important

This upgrade involves moving to a newer version of RabbitMQ

Backup your existing system

  1. Backup $MOOGSOFT_HOME.
  2. Take a snapshot (for VMs).
  3. Backup MySQL.

Stop Services and Processes

  1. moog_farmd: 
    • service moogfarmd stop
  2. apache-tomcat: 
    • service apache-tomcat stop
  3. LAMs: Either via their service scripts:
    • service <lam_service_name> stop
    • or by stopping their running processes using:
      • kill <pid>
  4. Ensure all the above processes have been stopped before continuing. 
    • service <service_name> status
  5. Disable the events_analyser and moog_indexer from running during the upgrade process:
    1. Run the following command to comment out the relevant lines in crontab:

      (crontab -l | sed -e 's/^\(.*moog_indexer.*\)$/#\1/' -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
    2. Run the following command to stop any running moog_indexer or events_analyser processes:

      ps -ef | grep java | egrep 'events_analyser|moog_indexer' | awk '{print $2}' | xargs kill 2>/dev/null

Upgrade Rabbit-MQ server


To upgrade a RabbitMQ cluster, follow the instructions in RabbitMQ documentation (http://www.rabbitmq.com/clustering.html#upgrading)

If RabbitMQ is configured in a clustered configuration across multiple hosts/nodes, all nodes must be stopped and upgraded at the same time


  1. Stop the RabbitMQ service (if running):


    service rabbitmq-server stop
  2. Add the official RabbitMQ yum repo:

    curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash


  3. Install the new version of RabbitMQ: 


    yum install rabbitmq-server-3.6.10
  4. Restart the RabbitMQ service:


    service rabbitmq-server start

Upgrade AIOps

Upgrade mechanism

  1. Single host from a remote repository:

    yum upgrade moogsoft-db-6.1.3 moogsoft-lams-6.1.3 moogsoft-mooms-6.1.3 moogsoft-search-6.1.3 moogsoft-server-6.1.3 moogsoft-ui-6.1.3 moogsoft-utils-6.1.3 moogsoft-common-6.1.3
  2. Single host from same machine (assumes the RPMs have been downloaded locally and are in the current folder):

    yum upgrade moogsoft-*6.1.3*.rpm
  3. Distributed installs: select the relevant update commands to run depending on which package(s) are installed on the current host. 
    For example:
    1. HOST1:

       yum upgrade moogsoft-server-6.1.3 moogsoft-ui-6.1.3
    2. HOST2:

       yum upgrade moogsoft-db-6.1.3 moogsoft-search-6.1.3
    3. HOST3:

       yum upgrade moogsoft-lams-6.1.3 moogsoft-utils-6.1.3 moogsoft-mooms-6.1.3

Merge latest config file changes

'.rpmsave' versions of files will need to be manually merged/compared with the new versions of those files (any new properties should be added to the older versions of the files).

An example of the process to do this:

  • Rename the new versions of the files (without the .rpmsave extension) to end with '.bak'
  • Merge the .rpmsave file with the new '.bak' file by adding new properties/config where needed so the structure matches the new version of the file
  • Rename the .rpmsave file to delete the '.rpmsave' extension

The 'configdiff' functionality is still available in the init scripts for helping merge system.conf, security.conf, and moogfarmd.conf if desired (only available in the moogsoft-server package):

$MOOGSOFT_HOME/bin/utils/moog_init_server.sh --configdiff

Upgrade the database 

  • Scripted (suitable for single host upgrade using the generic automated script):

    $MOOGSOFT_HOME/bin/utils/moog_init_db.sh -m /usr/share/moogsoft/etc/moog/schema_upgrades/6.1.2-6.1.3/613_moogdb_upgrade_all.sql -r /usr/share/moogsoft/etc/moog/schema_upgrades/6.1.2-6.1.3/613_moog_reference_upgrade_all.sql 
  • Manual (replace moogdb and moog_reference database names as appropriate):

    mysql -h localhost -P 3306 -u root moogdb < $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.1.2-6.1.3/613_moogdb_upgrade_all.sql
    mysql -h localhost -P 3306 -u root moog_reference < $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.1.2-6.1.3/613_moog_reference_upgrade_all.sql
    cat <(echo "SET FOREIGN_KEY_CHECKS=0; ") $MOOGSOFT_HOME/etc/moog/moog_sigdb/stored_procedures/*.sql | mysql -h localhost -P 3306 -u root moogdb
    cat <(echo "SET FOREIGN_KEY_CHECKS=0; ") $MOOGSOFT_HOME/etc/moog/moog_sigdb/triggers/*.sql | mysql -h localhost -P 3306 -u root moogdb
    cat <(echo "SET FOREIGN_KEY_CHECKS=0; ") $MOOGSOFT_HOME/etc/moog/moog_referencedb/stored_procedures/*.sql | mysql -h localhost -P 3306 -u root moog_reference

Run the data migration utility

Run the following command to migrate system config data into V6.1.3:

$MOOGSOFT_HOME/etc/moog/schema_upgrades/6.1.2-6.1.3/613_data_migrator

The migration utility updates the 'Default Cookbook recipes' as follows:

  • If the Default Cookbook or any of the contained Recipes have been edited in any way, they will not be changed by this utility.
  • If the Default Cookbook or any of the contained Recipes have not been edited in any way, the Recipes will be updated to remove their 'Exclusion filter' and an additional Clustering field will be added to each one: Agent matching 100%

When the utility is run, the output should look like this:

Starting V6.1.3 Data Migration...

    Starting migration of system config...

    ... Successfully migrated system config

... Successfully finished V6.1.3 Data Migration


Upgrade the Apache-Tomcat webapps

Stop apache-tomcat:

service apache-tomcat stop

Check the tomcat process has stopped before continuing:

service apache-tomcat status
  • If the service/systemctl command hangs on RHEL7, you will need to kill the apache-tomcat java process manually

Scripted (recommended for standard/single-host installs): 

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  • If tomcat startup/shutdown failures are reported (such as: Web apps are not rebuilding, please check catalina.out) when running the above command then tomcat should be forcibly shutdown (using kill -9 on the apache-tomcat java PID) then the step repeated

Alternative manual steps:

  • Stop apache-tomcat: service apache-tomcat stop
  • Check the apache-tomcat java process is no longer running: service apache-tomcat status
  • Delete the existing extracted webapps: rm -rf /usr/share/apache-tomcat/webapps/{moogpoller,moogsvr,toolrunner,graze,events}
  • Copy in the new webapp WAR files (and replace the old ones): cp -f $MOOGSOFT_HOME/lib/{moogpoller,moogsvr,toolrunner,graze,events}.war $APPSERVER_HOME/webapps/
  • Restart apache-tomcat: service apache-tomcat start

Integrations

The following command needs to be run to extract Integrations and SolutionPaks:

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -x


Confirm all required services are running and start moogfarmd

service apache-tomcat status
service moogfarmd start
service nginx status
service elasticsearch status

Important: Before using the UI it is important to clear out your browser cache and local storage - otherwise the browser may be using old cached versions of files which will lead to problems.

Re-enable the moog_indexer and events_analyser cronjobs using the following command:

(crontab -l | sed -e 's/^\#\+\(.*moog_indexer.*\)/\1/' -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -

Verification

Manual Verification of Upgrade

Carry out the following basic steps to ensure the upgrade to 6.1.3 was successful:

  • Ensure the $MOOGSOFT_HOME/etc/VERSION.txt file contains just "6.1.3"
  • Check that the last entry in the moogdb.schema_upgrades table in MySQL is the 6.1.3 upgrade entry
  • Check that the UI login page displays "Version 6.1.3" at the top
  • Check that the UI "Support Information" window correctly indicates the current version as 6.1.3 and shows the correct schema upgrade history

Automated Verification of Upgrade

Confirm all Moog files have been deployed correctly within $MOOGSOFT_HOME using this utility:


$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Confirm all apache-tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:


$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh

Confirm that all the steps are successful. If there are some webapp differences then this can be resolved using moog_init_ui.sh -w which will extract the webapps with the right files:


$MOOGSOFT_HOME/bin/utils/mysql_install_validator.sh


Confirm that all the steps are successful. If there are some schema differences, they may be valid (e.g.: custom_info, or 'view' related) but if there are more substantial differences then it would be worth investigating further to check if all the pre-requisite upgrade scripts have been applied in the right order:


mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"
  • No labels