Page tree
Skip to end of metadata
Go to start of metadata

We do not support or recommend the upgrade of a production AIOps deployment to an Edge release.

Backup Existing System

  1. Backup $MOOGSOFT_HOME.
  2. Take a snapshot (for VMs).
  3. Backup MySQL.

Stop Services and Processes

  1. moog_farmd: 

    service moogfarmd stop
  2. apache-tomcat: 

    service apache-tomcat stop
  3. LAMs. This can be done by stopping their service scripts:

    service <lam_service_name> stop

    Alternatively, stop their running processes using:

    kill <pid>
  4.  Ensure all the above processes have been stopped before continuing:

    service <service_name> status
  5. Disable the events_analyser from running during the upgrade process:

    1. Run the following command to comment out the relevant lines in crontab:

      (crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
    2. Run the following command to stop any running events_analyser processes:

      ps -ef | grep java | egrep 'events_analyser|moog_indexer' | awk '{print $2}' | xargs kill 2>/dev/null

Upgrade Erlang

Important: Stop all RabbitMQ nodes before performing this step: 

service rabbitmq-server stop
  • If you are on RHEL6/CentOS6, run the following commands to switch the current version of Erlang to v20.1.7. For offline upgrades see the information panel at the bottom of this section: 

    echo 'remove erlang-*' > erlang.sh
    echo 'install https://github.com/rabbitmq/erlang-rpm/releases/download/v20.1.7/erlang-20.1.7-1.el6.x86_64.rpm' >> erlang.sh
    echo 'run' >> erlang.sh
    
    cat erlang.sh | yum shell -y
  • If you are on RHEL7/CentOS7 please run the following commands to switch the current version of erlang to v20.1.7. For offline upgrades see the information panel at the bottom of this section: 

    echo 'remove erlang-*' > erlang.sh
    echo 'install https://github.com/rabbitmq/erlang-rpm/releases/download/v20.1.7/erlang-20.1.7-1.el7.centos.x86_64.rpm' >> erlang.sh
    echo 'run' >> erlang.sh
    
    cat erlang.sh | yum shell -y

    If the upgrade is being done using the offline tarballs, replace the URL to the erlang RPM in the erlang.sh script with 'erlang-20.1.7'

Upgrade AIOps

Target the AIOps Edge yum Repository

To upgrade to an AIOps Edge Release, you must configure the baseurl AIOps yum repository to point to the edges endpoint:

  1. Edit /etc/yum.repos.d/moogsoft-aiops.repo.
  2. Configure the baseurl as follows:
    baseurl=https://<login>:<password>@speedy.moogsoft.com/repo/aiops/edge

    Contact support@moogsoft.com with questions rearding regarding access to speedy.moogsoft.com.

Upgrade Mechanism

  1. Single host from a remote repository:

    yum upgrade moogsoft-db-6.3.1 \
                moogsoft-lams-6.3.1 \
                moogsoft-mooms-6.3.1 \
                moogsoft-search-6.3.1 \
                moogsoft-server-6.3.1 \
                moogsoft-ui-6.3.1 \
                moogsoft-utils-6.3.1 \
                moogsoft-common-6.3.1
  2. Single host from same machine (assumes the RPMs have been downloaded locally and are in the current folder):

    yum upgrade moogsoft-*6.3.1*.rpm
  3. Distributed installs: select the relevant update commands to run depending on which package(s) are installed on the current host. For example:

    1. HOST1:

       yum upgrade moogsoft-server-6.3.1 moogsoft-ui-6.3.1
    2. HOST2:

       yum upgrade moogsoft-db-6.3.1 moogsoft-search-6.3.1
    3. HOST3:

       yum upgrade moogsoft-lams-6.3.1 moogsoft-utils-6.3.1 moogsoft-mooms-6.3.1

Restart RabbitMQ:

service rabbitmq-server start

Merge Latest Config File Changes

'.rpmsave' versions of files will need to be manually merged/compared with the new versions of those files (any new properties should be added to the older versions of the files).

An example of the process to do this:

  1. Rename the new versions of the files (without the .rpmsave extension) to end with '.bak'.
  2. Merge the .rpmsave file with the new '.bak' file by adding new properties/config where needed so the structure matches the new version of the file.
  3. Rename the .rpmsave file to delete the '.rpmsave' extension.

[Optional - but recommended] Update /etc/my.cnf with new guidance on innodb-buffer-pool performance tuning:

echo "# Important: innodb-buffer-pool performance tuning" > tmpfile.txt
echo "# On servers with >= 16GB RAM that run MySQL and Moog applications, consider setting" >> tmpfile.txt
echo "# On servers where only MySQL is running, consider setting innodb-buffer-pool-size" >> tmpfile.txt
echo "# to 80% of system RAM." >> tmpfile.txt
echo "# If innodb-buffer-pool-size > 8G then increase innodb_buffer_pool_instances from its" >> tmpfile.txt
echo "# default of 8 to divide buffer pool into 1G chunks " >> tmpfile.txt
echo "# e.g: innodb-buffer-pool-size=64G innodb_buffer_pool_instances=64" >> tmpfile.txt
echo "#innodb_buffer_pool_instances   = 8" >> tmpfile.txt
sed -i  '/innodb-buffer-pool-size*/r tmpfile.txt' /etc/my.cnf
rm -f tmpfile.txt


Upgrade the AIOps schema 

  • Scripted (suitable for single host upgrade): 

    $MOOGSOFT_HOME/bin/utils/moog_init_db.sh -m $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.3.0-6.3.1/631_moogdb_upgrade_all.sql -r $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.3.0-6.3.1/631_moog_reference_upgrade_all.sql
  • OR Manual (replace 'moogdb' and 'moog_reference' database names as appropriate): 

    mysql -h localhost -P 3306 -u root moogdb < $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.3.0-6.3.1/631_moogdb_upgrade_all.sql
    mysql -h localhost -P 3306 -u root moog_reference < $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.3.0-6.3.1/631_moog_reference_upgrade_all.sql

Upgrade the Apache-Tomcat Webapps

Scripted (recommended for standard/single-host installs): 

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  • If tomcat startup/shutdown failures are reported (such as: Web apps are not rebuilding, please check catalina.out) when running the above command then tomcat should be forcibly shutdown (using kill -9 on the apache-tomcat java PID) then the step repeated

Alternative manual steps:

  1. Stop apache-tomcat: 

    service apache-tomcat stop
  2. Check the apache-tomcat java process is no longer running: 

    service apache-tomcat status
  3. Delete the existing extracted webapps: 

    rm -rf /usr/share/apache-tomcat/webapps/{moogpoller,moogsvr,toolrunner,graze,events}
  4. Copy in the new webapp WAR files (and replace the old ones):

    cp -f $MOOGSOFT_HOME/lib/{moogpoller,moogsvr,toolrunner,graze,events}.war $APPSERVER_HOME/webapps/
  5. Restart apache-tomcat:

    service apache-tomcat start

Integrations

The following command needs to be run to extract Integrations and SolutionPaks:

bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN

OPTIONAL: If the images for some Integrations are not visible post-upgrade, the following script can be run to replace the broken image links in the database:

bash $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.1.0_ESR5-6.2.0_ESR6/fix_integration_tmp_image_paths.sh

Restart moogfarmd and Verify Services

Check all required services are running and start moogfarmd by entering these commands:

service apache-tomcat status
service moogfarmd start
service nginx status
service elasticsearch status

Re-enable the events_analyser cronjobs using the following command:

(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -

Any previously running integrations can be restarted using the following command:

service moogstartupd restart

Verification

Manual Verification of Upgrade

Carry out the following basic steps to ensure the upgrade to 6.3.1 was successful:

  • Ensure the $MOOGSOFT_HOME/etc/VERSION.txt file contains just "6.3.1"
  • Check that the last entry in the moogdb.schema_upgrades table in MySQL is the 6.3.1 upgrade entry
  • Check that the UI login page displays "Version 6.3.1" at the top
  • Check that the UI "Support Information" window correctly indicates the current version as 6.3.1 and shows the correct schema upgrade history
  • Integrations: If any of the Integrations images are not displaying, please see the 'Integrations' section above and run the final optional step/command.

Automated Verification of Upgrade

Confirm all Moog files have been deployed correctly within $MOOGSOFT_HOME using this utility:


$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Confirm all apache-tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:


$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh

Confirm that all the steps are successful. If there are some webapp differences then this can be resolved using moog_init_ui.sh -w which will extract the webapps with the right files.

If there are only hostname-related differences in the apache-tomcat server.xml file then these can be safely ignored.

Confirm the schema has been upgraded successfully:


$MOOGSOFT_HOME/bin/utils/mysql_install_validator.sh

Confirm that all the steps are successful.

If there are some schema differences, they may be valid (e.g.: custom_info related) but if there are more substantial differences then it would be worth investigating further to check if all the pre-requisite upgrade scripts have been applied in the right order:


mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have an issues please refer to Troubleshooting.
  • No labels