Page tree
Skip to end of metadata
Go to start of metadata


Important Notice

Please note:

  1. Users must first upgrade to V6.0.0, V6.1.0, or V6.2.0 before they can upgrade to V6.3.0
  2. This release is considered an 'Edge' release so please ensure the Moogsoft yum repository is configured to point at 'edge' instead of 'esr'

Backup your existing system

  1. Backup $MOOGSOFT_HOME.
  2. Take a snapshot (for VMs).
  3. Backup MySQL.

Stop Services and Processes

  1. moog_farmd: 

    service moogfarmd stop
  2. apache-tomcat: 

    service apache-tomcat stop
  3. LAMs. 

    Either via their service scripts:

    service <lam_service_name> stop

    Alternatively, stop their running processes using:

    kill <pid>
  4. Ensure all the above processes have been stopped before continuing:

    service <service_name> status
  5. Disable the events_analyser and moog_indexer from running during the upgrade process:

    1. Run the following command to comment out the relevant lines in crontab:

      (crontab -l | sed -e 's/^\(.*moog_indexer.*\)$/#\1/' -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
    2. Run the following command to stop any running moog_indexer or events_analyser processes:

      ps -ef | grep java | egrep 'events_analyser|moog_indexer' | awk '{print $2}' | xargs kill 2>/dev/null

Run Manual Delta Index

In order to support a change to our moog_indexer, we require that all alerts and situations are indexed before continuing the upgrade. Please run the following command (add the -p flag if Private Teams are being used):

$MOOGSOFT_HOME/bin/utils/moog_indexer -b 1000 -t 5

...to perform a final delta index. MySQL and ElasticSearch should obviously be running for this to be successful.

Prepare MySQL for the upgrade

Official instructions for upgrading from 5.6.26 to 5.7.19 are here: https://dev.mysql.com/doc/refman/5.7/en/upgrading.html and we summarise the steps below:

We recommend you perform a backup of the database before continuing.

Add the MySQL 5.7 yum repo package (and remove the 5.6 package first):

For RHEL6/CentOS6:


yum -y remove mysql-community-release
yum -y install http://repo.mysql.com/mysql57-community-release-el6.rpm


For RHEL7/CentOS7:


yum -y remove mysql-community-release
yum -y install http://repo.mysql.com/mysql57-community-release-el7.rpm


Stop the mysql service (change the password as appropriate):

mysql -u root --password='' --execute="SET GLOBAL innodb_fast_shutdown=0"
service mysqld stop

Upgrade erlang

If you are on RHEL6/CentOS6 please run the following commands to switch the current version of erlang to v20.1.4 (for offline upgrades see the info block at the bottom of this section):

echo 'remove erlang-*' > erlang.sh
echo 'install https://github.com/rabbitmq/erlang-rpm/releases/download/v20.1.4/erlang-20.1.4-1.el6.x86_64.rpm' >> erlang.sh
echo 'run' >> erlang.sh

cat erlang.sh | yum shell -y


If you are on RHEL7/CentOS7 please run the following commands to switch the current version of erlang to v20.1.4 (for offline upgrades see the info block at the bottom of this section):

echo 'remove erlang-*' > erlang.sh
echo 'install https://github.com/rabbitmq/erlang-rpm/releases/download/v20.1.4/erlang-20.1.4-1.el7.centos.x86_64.rpm' >> erlang.sh
echo 'run' >> erlang.sh

cat erlang.sh | yum shell -y

If the upgrade is being done using the offline tarballs, replace the URL to the erlang RPM in the erlang.sh script with 'erlang-20.1.4'


Upgrade RabbitMQ Server

To upgrade a RabbitMQ cluster, follow the instructions in RabbitMQ documentation (http://www.rabbitmq.com/clustering.html#upgrading)

If RabbitMQ is configured in a clustered configuration across multiple hosts/nodes, all nodes must be stopped and upgraded at the same time.

  1. Stop the RabbitMQ service (if running): 

    service rabbitmq-server stop
  2. Add the official RabbitMQ yum repo: 

    curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash


  3. Install the new version of RabbitMQ: 

    yum -y install rabbitmq-server-3.6.12
  4. Restart the RabbitMQ service: 

    service rabbitmq-server start

Create the nginx yum repo file

Install the nginx yum repo by creating a script (e.g.: create_nginx_repo.sh) containing the following contents and then executing it. e.g.: bash create_nginx_repo.sh

#!/bin/bash
 
echo '[nginx]' > /etc/yum.repos.d/nginx.repo
echo 'name=nginx repo' >> /etc/yum.repos.d/nginx.repo
echo 'baseurl=http://nginx.org/packages/OS/OSRELEASE/$basearch/' >> /etc/yum.repos.d/nginx.repo
echo 'gpgcheck=0' >> /etc/yum.repos.d/nginx.repo
echo 'enabled=1' >> /etc/yum.repos.d/nginx.repo
 
OS_VERSION=$(cat /etc/system-release)
case "$OS_VERSION" in
    CentOS*release\ 6* )
            sed -i -e 's/OS/centos/' -e 's/OSRELEASE/6/' /etc/yum.repos.d/nginx.repo;;
    CentOS*release\ 7* )
            sed -i -e 's/OS/centos/' -e 's/OSRELEASE/7/' /etc/yum.repos.d/nginx.repo;;
    Red\ Hat*release\ 6* )
            sed -i -e 's/OS/rhel/' -e 's/OSRELEASE/6/' /etc/yum.repos.d/nginx.repo;;
    Red\ Hat*release\ 7* )
            sed -i -e 's/OS/rhel/' -e 's/OSRELEASE/7/' /etc/yum.repos.d/nginx.repo;;
esac

Upgrade nginx

The following commands need to be run to upgrade nginx to the latest version (1.12.2):

echo 'remove nginx-filesystem' > upgrade_nginx.sh
echo 'remove nginx-all-modules' >> upgrade_nginx.sh
echo 'remove nginx-mod-http-xslt-filter' >> upgrade_nginx.sh
echo 'remove nginx-mod-http-image-filter' >> upgrade_nginx.sh
echo 'remove nginx-mod-http-perl' >> upgrade_nginx.sh
echo 'remove nginx-mod-http-geoip' >> upgrade_nginx.sh
echo 'remove nginx-mod-stream' >> upgrade_nginx.sh
echo 'remove nginx-mod-mail' >> upgrade_nginx.sh
echo 'install nginx-module-xslt-1.12.2' >> upgrade_nginx.sh
echo 'install nginx-module-image-filter-1.12.2' >> upgrade_nginx.sh
echo 'install nginx-module-perl-1.12.2' >> upgrade_nginx.sh
echo 'install nginx-module-geoip-1.12.2' >> upgrade_nginx.sh
echo 'upgrade nginx-1.12.2' >> upgrade_nginx.sh
echo 'run' >> upgrade_nginx.sh
cat upgrade_nginx.sh | yum shell -y

Then restart nginx:

service nginx restart

Upgrade AIOps

Upgrade Mechanism

Please note that there is an additional moogsoft package (moogsoft-common) which is required by AIOps v6.3.0. As a result, it is required to perform an install of that package and an upgrade of the other packages at the same time.

This is performed using a 'yum shell' script.


  1. Single host from a remote repository. Please run the following commands to perform the upgrade:

    echo 'install moogsoft-common-6.3.0' > upgrade_to_630.sh
    echo 'upgrade moogsoft-db-6.3.0' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-eula-20171109' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-lams-6.3.0' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-mooms-6.3.0' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-search-6.3.0' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-server-6.3.0' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-ui-6.3.0' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-utils-6.3.0' >> upgrade_to_630.sh
    echo 'run' >> upgrade_to_630.sh
    cat upgrade_to_630.sh | yum shell -y
  2. Single host from same machine (assumes the RPMs have been downloaded locally and are in the current folder). Please run the following commands to perform the upgrade:

    echo 'install moogsoft-common-6.3.0-beta.23.x86_64.rpm' > upgrade_to_630.sh
    echo 'upgrade moogsoft-eula-20171109-beta.23.noarch.rpm' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-lams-6.3.0-beta.23.x86_64.rpm' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-mooms-6.3.0-beta.23.x86_64.rpm' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-search-6.3.0-beta.23.x86_64.rpm' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-server-6.3.0-beta.23.x86_64.rpm' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-ui-6.3.0-beta.23.x86_64.rpm' >> upgrade_to_630.sh
    echo 'upgrade moogsoft-utils-6.3.0-beta.23.x86_64.rpm' >> upgrade_to_630.sh
    echo 'run' >> upgrade_to_630.sh
    cat upgrade_to_630.sh | yum shell -y
  3. Distributed installs: select the relevant update commands to run depending on which package(s) are installed on the current host. For example:

    1. HOST1:

      echo 'install moogsoft-common-6.3.0' > upgrade_to_630.sh
      echo 'upgrade moogsoft-eula-20171109' >> upgrade_to_630.sh
      echo 'upgrade moogsoft-server-6.3.0' >> upgrade_to_630.sh
      echo 'upgrade moogsoft-ui-6.3.0' >> upgrade_to_630.sh
      echo 'run' >> upgrade_to_630.sh
      cat upgrade_to_630.sh | yum shell -y
    2. HOST2:

      echo 'install moogsoft-common-6.3.0' > upgrade_to_630.sh
      echo 'upgrade moogsoft-db-6.3.0' >> upgrade_to_630.sh
      echo 'upgrade moogsoft-eula-20171109' >> upgrade_to_630.sh
      echo 'upgrade moogsoft-search-6.3.0' >> upgrade_to_630.sh
      echo 'run' >> upgrade_to_630.sh
      cat upgrade_to_630.sh | yum shell -y
    3. HOST3:

      echo 'install moogsoft-common-6.3.0' > upgrade_to_630.sh
      echo 'upgrade moogsoft-eula-20171109' >> upgrade_to_630.sh
      echo 'upgrade moogsoft-lams-6.3.0' >> upgrade_to_630.sh
      echo 'upgrade moogsoft-mooms-6.3.0' >> upgrade_to_630.sh
      echo 'upgrade moogsoft-utils-6.3.0' >> upgrade_to_630.sh
      echo 'run' >> upgrade_to_630.sh
      cat upgrade_to_630.sh | yum shell -y

Merge Latest Config File Changes

'.rpmsave' versions of files will need to be manually merged/compared with the new versions of those files (any new properties should be added to the older versions of the files).

To find files which have been changed/moved/deleted run these command: 

find $MOOGSOFT_HOME -name '*.rpmsave'
find /etc/init.d/ -name '*.rpmsave'


An example of the process to merge the differences:

  1. Rename the new versions of the files (without the .rpmsave extension) to end with '.bak'.
  2. Merge the .rpmsave file with the new '.bak' file by adding new properties/config where needed so the structure matches the new version of the file.
  3. Rename the .rpmsave file to delete the '.rpmsave' extension.

Upgrade MySQL and the AIOps schema

  1. Prepare the system for MySQL 5.7: 

    sed -i 's/myisam-recover\s*=\s*FORCE,BACKUP/myisam-recover-options = FORCE,BACKUP/' /etc/my.cnf;
    sed -i 's/user\s*=\s*mysql/user = mysql\nsql-mode = NO_ENGINE_SUBSTITUTION/' /etc/my.cnf;
    sed -i 's/max-allowed-packet\s*=\s*16M/max-allowed-packet = 128M/' /etc/my.cnf
    sed -i 's/#innodb-log-file-size\s*=\s*128M/innodb-log-file-size = 1024M/' /etc/my.cnf
    mkdir /var/log/mysql;
    touch /var/log/mysql/mysql-error.log;
    touch /var/log/mysql/mysql-slow.log;
    chown -R mysql:mysql /var/log/mysql;
  2. Restart the mysql service: 

    service mysqld restart
  3. If mysql is NOT configured to run with --gtid-mode=ON (regular mysql installs do not have this enabled, but Percona/Galera may do), run the mysql upgrade utility (provide mysql root password when prompted or just press <return> if no password set): 

    mysql_upgrade -u root -p

    More information on GTID and mysql_upgrade is here: https://dev.mysql.com/doc/refman/5.7/en/replication-gtids-restrictions.html#replication-gtids-restrictions-mysql_upgrade


  4. Restart mysql to ensure any changes to system tables are saved: 

    service mysqld restart
  5. Now apply the SQL schema upgrades:

    bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 6.3.0

Upgrade Apache-tomcat configuration files

The following lines need to be run to update the $APPSERVER_HOME/conf/context.xml file:

sed -i $'s;</Context>;<Resources className="org.apache.catalina.webresources.StandardRoot">\\\n<JarResources className="org.apache.catalina.webresources.DirResourceSet"\\\nbase="MOOGSOFT_HOME/lib"\\\nwebAppMount="/WEB-INF/lib">\\\n</JarResources>\\\n</Resources>\\\n\\\n</Context>;g' $APPSERVER_HOME/conf/context.xml
sed -i "s;MOOGSOFT_HOME;${MOOGSOFT_HOME};g"  $APPSERVER_HOME/conf/context.xml
LINE_NUM=$(cat -nu $APPSERVER_HOME/conf/context.xml | grep '<Resources' | tail -1 | awk '{print $1}'); LINE_NUM=$((LINE_NUM-=1))
if [[ $(awk "NR==${LINE_NUM}" $APPSERVER_HOME/conf/context.xml) =~ ^\s*$ ]]; then sed -i "${LINE_NUM}d" $APPSERVER_HOME/conf/context.xml; fi

Or add these lines into $APPSERVER_HOME/conf/context.xml:

<Resources className="org.apache.catalina.webresources.StandardRoot">
	<JarResources className="org.apache.catalina.webresources.DirResourceSet"
		base="/usr/share/moogsoft/lib"
		webAppMount="/WEB-INF/lib">
	</JarResources>
</Resources>

Upgrade the Apache-Tomcat webapps

Stop apache-tomcat:

service apache-tomcat stop

Check the tomcat process has stopped before continuing. If the service/systemctl command hangs or reports that the process is still running, you will need to kill the apache-tomcat java process manually.

Scripted (recommended for standard/single-host installs): 

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  • If tomcat startup/shutdown failures are reported (such as: Web apps are not rebuilding, please check catalina.out) when running the above command then tomcat should be forcibly shutdown (using kill -9 on the apache-tomcat java PID) then the step repeated

Alternative manual steps:

  • Stop apache-tomcat: service apache-tomcat stop
  • Check the apache-tomcat java process is no longer running: service apache-tomcat status
  • Delete the existing extracted webapps: rm -rf /usr/share/apache-tomcat/webapps/{moogpoller,moogsvr,toolrunner,graze,events}
  • Copy in the new webapp WAR files (and replace the old ones): cp -f $MOOGSOFT_HOME/lib/{moogpoller,moogsvr,toolrunner,graze,events}.war $APPSERVER_HOME/webapps/
  • Restart apache-tomcat: service apache-tomcat start

Integrations

The following command needs to be run to extract Integrations and SolutionPaks:

rm -f $MOOGSOFT_HOME/etc/integrations/sample_data_lam*
bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN


OPTIONAL
: If the images for some Integrations are not visible post-upgrade, the following script can be run to replace the broken image links in the database:

bash $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.1.0_ESR5-6.2.0_ESR6/fix_integration_tmp_image_paths.sh

Upgrade the nginx configuration file

The following commands need to be run to update the nginx configuration file to enable additional security headers:

sed -i 's/\(http:\/\/localhost:8085\/moogsvr;\).*/\1\nclient_max_body_size 10m;/' /etc/nginx/conf.d/moog-ssl.conf
sed -i 's/\(proxy_pass.*;\).*/\1\nadd_header Cache-Control no-store;\nadd_header Pragma no-cache;\n/' /etc/nginx/conf.d/moog-ssl.conf
sed -i 's/\(ssl_protocols.*;\).*/\1\nadd_header X-Frame-Options SAMEORIGIN;\nadd_header Strict-Transport-Security "max-age=63072000; includeSubDomains; always";\nunderscores_in_headers on;\nserver_tokens off;/' /etc/nginx/conf.d/moog-ssl.conf

Finally, restart nginx using:

service nginx restart

Apply Alert Indexing Change in MySQL

Run the following mysql command:

mysql -u root moogdb -e "REPLACE INTO moog_indexer (indexer_name, last_updated) VALUES ('com.moogsoft.searchengine.CAlertIndexer', (select max(event_id) from events));"

...to modify alert indexing to use the new improved method under MOOG-6937.

Note: Replace <moogdb> with different database name if using a custom name.

Confirm all services are running and start moogfarmd

To confirm all required Services are running and start moogfarmd, run the following commands:

service apache-tomcat status
service moogfarmd start
service nginx status
service elasticsearch status

Re-enable the moog_indexer and events_analyser cronjobs using the following command:

(crontab -l | sed -e 's/^\#\+\(.*moog_indexer.*\)/\1/' -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -


OPTIONAL
: In v6.2.0, the events_analyser and moog_indexer cronjobs now redirect their log output into the /var/log/moogsoft/ folder. If this is desired the cronjobs can be editing using:

crontab -e

Then the entry can be edited using a 'vi' type editor. Save and exit to commit the changes.

Example edited crontabs:

0	*	*	*	* $MOOGSOFT_HOME/bin/events_analyser --incremental >> /var/log/moogsoft/events_analyser_incremental.log 2>&1
0	3	*	*	0 $MOOGSOFT_HOME/bin/events_analyser >> /var/log/moogsoft/events_analyser.log 2>&1
*	*	*	*	* $MOOGSOFT_HOME/bin/utils/moog_indexer -b 1000 -t 5 >> /var/log/moogsoft/moog_indexer.log 2>&1

Any previously running integrations can be restarting using the following command:

service moogstartupd restart

Verification

Manual Verification of Upgrade

Carry out the following basic steps to ensure the upgrade to 6.3.0 was successful:

  • Ensure the $MOOGSOFT_HOME/etc/VERSION.txt file contains just "6.3.0.beta"
  • Check that the last entry in the moogdb.schema_upgrades table in MySQL is the 6.3.0 upgrade entry
  • Check that the UI login page displays "Version 6.3.0.beta" at the top
  • Check that the UI "Support Information" window correctly indicates the current version as 6.3.0 and shows the correct schema upgrade history
  • Integrations: If any of the Integrations images are not displaying, please see the 'Integrations' section above and run the final optional step/command.

Automated Verification of Upgrade

Confirm all Moog files have been deployed correctly within $MOOGSOFT_HOME using this utility:


$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Confirm all apache-tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:


$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh

Confirm that all the steps are successful. If there are some webapp differences then this can be resolved using moog_init_ui.sh -w which will extract the webapps with the right files.

If there is a difference reported in the checksum of context.xml, this can be ignored. The issue will be fixed in the GA release of 6.3.0.


Validate the schema upgrade was successful:

$MOOGSOFT_HOME/bin/utils/mysql_install_validator.sh

Confirm that all the steps are successful. If there are some schema differences, they may be valid (e.g.: custom_info related) but if there are more substantial differences then it would be worth investigating further to check if all the pre-requisite upgrade scripts have been applied in the right order:


mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have an issues please refer to Troubleshooting.

  • No labels