Moogsoft Docs

Upgrade v6.0.0 - v6.5.0

This topic describes the upgrade procedure from Moogsoft AIOps v6.0.0 to Moogosft AIOps v6.5.0. Refer to Upgrade Moogsoft AIOps - RPM Deployments for general information and links to upgrades for other versions.

When preparing for your upgrade, consider the following:

  • Red Hat Enterprise Linux 6/CentOS 6 is deprecated and will no longer be supported after this release. See Deprecation Notice: RHEL 6 .
  • Moogsoft AIOps no longer requires the moogsoft-eula RPM package. You accept the  End User License Agreement (EULA) when you run one of the moog_init commands during the upgrade proces.  You must accept the EULA to continue with the upgrade.

Preparations required for an offline upgrade

To prepare for an offline upgrade (where the AIOps packages are shipped in a 'local' yum repository), follow the steps 1-8 in this guide Pre-Install Offline RPMs . Once those preparations are completed, this guide can be used to perform the actual upgrade whilst taking into account any step-specific notes regarding Offline upgrades.

Back Up the Current System

  1. Back up $MOOGSOFT_HOME.
  2. Take a snapshot (for VMs).
  3. Back up MySQL.

Stop Services and Processes

  1. moog_farmd:

    service moogfarmd stop
  2. apache-tomcat:

    service apache-tomcat stop
  3. LAMs.

    Either via their service scripts:

    service <lam_service_name> stop

    Alternatively, stop their running processes using:

    kill <pid>
  4. Ensure all the above processes have been stopped before continuing:

    service <service_name> status
  5. Disable the events_analyser and moog_indexer from running during the upgrade process:

    1. Run the following command to comment out the relevant lines in crontab:

      (crontab -l | sed -e 's/^\(.*moog_indexer.*\)$/#\1/' -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
    2. Run the following command to stop any running moog_indexer or events_analyser processes:

      ps -ef | grep java | egrep 'events_analyser|moog_indexer' | awk '{print $2}' | xargs kill 2>/dev/null

Remove the logrotate cronjob

The logrotate utility is no longer used by AIOps and is replaced by Log4j. The following command should be run to remove the logrotate cronjob entry:

(crontab -l | grep -v logrotate) | crontab -

Run Manual Delta Index

In order to support a change to our moog_indexer, we require that all alerts and situations are indexed before continuing the upgrade. Please run the following command (add the -p flag if Private Teams are being used):

$MOOGSOFT_HOME/bin/utils/moog_indexer -b 1000 -t 5

...to perform a final delta index. MySQL and ElasticSearch need to be running for this to be successful.

Prepare MySQL for the upgrade

Check if MySQL is configured to run with --gtid-mode=ON using the following command in the MySQL CLI:

show variables like '%gtid%';

Remember what the value is - it will be needed later.

Official instructions for upgrading from 5.6.26 to 5.7.22 are here: https://dev.mysql.com/doc/refman/5.7/en/upgrading.html and we summarise the steps below. We recommend you perform a backup of the database before continuing.

Remove the mysql yum repo package and install the mysql 5.7 yum repo package:

Note

This step does not need to be run for 'Offline' upgrades.

  • For RHEL6/CentOS6:

    yum -y remove mysql-community-release
    yum -y install http://repo.mysql.com/mysql57-community-release-el6.rpm
  • For RHEL7/CentOS7:

    yum -y remove mysql-community-release
    yum -y install http://repo.mysql.com/mysql57-community-release-el7.rpm

Finally, stop the mysql service (change the password as appropriate):

mysql -u root --password='' --execute="SET GLOBAL innodb_fast_shutdown=0"
service mysqld stop

Check Erlang version and upgrade if required

Run the following commend to identify the version of erlang installed on the server:

rpm  -qa --qf '%{NAME}_%{VERSION}\n' | egrep 'erlang_'

If the version is equal to or greater than 20.1.7, then this section can be skipped. If the version is less than 20.1.7, follow the steps below.

Stop all RabbitMQ nodes:

service rabbitmq-server stop

If the upgrade is being done using the offline RPM tarballs, replace the URL to the erlang RPM in the below erlang.sh script with 'erlang-20.1.7'

If you are on RHEL6/CentOS6, check the installed version of Erlang and upgrade v20.1.7 if required. For offline upgrades see the information panel at the bottom of this section.

echo 'remove erlang-*' > erlang.sh
echo 'install https://github.com/rabbitmq/erlang-rpm/releases/download/v20.1.7/erlang-20.1.7-1.el6.x86_64.rpm' >> erlang.sh
echo 'run' >> erlang.sh
cat erlang.sh | yum shell -y;

If you are on RHEL7/CentOS7, check the installed version of Erlang and upgrade v20.1.7 if required. For offline upgrades see the information panel at the bottom of this section.

echo 'remove erlang-*' > erlang.sh
echo 'install https://github.com/rabbitmq/erlang-rpm/releases/download/v20.1.7/erlang-20.1.7-1.el7.centos.x86_64.rpm' >> erlang.sh
echo 'run' >> erlang.sh
cat erlang.sh | yum shell -y;

Note

If you see an error mentioning libcrypto (like below), you need to opdate openssl then retry the 'yum shell' command :

--> Finished Dependency Resolution
Error: Package: 1:nginx-1.14.0-1.el7_4.ngx.x86_64 (nginx)
    Requires: libcrypto.so.10(OPENSSL_1.0.2)(64bit)
> Leaving Shell

Update openssl:

yum update openssl

Then retry the 'yum shell' command:

cat erlang.sh | yum shell -y

Add the RabbitMQ yum repo

To upgrade a RabbitMQ cluster, follow the instructions in RabbitMQ documentation ( http://www.rabbitmq.com/clustering.html#upgrading ).

If RabbitMQ is configured in a clustered configuration across multiple hosts/nodes, all nodes must be stopped and upgraded at the same time.
  1. Stop the RabbitMQ service (if running):

    service rabbitmq-server stop
  2. This step does not need to be run for 'Offline' upgrades. Add the official RabbitMQ yum repo:

    curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash

Create the Nginx yum repo file

This step does not need to be run for 'Offline' upgrades. Install the Nginx yum repo by creating a script (e.g.: create_nginx_repo.sh ) containing the following contents.

#!/bin/bash
 
echo '[nginx]' > /etc/yum.repos.d/nginx.repo
echo 'name=nginx repo' >> /etc/yum.repos.d/nginx.repo
echo 'baseurl=http://nginx.org/packages/OS/OSRELEASE/$basearch/' >> /etc/yum.repos.d/nginx.repo
echo 'gpgcheck=0' >> /etc/yum.repos.d/nginx.repo
echo 'enabled=1' >> /etc/yum.repos.d/nginx.repo
 
OS_VERSION=$(cat /etc/system-release)
case "$OS_VERSION" in
    CentOS*release\ 6* )
            sed -i -e 's/OS/centos/' -e 's/OSRELEASE/6/' /etc/yum.repos.d/nginx.repo;;
    CentOS*release\ 7* )
            sed -i -e 's/OS/centos/' -e 's/OSRELEASE/7/' /etc/yum.repos.d/nginx.repo;;
    Red\ Hat*release\ 6* )
            sed -i -e 's/OS/rhel/' -e 's/OSRELEASE/6/' /etc/yum.repos.d/nginx.repo;;
    Red\ Hat*release\ 7* )
            sed -i -e 's/OS/rhel/' -e 's/OSRELEASE/7/' /etc/yum.repos.d/nginx.repo;;
esac

Now execute the script:

bash create_nginx_repo.sh

Check Nginx version and upgrade if required

Nginx version 1.14.0 or greater is required. Check the version already installed:

rpm  -qa --qf '%{NAME}_%{VERSION}\n' | egrep 'nginx_'

If this returns 1.14.0 or greater, skip the rest of this section.

Otherwise, upgrade Nginx to 1.14.0:

echo 'remove nginx-filesystem' > upgrade_nginx.sh
echo 'remove nginx-all-modules' >> upgrade_nginx.sh
echo 'remove nginx-mod-http-xslt-filter' >> upgrade_nginx.sh
echo 'remove nginx-mod-http-image-filter' >> upgrade_nginx.sh
echo 'remove nginx-mod-http-perl' >> upgrade_nginx.sh
echo 'remove nginx-mod-http-geoip' >> upgrade_nginx.sh
echo 'remove nginx-mod-stream' >> upgrade_nginx.sh
echo 'remove nginx-mod-mail' >> upgrade_nginx.sh
echo 'install nginx-module-xslt-1.14.0' >> upgrade_nginx.sh
echo 'install nginx-module-image-filter-1.14.0' >> upgrade_nginx.sh
echo 'install nginx-module-perl-1.14.0' >> upgrade_nginx.sh
echo 'install nginx-module-geoip-1.14.0' >> upgrade_nginx.sh
echo 'upgrade nginx-1.14.0' >> upgrade_nginx.sh
echo 'run' >> upgrade_nginx.sh

If you are performing an online (internet connected / epel yum repository present) upgrade, run the following command to upgrade nginx:

cat upgrade_nginx.sh | yum shell -y --disablerepo epel

Alternatively, if an offline upgrade is being performed, run the following command to upgrade nginx:

cat upgrade_nginx.sh | yum shell -y

Note

If you see an error mentioning libcrypto (like below), you need to opdate openssl then retry the 'yum shell' command :

--> Finished Dependency Resolution
Error: Package: 1:nginx-1.14.0-1.el7_4.ngx.x86_64 (nginx)
    Requires: libcrypto.so.10(OPENSSL_1.0.2)(64bit)
> Leaving Shell

Update openssl:

yum update openssl

Then retry the 'yum shell' command:

cat upgrade_nginx.sh | yum shell -y

Finally, restart Nginx:

service nginx restart

Upgrade AIOps

Note

Please note that there are some new moogsoft packages (moogsoft-common, moogsoft-integrations, moogsoft-integrations-ui) which are required by AIOps v6.5.0. As a result, it is required to perform an install of those packages and an upgrade of the other packages at the same time.

This is performed using a 'yum shell' script.

If you are performing an offline upgrade, the new MySQL packages must be downloading manually before continuing as specified in Pre-Install Offline RPMs .

  • Single host from a remote repository. Run the following commands to perform the upgrade:

    echo 'install moogsoft-common-6.5.0' > upgrade_to_650.sh
    echo 'upgrade moogsoft-db-6.5.0' >> upgrade_to_650.sh
    echo 'install moogsoft-integrations-6.5.0' >> upgrade_to_650.sh
    echo 'install moogsoft-integrations-ui-6.5.0' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-mooms-6.5.0' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-search-6.5.0' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-server-6.5.0' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-ui-6.5.0' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-utils-6.5.0' >> upgrade_to_650.sh
    echo 'remove moogsoft-lams' >> upgrade_to_650.sh
    echo 'run' >> upgrade_to_650.sh
    cat upgrade_to_650.sh | yum shell -y

    If this is an offline upgrade and the MySQL rpms are in the current directory, the following script should be used instead (replace el7 with el6 as appropriate for your architecture):

    echo 'install moogsoft-common-6.5.0' > upgrade_to_650.sh
    echo 'upgrade moogsoft-db-6.5.0' >> upgrade_to_650.sh
    echo 'install moogsoft-integrations-6.5.0' >> upgrade_to_650.sh
    echo 'install moogsoft-integrations-ui-6.5.0' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-mooms-6.5.0' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-search-6.5.0' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-server-6.5.0' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-ui-6.5.0' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-utils-6.5.0' >> upgrade_to_650.sh
    echo 'remove moogsoft-lams' >> upgrade_to_650.sh
    echo 'install mysql-community-libs-compat-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_650.sh
    echo 'upgrade mysql-community-server-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_650.sh
    echo 'upgrade mysql-community-client-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_650.sh
    echo 'upgrade mysql-community-common-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_650.sh
    echo 'upgrade mysql-community-libs-5.7.22-1.el7.x86_64.rpm' >> upgrade_to_650.sh
    echo 'run' >> upgrade_to_650.sh
    cat upgrade_to_650.sh | yum shell -y
  • Single host from same machine (assumes the RPMs have been downloaded locally and are in the current folder). Run the following commands to perform the upgrade:

    echo 'install moogsoft-common-6.5.0-116.x86_64.rpm' > upgrade_to_650.sh
    echo 'install moogsoft-integrations-6.5.0-19744.x86_64.rpm' >> upgrade_to_650.sh
    echo 'install moogsoft-integrations-ui-6.5.0-19744.x86_64.rpm' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-db-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-mooms-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-search-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-server-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-ui-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
    echo 'upgrade moogsoft-utils-6.5.0-116.x86_64.rpm' >> upgrade_to_650.sh
    echo 'remove moogsoft-lams' >> upgrade_to_650.sh
    echo 'run' >> upgrade_to_650.sh
    cat upgrade_to_650.sh | yum shell -y
  • Distributed installs: select the relevant update commands to run depending on which package(s) are installed on the current host. For example:

    1. HOST1 :

      echo 'install moogsoft-common-6.5.0' > upgrade_to_650.sh
      echo 'upgrade moogsoft-server-6.5.0' >> upgrade_to_650.sh
      echo 'upgrade moogsoft-ui-6.5.0' >> upgrade_to_650.sh
      echo 'install moogsoft-integrations-ui-6.5.0' >> upgrade_to_650.sh
      echo 'run' >> upgrade_to_650.sh
      cat upgrade_to_650.sh | yum shell -y
    2. HOST2 :

      echo 'install moogsoft-common-6.5.0' > upgrade_to_650.sh
      echo 'upgrade moogsoft-db-6.5.0' >> upgrade_to_650.sh
      echo 'upgrade moogsoft-search-6.5.0' >> upgrade_to_650.sh
      echo 'run' >> upgrade_to_650.sh
      cat upgrade_to_650.sh | yum shell -y
    3. HOST3 :

      echo 'install moogsoft-common-6.5.0' > upgrade_to_650.sh
      echo 'install moogsoft-integrations-6.5.0' >> upgrade_to_650.sh
      echo 'upgrade moogsoft-mooms-6.5.0' >> upgrade_to_650.sh
      echo 'upgrade moogsoft-utils-6.5.0' >> upgrade_to_650.sh
      echo 'remove moogsoft-lams' >> upgrade_to_650.sh
      echo 'run' >> upgrade_to_650.sh
      cat upgrade_to_650.sh | yum shell -y

Merge Latest Config File Changes

'.rpmsave' versions of files will need to be manually merged/compared with the new versions of those files (any new properties should be added to the older versions of the files).

To find files which have been changed/moved/deleted run these commands:

find $MOOGSOFT_HOME -name '*.rpmsave'
find /etc/init.d/ -name '*.rpmsave'

An example command to see what differences are present in the $MOOGSOFT_HOME/config/system.conf file:

diff -u $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.rpmsave


An example of the process to merge the differences:

  1. Rename the new versions of the files (without the .rpmsave extension) to end with '.bak'.
  2. Merge the .rpmsave file with the new '.bak' file by adding new properties/config where needed so the structure matches the new version of the file.
  3. Rename the .rpmsave file to delete the '.rpmsave' extension.

Download the mysql-connector JAR

Download the mysql-connector-5.1.45 jar file using this command (unless the file has already been downloaded as part of the 'offline Pre-Reqs' guide):

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.45.tar.gz

Extract, rename and move the file to $MOOGSOFT_HOME/lib/cots/nonDist/ using this command:

tar --strip-components 1 -xvf mysql-connector-java-5.1.45.tar.gz -C $MOOGSOFT_HOME/lib/cots/nonDist/ mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar --transform 's/-bin//'

Restart RabbitMQ and ElasticSearch

The following command needs to be run to restart RabbitMQ now that it has been upgraded:

service rabbitmq-server restart

The following command should be run now that ElasticSearch has been upgraded:

service elasticsearch restart

Upgrade MySQL and the AIOps schema

  1. Prepare the system for MySQL 5.7:

    # Switch the myisam-recover flag for myisam-recover-options
    sed -i 's/myisam-recover\s*=\s*FORCE,BACKUP/myisam-recover-options = FORCE,BACKUP/' /etc/my.cnf;
    # Add the sql-mode flag and set it to NO_ENGINE_SUBSTITUTION only
    sed -i 's/user\s*=\s*mysql/user = mysql\nsql-mode = NO_ENGINE_SUBSTITUTION/' /etc/my.cnf;
    # Increase the max-allowed-packet
    sed -i 's/max-allowed-packet\s*=\s*16M/max-allowed-packet = 128M/' /etc/my.cnf
    # Uncomment and increase the innodb-log-file-size
    sed -i 's/#innodb-log-file-size\s*=\s*128M/innodb-log-file-size = 1024M/' /etc/my.cnf
    sed -i 's/\(table-open-cache.*\)/\1\nmax_prepared_stmt_count = 1048576\ngroup_concat_max_len = 1048576/' /etc/my.cnf
    mkdir /var/log/mysql;
    touch /var/log/mysql/mysql-error.log;
    touch /var/log/mysql/mysql-slow.log;
    chown -R mysql:mysql /var/log/mysql;
  2. Restart the mysql service:

    service mysqld restart
  3. If the gtid-mode was OFF (based on the command run earlier in the upgrade), run the mysql upgrade utility (provide mysql root password when prompted or just press <return> if no password set), then restart mysql to ensure any changes to system tables are saved:

    mysql_upgrade -u root -p
    service mysqld restart

    More information on GTID and mysql_upgrade is here: https://dev.mysql.com/doc/refman/5.7/en/replication-gtids-restrictions.html#replication-gtids-restrictions-mysql_upgrade


  4. Upgrading the AIOps database requires providing the moog_db_auto_upgrader utility with the credentials of a database user with super privileges.

    For single-host installs where mysql was installed as part of the AIOps deployment, the default 'root' user can be used.

    The following command must be run after substituting the <MySQL-SuperUsername> argument. It will then ask for the password for that user. The password can also be provided to the utility with the ' -p' flag but this is not recommended in non-test deployments for security reasons.


    bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 6.5.0 -u <MySQL-SuperUsername>
  5. OPTIONAL : If the  ability to connect AIOps components to MySQL 5.7 using SSL is to be used, the MySQL certificate files (which are needed for this feature) can be generated using the following command:

    mysql_ssl_rsa_setup

Upgrade Apache-Tomcat

AIOps v6.5.0 comes with a new version of apache-tomcat (v8.0.48).

Run the following commands to backup the listed files and any others that you have customized before you start the apache-tomcat upgrade (the $APPSERVER_HOME default value is /usr/share/apache-tomcat):

cp -f $APPSERVER_HOME/conf/context.xml ~/context.xml.orig
cp -f $APPSERVER_HOME/conf/server.xml ~/server.xml.orig
cp -f /etc/init.d/apache-tomcat ~/apache-tomcat.orig

Ensure apache-tomcat is stopped:

service apache-tomcat stop

Remove the old version of tomcat:

rm -rf /usr/share/apache-tomcat/
rm -f /etc/init.d/apache-tomcat

Now you can install the new version of apache-tomcat and make configuration changes in the moog_init_ui.sh script:

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -tf

If you made customizations to your backed-up context.xml/server.xml/apache-tomcat service script files, you need to manually merge changes into the new versions of the files.

The upgrade adds the following parameter to the apache-tomcat startup script at /etc/init.d/apache-tomcat:

'-Dorg.apache.tomcat.websocket.DISABLE_BUILTIN_EXTENSIONS=true'

If you are merging in customizations to the startup script from your backed-up files, be sure to include this parameter.

Run the following command to confirm the apache-tomcat was successful:

bash /usr/share/apache-tomcat/bin/catalina.sh version

The 'Server number' in the output should be '8.0.48.0'.

Upgrade the Apache-Tomcat webapps

Stop apache-tomcat:

service apache-tomcat stop

Check the tomcat process has stopped before continuing. If the service/systemctl command hangs or reports that the process is still running, you will need to kill the apache-tomcat java process manually.

Scripted command to update the webapps (recommended for standard/single-host installs):

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  • If tomcat startup/shutdown failures are reported (such as: Web apps are not rebuilding, please check catalina.out ) when running the above command then tomcat should be forcibly shutdown (using kill -9 on the apache-tomcat java PID) then the step repeated

Alternative manual steps:

  1. Stop apache-tomcat: service apache-tomcat stop
  2. Check the apache-tomcat java process is no longer running: service apache-tomcat status
  3. Delete the existing extracted webapps: rm -rf /usr/share/apache-tomcat/webapps/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}
  4. Copy in the new webapp WAR files (and replace the old ones): cp -f $MOOGSOFT_HOME/lib/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}.war $APPSERVER_HOME/webapps/
  5. Restart apache-tomcat: service apache-tomcat start

Upgrade the Nginx configuration file

If no deployment-specific customisations have been made to the /etc/nginx/conf.d/moog-ssl.conf file, then the new version of the file can replace the old one using the following command:

cp -f $MOOGSOFT_HOME/etc/cots/nginx/moog-ssl.conf /etc/nginx/conf.d/moog-ssl.conf

If customisations had been made, please make a backup of that file before running the above step.

Then, ensure the new configuration file is updated for the system by running the following commands:

sed -i -e "s|# Load proxy configuration file for additional endpoints/lams|# Load proxy configuration file for additional endpoints/lams\n    include $MOOGSOFT_HOME/etc/cots/nginx/moog-proxies.conf;|g" /etc/nginx/conf.d/moog-ssl.conf
sed -i "s|__MOOGSOFT_HOME__|$MOOGSOFT_HOME|g" /etc/nginx/conf.d/moog-ssl.conf
sed -i "s|__NGINX_SSL_|/etc/nginx/ssl|g" /etc/nginx/conf.d/moog-ssl.conf
sed -i "s|__SERVER_HOST__|$(hostname)|g" /etc/nginx/conf.d/moog-ssl.conf
sed -i "s|__NGINX_PORT_SSL__|443|g" /etc/nginx/conf.d/moog-ssl.conf

Any deployment-specific customisations will now need to be re-applied to /etc/nginx/conf.d/moog-ssl.conf

Finally, restart Nginx:

service nginx restart

Integrations

The following command needs to be run to extract Integrations and SolutionPaks:

rm -f $MOOGSOFT_HOME/etc/integrations/sample_data_lam*
bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN


Additionally, the AppDynamics and SolarWinds UI Integrations (SolutionPaks) have been updated and come with new Lambots. If any customisations had been made to the Lambots pre-upgrade, these changes will need to be copied from the backed-up files (e.g. AppDynamicsLam.js.rpmsave) into the new Lambots which are named as follows:

  • AppDynamicsLam.js
  • SolarWindsLam.js

Additionally, any of the following Ticketing integrations present before this upgrade will need to be un-installed and re-installed after the upgrade is complete:

  • Cherwell
  • JIRA Service Desk
  • JIRA Software
  • Remedy
  • ServiceNow


Note

Some integrations have been upgraded since the previous version of AIOps. If an Integration has been installed before the upgrade, it will still be present but images present in the instructions for it may no longer be present. Removing and re-adding the integration will deploy the newest version of the Integration and as a result, the docs.

For JIRA (and JIRA Service Desk) and AWS, these integrations have been upgraded; for the new functionality to take effect, please remove the older version and re-create the newer version.


OPTIONAL : If the images for some Integrations are not visible post-upgrade, the following script can be run to replace the broken image links in the database:


bash $MOOGSOFT_HOME/etc/moog/schema_upgrades/6.1.0_ESR5-6.2.0_ESR6/fix_integration_tmp_image_paths.sh


Apply Alert Indexing Change in MySQL

Run the following mysql command:

mysql -u root moogdb -e "REPLACE INTO moog_indexer (indexer_name, last_updated) VALUES ('com.moogsoft.searchengine.CAlertIndexer', (select max(event_id) from events));"

...to modify alert indexing to use the new improved method under MOOG-6937.

Note: Replace <moogdb> with different database name if using a custom name.

Confirm all services are running and start moogfarmd

To confirm all required Services are running and start moogfarmd, run the following commands:

service apache-tomcat status
service moogfarmd start
service nginx status
service elasticsearch status

Re-enable the events_analyser cronjobs using the following command:

(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -


OPTIONAL
: Since v6.2.0, the events_analyser cronjobs now redirect it's log output into the /var/log/moogsoft/ folder. If this is desired the cronjobs can be editing using:

crontab -e

Then the entry can be edited using a 'vi' type editor. Save and exit to commit the changes.

Example edited crontabs:

0	*	*	*	* $MOOGSOFT_HOME/bin/events_analyser --incremental >> /var/log/moogsoft/events_analyser_incremental.log 2>&1
0	3	*	*	0 $MOOGSOFT_HOME/bin/events_analyser >> /var/log/moogsoft/events_analyser.log 2>&1

Any previously running integrations can be restarting using the following command:

service moogstartupd restart

Elasticsearch Re-Index

The following command must be run on the moogsoft-search/Elasticsearch server to remove the old unsortable Elasticsearch indexes. This is needed to allow search results to be sorted.

curl -XDELETE 'http://localhost:9200/alerts/' && curl -XDELETE 'http://localhost:9200/situations/'

If the command completes successfully, the following message should be shown on screen:

{"acknowledged":true}{"acknowledged":true}

To re-index the Alerts and Situations in moogfarmd run the following command (requires moogfarmd to be running):

$MOOGSOFT_HOME/bin/utils/moog_indexer --now

The re-index occurs in the background and may take a while to complete. This will depend on the number of Alerts and Situations in your system.

Situation and Alert Client Tool changes

If the ServiceNow UI integration has been installed, the 'Open ServiceNow Ticket' Situation Client Tool must be updated to use the following URL (please replace <your-configured-now-host> with your ServiceNow URL):

<your-configured-snow-host>/incident.do?sys_id=-1&sysparm_query=x_moogs_incident_m_moog_situation_id=$sig_id^opened_by=javascript:gs.getUserID()^description=Total Alerts:$total_alerts  CIs Affected:$affected_entities  Category: $category^short_description=Situation: $sig_id  $description^caller_id=javascript:gs.getUserID()

Similarly, if any Situation or Alert Client Tools were present with a URL-encoded 'Content' field (i.e. The string in that field was already 'escaped'), this must be un-encoded, as v6.5.0 will perform the URL-encoding automatically.

Java Keystore migration

As the Java runtime has been upgraded, any certificates stored in the old Java keystore need to be migrated into the new JAVA_HOME keystore.

If no certificates were manually added into the old store, no action is needed.

Verification

Manual Verification of Upgrade

Carry out the following basic steps to ensure the upgrade to 6.5.0 was successful:

  • Ensure the $MOOGSOFT_HOME/etc/VERSION.txt file contains just " 6.5.0 "
  • Check that the last entry in the moogdb.schema_upgrades table in MySQL is the 6.5.0 upgrade entry
  • Check that the UI login page displays " Version 6.5.0 " at the top
  • Check that the UI "Support Information" window correctly indicates the current version as 6.5.0 and shows the correct schema upgrade history
  • Integrations : If any of the Integrations images are not displaying, please see the 'Integrations' section above and run the final optional step/command.

Automated Verification of Upgrade

Confirm all Moog files have been deployed correctly within $MOOGSOFT_HOME using this utility:


$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Confirm all apache-tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:


$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh

Confirm that all the steps are successful. If there are some webapp differences then this can be resolved using moog_init_ui.sh -w which will extract the webapps with the right files.


Validate the schema upgrade was successful:

$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

Confirm that all the steps are successful. If there are some schema differences, they may be valid (e.g.: custom_info related) but if there are more substantial differences then it would be worth investigating further to check if all the pre-requisite upgrade scripts have been applied in the right order:


mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"

Troubleshooting

If you have any issues refer to Troubleshooting .