Moogsoft Docs

Upgrade v6.3.0 - v6.5.0

This topic describes the upgrade procedure from Moogsoft AIOps v6.3.0 to Moogsoft AIOps v6.5.0. Refer to Upgrade Moogsoft AIOps - RPM Deployments for general information and links to upgrades for other versions.

When preparing for your upgrade, consider the following:

  • Red Hat Enterprise Linux 6/CentOS 6 is deprecated and will no longer be supported after this release. See Deprecation Notice: RHEL 6 .
  • Moogsoft AIOps no longer requires the moogsoft-eula RPM package. You accept the  End User License Agreement (EULA) when you run one of the moog_init commands during the upgrade proces.  You must accept the EULA to continue with the upgrade.

Preparations required for an offline upgrade

To prepare for an offline upgrade (where the AIOps packages are shipped in a 'local' yum repository), follow the steps 1-8 in this guide Pre-Install Offline RPMs . Once those preparations are completed, this guide can be used to perform the actual upgrade whilst taking into account any step-specific notes regarding Offline upgrades.

Back up your existing system

  1. Back up $MOOGSOFT_HOME.
  2. Take a snapshot (for VMs).
  3. Back up MySQL.

Stop Services and Processes

  1. moog_farmd:

    service moogfarmd stop
  2. apache-tomcat:

    service apache-tomcat stop
  3. LAMs. Either via their service scripts:

    service <lam_service_name> stop

    Alternatively, stop their running processes using:

    kill <pid>
  4. Ensure all the above processes have been stopped before continuing:

    service <service_name> status
  5. Disable the events_analyser from running during the upgrade process:

    1. Run the following command to comment out the relevant lines in crontab:

      (crontab -l | sed -e 's/^\(.*events_analyser.*\)$/#\1/') | crontab -
    2. Run the following command to stop any running events_analyser processes:

      ps -ef | grep java | egrep 'events_analyser|moog_indexer' | awk '{print $2}' | xargs kill 2>/dev/null

Remove the logrotate cronjob

The logrotate utility is no longer used by AIOps and is replaced by Log4j. The following command should be run to remove the logrotate cronjob entry:

(crontab -l | grep -v logrotate) | crontab -

Prepare MySQL for the upgrade

Check if MySQL is configured to run with --gtid-mode=ON using the following command in the MySQL CLI:

show variables like '%gtid%';

Remember what the value is - it will be needed later.

Official instructions for upgrading from 5.6.26 to 5.7.22 are here: and we summarise the steps below. We recommend you perform a backup of the database before continuing.

Stop the mysql service (change the password as appropriate):

mysql -u root --password='' --execute="SET GLOBAL innodb_fast_shutdown=0"
service mysqld stop

Check Erlang version and upgrade if required

Run the following commend to identify the version of erlang installed on the server:

rpm  -qa --qf '%{NAME}_%{VERSION}\n' | egrep 'erlang_'

If the version is equal to or greater than 20.1.7, then this section can be skipped. If the version is less than 20.1.7, follow the steps below.

Stop all RabbitMQ nodes:

service rabbitmq-server stop

If the upgrade is being done using the offline RPM tarballs, replace the URL to the erlang RPM in the below script with 'erlang-20.1.7'

If you are on RHEL6/CentOS6, check the installed version of Erlang and upgrade v20.1.7 if required. For offline upgrades see the information panel at the bottom of this section.

echo "remove erlang-*" >
echo "install" >>
echo "run" >>
cat | yum shell -y;

If you are on RHEL7/CentOS7, check the installed version of Erlang and upgrade v20.1.7 if required. For offline upgrades see the information panel at the bottom of this section.

echo "remove erlang-*" >
echo "install" >>
echo "run" >>
cat | yum shell -y;


If you see an error mentioning libcrypto (like below), you need to opdate openssl then retry the 'yum shell' command :

--> Finished Dependency Resolution
Error: Package: 1:nginx-1.14.0-1.el7_4.ngx.x86_64 (nginx)
> Leaving Shell

Update openssl:

yum update openssl

Then retry the 'yum shell' command:

cat | yum shell -y

Add the RabbitMQ yum repo

To upgrade a RabbitMQ cluster, follow the instructions in RabbitMQ documentation .

If RabbitMQ is configured in a clustered configuration across multiple hosts/nodes, all nodes must be stopped and upgraded at the same time.

  1. Stop the RabbitMQ service (if running):

    service rabbitmq-server stop
  2. This step does not need to be run for 'Offline' upgrades. Add the official RabbitMQ yum repo:

    curl -s | sudo bash

Upgrade AIOps


Please note that there are some new moogsoft packages (moogsoft-integrations, moogsoft-integrations-ui) which are required by AIOps v6.5.0. As a result, it is required to perform an install of those packages and an upgrade of the other packages at the same time.

This is performed using a 'yum shell' script.

If you are performing an offline upgrade, the new MySQL packages must be downloading manually before continuing as specified in Offline Installation (DEPRECATED - to remove) .

  • Single host from a remote repository. Run the following commands to perform the upgrade:

    echo "upgrade moogsoft-common-6.5.0" >
    echo "upgrade moogsoft-db-6.5.0" >>
    echo "install moogsoft-integrations-6.5.0" >>
    echo "install moogsoft-integrations-ui-6.5.0" >>
    echo "upgrade moogsoft-mooms-6.5.0" >>
    echo "upgrade moogsoft-search-6.5.0" >>
    echo "upgrade moogsoft-server-6.5.0" >>
    echo "upgrade moogsoft-ui-6.5.0" >>
    echo "upgrade moogsoft-utils-6.5.0" >>
    echo "remove moogsoft-lams" >>
    echo "run" >>
    cat | yum shell -y

    If this is an offline upgrade and the MySQL rpms are in the current directory, the following script should be used instead (replace el7 with el6 as appropriate for your architecture):

    echo "upgrade moogsoft-common-6.5.0" >
    echo "upgrade moogsoft-db-6.5.0" >>
    echo "install moogsoft-integrations-6.5.0" >>
    echo "install moogsoft-integrations-ui-6.5.0" >>
    echo "upgrade moogsoft-mooms-6.5.0" >>
    echo "upgrade moogsoft-search-6.5.0" >>
    echo "upgrade moogsoft-server-6.5.0" >>
    echo "upgrade moogsoft-ui-6.5.0" >>
    echo "upgrade moogsoft-utils-6.5.0" >>
    echo "remove moogsoft-lams" >>
    echo "upgrade mysql-community-libs-compat-5.7.22-1.el7.x86_64.rpm" >>
    echo "upgrade mysql-community-server-5.7.22-1.el7.x86_64.rpm" >>
    echo "upgrade mysql-community-client-5.7.22-1.el7.x86_64.rpm" >>
    echo "upgrade mysql-community-common-5.7.22-1.el7.x86_64.rpm" >>
    echo "upgrade mysql-community-libs-5.7.22-1.el7.x86_64.rpm" >>
    echo "run" >>
    cat | yum shell -y
  • Single host from same machine (assumes the RPMs have been downloaded locally and are in the current folder). Run the following commands to perform the upgrade:

    echo "upgrade moogsoft-common-6.5.0-116.x86_64.rpm" >
    echo "install moogsoft-integrations-6.5.0-19744.x86_64.rpm" >>
    echo "install moogsoft-integrations-ui-6.5.0-19744.x86_64.rpm" >>
    echo "upgrade moogsoft-db-6.5.0-116.x86_64.rpm" >>
    echo "upgrade moogsoft-mooms-6.5.0-116.x86_64.rpm" >>
    echo "upgrade moogsoft-search-6.5.0-116.x86_64.rpm" >>
    echo "upgrade moogsoft-server-6.5.0-116.x86_64.rpm' >>
    echo "upgrade moogsoft-ui-6.5.0-116.x86_64.rpm" >>
    echo "upgrade moogsoft-utils-6.5.0-116.x86_64.rpm" >>
    echo "remove moogsoft-lams" >>
    echo "run" >>
    cat | yum shell -y
  • Distributed installs: select the relevant update commands to run depending on which package(s) are installed on the current host. For example:

    1. HOST1 :

      echo "upgrade moogsoft-common-6.5.0" >
      echo "upgrade moogsoft-server-6.5.0" >>
      echo "upgrade moogsoft-ui-6.5.0" >>
      echo "install moogsoft-integrations-ui-6.5.0" >>
      echo "run" >>
      cat | yum shell -y
    2. HOST2 :

      echo "upgrade moogsoft-common-6.5.0" >
      echo "upgrade moogsoft-db-6.5.0" >>
      echo "upgrade moogsoft-search-6.5.0" >>
      echo "run" >>
      cat | yum shell -y
    3. HOST3 :

      echo "upgrade moogsoft-common-6.5.0" >
      echo "install moogsoft-integrations-6.5.0" >>
      echo "remove moogsoft-lams" >>
      echo "upgrade moogsoft-mooms-6.5.0" >>
      echo "upgrade moogsoft-utils-6.5.0" >>
      echo "run" >>
      cat | yum shell -y

Merge Latest Config File Changes

'.rpmsave' versions of files will need to be manually merged/compared with the new versions of those files (any new properties should be added to the older versions of the files).

To find files which have been changed/moved/deleted run these commands:

find $MOOGSOFT_HOME -name '*.rpmsave'
find /etc/init.d/ -name '*.rpmsave'

An example command to see what differences are present in the $MOOGSOFT_HOME/config/system.conf file:

diff -u $MOOGSOFT_HOME/config/system.conf $MOOGSOFT_HOME/config/system.conf.rpmsave

An example of the process to merge the differences:

  1. Rename the new versions of the files (without the .rpmsave extension) to end with '.bak'.
  2. Merge the .rpmsave file with the new '.bak' file by adding new properties/config where needed so the structure matches the new version of the file.
  3. Rename the .rpmsave file to delete the '.rpmsave' extension.

Download the mysql-connector JAR

Download the mysql-connector-5.1.45 jar file using this command (unless the file has already been downloaded as part of the 'offline Pre-Reqs' guide):


Extract, rename and move the file to $MOOGSOFT_HOME/lib/cots/nonDist/ using this command:

tar --strip-components 1 -xvf mysql-connector-java-5.1.45.tar.gz -C $MOOGSOFT_HOME/lib/cots/nonDist/ mysql-connector-java-5.1.45/mysql-connector-java-5.1.45-bin.jar --transform 's/-bin//'

Restart RabbitMQ and ElasticSearch

The following command needs to be run to restart RabbitMQ now that it has been upgraded:

service rabbitmq-server restart

The following command needs to be run to restart ElasticSearch now that it has been upgraded:

service elasticsearch restart

Upgrade MySQL and the AIOps schema

  1. Prepare the system for MySQL 5.7.22 by updating /etc/my.cnf:

    sed -i 's/\(table-open-cache.*\)/\1\nmax_prepared_stmt_count = 1048576\ngroup_concat_max_len = 1048576/' /etc/my.cnf
  2. Restart mysql:

    service mysqld restart
  3. If the gtid-mode was OFF (based on the command run earlier in the upgrade), run the mysql upgrade utility (provide mysql root password when prompted or just press <return> if no password set), then restart mysql to ensure any changes to system tables are saved:

    mysql_upgrade -u root -p
    service mysqld restart

    More information on GTID and mysql_upgrade is here:

  4. Upgrading the AIOps database requires providing the moog_db_auto_upgrader utility with the credentials of a database user with super user privileges.
    • For single-host installs where MySQL was installed as part of the AIOps deployment, the default 'ermintrude' user can be used.
    • The following command must be run after substituting the <MySQL-SuperUsername> argument. It will then ask for the password for that user. The password can also be provided to the utility with the '-p' flag but this is not recommended in non-test deployments for security reasons.

      bash $MOOGSOFT_HOME/bin/utils/moog_db_auto_upgrader -t 6.5.0 -u <MySQL-SuperUsername>

Upgrade Apache-Tomcat

AIOps v6.5.0 comes with a new version of apache-tomcat (v8.0.48).

Run the following commands to backup the listed files and any others that you have customized before you start the apache-tomcat upgrade (the $APPSERVER_HOME default value is /usr/share/apache-tomcat):

cp -f $APPSERVER_HOME/conf/context.xml ~/context.xml.orig
cp -f $APPSERVER_HOME/conf/server.xml ~/server.xml.orig
cp -f /etc/init.d/apache-tomcat ~/apache-tomcat.orig

Ensure apache-tomcat is stopped:

service apache-tomcat stop

Remove the old version of tomcat:

rm -rf /usr/share/apache-tomcat/
rm -f /etc/init.d/apache-tomcat

Now you can install the new version of apache-tomcat and make configuration changes in the script:

$MOOGSOFT_HOME/bin/utils/ -tf

If you made customizations to your backed-up context.xml/server.xml/apache-tomcat service script files, you need to manually merge changes into the new versions of the files.

The upgrade adds the following parameter to the apache-tomcat startup script at /etc/init.d/apache-tomcat:


If you are merging in customizations to the startup script from your backed-up files, be sure to include this parameter.

Run the following command to confirm the apache-tomcat was successful:

bash /usr/share/apache-tomcat/bin/ version

The 'Server number' in the output should be ''.

Upgrade the Apache-Tomcat Webapps

Scripted (recommended for standard/single-host installs):

$MOOGSOFT_HOME/bin/utils/ -w
  • If tomcat startup/shutdown failures are reported (such as: Web apps are not rebuilding, please check catalina.out ) when running the above command then tomcat should be forcibly shutdown (using kill -9 on the apache-tomcat java PID) then the step repeated:

Alternative manual steps:

  1. Stop apache-tomcat: service apache-tomcat stop
  2. Check the apache-tomcat java process is no longer running: service apache-tomcat status
  3. Delete the existing extracted webapps: rm -rf /usr/share/apache-tomcat/webapps/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}
  4. Copy in the new webapp WAR files (and replace the old ones): cp -f $MOOGSOFT_HOME/lib/{moogpoller,moogsvr,toolrunner,graze,events,situation_similarity}.war $APPSERVER_HOME/webapps/
  5. Restart apache-tomcat: service apache-tomcat start

Upgrade the Nginx configuration file

If no deployment-specific customisations have been made to the /etc/nginx/conf.d/moog-ssl.conf file, then the new version of the file can replace the old one using the following command:

cp -f $MOOGSOFT_HOME/etc/cots/nginx/moog-ssl.conf /etc/nginx/conf.d/moog-ssl.conf

If customisations had been made, please make a backup of that file before running the above step.

Then, ensure the new configuration file is updated for the system by running the following commands:

sed -i -e "s|# Load proxy configuration file for additional endpoints/lams|# Load proxy configuration file for additional endpoints/lams\n    include $MOOGSOFT_HOME/etc/cots/nginx/moog-proxies.conf;|g" /etc/nginx/conf.d/moog-ssl.conf
sed -i "s|__MOOGSOFT_HOME__|$MOOGSOFT_HOME|g" /etc/nginx/conf.d/moog-ssl.conf
sed -i "s|__NGINX_SSL_|/etc/nginx/ssl|g" /etc/nginx/conf.d/moog-ssl.conf
sed -i "s|__SERVER_HOST__|$(hostname)|g" /etc/nginx/conf.d/moog-ssl.conf
sed -i "s|__NGINX_PORT_SSL__|443|g" /etc/nginx/conf.d/moog-ssl.conf

Any deployment-specific customisations will now need to be re-applied to /etc/nginx/conf.d/moog-ssl.conf

Finally, restart Nginx:

service nginx restart


The following command needs to be run to extract Integrations and SolutionPaks:

rm -f $MOOGSOFT_HOME/etc/integrations/sample_data_lam*
bash $MOOGSOFT_HOME/bin/utils/integration_installer -a -l WARN

Additionally, the AppDynamics and SolarWinds UI Integrations (Solution Paks) have been updated and come with new Lambots. If any customizations had been made to the Lambots pre-upgrade, these changes will need to be copied from the backed-up files (e.g. AppDynamicsLam.js.rpmsave) into the new Lambots which are named as follows:

  • AppDynamicsLam.js
  • SolarWindsLam.js

Additionally, any of the following Ticketing integrations present before this upgrade will need to be un-installed and re-installed after the upgrade is complete:

  • Cherwell
  • JIRA Service Desk
  • JIRA Software
  • Remedy
  • ServiceNow


Some integrations have been upgraded since the previous version of AIOps. If an Integration has been installed before the upgrade, it will still be present but images present in the instructions for it may no longer be present. Removing and re-adding the integration will deploy the newest version of the Integration and as a result, the docs.

For JIRA (and JIRA Service Desk) and AWS, these integrations have been upgraded; for the new functionality to take effect, please remove the older version and re-create the newer version.

Confirm all services are running and start moogfarmd

Check all required services are running and start moogfarmd by entering these commands:

service apache-tomcat status
service moogfarmd start
service nginx status
service elasticsearch status

Re-enable the events_analyser cronjobs using the following command:

(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -

Any previously running integrations can be restarted using the following command:

service moogstartupd restart

Elasticsearch Re-Index

The following command must be run on the moogsoft-search/Elasticsearch server to remove the old unsortable Elasticsearch indexes. This is needed to allow search results to be sorted.

curl -XDELETE 'http://localhost:9200/alerts/' && curl -XDELETE 'http://localhost:9200/situations/'

If the command completes successfully, the following message should be shown on screen:


To re-index the Alerts and Situations in moogfarmd run the following command (requires moogfarmd to be running):

$MOOGSOFT_HOME/bin/utils/moog_indexer --now

The re-index occurs in the background and may take a while to complete. This will depend on the number of Alerts and Situations in your system.

Situation and Alert Client Tool changes

If the ServiceNow UI integration has been installed, the 'Open ServiceNow Ticket' Situation Client Tool must be updated to use the following URL (please replace <your-configured-now-host> with your ServiceNow URL):

<your-configured-snow-host>/$sig_id^opened_by=javascript:gs.getUserID()^description=Total Alerts:$total_alerts  CIs Affected:$affected_entities  Category: $category^short_description=Situation: $sig_id  $description^caller_id=javascript:gs.getUserID()

Similarly, if any Situation or Alert Client Tools were present with a URL-encoded 'Content' field (i.e. The string in that field was already 'escaped'), this must be un-encoded, as v6.5.0 will perform the URL-encoding automatically.

Java Keystore migration

As the Java runtime has been upgraded, any certificates stored in the old Java keystore need to be migrated into the new JAVA_HOME keystore.

If no certificates were manually added into the old store, no action is needed.


Manual Verification of Upgrade

Carry out the following basic steps to ensure the upgrade to 6.5.0 was successful:

  • Ensure the $MOOGSOFT_HOME/etc/VERSION.txt file contains just " 6.5.0 "
  • Check that the last entry in the moogdb.schema_upgrades table in MySQL is the 6.5.0 upgrade entry
  • Check that the UI login page displays " Version 6.5.0 " at the top
  • Check that the UI "Support Information" window correctly indicates the current version as 6.5.0 and shows the correct schema upgrade history
  • Integrations : If any of the Integrations images are not displaying, please see the 'Integrations' section above and run the final optional step/command.

Automated Verification of Upgrade

Confirm all Moog files have been deployed correctly within $MOOGSOFT_HOME using this utility:


Confirm all apache-tomcat files have been deployed correctly within $MOOGSOFT_HOME using this utility:


Confirm that all the steps are successful. If there are some webapp differences then this can be resolved using -w which will extract the webapps with the right files.

Confirm the schema has been upgraded successfully:


Confirm that all the steps are successful.

If there are some schema differences, they may be valid (e.g.: custom_info related) but if there are more substantial differences then it would be worth investigating further to check if all the pre-requisite upgrade scripts have been applied in the right order:

mysql -u root <moogdb_database_name> -e "select * from schema_upgrades;"
mysql -u root <moog_reference_database_name> -e "select * from schema_upgrades;"


If you have an issues please refer to Troubleshooting .