Finalize and validate the upgrade
Follow these steps to perform the final tasks for an upgrade to Moogsoft AIOps v7.3.x from v7.0.x, v7.1,.x or 7.2.x.
Refer to Upgrade Moogsoft AIOps for general information and upgrade instructions for other components and versions.
Configure the ServiceNow MID server to use Java 8
Note
Only perform this step if you are upgrading from Moogsoft AIOps v7.0.x or v7.1.x.
If you are using a ServiceNow MID server installed on the same host as Moogsoft AIOps, you must configure it to point to Java 8. The MID server requires Java 8 (update 152 or later). It will not work with Java 9+. To do this:
Install the latest version of Java 8. See the ServiceNow MID server system requirements for more information.
Stop the MID server by running the appropriate command. For example:
kill -9 $(ps -ef | grep mid_server | grep -v grep | awk '{print $2}')Configure the
wrapper.java.commandproperty to point to the Java 8 binary in the following file:/usr/local/servicenow/moog_mid_server/agent/conf/wrapper-override.confFor example:
wrapper.java.command=/usr/java/jre1.8.0_171-amd64/bin/java
Please note that in v7.3.0, the MidServer is not deployed or managed by Apache Tomcat and all configuration of the MidServer is now a manual task.
Restart services and processes
It is important to restart processes in the correct order. Follow each section below on the relevant servers, and run the commands in the specified order.
Database
This step only needs to be run if the Database is not running for any reason. If the documented upgrade process has been followed until this point, it should be running.
On the Database server (and any node where the database is configured to run), run the command appropriate to your deployment type:
RPM:
service mysqld restart
Tarball:
$MOOGSOFT_HOME/bin/utils/process_cntl mysql restart
Message Bus (RabbitMQ)
On the Core server (or any node where RabbitMQ is configured), run the command appropriate to your deployment type:
RPM:
service rabbitmq-server restart
Tarball:
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart
Elasticsearch
On the Core server (or where Elasticsearch is configured), run the command appropriate to your deployment type:
RPM:
service elasticsearch restart
Tarball:
$MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch restart
Moogfarmd
On the Core server (where Moogfarmd is configured), run the command appropriate to your deployment type:
RPM:
service moogfarmd restart
Tarball:
$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd restart
Data Ingestion
On the Data Ingestion server (where back-end LAMs are configured), run the commands appropriate to your deployment type:
RPM:
Run the relevant
servicecommand for each of the LAMs deployed on the server. Follow this format replacing<LAM_service_name>as appropriate:service <LAM_service_name> restart
For example:
service restlamd restart
Tarball:
Run the relevant
process_cntlcommand for each of the LAMs deployed on the server. Follow this format replacing<LAM_instance_name>as appropriate:$MOOGSOFT_HOME/bin/utils/process_cntl <LAM_instance_name> restart
For example:
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart
UI
On the UI server, run the commands appropriate to your deployment type:
RPM:
Rebuild the webapps and start Apache Tomcat:
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
Restart Nginx:
service nginx restart
Tarball:
Rebuild the webapps and start Apache Tomcat:
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
Restart Nginx:
$MOOGSOFT_HOME/bin/utils/process_cntl nginx restart
Finalize the Events Analyser configuration
Note
Complete this step on the server on which the Events Analyser process is configured to run (if you are using alert entropy in your deployment). You must complete this operation before any events are sent into the system to ensure events are correctly clustered into Situations.
Run the Events Analyser utility. See Run Events Analyser for more information on the Events Analyser command line options.
$MOOGSOFT_HOME/bin/events_analyser --readage 2w
Once complete, re-enable the regular Events Analyser cronjob:
(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -
Reindex Elasticsearch
Note
Run the following command on the server that houses the Core components. See Upgrade Moogsoft AIOps for a description of the component groups.
Ensure that Moogfarmd is running, then run the following command to reindex alerts and Situations:
$MOOGSOFT_HOME/bin/utils/moog_indexer --now
The reindex process occurs in the background and may take a while to complete, depending on the number of alerts and Situations in your system.
Reconfigure UI integrations
Note
Only perform this step if you are upgrading from Moogsoft AIOps v7.0.x or v7.1.x.
If you are using any of the following UI integrations and are upgrading from v7.0.x or v7.1.x, reconfigure them using the details you noted prior to the upgrade. This is required due to UI changes in the new version.
AWS CloudWatch
Cherwell
JIRA Service Desk and JIRA Software
JMS
New Relic
Remedy
ServiceNow
SevOne
Slack
SolarWinds
VMware vCenter and vSphere
vRealize Log Insight
WebSphere MQ
xMatters
Validate the upgrade: UI components
Note
Run the following commands/tasks on the server that houses the UI components. See Upgrade Moogsoft AIOps for a description of the component groups.
Run this utility to confirm that all Apache Tomcat files were deployed correctly in $MOOGSOFT_HOME:
$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh
If there are webapp differences, run the following command to extract the webapps with the correct files:
$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
If you set the file_only_config property to false (or the property is missing) when you merged the latest configuration file changes, Moogfarmd will attempt to import the following items from your Moogfarmd configuration into the MoogDb database:
Cookbooks (including value and bot Recipes)
Tempus configurations
The default Merge Group and any custom Merge Groups
After the import operation, Moogfarmd backs up the entire $MOOGSOFT_HOME/config folder to $MOOGSOFT_HOME/config_backup_720.tgz. The import process did not remove any entries from the configuration files on disk. You can check that the import process completed successfully as follows:
Tempus configurations:
The Moogsoft AIOps v7.3.0 UI has a new UI System Settings panel for Tempus which displays the same settings that were used by your file-based Tempus configurations.
If more than one Tempus is defined or referenced in your Moogfarmd configuration, they are all imported, but the UI is only designed to display the configuration for one. You can use the Graze API endpoint getTempus to confirm that the configuration is correct.
Run the
ha_cntl -vutility to confirm whether the correctly-named Tempus is running.
Cookbooks:
Run the
ha_cntl -vutility to confirm whether the correctly-named Cookbook is running.Run the new Graze API endpoints getCookbooks and getRecipes to check whether the Cookbooks and Recipes were imported successfully.
Default Merge Group:
Run the new Graze API endpoint getDefaultMergeGroup to check whether the default merge group was successfully imported.
Custom Merge Groups:
Run the new Graze API endpoint getMergeGroups to check whether the custom merge groups were imported successfully.
If the import was successful and the database-based Sigalisers and merge groups work as expected over a period of time, you can delete their entries in $MOOGSOFT_HOME/config/moog_farmd.conf. Moogfarmd will no longer attempt to import or run the Sigalisers or merge groups defined in the configuration filter after the import.
If one or more of the configuration items did not successfully migrate or are not present at all:
Confirm that the
moog_farmd.confpropertyfile_only_configis not set totrue.Confirm that your Sigalisers and Merge Groups are defined in the upgraded moog_farmd.conf file, or imported via 'included' Moolet configuration files.
Look for any startup errors in the Moogfarmd log file (the default log file is MOO.moog_farmd.log).
You can attempt the import again by truncating the
config_migrationtable in the MoogDb database. To do this, run the following command:$MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "truncate config_migration"
Restart Moogfarmd.
Re-validate the import and check the Moogfarmd log for any errors on startup.
Note
If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.
Run the Install Validator utility to ensure that all Moogsoft AIOps files were deployed correctly in $MOOGSOFT_HOME:
$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
Validate the upgrade: Core components
Note
Run the following command on the server that houses the core components. See Upgrade Moogsoft AIOps for a description of the component groups.
Note
If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.
Run the Install Validator utility to ensure that all Moogsoft AIOps files were deployed correctly in $MOOGSOFT_HOME:
$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
Validate the upgrade: Data ingestion components
Note
Run the following command on the server that houses the data ingestion components. See Upgrade Moogsoft AIOps for a description of the component groups.
Note
If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.
Run the Install Validator utility to ensure that all Moogsoft AIOps files were deployed correctly in $MOOGSOFT_HOME:
$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh
Validate the upgrade: Database components
Note
Run the following command on the server that houses the database components. See Upgrade Moogsoft AIOps for a description of the component groups.
Note
If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.
Run the Database Validator utility to validate the database schema:
$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh
Note
Some schema differences are valid, for example those related to custom_info (new columns added etc).
An additional required schema upgrade step is documented on the Post-upgrade steps page. Until this has been run, you should expect to see the following differences in the output of the Database Validator utility:
Differences found in 'historic_moogdb' tables: 41,49c41,43 < primary key (`alert_id`), < unique key `idx_signature` (`signature`), < key `idx_first_event_time` (`first_event_time`), < key `idx_state_last` (`state`,`last_state_change`), < key `idx_severity` (`severity`,`state`), < key `idx_agent` (`agent`(12)), < key `idx_source` (`source`(12)), < key `idx_type` (`type`(12)), < key `idx_manager` (`manager`(12)) --- > primary key (`signature`), > key `alert_id` (`alert_id`), > key `first_event_time` (`first_event_time`,`alert_id`) 93,94c87 < key `timestamp` (`timestamp`,`type`), < key `idx_type_time` (`type`,`timestamp`) --- > key `timestamp` (`timestamp`,`type`) 241,242c234 < key `sig_id` (`sig_id`,`action_code`,`timestamp`), < key `idx_action_sig` (`action_code`,`sig_id`) --- > key `sig_id` (`sig_id`,`action_code`,`timestamp`)
The differences above will not have any functional impact, but you must complete the rest of the upgrade to ensure the system is performant and the schema is ready for future upgrades.
If you have performed an upgrade and you see errors similar to the following:
Differences found in 'moogdb' tables:
57a58
> key 'filter_id' ('filter_id'),
194a196
> key 'enrichment_static_mappings_ibfk_1' ('eid'),
1196a1199
> key 'sig_id' ('sig_id'),
1325a1329
> key 'filter_id' ('filter_id'),Run the following commands to resolve these index-related problems:
mysql moogdb -u root -e "alter table alert_filters_access drop key filter_id" mysql moogdb -u root -e "alter table situation_filters_access drop key filter_id" mysql moogdb -u root -e "alter table enrichment_static_mappings drop key enrichment_static_mappings_ibfk_1" mysql moogdb -u root -e "alter table sig_stats_cache drop key sig_id"
Now that the upgrade is complete, follow the instructions in v7.3.x - Post upgrade steps.