Finalize and validate the upgrade

Follow these steps to perform the final tasks for an upgrade to Moogsoft Enterprise v8.0.x from v7.0.x, v7.1,.x or 7.2.x.

Refer to Upgrade Moogsoft Enterprise for general information and upgrade instructions for other components and versions.

Configure the ServiceNow MID server to use Java 8

If you are using a ServiceNow MID server installed on the same host as Moogsoft Enterprise, you must configure it to point to Java 8. The MID server requires Java 8 (update 152 or later). It will not work with Java 9+. To do this:

Install the latest version of Java 8. See the ServiceNow MID server system requirements for more information.

Stop the MID server by running the appropriate command. For example:

kill -9 $(ps -ef | grep mid_server | grep -v grep | awk '{print $2}')

Configure the wrapper.java.command property to point to the Java 8 binary in the following file:
/usr/local/servicenow/moog_mid_server/agent/conf/wrapper-override.conf
For example:
```
wrapper.java.command=/usr/java/jre1.8.0_171-amd64/bin/java
```

Please note that in v7.3, the MID Server is not deployed or managed by Apache Tomcat and all MID Server configuration is now a manual task.

Configure Elasticsearch heap size

The minimum and maximum JVM heap sizes must be large enough to ensure that Elasticsearch starts.

To set the minimum and maximum JVM heap sizes:

For RPM, edit the /etc/elasticsearch/jvm.options file.
For Tarball, edit the $MOOGSOFT_HOME/cots/elasticsearch/config/jvm.options file.

These heap sizes must be the same value. For example, to set the heap to 4 GB:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms4g
-Xmx4g

If you change the heap size, you must restart Elasticsearch:

For RPM, run service elasticsearch restart.
For Tarball, run $MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch restart.

Upgrade Moogsoft Add-ons

Moogsoft periodically releases add-ons to extend and enhance the core Moogsoft Enterprise functionality. For example, new Workflow Engine functions, new Workflow Engines, or Integrations tiles. All add-ons releases are cumulative and include the fixes from previous releases.

Upgrading the Moogsoft Enterprise core components replaces any manually installed add-ons with versions of those add-ons that come with the upgrade by default. These default versions can be an earlier version than the manually installed versions of the add-ons.

Once you have finished upgrading the core components, you should install the Moogsoft Enterprise add-ons to ensure you have the latest versions of these add-ons.

See Install Moogsoft Add-ons for more information on how to install the Moogsoft Enterprise add-ons.

Restart services and processes

It is important to restart processes in the correct order. Follow each section below on the relevant servers, and run the commands in the specified order.

Database
Only do this step if the database is not running. If you have followed the documented upgrade process until this point, the database should be running.
On the database server (and any node where the database is configured to run), run the command appropriate to your deployment type:
- RPM:
```
service mysqld restart
```
- Tarball:
```
$MOOGSOFT_HOME/bin/utils/process_cntl mysql restart
```
Message Bus (RabbitMQ)
On the Core server (or any node where RabbitMQ is configured), run the command appropriate to your deployment type:
- RPM:
```
service rabbitmq-server restart
```
- Tarball:
```
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart
```
Elasticsearch
On the Core server (or where Elasticsearch is configured), run the command appropriate to your deployment type:
- RPM:
```
service elasticsearch restart
```
- Tarball:
```
$MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch restart
```
Moogfarmd
On the Core server (where Moogfarmd is configured), run the command appropriate to your deployment type:
- RPM:
```
service moogfarmd restart
```
- Tarball:
```
$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd restart
```
Warning
When you upgrade to Moogsoft Enterprise v8.0.x, Moogfarmd may not run if you are using Workflow Engine actions that have either changed from v7.x or are not in v8.0.x.
When you restart Moogfarmd, check the moog_farmd.log file for errors such as the following which indicates a missing function:
```
ERROR: [main][20200416 17:07:03.791 +0100] [CMooletCntr.java:1664] +|Error during initialization: [No such function: 'myWFEdoSomethingFunction' in the bot file of moolet: 'Situation Workflows']|+
```
If the log reports that Moogfarmd is not running or there is a missing function, contact Moogsoft support for more information.
Data Ingestion
On the Data Ingestion server (where back-end LAMs are configured), run the commands appropriate to your deployment type:
- RPM:
  Run the relevant service command for each of the LAMs deployed on the server. Follow this format replacing <LAM_service_name> as appropriate:
```
service <LAM_service_name> restart
```
  For example:
```
service restlamd restart
```
- Tarball:
  Run the relevant process_cntl command for each of the LAMs deployed on the server. Follow this format replacing <LAM_instance_name> as appropriate:
```
$MOOGSOFT_HOME/bin/utils/process_cntl <LAM_instance_name> restart
```
  For example:
```
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart
```
UI
On the UI server, run the commands appropriate to your deployment type:
- RPM:
  1. Rebuild the webapps and start Apache Tomcat:
    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  2. Restart Nginx:
    service nginx restart
- Tarball:
  1. Rebuild the webapps and start Apache Tomcat:
    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  2. Restart Nginx:
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx restart

Finalize the Events Analyser configuration

You can configure the Events Analyser in v8.0.x in two ways:

Use the Events Analyser database configuration and schedule. This is the default method for Moogsoft Enterprise v8.0.x.
Use the event_analyser.conf file (for example in the case of significant customisations) and schedule through cronjob. This is the pre-v8.0.x method. See the 'Important' section below for steps if this is needed.

We recommend that you use the default Events Analyser database configuration and schedule method. Upgrading to Moogsoft Enterprise v8.0.x automatically creates the database configuration and schedule.

After you have finished the upgrade, you must update the cronjob to check each minute if the job needs to run. Remove any existing cronjobs by running the following:

(crontab -l | grep -v events_analyser) | crontab -

Add the new cronjob:

$MOOGSOFT_HOME/bin/utils/moog_init_server.sh -e

Important

If you are using a significantly customised Events Analyser configuration file or are running the Events Analyser using the stream or partition flags, you should continue to use the pre-v8.0.x method.

Uncomment the existing events_analyser cronjob(s):

(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -

Use the crontab -e command (a 'vi'-like editor) to update any events_analyser cronjobs.

Specify the events_analyser configuration file at the end of each events_analyser command. The following example schedules a run every day at 3am, and each run will analyze two weeks of data. We recommend that you use this schedule if you do not have an existing schedule that works for your deployment.

03 * * * /usr/share/moogsoft/bin/events_analyser --incremental --readage 2w --config /usr/share/moogsoft/config/events_analyser.conf

If you want to migrate from the pre-v8.0.x method to the v8.0.x database method, contact Moogsoft Support.

Reindex Elasticsearch

Note

Run the following command on the server that houses the Core components. See Upgrade Moogsoft Enterprise for a description of the component groups.

Ensure that Moogfarmd is running, then run the following command to reindex alerts and Situations:

$MOOGSOFT_HOME/bin/utils/moog_indexer -f -n

The reindex process occurs in the background and may take a while to complete, depending on the number of alerts and Situations in your system.

Reconfigure UI integrations

If you have configured an xMatters or Dynatrace Synthetic UI integration, then these have are deprecated in 8.0.x. If you do not have these UI integrations installed, the section below can be skipped.

Important

This section is for offline deployments only, which had a configured xMatters or Dynatrace Synthetic UI integration installed.

These steps must be run on the UI server only

Download the following two files and place them in $MOOGSOFT_HOME/etc/integrations/deprecated to be able to view the configuration.

https://integrations-downloads.s3.amazonaws.com/deprecated/dynatrace_synthetic_lam.zip
https://integrations-downloads.s3.amazonaws.com/deprecated/xMatters.zip

Run the following commands to ensure the old integrations can be safely removed:

mkdir -p $MOOGSOFT_HOME/etc/integrations/deprecated
chown -R moogsoft:moogsoft $MOOGSOFT_HOME/etc/integrations/deprecated
cp dynatrace_synthetic_lam.zip $MOOGSOFT_HOME/etc/integrations/deprecated
cp xMatters.zip $MOOGSOFT_HOME/etc/integrations/deprecated

Restart apache-tomcat:

For non-root tarball deployments: $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat restart
For RPM deployments: service apache-tomcat restart

The xMatters and Dynatrace Synthetic UI integrations should now be uninstalled from using the UI.

Validate the upgrade: UI components

Note

Run the following commands/tasks on the server that houses the UI components. See Upgrade Moogsoft Enterprise for a description of the component groups.

Run this utility to confirm that all Apache Tomcat files were deployed correctly in $MOOGSOFT_HOME:

$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh

If there are webapp differences, run the following command to extract the webapps with the correct files:

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w

If you set the file_only_config property to false (or the property is missing) in $MOOGSOFT_HOME/config/moog_farmd.conf when you merged the latest configuration file changes, Moogfarmd will attempt to import the following items from your Moogfarmd configuration into the MoogDb database:

Cookbooks (including value and bot Recipes)
Tempus configurations
The default Merge Group and any custom Merge Groups

After the import operation, Moogfarmd backs up the entire $MOOGSOFT_HOME/config folder to $MOOGSOFT_HOME/config_backup_720.tgz. The version number at the end will change depending on version being upgraded to. The import process did not remove any entries from the configuration files on disk. You can check that the import process completed successfully as follows:

Tempus configurations:

The Moogsoft Enterprise UI now has a new UI System Settings panel for Tempus which displays the same settings that were used by your file-based Tempus configurations.
If more than one Tempus is defined or referenced in your Moogfarmd configuration, they are all imported, but the UI is only designed to display the configuration for one. You can use the Graze API endpoint getTempus to confirm that the configuration is correct.
Run the ha_cntl -v utility to confirm whether the correctly-named Tempus is running.

Cookbooks:

Run the ha_cntl -v utility to confirm whether the correctly-named Cookbook is running.
Run the new Graze API endpoints getCookbooks and getRecipes to check whether the Cookbooks and Recipes were imported successfully.

Default Merge Group:

Run the new Graze API endpoint getDefaultMergeGroup to check whether the default merge group was successfully imported.

Custom Merge Groups:

Run the new Graze API endpoint getMergeGroups to check whether the custom merge groups were imported successfully.

If the import was successful and the database-based Sigalisers and merge groups work as expected over a period of time, you can delete their entries in $MOOGSOFT_HOME/config/moog_farmd.conf. Moogfarmd will no longer attempt to import or run the Sigalisers or merge groups defined in the configuration filter after the import.

If one or more of the configuration items did not successfully migrate or are not present at all:

Confirm that the moog_farmd.conf property file_only_config is not set to true.
Confirm that your Sigalisers and Merge Groups are defined in the upgraded moog_farmd.conf file, or imported via 'included' Moolet configuration files.
Look for any startup errors in the Moogfarmd log file (the default log file is MOO.moog_farmd.log).
You can attempt the import again by truncating the config_migration table in the MoogDb database. To do this, run the following command:
```
$MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "truncate config_migration"
```
Restart Moogfarmd.
Re-validate the import and check the Moogfarmd log for any errors on startup.

Note

If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.

Run the Install Validator utility to ensure that all Moogsoft Enterprise files were deployed correctly in $MOOGSOFT_HOME:

$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Validate the upgrade: Core components

Note

Run the following command on the server that houses the core components. See Upgrade Moogsoft Enterprise for a description of the component groups.

Note

If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.

Run the Install Validator utility to ensure that all Moogsoft Enterprise files were deployed correctly in $MOOGSOFT_HOME:

$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Validate the upgrade: Data ingestion components

Note

Run the following command on the server that houses the data ingestion components. See Upgrade Moogsoft Enterprise for a description of the component groups.

Note

If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.

Run the Install Validator utility to ensure that all Moogsoft Enterprise files were deployed correctly in $MOOGSOFT_HOME:

$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Validate the upgrade: Database components

Note

Run the following command on the server that houses the database components. See Upgrade Moogsoft Enterprise for a description of the component groups.

Note

If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.

Run the Database Validator utility to validate the database schema:

$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

Note

Some schema differences are valid, for example those related to custom_info (new columns added etc).

An additional schema upgrade step is required if you are upgrading from v7.0.x, v7.1.x, or v7.2.x.

This additional step is documented on the Post-upgrade steps page.

Until you have complete this step, you should expect to see the following differences in the output of the Database Validator utility:

Differences found in 'historic_moogdb' tables:
41,49c41,43
<   primary key (`alert_id`),
<   unique key `idx_signature` (`signature`),
<   key `idx_first_event_time` (`first_event_time`),
<   key `idx_state_last` (`state`,`last_state_change`),
<   key `idx_severity` (`severity`,`state`),
<   key `idx_agent` (`agent`(12)),
<   key `idx_source` (`source`(12)),
<   key `idx_type` (`type`(12)),
<   key `idx_manager` (`manager`(12))
---
>   primary key (`signature`),
>   key `alert_id` (`alert_id`),
>   key `first_event_time` (`first_event_time`,`alert_id`)
93,94c87
<   key `timestamp` (`timestamp`,`type`),
<   key `idx_type_time` (`type`,`timestamp`)
---
>   key `timestamp` (`timestamp`,`type`)
241,242c234
<   key `sig_id` (`sig_id`,`action_code`,`timestamp`),
<   key `idx_action_sig` (`action_code`,`sig_id`)
---
>   key `sig_id` (`sig_id`,`action_code`,`timestamp`)

The differences above will not have any functional impact, but you must complete the rest of the upgrade to ensure the system is performant and the schema is ready for future upgrades.

If you have performed an upgrade and you see errors similar to the following:

Differences found in 'moogdb' tables:
57a58
>  key 'filter_id' ('filter_id'),
194a196
>  key 'enrichment_static_mappings_ibfk_1' ('eid'),
1196a1199
>  key 'sig_id' ('sig_id'),
1325a1329
>  key 'filter_id' ('filter_id'),

Run the following commands to resolve these index-related problems:

mysql moogdb -u root -e "alter table alert_filters_access drop key filter_id"
mysql moogdb -u root -e "alter table situation_filters_access drop key filter_id"
mysql moogdb -u root -e "alter table enrichment_static_mappings drop key enrichment_static_mappings_ibfk_1"
mysql moogdb -u root -e "alter table sig_stats_cache drop key sig_id"

Add Tempus (optional)

A new install of Moogsoft Enterprise v8.0.x includes the Tempus clustering algorithm by default. You will need to add Tempus manually when you upgrade to v8.0.x if you did not already add Tempus to an earlier version of your Moogsoft Enterprise deployment.

Run the following command to add the default configuration of Tempus to Moogsoft Enterprise. Replace the following parameters' default values if required:

run_on_startup: Set this value to true to enable Tempus.
You can also enable Tempus by going to System Settings in the UI after you have run this command.
localhost: If you run the command from a different server to your UI server, replace localhost with the hostname of your UI server.
graze:graze: Replace graze:graze with the credentials of a user with the grazer role in your deployment if needed.
MaintenanceWindowManager: If the non-Tempus clustering algorithms do not process events from the Maintenance Window Manager Moolet, replace this default value with the Moolet that the non-Tempus clustering algorithms process events from.
See Clustering Algorithm Guide for more information on the non-Tempus clustering algorithms.
See Configure Alert Behavior During a Maintenance Window for more information on this Moolet.

curl -X POST -u graze:graze -k "https://localhost/graze/v1/addTempus" -H "Content-Type: application/json; charset=UTF-8" --data '{
"name" : "Time Based (Tempus)",
"description" : "A Tempus Situation",
"run_on_startup" : false,
"entropy_threshold" : 0.0,
"threshold_type" : "global",
"process_output_of" : [ "MaintenanceWindowManager" ],
"execution_interval" : 60,
"window_size" : 1200,
"bucket_size" : 5,
"arrival_spread" : 15,
"minimum_arrival_similarity" : 0.6667,
"alert_threshold" : 4,
"pre_partition" : null,
"partition_by" : null,
"significance_test" : "Poisson1",
"significance_threshold" : 1,
"detection_algorithm" : "Louvain"
}'

See the Time-based Clustering with Tempus for more information on this clustering algorithm.

Next steps

Now that the upgrade is complete, follow the instructions in Post upgrade steps.

In this section:

Finalize and validate the upgrade

Configure the ServiceNow MID server to use Java 8

Note

Configure Elasticsearch heap size

Upgrade Moogsoft Add-ons

Restart services and processes

Warning

Finalize the Events Analyser configuration

Important

Reindex Elasticsearch

Note

Reconfigure UI integrations

Note

Important

Validate the upgrade: UI components

Note

Note

Validate the upgrade: Core components

Note

Note

Validate the upgrade: Data ingestion components

Note

Note

Validate the upgrade: Database components

Note

Note

Note

Add Tempus (optional)

Next steps