Finalize and validate the upgrade

Follow these steps to perform the final tasks for an upgrade to Moogsoft AIOps v7.3.x from v7.0.x, v7.1,.x or 7.2.x.

Refer to Upgrade Moogsoft AIOps for general information and upgrade instructions for other components and versions.

Configure the ServiceNow MID server to use Java 8

If you are using a ServiceNow MID server installed on the same host as Moogsoft AIOps, you must configure it to point to Java 8. The MID server requires Java 8 (update 152 or later). It will not work with Java 9+. To do this:

Install the latest version of Java 8. See the ServiceNow MID server system requirements for more information.

Stop the MID server by running the appropriate command. For example:

kill -9 $(ps -ef | grep mid_server | grep -v grep | awk '{print $2}')

Configure the wrapper.java.command property to point to the Java 8 binary in the following file:
/usr/local/servicenow/moog_mid_server/agent/conf/wrapper-override.conf
For example:
```
wrapper.java.command=/usr/java/jre1.8.0_171-amd64/bin/java
```

Please note that in v7.3, the MID Server is not deployed or managed by Apache Tomcat and all MID Server configuration is now a manual task.

Configure Elasticsearch heap size

The minimum and maximum JVM heap sizes must be large enough to ensure that Elasticsearch starts.

To set the minimum and maximum JVM heap sizes:

For RPM, edit the /etc/elasticsearch/jvm.options file.
For Tarball, edit the $MOOGSOFT_HOME/etc/cots/elasticsearch/config/jvm.options file.

These heap sizes must be the same value. For example, to set the heap to 4 GB:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms4g
-Xmx4g

If you change the heap size, you must restart Elasticsearch:

For RPM, run service elasticsearch restart.
For Tarball, run $MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch restart.

Restart services and processes

It is important to restart processes in the correct order. Follow each section below on the relevant servers, and run the commands in the specified order.

Database
This step only needs to be run if the Database is not running for any reason. If the documented upgrade process has been followed until this point, it should be running.
On the Database server (and any node where the database is configured to run), run the command appropriate to your deployment type:
- RPM:
```
service mysqld restart
```
- Tarball:
```
$MOOGSOFT_HOME/bin/utils/process_cntl mysql restart
```
Message Bus (RabbitMQ)
On the Core server (or any node where RabbitMQ is configured), run the command appropriate to your deployment type:
- RPM:
```
service rabbitmq-server restart
```
- Tarball:
```
$MOOGSOFT_HOME/bin/utils/process_cntl rabbitmq restart
```
Elasticsearch
On the Core server (or where Elasticsearch is configured), run the command appropriate to your deployment type:
- RPM:
```
service elasticsearch restart
```
- Tarball:
```
$MOOGSOFT_HOME/bin/utils/process_cntl elasticsearch restart
```
Moogfarmd
On the Core server (where Moogfarmd is configured), run the command appropriate to your deployment type:
- RPM:
```
service moogfarmd restart
```
- Tarball:
```
$MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd restart
```
Data Ingestion
On the Data Ingestion server (where back-end LAMs are configured), run the commands appropriate to your deployment type:
- RPM:
  Run the relevant service command for each of the LAMs deployed on the server. Follow this format replacing <LAM_service_name> as appropriate:
```
service <LAM_service_name> restart
```
  For example:
```
service restlamd restart
```
- Tarball:
  Run the relevant process_cntl command for each of the LAMs deployed on the server. Follow this format replacing <LAM_instance_name> as appropriate:
```
$MOOGSOFT_HOME/bin/utils/process_cntl <LAM_instance_name> restart
```
  For example:
```
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam restart
```
UI
On the UI server, run the commands appropriate to your deployment type:
- RPM:
  1. Rebuild the webapps and start Apache Tomcat:
    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  2. Restart Nginx:
    service nginx restart
- Tarball:
  1. Rebuild the webapps and start Apache Tomcat:
    $MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w
  2. Restart Nginx:
    $MOOGSOFT_HOME/bin/utils/process_cntl nginx restart

Finalize the Events Analyser configuration

Note

You must complete the following step if you are using alert entropy in your deployment.

Re-enable the regular Events Analyser cronjob:

(crontab -l | sed -e 's/^\#\+\(.*events_analyser.*\)/\1/') | crontab -

Reindex Elasticsearch

Note

Run the following command on the server that houses the Core components. See Upgrade Moogsoft AIOps for a description of the component groups.

Ensure that Moogfarmd is running, then run the following command to reindex alerts and Situations:

$MOOGSOFT_HOME/bin/utils/moog_indexer --now

The reindex process occurs in the background and may take a while to complete, depending on the number of alerts and Situations in your system.

Reconfigure UI integrations

Validate the upgrade: UI components

Note

Run the following commands/tasks on the server that houses the UI components. See Upgrade Moogsoft AIOps for a description of the component groups.

Run this utility to confirm that all Apache Tomcat files were deployed correctly in $MOOGSOFT_HOME:

$MOOGSOFT_HOME/bin/utils/tomcat_install_validator.sh

If there are webapp differences, run the following command to extract the webapps with the correct files:

$MOOGSOFT_HOME/bin/utils/moog_init_ui.sh -w

If you set the file_only_config property to false (or the property is missing) when you merged the latest configuration file changes, Moogfarmd will attempt to import the following items from your Moogfarmd configuration into the MoogDb database:

Cookbooks (including value and bot Recipes)
Tempus configurations
The default Merge Group and any custom Merge Groups

After the import operation, Moogfarmd backs up the entire $MOOGSOFT_HOME/config folder to $MOOGSOFT_HOME/config_backup_720.tgz. The import process did not remove any entries from the configuration files on disk. You can check that the import process completed successfully as follows:

Tempus configurations:

The Moogsoft AIOps v7.3 UI has a new UI System Settings panel for Tempus which displays the same settings that were used by your file-based Tempus configurations.
If more than one Tempus is defined or referenced in your Moogfarmd configuration, they are all imported, but the UI is only designed to display the configuration for one. You can use the Graze API endpoint getTempus to confirm that the configuration is correct.
Run the ha_cntl -v utility to confirm whether the correctly-named Tempus is running.

Cookbooks:

Run the ha_cntl -v utility to confirm whether the correctly-named Cookbook is running.
Run the new Graze API endpoints getCookbooks and getRecipes to check whether the Cookbooks and Recipes were imported successfully.

Default Merge Group:

Run the new Graze API endpoint getDefaultMergeGroup to check whether the default merge group was successfully imported.

Custom Merge Groups:

Run the new Graze API endpoint getMergeGroups to check whether the custom merge groups were imported successfully.

If the import was successful and the database-based Sigalisers and merge groups work as expected over a period of time, you can delete their entries in $MOOGSOFT_HOME/config/moog_farmd.conf. Moogfarmd will no longer attempt to import or run the Sigalisers or merge groups defined in the configuration filter after the import.

If one or more of the configuration items did not successfully migrate or are not present at all:

Confirm that the moog_farmd.conf property file_only_config is not set to true.
Confirm that your Sigalisers and Merge Groups are defined in the upgraded moog_farmd.conf file, or imported via 'included' Moolet configuration files.
Look for any startup errors in the Moogfarmd log file (the default log file is MOO.moog_farmd.log).
You can attempt the import again by truncating the config_migration table in the MoogDb database. To do this, run the following command:
```
$MOOGSOFT_HOME/bin/utils/moog_mysql_client -e "truncate config_migration"
```
Restart Moogfarmd.
Re-validate the import and check the Moogfarmd log for any errors on startup.

Note

If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.

Run the Install Validator utility to ensure that all Moogsoft AIOps files were deployed correctly in $MOOGSOFT_HOME:

$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Validate the upgrade: Core components

Note

Run the following command on the server that houses the core components. See Upgrade Moogsoft AIOps for a description of the component groups.

Note

If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.

Run the Install Validator utility to ensure that all Moogsoft AIOps files were deployed correctly in $MOOGSOFT_HOME:

$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Validate the upgrade: Data ingestion components

Note

Run the following command on the server that houses the data ingestion components. See Upgrade Moogsoft AIOps for a description of the component groups.

Note

If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.

Run the Install Validator utility to ensure that all Moogsoft AIOps files were deployed correctly in $MOOGSOFT_HOME:

$MOOGSOFT_HOME/bin/utils/moog_install_validator.sh

Validate the upgrade: Database components

Note

Run the following command on the server that houses the database components. See Upgrade Moogsoft AIOps for a description of the component groups.

Note

If you have already completed this step previously (as part of this upgrade process) on the current host, you can skip this step.

Run the Database Validator utility to validate the database schema:

$MOOGSOFT_HOME/bin/utils/moog_db_validator.sh

Note

Some schema differences are valid, for example those related to custom_info (new columns added etc).

An additional required schema upgrade step is documented on the Post-upgrade steps page. Until this has been run, you should expect to see the following differences in the output of the Database Validator utility:

Differences found in 'historic_moogdb' tables:
41,49c41,43
<   primary key (`alert_id`),
<   unique key `idx_signature` (`signature`),
<   key `idx_first_event_time` (`first_event_time`),
<   key `idx_state_last` (`state`,`last_state_change`),
<   key `idx_severity` (`severity`,`state`),
<   key `idx_agent` (`agent`(12)),
<   key `idx_source` (`source`(12)),
<   key `idx_type` (`type`(12)),
<   key `idx_manager` (`manager`(12))
---
>   primary key (`signature`),
>   key `alert_id` (`alert_id`),
>   key `first_event_time` (`first_event_time`,`alert_id`)
93,94c87
<   key `timestamp` (`timestamp`,`type`),
<   key `idx_type_time` (`type`,`timestamp`)
---
>   key `timestamp` (`timestamp`,`type`)
241,242c234
<   key `sig_id` (`sig_id`,`action_code`,`timestamp`),
<   key `idx_action_sig` (`action_code`,`sig_id`)
---
>   key `sig_id` (`sig_id`,`action_code`,`timestamp`)

The differences above will not have any functional impact, but you must complete the rest of the upgrade to ensure the system is performant and the schema is ready for future upgrades.

If you have performed an upgrade and you see errors similar to the following:

Differences found in 'moogdb' tables:
57a58
>  key 'filter_id' ('filter_id'),
194a196
>  key 'enrichment_static_mappings_ibfk_1' ('eid'),
1196a1199
>  key 'sig_id' ('sig_id'),
1325a1329
>  key 'filter_id' ('filter_id'),

Run the following commands to resolve these index-related problems:

mysql moogdb -u root -e "alter table alert_filters_access drop key filter_id"
mysql moogdb -u root -e "alter table situation_filters_access drop key filter_id"
mysql moogdb -u root -e "alter table enrichment_static_mappings drop key enrichment_static_mappings_ibfk_1"
mysql moogdb -u root -e "alter table sig_stats_cache drop key sig_id"

Now that the upgrade is complete, follow the instructions in v7.3.x - Post upgrade steps.

In this section: