Patch Moogsoft Onprem RPM v9.2.0 to v9.2.0.1
This topic describes how to patch an RPM-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.2.0
Important
This process requires the presence of a third Moogsoft Onprem server to act a redundancy server as per : Fully Distributed HA Installation
In the process below, Server 1 and Server 2 are servers running MoogFarmD, Apache-Tomcat, RabbitMQ etc
In the process below, Server 3 is the Redundancy Server running just RabbitMQ and possibly OpenSearch too
OPTIONAL (but recommended) : On any server - perform cluster health checks
Confirm the RabbitMQ cluster is healthy:
rabbitmqctl cluster_status
This output should show three 'Running nodes' in the cluster
Confirm the Percona cluster is healthy:
moog_mysql_client -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
The above command should return 3 for a 3-node cluster etc.
Now check all nodes are synced:
for CLUSTER_NODE in $(moog_mysql_client -sNe "SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME='wsrep_incoming_addresses'" 2>/dev/null | tr ',' '\n' | cut -d: -f1); do echo -n "${CLUSTER_NODE}: "; curl -s http://${CLUSTER_NODE}:9198 doneThis command show show 'Percona XtraDB Cluster Node is synced' for every node
Confirm the OpenSearch cluster is healthy:
curl -XGET 'https://localhost:9200/_cluster/health?pretty' -u moog_opensearchuser:$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k search.password)This should report: "number_of_nodes" : 3
One-time multi-server upgrade step
Update all Percona nodes to the latest version using the instructions here: Percona Cluster 8.4 RPM Minor Version Upgrade
One-time multi-server upgrade step
On Server 1
Stop MoogFarmd to pause event processing (events will buffer in RabbitMQ)
service moogfarmd stop
On Server 2
Stop MoogFarmD to pause event processing (events will buffer in RabbitMQ)
service moogfarmd stop
On Server 3 (Redundancy Server)
Stop all services
systemctl stop opensearch rabbitmq-upgrade drain systemctl stop rabbitmq-server
Upgrade erlang
Online deployments:
yum -y upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v27.3.4.13/erlang-27.3.4.13-1.el8.x86_64.rpm
Offline deployments (assumes offline yum repository is present)::
yum -y upgrade erlang-27.3.4.13
Upgrade the packages on the server
Restart the RabbitMQ service and rejoin the cluster:
systemctl restart rabbitmq-server rabbitmq-upgrade revive rabbitmq-queues rebalance quorum
Confirm the RabbitMQ node has rejoined the existing cluster:
rabbitmqctl cluster_status
Three nodes should be listed
On Server 1
Stop all services (update the commands below as needed)
service apache-tomcat stop service opensearch stop service restlamd stop service socketlamd stop kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null; rabbitmq-upgrade drain service rabbitmq-server stop service nginx stopUpgrade Erlang:
Online deployments:
yum -y upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v27.3.4.13/erlang-27.3.4.13-1.el8.x86_64.rpm
Offline deployments (assumes offline yum repository is present):
yum -y upgrade erlang-27.3.4.13
Upgrade the packages on the server
If the Enrichment Database (Create the Enrichment API Data Store) is being used, ensure the schema is updated using the schema update command on that page.
Restart RabbitMQ to rejoin the cluster:
service rabbitmq-server restart rabbitmq-upgrade revive rabbitmq-queues rebalance quorum
Confirm the RabbitMQ node has rejoined the existing cluster:
rabbitmqctl cluster_status
Three nodes should be listed
Upgrade Apache-Tomcat
Perform final checks
Start the LAMs (update the commands below as needed)
service restlamd restart service socketlamd restart
Restart Nginx:
service nginx restart
On Server 2
Stop all services (update the commands below as needed)
service apache-tomcat stop service restlamd stop service socketlamd stop kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null; rabbitmq-upgrade drain service rabbitmq-server stop service nginx stopUpgrade erlang:
Online deployments:
yum -y upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v27.3.4.13/erlang-27.3.4.13-1.el8.x86_64.rpm
Offline deployments (assumes offline yum repository is present):
yum -y upgrade erlang-27.3.4.13
Upgrade the packages on the server
Restart RabbitMQ to rejoin the cluster:
service rabbitmq-server restart rabbitmq-upgrade revive rabbitmq-queues rebalance quorum
Confirm the RabbitMQ node has rejoined the existing cluster:
rabbitmqctl cluster_status
Three nodes should be listed
Upgrade Apache-Tomcat
Perform final checks
Enable all RabbitMQ feature flags:
rabbitmqctl enable_feature_flag all
Start the LAMs (update the commands below as needed)
service restlamd restart service socketlamd restart
Start Nginx:
service nginx restart
The OpenSearch Cluster now needs to be recreated: Opensearch Clustering Guide - RPM
OPTIONAL (but recommended) : On any server - perform cluster health checks
Confirm the RabbitMQ cluster is healthy:
rabbitmqctl cluster_status
This output should show three 'Running nodes' in the cluster
Confirm the Percona cluster is healthy:
moog_mysql_client -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
The above command should return 3 for a 3-node cluster etc.
Now check all nodes are synced:
for CLUSTER_NODE in $(moog_mysql_client -sNe "SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME='wsrep_incoming_addresses'" 2>/dev/null | tr ',' '\n' | cut -d: -f1); do echo -n "${CLUSTER_NODE}: "; curl -s http://${CLUSTER_NODE}:9198 doneThis command show show 'Percona XtraDB Cluster Node is synced' for every node
Confirm the OpenSearch cluster is healthy:
curl -XGET 'https://localhost:9200/_cluster/health?pretty' -u moog_opensearchuser:$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k search.password)This should report: "number_of_nodes" : 3
On Server 1
Start MoogFarmD
service moogfarmd restart
On Server 2
Start MoogFarmD
service moogfarmd restart
Restart any event feeds if they were stopped
Upgrade the Moogsoft Bridge Servers