Upgrade Moogsoft Onprem RPM v9.0.x or v9.1.x to v9.2.0.1
This topic describes how to patch an RPM-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.0.0 or 9.0.0.x or 9.0.1.x, 9.1.0.x, or 9.1.1.x
Important
This process requires the presence of a third Moogsoft Onprem server to act a redundancy server as per : Fully Distributed HA Installation
If this server is not already provisioned and running, it must be provisioned before starting this process, and a clean installation of Moogsoft Onprem v9.2.0 performed on it, with just the RabbitMQ and OpenSearch processes left running
In the process below, Server 1 and Server 2 are existing servers running MoogFarmD, Apache-Tomcat, RabbitMQ etc
In the process below, Server 3 is the Redundancy Server running just RabbitMQ and possibly OpenSearch too
One-time multi-server upgrade step
Update all Percona nodes to the latest version using the instructions here: Percona Cluster 8.4 RPM Minor Version Upgrade
On Server 1 or Server 2
IMPORTANT: Make a note of the existing RabbitMQ Events.HA queue name by running this command:
rabbitmqctl -p <YOUR_VHOST> list_queues | grep Events.HA | awk '{print $1}' | head -n 1This queue name is needed later in the upgrade process
On Server 3
Stop RabbitMQ
service rabbitmq-server stop
On Server 1
If Server 3 existed before the upgrade and was part of the RabbitMQ cluster, remove the Server 3 RabbitMQ node from the cluster:
rabbitmqctl forget_cluster_node rabbit@<SERVER_3_IP_OR_HOSTNAME>
Comment out any reference to the Server 1 RabbitMQ broker from $MOOGSOFT_HOME/config/system.conf
The entry under mooms -> brokers should be commented out
Comment out any reference to the Server 3 RabbitMQ broker from $MOOGSOFT_HOME/config/system.conf
The entry under mooms -> brokers should be commented out
Stop MoogFarmd to pause event processing
service moogfarmd stop
Restart all other Moogsoft Onprem services (excluding MoogFarmD)
service apache-tomcat restart service restlamd restart service socketlamd restart
On Server 2
Comment out any reference to the Server 1 and the Server 3 RabbitMQ brokers from $MOOGSOFT_HOME/config/system.conf
The entries under mooms -> brokers should be commented out
Stop MoogFarmD to pause event processing
service moogfarmd stop
Restart all other Moogsoft Onprem Services (excluding MoogFarmD)
service apache-tomcat restart service restlamd restart service socketlamd restart
On Server 3 (Redundancy Server)
Stop all services
service opensearch stop
Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade
rm -rf /var/lib/rabbitmq/mnesia/*
Upgrade the packages on the server
Create the RabbitMQ vhost and Events.HA quorum queue on that node (to allow events to queue here during the upgrade)
VHOST=<your_vhost/zone_name>; $MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -pz "${VHOST}"; QUEUE_NAME="<YOUR_EVENTS_HA_QUEUE_NAME>"; chmod u+x /usr/lib/rabbitmq/lib/rabbitmq_server-4.*/plugins/rabbitmq_management-4.*/priv/www/cli/rabbitmqadmin; /usr/lib/rabbitmq/lib/rabbitmq_server-4.*/plugins/rabbitmq_management-4.*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft --vhost=${VHOST} declare queue name=${QUEUE_NAME} durable=true arguments='{"x-queue-type":"quorum"}'; /usr/lib/rabbitmq/lib/rabbitmq_server-4.*/plugins/rabbitmq_management-4.*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft --vhost=${VHOST} declare exchange name=MOOMS.GENERAL.TOPIC_EXCHANGE type=topic durable=true /usr/lib/rabbitmq/lib/rabbitmq_server-4.*/plugins/rabbitmq_management-4.*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft -V ${VHOST} declare binding source="MOOMS.GENERAL.TOPIC_EXCHANGE" destination_type="queue" destination="${QUEUE_NAME}" routing_key="events"
On Server 2
Confirm that the RabbitMQ HA policy is still present on the Events queue:
rabbitmqctl list_queues name durable policy -p <YOUR_VHOST> | grep Events.HA
This command should return a value. If it does not, the policy should be reapplied with this command:
rabbitmqctl set_policy -p <YOUR_ZONE> ha-all ".+.HA" '{"ha-mode":"all"}'Create a RabbitMQ shovel from the existing Events.HA queue to the Server 3 Events.HA queue to buffer events during the upgrade. Update the commands below with accurate variable values for the hostnames, v-hosts, and credentials.
SERVER_2_RABBITMQ_CREDS="moogsoft:m00gs0ft"; SERVER_2_RABBITMQ_VHOST="<YOUR_VHOST_NAME>"; SERVER_2_RABBITMQ_HOST="127.0.0.1"; QUEUE="<YOUR_EVENTS_HA_QUEUE_NAME>"; SERVER_3_RABBITMQ_CREDS="moogsoft:m00gs0ft"; SERVER_3_RABBITMQ_VHOST="<YOUR_VHOST_NAME>"; SERVER_3_RABBITMQ_HOST="<YOUR_SERVER_3_IP_OR_HOSTNAME>"; rabbitmq-plugins enable rabbitmq_shovel; rabbitmq-plugins enable rabbitmq_shovel_management; rabbitmqctl -p "${SERVER_2_RABBITMQ_VHOST}" set_parameter shovel my_shovel "$(cat <<EOF { "src-uri": "amqp://${SERVER_2_RABBITMQ_CREDS}@${SERVER_2_RABBITMQ_HOST}/${SERVER_2_RABBITMQ_VHOST}", "src-queue": "${QUEUE}", "dest-uri": "amqp://${SERVER_3_RABBITMQ_CREDS}@${SERVER_3_RABBITMQ_HOST}/${SERVER_3_RABBITMQ_VHOST}", "add-forward-headers": false, "dest-exchange": "MOOMS.GENERAL.TOPIC_EXCHANGE", "ack-mode": "on-confirm", "delete-after": "never" } EOF )"Confirm that the Shovel was created successfully by running the command below.
The "state" must show "running" in this output. If the Shovel is not running at this stage, events may be lost depending on the overall process being carried out.
rabbitmqctl -p <YOUR_VHOST> shovel_status
Assuming events are being ingested via LAMs and UI Integrations, the count for the Events.HA queue on the Server 3 node should now be increasing.
You can check this using the CLI command below:
rabbitmqctl -p <YOUR_VHOST> list_queues
On Server 1
Confirm that the RabbitMQ HA policy is still present on the Events queue:
VHOST=$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k mooms.zone) rabbitmqctl list_queues name durable policy -p "${VHOST}" | grep Events.HAThis command should return a value. If it does not, the policy should be reapplied with this command:
rabbitmqctl set_policy -p "${VHOST}" ha-all ".+.HA" '{"ha-mode":"all"}'Stop RabbitMQ
service rabbitmq-server stop
On Server 2
Remove the Server 1 RabbitMQ node from the cluster:
rabbitmqctl forget_cluster_node rabbit@<SERVER_1_IP_OR_HOSTNAME>
On Server 1
Stop all services (update the commands below as needed)
service apache-tomcat stop service opensearch stop service restlamd stop service socketlamd stop kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null;Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade
rm -rf /var/lib/rabbitmq/mnesia/*
Upgrade the packages on the server
Upgrade the database schema
If the Enrichment Database (Create the Enrichment API Data Store) is being used, ensure the schema is updated using the schema update command on that page.
Cluster RabbitMQ with the Node 3 RabbitMQ
Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 3
The entries under mooms -> brokers should be added or uncommented as needed
Upgrade Apache-Tomcat
Update the NginX configuration file to ensure the UI Integrations tab is accessible:
sed -i 's;location ^~ /integrations {;location ^~ /integrations/ {;' /etc/nginx/conf.d/moog-ssl.confThen reload NginX:
service nginx reload
Perform final checks
Start the LAMs (update the commands below as needed)
service restlamd restart service socketlamd restart
On Server 2
Stop Tomcat and LAMs (update the commands below as needed)
service apache-tomcat stop service restlamd stop service socketlamd stop kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null;Wait for the old Events.HA RabbitMQ queue to drain via the shovel to the new nodes. The command below needs to return a value of 0 before proceeding:
VHOST=$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k mooms.zone) rabbitmqctl -p "${VHOST}" list_queues name messages | grep Events.HAStop RabbitMQ
service rabbitmq-server stop
Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade
rm -rf /var/lib/rabbitmq/mnesia/*
Upgrade the packages on the server
Cluster RabbitMQ with Node 3 RabbitMQ
Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 2, and Server 3
The entries under mooms -> brokers should be added or uncommented as needed
Upgrade Apache-Tomcat
Update the NginX configuration file to ensure the UI Integrations tab is accessible:
sed -i 's;location ^~ /integrations {;location ^~ /integrations/ {;' /etc/nginx/conf.d/moog-ssl.confThen reload NginX:
service nginx reload
Perform final checks
Start the LAMs (update the commands below as needed)
service restlamd restart service socketlamd restart
On Server 1
Ensure all RabbitMQ quorum queues are updated to have the right number of replicas:
rabbitmq-queues grow rabbit@<SERVER1_HOSTNAME_OR_IP> all; rabbitmq-queues grow rabbit@<SERVER2_HOSTNAME_OR_IP> all; rabbitmq-queues grow rabbit@<SERVER3_HOSTNAME_OR_IP> all;
Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 2, and Server 3
The entries under mooms -> brokers should be added or uncommented as needed
Start MoogFarmD
service moogfarmd restart
Restart Apache-Tomcat
service apache-tomcat restart
On Server 2
Start MoogFarmD
service moogfarmd restart
Upgrade the Moogsoft Bridge Servers