Skip to main content

Upgrade Moogsoft Onprem RPM v9.0.x or v9.1.x to v9.2.0.1

This topic describes how to patch an RPM-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.0.0 or 9.0.0.x or 9.0.1.x, 9.1.0.x, or 9.1.1.x

Important

This process requires the presence of a third Moogsoft Onprem server to act a redundancy server as per : Fully Distributed HA Installation

If this server is not already provisioned and running, it must be provisioned before starting this process, and a clean installation of Moogsoft Onprem v9.2.0 performed on it, with just the RabbitMQ and OpenSearch processes left running

In the process below, Server 1 and Server 2 are existing servers running MoogFarmD, Apache-Tomcat, RabbitMQ etc

In the process below, Server 3 is the Redundancy Server running just RabbitMQ and possibly OpenSearch too

  1. One-time multi-server upgrade step

    Update all Percona nodes to the latest version using the instructions here: Percona Cluster 8.4 RPM Minor Version Upgrade

  2. On Server 1 or Server 2

    1. IMPORTANT: Make a note of the existing RabbitMQ Events.HA queue name by running this command:

      rabbitmqctl -p <YOUR_VHOST> list_queues | grep Events.HA | awk '{print $1}' | head -n 1

      This queue name is needed later in the upgrade process

  3. On Server 3

    1. Stop RabbitMQ

      service rabbitmq-server stop
  4. On Server 1

    1. If Server 3 existed before the upgrade and was part of the RabbitMQ cluster, remove the Server 3 RabbitMQ node from the cluster:

      rabbitmqctl forget_cluster_node rabbit@<SERVER_3_IP_OR_HOSTNAME>
    2. Comment out any reference to the Server 1 RabbitMQ broker from $MOOGSOFT_HOME/config/system.conf

      The entry under mooms -> brokers should be commented out

    3. Comment out any reference to the Server 3 RabbitMQ broker from $MOOGSOFT_HOME/config/system.conf

      The entry under mooms -> brokers should be commented out

    4. Stop MoogFarmd to pause event processing

      service moogfarmd stop
    5. Restart all other Moogsoft Onprem services (excluding MoogFarmD)

      service apache-tomcat restart
      service restlamd restart
      service socketlamd restart
  5. On Server 2

    1. Comment out any reference to the Server 1 and the Server 3 RabbitMQ brokers from $MOOGSOFT_HOME/config/system.conf

      The entries under mooms -> brokers should be commented out

    2. Stop MoogFarmD to pause event processing

      service moogfarmd stop
    3. Restart all other Moogsoft Onprem Services (excluding MoogFarmD)

      service apache-tomcat restart
      service restlamd restart
      service socketlamd restart
  6. On Server 3 (Redundancy Server)

    1. Stop all services

      service opensearch stop
    2. Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade

      rm -rf /var/lib/rabbitmq/mnesia/*
    3. Upgrade the packages on the server

    4. Create the RabbitMQ vhost and Events.HA quorum queue on that node (to allow events to queue here during the upgrade)

      VHOST=<your_vhost/zone_name>;
      $MOOGSOFT_HOME/bin/utils/moog_init_mooms.sh -pz "${VHOST}";
      QUEUE_NAME="<YOUR_EVENTS_HA_QUEUE_NAME>";
      chmod u+x /usr/lib/rabbitmq/lib/rabbitmq_server-4.*/plugins/rabbitmq_management-4.*/priv/www/cli/rabbitmqadmin;
      /usr/lib/rabbitmq/lib/rabbitmq_server-4.*/plugins/rabbitmq_management-4.*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft --vhost=${VHOST} declare queue name=${QUEUE_NAME} durable=true arguments='{"x-queue-type":"quorum"}';
      /usr/lib/rabbitmq/lib/rabbitmq_server-4.*/plugins/rabbitmq_management-4.*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft --vhost=${VHOST} declare exchange name=MOOMS.GENERAL.TOPIC_EXCHANGE type=topic durable=true
      /usr/lib/rabbitmq/lib/rabbitmq_server-4.*/plugins/rabbitmq_management-4.*/priv/www/cli/rabbitmqadmin -u moogsoft -p m00gs0ft -V ${VHOST} declare binding source="MOOMS.GENERAL.TOPIC_EXCHANGE" destination_type="queue" destination="${QUEUE_NAME}" routing_key="events"
  7. On Server 2

    1. Confirm that the RabbitMQ HA policy is still present on the Events queue:

      rabbitmqctl list_queues name durable policy -p <YOUR_VHOST> | grep Events.HA

      This command should return a value. If it does not, the policy should be reapplied with this command:

      rabbitmqctl set_policy -p <YOUR_ZONE> ha-all ".+.HA" '{"ha-mode":"all"}'
    2. Create a RabbitMQ shovel from the existing Events.HA queue to the Server 3 Events.HA queue to buffer events during the upgrade. Update the commands below with accurate variable values for the hostnames, v-hosts, and credentials.

      SERVER_2_RABBITMQ_CREDS="moogsoft:m00gs0ft";
      SERVER_2_RABBITMQ_VHOST="<YOUR_VHOST_NAME>";
      SERVER_2_RABBITMQ_HOST="127.0.0.1";
      QUEUE="<YOUR_EVENTS_HA_QUEUE_NAME>";
      SERVER_3_RABBITMQ_CREDS="moogsoft:m00gs0ft";
      SERVER_3_RABBITMQ_VHOST="<YOUR_VHOST_NAME>";
      SERVER_3_RABBITMQ_HOST="<YOUR_SERVER_3_IP_OR_HOSTNAME>";
      rabbitmq-plugins enable rabbitmq_shovel;
      rabbitmq-plugins enable rabbitmq_shovel_management;
      rabbitmqctl -p "${SERVER_2_RABBITMQ_VHOST}" set_parameter shovel my_shovel "$(cat <<EOF
      {
         "src-uri": "amqp://${SERVER_2_RABBITMQ_CREDS}@${SERVER_2_RABBITMQ_HOST}/${SERVER_2_RABBITMQ_VHOST}",
         "src-queue": "${QUEUE}",
         "dest-uri": "amqp://${SERVER_3_RABBITMQ_CREDS}@${SERVER_3_RABBITMQ_HOST}/${SERVER_3_RABBITMQ_VHOST}",
         "add-forward-headers": false,
         "dest-exchange": "MOOMS.GENERAL.TOPIC_EXCHANGE",
         "ack-mode": "on-confirm",
         "delete-after": "never"
      }
      EOF
      )"
      
    3. Confirm that the Shovel was created successfully by running the command below.

      The "state" must show "running" in this output. If the Shovel is not running at this stage, events may be lost depending on the overall process being carried out.

      rabbitmqctl -p <YOUR_VHOST> shovel_status
    4. Assuming events are being ingested via LAMs and UI Integrations, the count for the Events.HA queue on the Server 3 node should now be increasing.

      You can check this using the CLI command below:

      rabbitmqctl -p <YOUR_VHOST> list_queues
  8. On Server 1

    1. Confirm that the RabbitMQ HA policy is still present on the Events queue:

      VHOST=$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k mooms.zone)
      rabbitmqctl list_queues name durable policy -p "${VHOST}" | grep Events.HA

      This command should return a value. If it does not, the policy should be reapplied with this command:

      rabbitmqctl set_policy -p "${VHOST}" ha-all ".+.HA" '{"ha-mode":"all"}'
    2. Stop RabbitMQ

      service rabbitmq-server stop
  9. On Server 2

    1. Remove the Server 1 RabbitMQ node from the cluster:

      rabbitmqctl forget_cluster_node rabbit@<SERVER_1_IP_OR_HOSTNAME>
  10. On Server 1

    1. Stop all services (update the commands below as needed)

      service apache-tomcat stop
      service opensearch stop
      service restlamd stop
      service socketlamd stop
      kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null;
    2. Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade

      rm -rf /var/lib/rabbitmq/mnesia/*
    3. Upgrade the packages on the server

    4. Upgrade the database schema

    5. If the Enrichment Database (Create the Enrichment API Data Store) is being used, ensure the schema is updated using the schema update command on that page.

    6. Cluster RabbitMQ with the Node 3 RabbitMQ

    7. Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 3

      The entries under mooms -> brokers should be added or uncommented as needed

    8. Upgrade Apache-Tomcat

    9. Update the NginX configuration file to ensure the UI Integrations tab is accessible:

      sed -i 's;location ^~ /integrations {;location ^~ /integrations/ {;' /etc/nginx/conf.d/moog-ssl.conf

      Then reload NginX:

      service nginx reload
    10. Perform final checks

    11. Start the LAMs (update the commands below as needed)

      service restlamd restart
      service socketlamd restart
  11. On Server 2

    1. Stop Tomcat and LAMs (update the commands below as needed)

      service apache-tomcat stop
      service restlamd stop
      service socketlamd stop
      kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null;
    2. Wait for the old Events.HA RabbitMQ queue to drain via the shovel to the new nodes. The command below needs to return a value of 0 before proceeding:

      VHOST=$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k mooms.zone)
      rabbitmqctl -p "${VHOST}" list_queues name messages | grep Events.HA
    3. Stop RabbitMQ

      service rabbitmq-server stop
    4. Delete the RabbitMQ mnesia dir to reset RabbitMQ to prepare for the upgrade

      rm -rf /var/lib/rabbitmq/mnesia/*
    5. Upgrade the packages on the server

    6. Cluster RabbitMQ with Node 3 RabbitMQ

    7. Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 2, and Server 3

      The entries under mooms -> brokers should be added or uncommented as needed

    8. Upgrade Apache-Tomcat

    9. Update the NginX configuration file to ensure the UI Integrations tab is accessible:

      sed -i 's;location ^~ /integrations {;location ^~ /integrations/ {;' /etc/nginx/conf.d/moog-ssl.conf

      Then reload NginX:

      service nginx reload
    10. Perform final checks

    11. Start the LAMs (update the commands below as needed)

      service restlamd restart
      service socketlamd restart
  12. On Server 1

    1. Ensure all RabbitMQ quorum queues are updated to have the right number of replicas:

      rabbitmq-queues grow rabbit@<SERVER1_HOSTNAME_OR_IP> all;
      rabbitmq-queues grow rabbit@<SERVER2_HOSTNAME_OR_IP> all;
      rabbitmq-queues grow rabbit@<SERVER3_HOSTNAME_OR_IP> all;
    2. Update the RabbitMQ brokers in $MOOGSOFT_HOME/config/system.conf to include Server 1 and Server 2, and Server 3

      The entries under mooms -> brokers should be added or uncommented as needed

    3. Start MoogFarmD

      service moogfarmd restart
    4. Restart Apache-Tomcat

      service apache-tomcat restart
  13. On Server 2

    1. Start MoogFarmD

      service moogfarmd restart
  14. Upgrade the Moogsoft Bridge Servers