Skip to main content

Patch Moogsoft Onprem RPM v9.2.0 to v9.2.0.1

This topic describes how to patch an RPM-based distribution of Moogsoft Onprem to v9.2.0.1 from v9.2.0

Important

This process requires the presence of a third Moogsoft Onprem server to act a redundancy server as per : Fully Distributed HA Installation

In the process below, Server 1 and Server 2 are servers running MoogFarmD, Apache-Tomcat, RabbitMQ etc

In the process below, Server 3 is the Redundancy Server running just RabbitMQ and possibly OpenSearch too

  1. OPTIONAL (but recommended) : On any server - perform cluster health checks

    1. Confirm the RabbitMQ cluster is healthy:

      rabbitmqctl cluster_status

      This output should show three 'Running nodes' in the cluster

    2. Confirm the Percona cluster is healthy:

      moog_mysql_client -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

      The above command should return 3 for a 3-node cluster etc.

      Now check all nodes are synced:

      for CLUSTER_NODE in $(moog_mysql_client -sNe "SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME='wsrep_incoming_addresses'" 2>/dev/null | tr ',' '\n' | cut -d: -f1); do
        echo -n "${CLUSTER_NODE}: "; curl -s http://${CLUSTER_NODE}:9198
      done

      This command show show 'Percona XtraDB Cluster Node is synced' for every node

    3. Confirm the OpenSearch cluster is healthy:

      curl -XGET 'https://localhost:9200/_cluster/health?pretty' -u moog_opensearchuser:$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k search.password)

      This should report: "number_of_nodes" : 3

  2. One-time multi-server upgrade step

    Update all Percona nodes to the latest version using the instructions here: Percona Cluster 8.4 RPM Minor Version Upgrade

  3. One-time multi-server upgrade step

  4. On Server 1

    1. Stop MoogFarmd to pause event processing (events will buffer in RabbitMQ)

      service moogfarmd stop
  5. On Server 2

    1. Stop MoogFarmD to pause event processing (events will buffer in RabbitMQ)

      service moogfarmd stop
  6. On Server 3 (Redundancy Server)

    1. Stop all services

      systemctl stop opensearch
      rabbitmq-upgrade drain
      systemctl stop rabbitmq-server
      
    2. Upgrade erlang

      • Online deployments:

        yum -y upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v27.3.4.13/erlang-27.3.4.13-1.el8.x86_64.rpm
      • Offline deployments (assumes offline yum repository is present)::

        yum -y upgrade erlang-27.3.4.13
    3. Upgrade the packages on the server

    4. Restart the RabbitMQ service and rejoin the cluster:

      systemctl restart rabbitmq-server
      rabbitmq-upgrade revive
      rabbitmq-queues rebalance quorum
      
    5. Confirm the RabbitMQ node has rejoined the existing cluster:

      rabbitmqctl cluster_status

      Three nodes should be listed

  7. On Server 1

    1. Stop all services (update the commands below as needed)

      service apache-tomcat stop
      service opensearch stop
      service restlamd stop
      service socketlamd stop
      kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null;
      rabbitmq-upgrade drain
      service rabbitmq-server stop
      service nginx stop
    2. Upgrade Erlang:

      • Online deployments:

        yum -y upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v27.3.4.13/erlang-27.3.4.13-1.el8.x86_64.rpm
      • Offline deployments (assumes offline yum repository is present):

        yum -y upgrade erlang-27.3.4.13
    3. Upgrade the packages on the server

    4. If the Enrichment Database (Create the Enrichment API Data Store) is being used, ensure the schema is updated using the schema update command on that page.

    5. Restart RabbitMQ to rejoin the cluster:

      service rabbitmq-server restart
      rabbitmq-upgrade revive
      rabbitmq-queues rebalance quorum
    6. Confirm the RabbitMQ node has rejoined the existing cluster:

      rabbitmqctl cluster_status

      Three nodes should be listed

    7. Upgrade Apache-Tomcat

    8. Perform final checks

    9. Start the LAMs (update the commands below as needed)

      service restlamd restart
      service socketlamd restart
    10. Restart Nginx:

      service nginx restart
  8. On Server 2

    1. Stop all services (update the commands below as needed)

      service apache-tomcat stop
      service restlamd stop
      service socketlamd stop
      kill -9 $(ps -ef | grep java | grep Broker | awk '{print $2}') 2>/dev/null;
      rabbitmq-upgrade drain
      service rabbitmq-server stop
      service nginx stop
    2. Upgrade erlang:

      • Online deployments:

        yum -y upgrade https://github.com/rabbitmq/erlang-rpm/releases/download/v27.3.4.13/erlang-27.3.4.13-1.el8.x86_64.rpm
      • Offline deployments (assumes offline yum repository is present):

        yum -y upgrade erlang-27.3.4.13
    3. Upgrade the packages on the server

    4. Restart RabbitMQ to rejoin the cluster:

      service rabbitmq-server restart
      rabbitmq-upgrade revive
      rabbitmq-queues rebalance quorum
    5. Confirm the RabbitMQ node has rejoined the existing cluster:

      rabbitmqctl cluster_status

      Three nodes should be listed

    6. Upgrade Apache-Tomcat

    7. Perform final checks

    8. Enable all RabbitMQ feature flags:

      rabbitmqctl enable_feature_flag all
    9. Start the LAMs (update the commands below as needed)

      service restlamd restart
      service socketlamd restart
    10. Start Nginx:

      service nginx restart
  9. The OpenSearch Cluster now needs to be recreated: Opensearch Clustering Guide - RPM

  10. OPTIONAL (but recommended) : On any server - perform cluster health checks

    1. Confirm the RabbitMQ cluster is healthy:

      rabbitmqctl cluster_status

      This output should show three 'Running nodes' in the cluster

    2. Confirm the Percona cluster is healthy:

      moog_mysql_client -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

      The above command should return 3 for a 3-node cluster etc.

      Now check all nodes are synced:

      for CLUSTER_NODE in $(moog_mysql_client -sNe "SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME='wsrep_incoming_addresses'" 2>/dev/null | tr ',' '\n' | cut -d: -f1); do
        echo -n "${CLUSTER_NODE}: "; curl -s http://${CLUSTER_NODE}:9198
      done

      This command show show 'Percona XtraDB Cluster Node is synced' for every node

    3. Confirm the OpenSearch cluster is healthy:

      curl -XGET 'https://localhost:9200/_cluster/health?pretty' -u moog_opensearchuser:$(${MOOGSOFT_HOME}/bin/utils/moog_config_reader -k search.password)

      This should report: "number_of_nodes" : 3

  11. On Server 1

    1. Start MoogFarmD

      service moogfarmd restart
  12. On Server 2

    1. Start MoogFarmD

      service moogfarmd restart
  13. Restart any event feeds if they were stopped

  14. Upgrade the Moogsoft Bridge Servers