Skip to main content

Percona Cluster 8.4 RPM Minor Version Upgrade

Use the following instructions to upgrade an RPM Percona XtraDB Cluster to v8.4.xx to avoid Moogsoft Onprem-impacting issues that have been identified in lower versions.

The official Percona upgrade documentation is also available here: https://docs.percona.com/percona-xtradb-cluster/8.4/upgrade-guide.html#upgrade-an-async-replication-replica-node

System Setup

This example uses the following sample server configuration for Moogsoft Onprem and the Percona Cluster:

  • Server1 - Percona Cluster Node1

  • Server2 - Percona Cluster Node2

  • Server3 - Percona Cluster Node3

  • Server4 - Moogsoft Onprem 9.2.x.x AppServer

    • If there are only three nodes in the deployment, for commands run from Server 4, run them from any of the other nodes instead

Assumptions

This procedure guides a user with root/sudo access through an RPM-based online installation.

Definitions

  • IST - Incremental State Transfer. Functionality which instead of whole state snapshot can catch up with the group by receiving the missing writesets, but only if the writeset is still in the donor’s writeset cache.

  • SST - State Snapshot Transfer is the full copy of data from one node to another. Used when a new node joins the cluster; it has to transfer data from an existing node.

Upgrade Steps

Note

The commands provided in this procedure assume that the rest_lam and socket_lam are in use. Other configurations are possible with additional LAMs and integrations.

Use this command to stop additional services

sudo systemctl stop <lam_service_name>

Step 1 - Stop all Moog services that connect to the database

sudo systemctl stop restlamd
sudo systemctl stop socketlamd
sudo systemctl stop moogfarmd
sudo systemctl stop apache-tomcat

Note

While it is technically possible to perform the cluster upgrade when the system is running and processing events, doing so greatly increases the chances of issues and/or SST occurrence later in the upgrade.

Step 2 - Make sure the three database nodes are synced

Perform the following check from any location (expected output is below). If any nodes do not report they are synced, do not proceed with the upgrade and check the mysqld.log for possible causes.

[root@server4]# curl http://server1:9198
Percona XtraDB Cluster Node is synced.
[root@server4]# curl http://server2:9198
Percona XtraDB Cluster Node is synced.
[root@server4]# curl http://server3:9198
Percona XtraDB Cluster Node is synced.

Step 3 - Perform the actions on Server2

  1. Stop the MySQL node (both commands are included below to include the two most common scenarios):

    sudo mysql -u root -p -e "SET GLOBAL innodb_fast_shutdown=0;"
sudo systemctl stop mysql
sudo systemctl stop mysql@bootstrap.service

  2. Update the packages:

    • Online

      sudo dnf -y module disable mysql;
sudo yum -y install https://repo.percona.com/yum/percona-release-latest.noarch.rpm;
sudo percona-release enable-only pxc-84-lts release;
sudo percona-release enable pxb-84-lts release;
sudo yum update percona-xtradb-cluster-*-8.4.8;
sudo yum remove percona-xtrabackup-80;
sudo yum install percona-xtrabackup-84-8.4.0;

    • Offline

      Ensure the Percona dependency RPMs have been downloaded to this server following these instructions: Moogsoft Onprem - Offline RPM pre-installation, then run the following commands:

      sudo dnf -y module disable mysql;
sudo yum update percona-xtradb-cluster-*-8.4.8*.rpm percona-telemetry-agent*.rpm percona_dependencies/nmap*.rpm;
sudo yum remove percona-xtrabackup-80;
sudo yum install percona-xtrabackup-84-8.4.0*.rpm;

  3. Backup the grastate.dat file. The command below assumes the default location (if it is not here, check the datadir folder which is specified in /etc/my.cnf).

    cp /var/lib/mysql/grastate.dat /var/lib/mysql/grastate.dat.bak

  4. Delete any references to service-startup-timeout from the /etc/my.cnf file:

    sed -i '/service-startup-timeout/d' /etc/my.cnf

  5. Update the /etc/my.cnf file to make it compatible with v8.4 using this command:

    sed -i -e 's/.*default_authentication_plugin.*/mysql-native-password = ON/' -e 's/\(.*innodb_autoinc_lock_mode.*\)/\1\ninnodb_numa_interleave = OFF/' -e "s/\(.*slow-query-log-file.*\)/\1\nlog_error_suppression_list = 'MY-013360'/" -e '/binlog_format/d' /etc/my.cnf

  6. Finally start the node normally:

    sudo systemctl start mysql

    It will either sync immediately or spend a short amount of time performing an IST from another node. See the example below from /var/log/mysql/mysqld.log:

    2020-08-20T11:12:45.559572-00:00 0 [Note] WSREP: Receiving IST... 
13.0% ( 96/739 events) complete.
2020-08-20T11:12:56.030910-00:00 0 [Note] WSREP: Receiving IST... 
23.8% (176/739 events) complete.
2020-08-20T11:13:06.922647-00:00 0 [Note] WSREP: Receiving IST... 
34.6% (256/739 events) complete.
2020-08-20T11:13:18.481076-00:00 0 [Note] WSREP: Receiving IST... 
45.5% (336/739 events) complete.
2020-08-20T11:13:30.433314-00:00 0 [Note] WSREP: Receiving IST... 
56.3% (416/739 events) complete.
2020-08-20T11:13:42.463826-00:00 0 [Note] WSREP: Receiving IST... 
67.1% (496/739 events) complete.
2020-08-20T11:13:52.673644-00:00 0 [Note] WSREP: Receiving IST... 
75.8% (560/739 events) complete.
2020-08-20T11:14:02.812737-00:00 0 [Note] WSREP: Receiving IST... 
84.4% (624/739 events) complete.
2020-08-20T11:14:06.210387-00:00 0 [Note] WSREP: Receiving IST...
100.0% (739/739 events) complete.
2020-08-20T11:14:06.211793-00:00 2 [Note] WSREP: IST 
received: beebd113-e080-11ea-8803-6f888df3319a:21784800
2020-08-20T11:14:06.212523-00:00 0 [Note] WSREP: 0.0 
(pxc-node-server2): State transfer from 2.0 
(pxc-node-server3) complete.
2020-08-20T11:14:06.212564-00:00 0 [Note] WSREP: SST leaving flow control
2020-08-20T11:14:06.212574-00:00 0 [Note] WSREP: Shifting 
JOINER -> JOINED (TO: 21784800)
2020-08-20T11:14:06.213148-00:00 0 [Note] WSREP: Member 0.0 
(pxc-node-server2) synced with group.
2020-08-20T11:14:06.213174-00:00 0 [Note] WSREP: Shifting JOINED 
-> SYNCED (TO: 21784800)
2020-08-20T11:14:06.213211-00:00 2 [Note] WSREP: Synchronized 
with group, ready for connections
2020-08-20T11:14:06.213234-00:00 2 [Note] WSREP: Setting 
wsrep_ready to true
2020-08-20T11:14:06.213254-00:00 2 [Note] WSREP: wsrep_notify_cmd 
is not defined, skipping notification.
2020-08-20T11:14:44.973932-00:00 0 [Note] InnoDB: Buffer 
pool(s) load completed at 200820 11:14:44

  7. Verify this node has synced before proceeding to the next node:

    [root@server4]# curl http://server2:9198
Percona XtraDB Cluster Node is synced.

Step 4 - Perform the actions on Server3

Repeat the same actions on Server3 as you performed on Server2 in Step 3.

Step 5 - Perform the actions on Server1

Repeat the same actions on Server1 as you performed on Server1 in Step 3.

Step 6 - Final Cluster Check

Check that all three database nodes are synced:

[root@server4]# curl http://server1:9198
Percona XtraDB Cluster Node is synced.
[root@server4]# curl http://server2:9198
Percona XtraDB Cluster Node is synced.
[root@server4]# curl http://server3:9198
Percona XtraDB Cluster Node is synced.

Step 7 - Resume Moog Services

sudo systemctl start apache-tomcat
sudo systemctl start moogfarmd
sudo systemctl start restlamd
sudo systemctl start socketlamd
In this section: