Use the following instructions to upgrade an RPM Percona XtraDB Cluster to v8.0.35 to avoid Moogsoft Onprem-impacting issues that have been identified in lower versions.
This example uses the following sample server configuration for Moogsoft Onprem and the Percona Cluster:
Server1 - Moogsoft Onprem 9.x.x.x
Server2 - Percona Cluster Node1
Server3 - Percona Cluster Node2
Server4 - Percona Cluster Node3
This procedure guides a user with root/sudo access through an RPM-based online installation.
IST - Incremental State Transfer. Functionality which instead of whole state snapshot can catch up with the group by receiving the missing writesets, but only if the writeset is still in the donor’s writeset cache.
SST - State Snapshot Transfer is the full copy of data from one node to another. Used when a new node joins the cluster; it has to transfer data from an existing node.
Note
The commands provided in this procedure assume that the rest_lam
and socket_lam
are in use. Other configurations are possible with additional LAMs and integrations.
Use this command to stop additional services
sudo systemctl stop <lam_service_name>
sudo systemctl stop restlamd
sudo systemctl stop socketlamd
sudo systemctl stop moogfarmd
sudo systemctl stop apache-tomcat
Note
While it is technically possible to perform the cluster upgrade when the system is running and processing events, doing so greatly increases the chances of issues and/or SST occurrence later in the upgrade.
Perform the following check from any location (expected output is below). If any nodes do not report they are synced, do not proceed with the upgrade and check the mysqld.log for possible causes.
[root@server1]# curl http://server2:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server3:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server4:9198
Percona XtraDB Cluster Node is synced.
Stop the MySQL node (both commands are included below to include the two most common scenarios):
sudo systemctl stop mysql sudo systemctl stop mysql@bootstrap.service
Update the packages:
sudo yum update percona-xtradb-cluster-*-8.0.35 percona-xtrabackup-80-8.0.35
Backup the grastate.dat file. The command below assumes the default location (if it is not here, check the datadir folder which is specified in /etc/my.cnf).
cp /var/lib/mysql/grastate.dat /var/lib/mysql/grastate.dat.bak
Delete any references to service-startup-timeout from the /etc/my.cnf file. Edit the file and remove any lines containing this string:
service-startup-timeout
An example command to do this is:
sed -i '/service-startup-timeout/d' /etc/my.cnf
Finally start the node normally:
sudo systemctl start mysql
It will either sync immediately or spend a short amount of time performing an IST from another node. See the example below from /var/log/mysql/mysqld.log:
2020-08-20T11:12:45.559572-00:00 0 [Note] WSREP: Receiving IST... 13.0% ( 96/739 events) complete. 2020-08-20T11:12:56.030910-00:00 0 [Note] WSREP: Receiving IST... 23.8% (176/739 events) complete. 2020-08-20T11:13:06.922647-00:00 0 [Note] WSREP: Receiving IST... 34.6% (256/739 events) complete. 2020-08-20T11:13:18.481076-00:00 0 [Note] WSREP: Receiving IST... 45.5% (336/739 events) complete. 2020-08-20T11:13:30.433314-00:00 0 [Note] WSREP: Receiving IST... 56.3% (416/739 events) complete. 2020-08-20T11:13:42.463826-00:00 0 [Note] WSREP: Receiving IST... 67.1% (496/739 events) complete. 2020-08-20T11:13:52.673644-00:00 0 [Note] WSREP: Receiving IST... 75.8% (560/739 events) complete. 2020-08-20T11:14:02.812737-00:00 0 [Note] WSREP: Receiving IST... 84.4% (624/739 events) complete. 2020-08-20T11:14:06.210387-00:00 0 [Note] WSREP: Receiving IST... 100.0% (739/739 events) complete. 2020-08-20T11:14:06.211793-00:00 2 [Note] WSREP: IST received: beebd113-e080-11ea-8803-6f888df3319a:21784800 2020-08-20T11:14:06.212523-00:00 0 [Note] WSREP: 0.0 (pxc-node-server2): State transfer from 2.0 (pxc-node-server3) complete. 2020-08-20T11:14:06.212564-00:00 0 [Note] WSREP: SST leaving flow control 2020-08-20T11:14:06.212574-00:00 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 21784800) 2020-08-20T11:14:06.213148-00:00 0 [Note] WSREP: Member 0.0 (pxc-node-server2) synced with group. 2020-08-20T11:14:06.213174-00:00 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 21784800) 2020-08-20T11:14:06.213211-00:00 2 [Note] WSREP: Synchronized with group, ready for connections 2020-08-20T11:14:06.213234-00:00 2 [Note] WSREP: Setting wsrep_ready to true 2020-08-20T11:14:06.213254-00:00 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2020-08-20T11:14:44.973932-00:00 0 [Note] InnoDB: Buffer pool(s) load completed at 200820 11:14:44
Verify this node has synced before proceeding to the next node:
[root@server1]# curl http://server2:9198 Percona XtraDB Cluster Node is synced.
Repeat the same actions on Server3 as you performed on Server2 in Step 3.
Repeat the same actions on Server4 as you performed on Server2 in Step 3.
Check that all three database nodes are synced:
[root@server1]# curl http://server2:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server3:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server4:9198
Percona XtraDB Cluster Node is synced.
sudo systemctl start apache-tomcat
sudo systemctl start moogfarmd
sudo systemctl start restlamd
sudo systemctl start socketlamd
When the upgraded node rejoins the cluster, it is important that it synchronizes with the cluster using IST. If an SST occurs, you may need to upgrade the data directory structure using mysql_upgrade
again to make sure it is compatible with the newer version of the binaries.
Note
Check the error log for statements like the following to determine if an SST occurred:
“Check if state gap can be serviced using IST ... State gap can’t be serviced using IST. Switching to SST” instead of “Receiving IST: ...” lines appropriate to IST synchronization.
Perform the following additional steps to upgrade the data directory structure after SST completes:
Shut down the node that rejoined the cluster using SST:
sudo systemctl stop mysql
Restart the node in standalone mode:
sudo mysqld --user=mysql --wsrep-provider='none'
Run
mysql-upgrade
in another terminal then kill the standalone node.Restart the node normally and ensure the node rejoins the cluster using IST:
sudo systemctl start mysql