Percona Cluster 8.4 Tarball Minor Version Upgrade
Use the following instructions to upgrade a tarball Percona XtraDB Cluster to v8.4.xx to avoid Moogsoft Onprem-impacting issues that have been identified in lower versions.
The official Percona upgrade documentation is also available here: https://docs.percona.com/percona-xtradb-cluster/8.4/upgrade-guide.html#upgrade-an-async-replication-replica-node
System Setup
This example uses the following sample server configuration for Moogsoft Onprem and the Percona Cluster:
Server1 - Percona Cluster Node1
Server2 - Percona Cluster Node2
Server3 - Percona Cluster Node3
Server4 - Moogsoft Onprem 9.2.x.x AppServer
If there are only three nodes in the deployment, for commands run from Server 4, run them from any of the other nodes instead
Assumptions
This procedure guides a user through a tarball-based installation. The same user that performed the Percona installation originally should perform the upgrade.
Definitions
IST - Incremental State Transfer. Functionality which instead of whole state snapshot can catch up with the group by receiving the missing writesets, but only if the writeset is still in the donor’s writeset cache.
SST - State Snapshot Transfer is the full copy of data from one node to another. Used when a new node joins the cluster; it has to transfer data from an existing node.
Upgrade Steps
Step 1 - Stop all Moog services that connect to the database
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam stop $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd stop $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat stop
Note
While it is technically possible to perform the cluster upgrade when the system is running and processing events, doing so greatly increases the chances of issues and/or SST occurrence later in the upgrade.
Step 2 - Make sure the three database nodes are synced
Perform the following check from any location:
[moogsoft@server4]# curl http://server1:9198 Percona XtraDB Cluster Node is synced. [moogsoft@server4]# curl http://server2:9198 Percona XtraDB Cluster Node is synced. [moogsoft@server4]# curl http://server3:9198 Percona XtraDB Cluster Node is synced.
Step 3 - Perform the actions on Server2
Stop the MySQL node:
If this is a database-only node (with no Moogsoft Onprem components installed), OR the
$MOOGSOFT_HOME/bin/utils/process_cntl mysql statuscommand does not report ‘running’, run the following commands:mysql -u root -p -e "SET GLOBAL innodb_fast_shutdown=0;" MYSQL_PID=$(ps -ef | grep mysql | egrep -v 'grep|mysqld_safe' | awk '{print $2}') kill -s TERM $MYSQL_PIDIf this database is on a node with Moogsoft Onprem installed AND the
$MOOGSOFT_HOME/bin/utils/process_cntl mysql statuscommand does report ‘running’, run the following commands:mysql -u root -p -e "SET GLOBAL innodb_fast_shutdown=0;" $MOOGSOFT_HOME/bin/utils/process_cntl mysql stop
Download the Percona files and copy them into the same folder where the existing Percona installation is located (default is ~/install):
cd ~/install; curl -L -O https://downloads.percona.com/downloads/Percona-XtraDB-Cluster-84/Percona-XtraDB-Cluster-8.4.8/binary/tarball/Percona-XtraDB-Cluster_8.4.8-8.1_Linux.x86_64.glibc2.28-minimal.tar.gz
Extract the files into the same folder where the existing Percona installation is located (default is ~/install):
tar -xf Percona-XtraDB-Cluster_8.4.8*.tar.gz;
Update the PATH variable in the
bashrcfile to reference the newer Percona packages:sed -i 's;Percona-XtraDB-Cluster_[^/]\+;Percona-XtraDB-Cluster_8.4.8-8.1_Linux.x86_64.glibc2.28-minimal;' ~/.bashrc; source ~/.bashrc;
Remove any old xtrabackup references in the PATH:
sed -i 's/export PATH/#export PATH/g' ~/.bashrc; NEW_PATH=$(echo "$PATH" | tr ':' '\n' | grep -v 'xtrabackup' | paste -sd ":" -) echo "export PATH=${NEW_PATH}" >> ~/.bashrc; source ~/.bashrc;Back-up and update the .my.cnf file:
cp ~/.my.cnf ~/.my.cnf$(date +%s); MYSQL_HOME=$(dirname $(dirname $(which mysqld))) sed -i "s;plugin_dir.*;plugin_dir = ${MYSQL_HOME}/lib/plugin;" ~/.my.cnf sed -i "s;wsrep_provider\s*=.*;wsrep_provider = ${MYSQL_HOME}/lib/libgalera_smm.so;" ~/.my.cnfBackup the
grastate.datfile as a safety measure.LINE_NUM=$(cat -un ~/.my.cnf | grep '\[mysqld\]' | awk '{ print $1 }') DATADIR=$(awk "NR>${LINE_NUM}" ~/.my.cnf | egrep '^\s*datadir' | head -1 | awk -F'=' '{ print $2 }' | tr -d '\t ') cp ${DATADIR}/grastate.dat ${DATADIR}/grastate.dat.bakUpdate the .my.cnf file to make it compatible with v8.4 using these commands:
sed -i -e 's/.*default_authentication_plugin.*/mysql-native-password = ON/' -e 's/\(.*innodb_autoinc_lock_mode.*\)/\1\ninnodb_numa_interleave = OFF/' -e "s/\(.*slow-query-log-file.*\)/\1\nlog_error_suppression_list = 'MY-013360'/" -e '/binlog_format/d' ~/.my.cnf sed -i '/service-startup-timeout/d' ~/.my.cnf
Finally, start the node normally:
If this is a database-only node (no Moogsoft Onprem components installed), run the following command:
(mysqld_safe > /dev/null 2>/dev/null)&
If this is a node with Moogsoft Onprem components installed, run the following command:
$MOOGSOFT_HOME/bin/utils/process_cntl mysql start
It will either sync immediately or spend a short amount of time performing an IST from another node. See the example below from mysqld.log:
2020-08-20T11:12:45.559572-00:00 0 [Note] WSREP: Receiving IST... 13.0% ( 96/739 events) complete. 2020-08-20T11:12:56.030910-00:00 0 [Note] WSREP: Receiving IST... 23.8% (176/739 events) complete. 2020-08-20T11:13:06.922647-00:00 0 [Note] WSREP: Receiving IST... 34.6% (256/739 events) complete. 2020-08-20T11:13:18.481076-00:00 0 [Note] WSREP: Receiving IST... 45.5% (336/739 events) complete. 2020-08-20T11:13:30.433314-00:00 0 [Note] WSREP: Receiving IST... 56.3% (416/739 events) complete. 2020-08-20T11:13:42.463826-00:00 0 [Note] WSREP: Receiving IST... 67.1% (496/739 events) complete. 2020-08-20T11:13:52.673644-00:00 0 [Note] WSREP: Receiving IST... 75.8% (560/739 events) complete. 2020-08-20T11:14:02.812737-00:00 0 [Note] WSREP: Receiving IST... 84.4% (624/739 events) complete. 2020-08-20T11:14:06.210387-00:00 0 [Note] WSREP: Receiving IST...100.0% (739/739 events) complete. 2020-08-20T11:14:06.211793-00:00 2 [Note] WSREP: IST received: beebd113-e080-11ea-8803-6f888df3319a:21784800 2020-08-20T11:14:06.212523-00:00 0 [Note] WSREP: 0.0 (pxc-node-ldev03): State transfer from 2.0 (pxc-node-ldev04) complete. 2020-08-20T11:14:06.212564-00:00 0 [Note] WSREP: SST leaving flow control 2020-08-20T11:14:06.212574-00:00 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 21784800) 2020-08-20T11:14:06.213148-00:00 0 [Note] WSREP: Member 0.0 (pxc-node-ldev03) synced with group. 2020-08-20T11:14:06.213174-00:00 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 21784800) 2020-08-20T11:14:06.213211-00:00 2 [Note] WSREP: Synchronized with group, ready for connections 2020-08-20T11:14:06.213234-00:00 2 [Note] WSREP: Setting wsrep_ready to true 2020-08-20T11:14:06.213254-00:00 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2020-08-20T11:14:44.973932-00:00 0 [Note] InnoDB: Buffer pool(s) load completed at 200820 11:14:44
Verify this node has synced before proceeding to the next node:
[root@server4]# curl http://server2:9198 Percona XtraDB Cluster Node is synced.
Step 4 - Perform the actions on Server3
Repeat the same actions on Server3 as you performed on Server2 in Step 3.
Step 5 - Perform the actions on Server1
Repeat the same actions on Server1 as you performed on Server2 in Step 3.
Step 6 - Final Cluster Check
Check that all three database nodes are synced:
[root@server4]# curl http://server1:9198 Percona XtraDB Cluster Node is synced. [root@server4]# curl http://server2:9198 Percona XtraDB Cluster Node is synced. [root@server4]# curl http://server3:9198 Percona XtraDB Cluster Node is synced.
Step 7 - Restart moog services
$MOOGSOFT_HOME/bin/utils/process_cntl rest_lam start $MOOGSOFT_HOME/bin/utils/process_cntl socket_lam start $MOOGSOFT_HOME/bin/utils/process_cntl moog_farmd start $MOOGSOFT_HOME/bin/utils/process_cntl apache-tomcat start