Percona Cluster 5.7 RPM Minor Version Upgrade

Use the following instructions to upgrade Percona XtraDB to 5.7.29 to avoid a potential issue that can cause the cluster to periodically lock up. This issue can affect versions of Percona XtraDB prior to 5.7.29, including the version of Percona shipping with Moogsoft AIOps version 7.3.x.

System Setup

This example uses the following sample server configuration for Moogsoft AIOps and the Percona Cluster:

  • Server1 - Moogsoft AIOps

  • Server2 - Percona Cluster Node1

  • Server3 - Percona Cluster Node2

  • Server4 - Percona Cluster Node3


This procedure guides a user with root/sudo access through an RPM-based online installation.


  • IST - Incremental State Transfer. Functionality which instead of whole state snapshot can catch up with the group by receiving the missing writesets, but only if the writeset is still in the donor’s writeset cache.

  • SST - State Snapshot Transfer is the full copy of data from one node to another. Used when a new node joins the cluster; it has to transfer data from an existing node.

Upgrade Steps


The commands provided in this procedure assume that the rest_lam and socket_lam are in use. Other configurations are possible with additional LAMs and integrations.

Use this command to stop additional services

sudo systemctl stop <lam_service_name>

Step 1 - Stop all Moog services that connect to the database

sudo systemctl stop restlamd
sudo systemctl stop socketlamd
sudo systemctl stop moogfarmd
sudo systemctl stop apache-tomcat


While it is technically possible to perform the cluster upgrade when the system is running and processing events, doing so greatly increases the chances of issues and/or SST occurrence later in the upgrade.

Step 2 - Make sure the three database nodes are synced

Perform the following check from any location (expected output is below). If any nodes do not report they are synced, do not proceed with the upgrade and check the mysqld.log for possible causes.

[root@server1]# curl http://server2:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server3:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server4:9198
Percona XtraDB Cluster Node is synced.

Step 3 - Perform the actions on Server2

  1. Stop the MySQL node (both commands are included below to include the two most common scenarios):

    sudo systemctl stop mysql
    sudo systemctl stop mysql@bootstrap.service
  2. Update the packages to 5.7.29:

    sudo yum update Percona-XtraDB-Cluster-*-57-5.7.29
  3. Backup the grastate.dat file. The command below assumes the default location (if it is not here, check the datadir folder which is specified in /etc/my.cnf).

    cp /var/lib/mysql/grastate.dat /var/lib/mysql/grastate.dat.bak
  4. Start the node in standalone mode which will leave the process running in the current terminal:

    sudo mysqld --user=mysql --wsrep-provider='none'

    There is no console output from this command. If the command returns immediately, check mysqld.log for potential reasons.

  5. Open another terminal on the server and run:


    The output will resemble the following:

    [root@server2 ~]# mysql_upgrade 
    Checking if update is needed.
    Checking server version.
    Running queries to upgrade MySQL server.
    Checking system database.
    mysql.columns_priv                                 OK
    mysql.db                                           OK
    mysql.engine_cost                                  OK
    mysql.event                                        OK
    mysql.func                                         OK
    mysql.general_log                                  OK
    mysql.gtid_executed                                OK
    mysql.help_category                                OK
    mysql.help_keyword                                 OK
    mysql.help_relation                                OK
    mysql.help_topic                                   OK
    mysql.innodb_index_stats                           OK
    mysql.innodb_table_stats                           OK
    mysql.ndb_binlog_index                             OK
    mysql.plugin                                       OK
    mysql.proc                                         OK
    mysql.procs_priv                                   OK
    mysql.proxies_priv                                 OK
    mysql.server_cost                                  OK
    mysql.servers                                      OK
    mysql.slave_master_info                            OK
    mysql.slave_relay_log_info                         OK
    mysql.slave_worker_info                            OK
    mysql.slow_log                                     OK
    mysql.tables_priv                                  OK
    mysql.time_zone                                    OK
    mysql.time_zone_leap_second                        OK
    mysql.time_zone_name                               OK
    mysql.time_zone_transition                         OK
    mysql.time_zone_transition_type                    OK
    mysql.user                                         OK
    Found outdated sys schema version 1.5.1.
    Upgrading the sys schema.
    Checking databases.
    historic_moogdb.alert_journal                      OK
    historic_moogdb.alerts                             OK
    historic_moogdb.alerts_marked_prc_by_user          OK
    historic_moogdb.comments                           OK
    historic_moogdb.event_codes                        OK                             OK                           OK
    historic_moogdb.invitations                        OK
    historic_moogdb.notification                       OK
    historic_moogdb.prc_earliest_highest_severity_event OK
    historic_moogdb.resolving_thread_entries           OK
    historic_moogdb.sig_alerts                         OK
    historic_moogdb.sig_correlation_info               OK
    historic_moogdb.sig_event_codes                    OK
    historic_moogdb.sig_journal                        OK
    historic_moogdb.sig_participants_in_room           OK
    historic_moogdb.sig_pinpoint                       OK
    historic_moogdb.sig_processes                      OK
    historic_moogdb.sig_rating                         OK
    historic_moogdb.sig_services                       OK
    historic_moogdb.sig_teams                          OK
    historic_moogdb.sig_view_by_user                   OK
    historic_moogdb.sigs                               OK
    historic_moogdb.snapshots                          OK
    historic_moogdb.thread_entries                     OK
    historic_moogdb.thread_entry_comments              OK
    historic_moogdb.thread_entry_feedback              OK
    historic_moogdb.threads                            OK
    moog_intdb.DATABASECHANGELOG                       OK
    moog_intdb.DATABASECHANGELOGLOCK                   OK
    moog_intdb.broker_profiles                         OK
    moog_intdb.broker_stage                            OK
    moog_intdb.enabled_features                        OK
    moog_intdb.expected_integration_state              OK
    moog_intdb.featured_integrations                   OK
    moog_intdb.installed_brokers                       OK
    moog_intdb.installed_integrations                  OK
    moog_intdb.integration_migration                   OK
    moog_intdb.jwt_tokens                              OK
    moog_reference.alert_properties                    OK
    moog_reference.event_streams                       OK
    moog_reference.execution_details                   OK
    moog_reference.incremental_token_data              OK
    moog_reference.one_hop_topo                        OK
    moog_reference.schema_upgrades                     OK
    moog_reference.schema_version                      OK
    moog_reference.token_data                          OK
    moog_reference.token_types                         OK
    moog_reference.topo_cache                          OK
    moog_reference.topo_nodes                          OK
    moog_reference.topo_updates                        OK
    moog_reference.vertex_entropy_max                  OK
    moogdb.action_states                               OK
    moogdb.actors                                      OK
    moogdb.alert_column_names                          OK
    moogdb.alert_filters                               OK
    moogdb.alert_filters_access                        OK
    moogdb.alert_journal                               OK
    moogdb.alerts                                      OK
    moogdb.alerts_custom_info_fields                   OK
    moogdb.alerts_marked_prc_by_user                   OK
    moogdb.assigned_ports                              OK
    moogdb.authentication_info                         OK
    moogdb.comments                                    OK
    moogdb.config_migration                            OK
    moogdb.default_filters                             OK
    moogdb.default_system_config                       OK
    moogdb.departments                                 OK
    moogdb.duplicate_historic_alerts                   OK
    moogdb.enrichment_static_data                      OK
    moogdb.enrichment_static_mappings                  OK
    moogdb.enrichments                                 OK
    moogdb.entity_catalog                              OK
    moogdb.event_codes                                 OK                                      OK
    moogdb.features                                    OK                                    OK
    moogdb.feedback_models                             OK
    moogdb.feedback_training_queues                    OK
    moogdb.functional_passwords                        OK
    moogdb.groups                                      OK
    moogdb.integration_properties                      OK
    moogdb.integration_url_tools                       OK
    moogdb.invitations                                 OK
    moogdb.invite_handlers                             OK
    moogdb.invite_states                               OK
    moogdb.known_sig_states                            OK
    moogdb.last_job_run_info                           OK
    moogdb.latest_insight_ids                          OK
    moogdb.licence_info                                OK
    moogdb.link_definitions                            OK
    moogdb.maintenance_windows                         OK                                       OK
    moogdb.moog_indexer                                OK
    moogdb.mtt_insights                                OK
    moogdb.mtt_team_insights                           OK
    moogdb.net_types                                   OK
    moogdb.notification                                OK
    moogdb.notification_type                           OK
    moogdb.prc_archived_training_alerts_data           OK
    moogdb.prc_archived_training_sig_alerts_data       OK
    moogdb.prc_archived_training_sigs_data             OK
    moogdb.prc_models                                  OK
    moogdb.processes                                   OK
    moogdb.product_deployment                          OK
    moogdb.raw_lam_events                              OK
    moogdb.recorded_product_actions                    OK
    moogdb.resolving_thread_entries                    OK
    moogdb.role_properties                             OK
    moogdb.roles                                       OK
    moogdb.room_post_comments                          OK
    moogdb.room_post_media                             OK
    moogdb.room_post_types                             OK
    moogdb.room_posts                                  OK
    moogdb.rooms                                       OK
    moogdb.run_on_startup                              OK
    moogdb.schema_upgrades                             OK
    moogdb.schema_version                              OK
    moogdb.security_realms                             OK
    moogdb.service_insights                            OK                                    OK
    moogdb.sessions                                    OK
    moogdb.severity                                    OK
    moogdb.severity_insights                           OK
    moogdb.sig_alerts                                  OK
    moogdb.sig_category                                OK
    moogdb.sig_correlation_info                        OK
    moogdb.sig_event_codes                             OK
    moogdb.sig_journal                                 OK
    moogdb.sig_participants_in_room                    OK
    moogdb.sig_pinpoint                                OK
    moogdb.sig_processes                               OK
    moogdb.sig_queue                                   OK
    moogdb.sig_rating                                  OK
    moogdb.sig_services                                OK
    moogdb.sig_state                                   OK
    moogdb.sig_stats_cache                             OK
    moogdb.sig_teams                                   OK
    moogdb.sig_view_by_user                            OK
    moogdb.significance                                OK
    moogdb.sigs                                        OK
    moogdb.sigs_custom_info_fields                     OK
    moogdb.sigs_lists_helper                           OK
    moogdb.sitroom_plugins                             OK
    moogdb.situation_column_names                      OK
    moogdb.situation_filters                           OK
    moogdb.situation_filters_access                    OK
    moogdb.snapshots                                   OK
    moogdb.split_alert_events                          OK
    moogdb.split_alerts                                OK
    moogdb.split_events                                OK
    moogdb.split_last_timestamp                        OK
    moogdb.split_sigs                                  OK
    moogdb.status                                      OK
    moogdb.status_insights                             OK
    moogdb.system_config                               OK
    moogdb.system_insights                             OK
    moogdb.team_insights                               OK
    moogdb.team_services                               OK
    moogdb.teams                                       OK
    moogdb.temporary_user_insights                     OK
    moogdb.thread_entries                              OK
    moogdb.thread_entry_comments                       OK
    moogdb.thread_entry_feedback                       OK
    moogdb.threads                                     OK
    moogdb.tools_access                                OK
    moogdb.transitions                                 OK
    moogdb.ui_statistics                               OK
    moogdb.user_insights                               OK
    moogdb.user_properties                             OK
    moogdb.user_roles                                  OK
    moogdb.user_status                                 OK
    moogdb.user_teams                                  OK
    moogdb.users                                       OK
    moogdb.users_realm_identifier                      OK
    moogdb.view_configs                                OK
    moogdb.workflow_alerts_accessed                    OK
    moogdb.workflow_moolets                            OK
    moogdb.workflow_situations_accessed                OK
    moogdb.workflows                                   OK
    sys.sys_config                                     OK
    Upgrade process completed successfully.
    Checking if update is needed.
  6. Kill the standalone node:

    sudo kill <pid of mysqld process>


    [root@server2 ~]# ps -ef|grep mysqld
    root       4200 191073  0 11:25 pts/0    
    00:00:00 sudo mysqld --user=mysql --wsrep-provider=none
    mysql      4201   4200 47 11:25 pts/0    
    00:00:14 mysqld --user=mysql --wsrep-provider=none
    root       4514   4394  0 11:25 pts/1    
    00:00:00 grep --color=auto mysqld
    [root@server2 ~]# sudo kill 4201
  7. Finally start the node normally:

    sudo systemctl start mysql

    It will either sync immediately or spend a short amount of time performing an IST from another node. See the example below from /var/log/mysql/mysqld.log:

    2020-08-20T11:12:45.559572-00:00 0 [Note] WSREP: Receiving IST... 
    13.0% ( 96/739 events) complete.
    2020-08-20T11:12:56.030910-00:00 0 [Note] WSREP: Receiving IST... 
    23.8% (176/739 events) complete.
    2020-08-20T11:13:06.922647-00:00 0 [Note] WSREP: Receiving IST... 
    34.6% (256/739 events) complete.
    2020-08-20T11:13:18.481076-00:00 0 [Note] WSREP: Receiving IST... 
    45.5% (336/739 events) complete.
    2020-08-20T11:13:30.433314-00:00 0 [Note] WSREP: Receiving IST... 
    56.3% (416/739 events) complete.
    2020-08-20T11:13:42.463826-00:00 0 [Note] WSREP: Receiving IST... 
    67.1% (496/739 events) complete.
    2020-08-20T11:13:52.673644-00:00 0 [Note] WSREP: Receiving IST... 
    75.8% (560/739 events) complete.
    2020-08-20T11:14:02.812737-00:00 0 [Note] WSREP: Receiving IST... 
    84.4% (624/739 events) complete.
    2020-08-20T11:14:06.210387-00:00 0 [Note] WSREP: Receiving IST...
    100.0% (739/739 events) complete.
    2020-08-20T11:14:06.211793-00:00 2 [Note] WSREP: IST 
    received: beebd113-e080-11ea-8803-6f888df3319a:21784800
    2020-08-20T11:14:06.212523-00:00 0 [Note] WSREP: 0.0 
    (pxc-node-server2): State transfer from 2.0 
    (pxc-node-server3) complete.
    2020-08-20T11:14:06.212564-00:00 0 [Note] WSREP: SST leaving flow control
    2020-08-20T11:14:06.212574-00:00 0 [Note] WSREP: Shifting 
    JOINER -> JOINED (TO: 21784800)
    2020-08-20T11:14:06.213148-00:00 0 [Note] WSREP: Member 0.0 
    (pxc-node-server2) synced with group.
    2020-08-20T11:14:06.213174-00:00 0 [Note] WSREP: Shifting JOINED 
    -> SYNCED (TO: 21784800)
    2020-08-20T11:14:06.213211-00:00 2 [Note] WSREP: Synchronized 
    with group, ready for connections
    2020-08-20T11:14:06.213234-00:00 2 [Note] WSREP: Setting 
    wsrep_ready to true
    2020-08-20T11:14:06.213254-00:00 2 [Note] WSREP: wsrep_notify_cmd 
    is not defined, skipping notification.
    2020-08-20T11:14:44.973932-00:00 0 [Note] InnoDB: Buffer 
    pool(s) load completed at 200820 11:14:44
  8. Verify this node has synced before proceeding to the next node:

    [root@server1]#  curl http://server2:9198
    Percona XtraDB Cluster Node is synced.

Step 4 - Perform the actions on Server3

Repeat the same actions on Server3 as you performed on Server2 in Step 3.

Step 5 - Server4 actions:

Repeat the same actions on Server4 as you performed on Server2 in Step 3.

Step 6 - Final Cluster Check

Check that all three database nodes are synced:

[root@server1]# curl http://server2:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server3:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server4:9198
Percona XtraDB Cluster Node is synced.

Step 7 - Resume Moog Services

sudo systemctl start apache-tomcat
sudo systemctl start moogfarmd
sudo systemctl start restlamd
sudo systemctl start socketlamd

Fixing the upgraded node if an SST occurs

When the upgraded node rejoins the cluster, it is important that it synchronizes with the cluster using IST. If an SST occurs, you may need to upgrade the data directory structure using mysql_upgrade again to make sure it is compatible with the newer version of the binaries.


Check the error log for statements like the following to determine if an SST occurred:

“Check if state gap can be serviced using IST ... State gap can’t be serviced using IST. Switching to SST” instead of “Receiving IST: ...” lines appropriate to IST synchronization.

Perform the following additional steps to upgrade the data directory structure after SST completes:

  1. Shut down the node that rejoined the cluster using SST:

    sudo systemctl stop mysql
  2. Restart the node in standalone mode:

    sudo mysqld --user=mysql --wsrep-provider='none'
  3. Run mysql-upgrade in another terminal then kill the standalone node.

  4. Restart the node normally and ensure the node rejoins the cluster using IST:

    sudo systemctl start mysql