Skip to main content

Percona Cluster 5.7 RPM Minor Version Upgrade

Use the following instructions to upgrade an RPM Percona XtraDB Cluster to v5.7.36 to avoid Moogsoft Enterprise-impacting issues that have been identified in lower versions.

System Setup

This example uses the following sample server configuration for Moogsoft Enterprise and the Percona Cluster:

  • Server1 - Moogsoft Enterprise 8.x.x.x

  • Server2 - Percona Cluster Node1

  • Server3 - Percona Cluster Node2

  • Server4 - Percona Cluster Node3

Assumptions

This procedure guides a user with root/sudo access through an RPM-based online installation.

Definitions

  • IST - Incremental State Transfer. Functionality which instead of whole state snapshot can catch up with the group by receiving the missing writesets, but only if the writeset is still in the donor’s writeset cache.

  • SST - State Snapshot Transfer is the full copy of data from one node to another. Used when a new node joins the cluster; it has to transfer data from an existing node.

Upgrade Steps

Note

The commands provided in this procedure assume that the rest_lam and socket_lam are in use. Other configurations are possible with additional LAMs and integrations.

Use this command to stop additional services

sudo systemctl stop <lam_service_name>

Step 1 - Stop all Moog services that connect to the database

sudo systemctl stop restlamd
sudo systemctl stop socketlamd
sudo systemctl stop moogfarmd
sudo systemctl stop apache-tomcat

Note

While it is technically possible to perform the cluster upgrade when the system is running and processing events, doing so greatly increases the chances of issues and/or SST occurrence later in the upgrade.

Step 2 - Make sure the three database nodes are synced

Perform the following check from any location (expected output is below). If any nodes do not report they are synced, do not proceed with the upgrade and check the mysqld.log for possible causes.

[root@server1]# curl http://server2:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server3:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server4:9198
Percona XtraDB Cluster Node is synced.

Step 3 - Perform the actions on Server2

  1. Stop the MySQL node (both commands are included below to include the two most common scenarios):

    sudo systemctl stop mysql
sudo systemctl stop mysql@bootstrap.service

  2. Update the packages to 5.7.36:

    sudo yum update Percona-XtraDB-Cluster-*-57-5.7.36 percona-xtrabackup-24-2.4.24

  3. Backup the grastate.dat file. The command below assumes the default location (if it is not here, check the datadir folder which is specified in /etc/my.cnf).

    cp /var/lib/mysql/grastate.dat /var/lib/mysql/grastate.dat.bak

  4. Start the node in standalone mode which will leave the process running in the current terminal:

    sudo mysqld --user=mysql --wsrep-provider='none'

    There is no console output from this command. If the command returns immediately, check mysqld.log for potential reasons.

  5. Open another terminal on the server and run:

    mysql_upgrade

    The output will resemble the following:

    [root@server2 ~]# mysql_upgrade 
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
Checking system database.
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.engine_cost                                  OK
mysql.event                                        OK
mysql.func                                         OK
mysql.general_log                                  OK
mysql.gtid_executed                                OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.server_cost                                  OK
mysql.servers                                      OK
mysql.slave_master_info                            OK
mysql.slave_relay_log_info                         OK
mysql.slave_worker_info                            OK
mysql.slow_log                                     OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
Found outdated sys schema version 1.5.1.
Upgrading the sys schema.
Checking databases.
historic_moogdb.alert_journal                      OK
historic_moogdb.alerts                             OK
historic_moogdb.alerts_marked_prc_by_user          OK
historic_moogdb.comments                           OK
historic_moogdb.event_codes                        OK
historic_moogdb.events                             OK
historic_moogdb.feedback                           OK
historic_moogdb.invitations                        OK
historic_moogdb.notification                       OK
historic_moogdb.prc_earliest_highest_severity_event OK
historic_moogdb.resolving_thread_entries           OK
historic_moogdb.sig_alerts                         OK
historic_moogdb.sig_correlation_info               OK
historic_moogdb.sig_event_codes                    OK
historic_moogdb.sig_journal                        OK
historic_moogdb.sig_participants_in_room           OK
historic_moogdb.sig_pinpoint                       OK
historic_moogdb.sig_processes                      OK
historic_moogdb.sig_rating                         OK
historic_moogdb.sig_services                       OK
historic_moogdb.sig_teams                          OK
historic_moogdb.sig_view_by_user                   OK
historic_moogdb.sigs                               OK
historic_moogdb.snapshots                          OK
historic_moogdb.thread_entries                     OK
historic_moogdb.thread_entry_comments              OK
historic_moogdb.thread_entry_feedback              OK
historic_moogdb.threads                            OK
moog_intdb.DATABASECHANGELOG                       OK
moog_intdb.DATABASECHANGELOGLOCK                   OK
moog_intdb.broker_profiles                         OK
moog_intdb.broker_stage                            OK
moog_intdb.enabled_features                        OK
moog_intdb.expected_integration_state              OK
moog_intdb.featured_integrations                   OK
moog_intdb.installed_brokers                       OK
moog_intdb.installed_integrations                  OK
moog_intdb.integration_migration                   OK
moog_intdb.jwt_tokens                              OK
moog_reference.alert_properties                    OK
moog_reference.event_streams                       OK
moog_reference.execution_details                   OK
moog_reference.incremental_token_data              OK
moog_reference.one_hop_topo                        OK
moog_reference.schema_upgrades                     OK
moog_reference.schema_version                      OK
moog_reference.token_data                          OK
moog_reference.token_types                         OK
moog_reference.topo_cache                          OK
moog_reference.topo_nodes                          OK
moog_reference.topo_updates                        OK
moog_reference.vertex_entropy_max                  OK
moogdb.action_states                               OK
moogdb.actors                                      OK
moogdb.alert_column_names                          OK
moogdb.alert_filters                               OK
moogdb.alert_filters_access                        OK
moogdb.alert_journal                               OK
moogdb.alerts                                      OK
moogdb.alerts_custom_info_fields                   OK
moogdb.alerts_marked_prc_by_user                   OK
moogdb.assigned_ports                              OK
moogdb.authentication_info                         OK
moogdb.comments                                    OK
moogdb.config_migration                            OK
moogdb.default_filters                             OK
moogdb.default_system_config                       OK
moogdb.departments                                 OK
moogdb.duplicate_historic_alerts                   OK
moogdb.enrichment_static_data                      OK
moogdb.enrichment_static_mappings                  OK
moogdb.enrichments                                 OK
moogdb.entity_catalog                              OK
moogdb.event_codes                                 OK
moogdb.events                                      OK
moogdb.features                                    OK
moogdb.feedback                                    OK
moogdb.feedback_models                             OK
moogdb.feedback_training_queues                    OK
moogdb.functional_passwords                        OK
moogdb.groups                                      OK
moogdb.integration_properties                      OK
moogdb.integration_url_tools                       OK
moogdb.invitations                                 OK
moogdb.invite_handlers                             OK
moogdb.invite_states                               OK
moogdb.known_sig_states                            OK
moogdb.last_job_run_info                           OK
moogdb.latest_insight_ids                          OK
moogdb.licence_info                                OK
moogdb.link_definitions                            OK
moogdb.maintenance_windows                         OK
moogdb.media                                       OK
moogdb.moog_indexer                                OK
moogdb.mtt_insights                                OK
moogdb.mtt_team_insights                           OK
moogdb.net_types                                   OK
moogdb.notification                                OK
moogdb.notification_type                           OK
moogdb.prc_archived_training_alerts_data           OK
moogdb.prc_archived_training_sig_alerts_data       OK
moogdb.prc_archived_training_sigs_data             OK
moogdb.prc_models                                  OK
moogdb.processes                                   OK
moogdb.product_deployment                          OK
moogdb.raw_lam_events                              OK
moogdb.recorded_product_actions                    OK
moogdb.resolving_thread_entries                    OK
moogdb.role_properties                             OK
moogdb.roles                                       OK
moogdb.room_post_comments                          OK
moogdb.room_post_media                             OK
moogdb.room_post_types                             OK
moogdb.room_posts                                  OK
moogdb.rooms                                       OK
moogdb.run_on_startup                              OK
moogdb.schema_upgrades                             OK
moogdb.schema_version                              OK
moogdb.security_realms                             OK
moogdb.service_insights                            OK
moogdb.services                                    OK
moogdb.sessions                                    OK
moogdb.severity                                    OK
moogdb.severity_insights                           OK
moogdb.sig_alerts                                  OK
moogdb.sig_category                                OK
moogdb.sig_correlation_info                        OK
moogdb.sig_event_codes                             OK
moogdb.sig_journal                                 OK
moogdb.sig_participants_in_room                    OK
moogdb.sig_pinpoint                                OK
moogdb.sig_processes                               OK
moogdb.sig_queue                                   OK
moogdb.sig_rating                                  OK
moogdb.sig_services                                OK
moogdb.sig_state                                   OK
moogdb.sig_stats_cache                             OK
moogdb.sig_teams                                   OK
moogdb.sig_view_by_user                            OK
moogdb.significance                                OK
moogdb.sigs                                        OK
moogdb.sigs_custom_info_fields                     OK
moogdb.sigs_lists_helper                           OK
moogdb.sitroom_plugins                             OK
moogdb.situation_column_names                      OK
moogdb.situation_filters                           OK
moogdb.situation_filters_access                    OK
moogdb.snapshots                                   OK
moogdb.split_alert_events                          OK
moogdb.split_alerts                                OK
moogdb.split_events                                OK
moogdb.split_last_timestamp                        OK
moogdb.split_sigs                                  OK
moogdb.status                                      OK
moogdb.status_insights                             OK
moogdb.system_config                               OK
moogdb.system_insights                             OK
moogdb.team_insights                               OK
moogdb.team_services                               OK
moogdb.teams                                       OK
moogdb.temporary_user_insights                     OK
moogdb.thread_entries                              OK
moogdb.thread_entry_comments                       OK
moogdb.thread_entry_feedback                       OK
moogdb.threads                                     OK
moogdb.tools_access                                OK
moogdb.transitions                                 OK
moogdb.ui_statistics                               OK
moogdb.user_insights                               OK
moogdb.user_properties                             OK
moogdb.user_roles                                  OK
moogdb.user_status                                 OK
moogdb.user_teams                                  OK
moogdb.users                                       OK
moogdb.users_realm_identifier                      OK
moogdb.view_configs                                OK
moogdb.workflow_alerts_accessed                    OK
moogdb.workflow_moolets                            OK
moogdb.workflow_situations_accessed                OK
moogdb.workflows                                   OK
sys.sys_config                                     OK
Upgrade process completed successfully.
Checking if update is needed.

  6. Kill the standalone node:

    sudo kill <pid of mysqld process>

    Example:

    [root@server2 ~]# ps -ef|grep mysqld
root       4200 191073  0 11:25 pts/0    
00:00:00 sudo mysqld --user=mysql --wsrep-provider=none
mysql      4201   4200 47 11:25 pts/0    
00:00:14 mysqld --user=mysql --wsrep-provider=none
root       4514   4394  0 11:25 pts/1    
00:00:00 grep --color=auto mysqld
[root@server2 ~]# sudo kill 4201

  7. Add the following property to the [mysqld_safe] section of the Percona Node's configuration file /etc/my.cnf to remove the restriction on SST duration:

    service-startup-timeout = -1

    e.g.:

    [mysqld_safe]
log-error = /var/log/mysqld.log
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
nice = 0
service-startup-timeout = -1

  8. Finally start the node normally:

    sudo systemctl start mysql

    It will either sync immediately or spend a short amount of time performing an IST from another node. See the example below from /var/log/mysql/mysqld.log:

    2020-08-20T11:12:45.559572-00:00 0 [Note] WSREP: Receiving IST... 
13.0% ( 96/739 events) complete.
2020-08-20T11:12:56.030910-00:00 0 [Note] WSREP: Receiving IST... 
23.8% (176/739 events) complete.
2020-08-20T11:13:06.922647-00:00 0 [Note] WSREP: Receiving IST... 
34.6% (256/739 events) complete.
2020-08-20T11:13:18.481076-00:00 0 [Note] WSREP: Receiving IST... 
45.5% (336/739 events) complete.
2020-08-20T11:13:30.433314-00:00 0 [Note] WSREP: Receiving IST... 
56.3% (416/739 events) complete.
2020-08-20T11:13:42.463826-00:00 0 [Note] WSREP: Receiving IST... 
67.1% (496/739 events) complete.
2020-08-20T11:13:52.673644-00:00 0 [Note] WSREP: Receiving IST... 
75.8% (560/739 events) complete.
2020-08-20T11:14:02.812737-00:00 0 [Note] WSREP: Receiving IST... 
84.4% (624/739 events) complete.
2020-08-20T11:14:06.210387-00:00 0 [Note] WSREP: Receiving IST...
100.0% (739/739 events) complete.
2020-08-20T11:14:06.211793-00:00 2 [Note] WSREP: IST 
received: beebd113-e080-11ea-8803-6f888df3319a:21784800
2020-08-20T11:14:06.212523-00:00 0 [Note] WSREP: 0.0 
(pxc-node-server2): State transfer from 2.0 
(pxc-node-server3) complete.
2020-08-20T11:14:06.212564-00:00 0 [Note] WSREP: SST leaving flow control
2020-08-20T11:14:06.212574-00:00 0 [Note] WSREP: Shifting 
JOINER -> JOINED (TO: 21784800)
2020-08-20T11:14:06.213148-00:00 0 [Note] WSREP: Member 0.0 
(pxc-node-server2) synced with group.
2020-08-20T11:14:06.213174-00:00 0 [Note] WSREP: Shifting JOINED 
-> SYNCED (TO: 21784800)
2020-08-20T11:14:06.213211-00:00 2 [Note] WSREP: Synchronized 
with group, ready for connections
2020-08-20T11:14:06.213234-00:00 2 [Note] WSREP: Setting 
wsrep_ready to true
2020-08-20T11:14:06.213254-00:00 2 [Note] WSREP: wsrep_notify_cmd 
is not defined, skipping notification.
2020-08-20T11:14:44.973932-00:00 0 [Note] InnoDB: Buffer 
pool(s) load completed at 200820 11:14:44

  9. Verify this node has synced before proceeding to the next node:

    [root@server1]#  curl http://server2:9198
Percona XtraDB Cluster Node is synced.

Step 4 - Perform the actions on Server3

Repeat the same actions on Server3 as you performed on Server2 in Step 3.

Step 5 - Server4 actions:

Repeat the same actions on Server4 as you performed on Server2 in Step 3.

Step 6 - Final Cluster Check

Check that all three database nodes are synced:

[root@server1]# curl http://server2:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server3:9198
Percona XtraDB Cluster Node is synced.
[root@server1]# curl http://server4:9198
Percona XtraDB Cluster Node is synced.

Step 7 - Resume Moog Services

sudo systemctl start apache-tomcat
sudo systemctl start moogfarmd
sudo systemctl start restlamd
sudo systemctl start socketlamd

Fixing the upgraded node if an SST occurs

When the upgraded node rejoins the cluster, it is important that it synchronizes with the cluster using IST. If an SST occurs, you may need to upgrade the data directory structure using mysql_upgrade again to make sure it is compatible with the newer version of the binaries.

Note

Check the error log for statements like the following to determine if an SST occurred:

“Check if state gap can be serviced using IST ... State gap can’t be serviced using IST. Switching to SST” instead of “Receiving IST: ...” lines appropriate to IST synchronization.

Perform the following additional steps to upgrade the data directory structure after SST completes:

  1. Shut down the node that rejoined the cluster using SST:

    sudo systemctl stop mysql

  2. Restart the node in standalone mode:

    sudo mysqld --user=mysql --wsrep-provider='none'

  3. Run mysql-upgrade in another terminal then kill the standalone node.

  4. Restart the node normally and ensure the node rejoins the cluster using IST:

    sudo systemctl start mysql
In this section: