I’m writing this article from the point of view of a NetEye system engineer. I want to share my findings from my path of improving the quality of the underlying Galera cluster architecture. The aim is to provide you with a practical guide for realizing non-blocking state transfers, Galera-aware health checks, database tuning and reliable connection management.

It’s not intended as a cookbook or a set of configurations to apply unchanged to every environment. Instead, it aims to raise awareness of the optimization strategies available, and encourage you to evaluate how the underlying database and cluster architecture can be adapted to your own workload, hardware, and operational requirements.

The Scenario

Imagine a typical production incident in a NetEye monitoring cluster: A temporary network problem causes one of the active MariaDB Galera nodes to restart. The node comes back online, a State Snapshot Transfer (SST) starts automatically, and the monitoring services appear to continue running. In practice however, the web interface may behave unpredictably. One page loads normally, while the next returns an SQL error. This instability can continue until the SST finishes.

The situation is particularly serious in a 2+1 cluster consisting of two active database nodes and one arbitration (voting) node. If a large database requires a full SST, the result can be a lengthy service interruption: the NetEye web interface becomes unavailable, and real-time infrastructure monitoring stops.

This article explains why that happens and presents a production-oriented approach to keeping node recovery controlled and predictable.

The 2+1 Architecture and Its Failure Modes

Using two active database nodes together with a dedicated arbitrator (garbd) is an efficient way to reduce the risk of a split-brain condition without adding unnecessary infrastructure. However, when one database node fails, two independent problems can occur at the same time. Let’s examine each one in turn.

Database Locking During an rsync SST

If the cluster uses the default rsync SST method, the donor node locks its tables to create a consistent snapshot. Write operations are blocked for as long as the lock is active. Because a 2+1 cluster has only two active database nodes, there is no third active node to handle incoming writes. The result can be a complete write outage for the monitoring platform until the transfer finishes.

Load Balancers Sending Traffic to Joining Nodes

Frontend TCP proxies such as open-source Nginx can provide excellent performance and reliability. However, a standard TCP health check typically verifies only that a port – such as port 3307 – is open. It doesn’t know whether the Galera node is a Joiner, Donor, Synced, or temporarily unable to serve production traffic.

Solution 1: Use Mariabackup for Non-Blocking SST

Galera uses synchronous replication during normal operation. When a node returns after a longer interruption, it may require a full SST. Rsync is commonly used because it’s widely available, does not require additional configuration, and is well-tested. However, creating a consistent snapshot requires the donor node to execute FLUSH TABLES WITH READ LOCK. While this lock is active, the node cannot process INSERT, UPDATE, or DELETE statements.

Mariabackup, also known as mariadb-backup, is a modern alternative for SST operations. It performs an online physical backup of InnoDB data and uses the transaction or redo log to track changes made during the backup. Because it doesn’t require a global table lock, the donor can continue processing read and write requests while synchronization is in progress.

The database therefore remains available, and the donor stays responsive from the load balancer’s perspective instead of appearing to be stalled during the transfer.

Setting up and Tuning SST Performance

Before enabling Mariabackup for SST, make sure that Mariabackup is installed and available on every MariaDB node that may act as a donor or joiner. The socat package is also required on all participating nodes because it is used to stream the backup from the donor to the joining node. Finally, configure the SST authentication account consistently across all affected nodes, and grant it the permissions required by the MariaDB and Galera configuration.

To do that, add the following configuration to the MariaDB configuration directory:

# /neteye/local/mariadb/my.cnf.d/sst.cnf
[mariadb]
wsrep_sst_method = mariabackup
wsrep_sst_auth = mariadbbackup:<SECURE_PASSWORD>

The account and password must be configured consistently across the cluster nodes. Store the password securely and restrict access to the configuration file. The SST method can be changed dynamically for the next transfer:

SET GLOBAL wsrep_sst_method = 'mariabackup';

Once Mariabackup is configured, you can tune its transfer performance. For databases ranging from hundreds of gigabytes to several terabytes, SST speed is usually limited by disk I/O and network bandwidth. On systems with multi-core CPUs and fast 10 or 20 Gbps network connections, the following settings can help use the available resources more effectively:

• parallel = 24: Uses multiple threads to read data from NVMe or enterprise SSD storage. Adjust the value to match the available CPU capacity and system load.
• zstd –fast=1: Enables fast compression with relatively low CPU overhead. This can reduce the amount of data sent over the network.
• -T24: Uses 24 CPU threads for compression and decompression.

Now add the performance settings to the same configuration file:

[mariabackup]
parallel = 24

[sst]
compressor = "zstd --fast=1 -T24"
decompressor = "zstd -d -T24"

Some advice: The values above are examples for a system with sufficient CPU resources, fast storage, and a high-bandwidth network connection. They should be adjusted and tested according to the actual hardware.

Solution 2: Galera-Aware Health Checks and Dynamic Traffic Management

When incoming database requests are load-balanced across Galera nodes, a node may receive traffic even if it isn’t actually ready yet to serve production requests. The open-source version of NGINX performs TCP-level availability checks: It will verify that the destination socket or port is reachable, but it won’t validate the application-level health or Galera synchronization state of the MariaDB node behind it. This solution applies to open-source NGINX implementations; it may not apply to the same extent to enterprise distributions NGINX offerings with more advanced health-check capabilities.

In practice, NGINX only checks whether it can establish a TCP connection to port 3307. If the connection succeeds, it immediately sends traffic to that node. This creates problems in two common situations:

• Degraded or Asynchronous Cluster State: MariaDB may still listen on port 3307 even when replication is broken or the node is no longer part of the cluster quorum. From NGINX’s perspective, the port is open, although the node may not be safe to use.
• Joiner State During an Active SST: A recovering node may open port 3307 before the SST has finished. NGINX sees an available port and begins routing queries to the node. Those requests fail because the node is still joining the cluster and is not ready for production traffic.

The clustercheck Daemon and Local Firewall Control

A lightweight monitoring script can run as a systemd service and check the local MariaDB node every two seconds (can be changed as you like) directly through its UNIX socket (no port needed). This avoids creating a network connection to the local database server. The script evaluates the Galera state and determines whether the node is ready to handle production traffic.

mysql --no-defaults --protocol=SOCKET \
  -u clustercheck -p"${pwd_mariadb_clustercheck}" \
  -S /var/lib/mysql/mysql-galera.sock \
  -e "SHOW GLOBAL VARIABLES LIKE 'read_only'; \
      SHOW GLOBAL VARIABLES LIKE 'wsrep_sst_method'; \
      SHOW GLOBAL STATUS LIKE 'wsrep_local_state';"

If the node isn’t ready, the script adds an iptables REJECT rule for port 3307, so that incoming TCP connection attempts are rejected immediately. NGINX detects the failed connection and quickly removes the node from the active upstream pool. When the node becomes healthy again, the script removes the firewall rule and makes the port available to NGINX.

Node state	Policy
Synced (wsrep_local_state = 4)	Keep port 3307 open. The node can serve read and write queries.
Donor using Mariabackup (state = 2)	Keep port 3307 open because Mariabackup performs a non-blocking backup.
Joiner, desynced, initializing, or another non-ready state	Add an iptables REJECT rule so the upstream proxy bypasses the node.
Any state with read_only = ON	Reject connections to prevent partial or failed write operations.

This gives the load balancer a simple signal: An open port means the node is ready, while an immediate rejection means it should not receive traffic. The approach combines Galera-aware health checking with the existing NGINX infrastructure without requiring a more complex proxy layer.

Galera and InnoDB Optimizations

After configuring SST and health checks, the next step is to consider optimizations in Galera and the underlying database engine. Both layers provide configuration options that can be adapted to the cluster’s workload and hardware.

All changes should be tested and validated in a representative environment before being applied in production!

Durability and Replication Apply Performance

innodb_flush_log_at_trx_commit = 2
wsrep_slave_threads = 16

innodb_flush_log_at_trx_commit = 2 flushes the redo log approximately once per second instead of after every commit. This can reduce disk I/O wait and improve write throughput. It also introduces a durability trade-off: Transactions committed shortly before a server or operating-system failure may not yet be present in the local redo log. Test this setting against the application’s durability requirements.

wsrep_slave_threads = 16 increases the number of parallel threads used to apply incoming replication events. This can reduce replication lag and the risk of Flow Control pauses during heavy write activity. The appropriate value depends on CPU resources, workload, and transaction patterns. More threads do not always improve performance, so validate the setting you use with production-like tests.

Buffer Pool Warm-Up

The following settings preserve frequently used data pages when MariaDB shuts down, and reload them during startup:

innodb_buffer_pool_dump_at_shutdown = 1
innodb_buffer_pool_load_at_startup = 1
innodb_buffer_pool_dump_pct = 75

This reduces the performance drop associated with a cold start and helps a node return to normal performance more quickly after rejoining the cluster.

NVMe-Optimized InnoDB I/O Settings

innodb_flush_method = O_DIRECT
innodb_flush_neighbors = 0
innodb_io_capacity = 8000
innodb_io_capacity_max = 16000
innodb_read_io_threads = 16
innodb_write_io_threads = 16

O_DIRECT helps avoid unnecessary double buffering. The setting innodb_flush_neighbors = 0 is appropriate for SSD and NVMe storage, which does not benefit from flushing neighboring pages together. Higher I/O capacity and parallel read/write threads allow InnoDB to more aggressively perform background flushing and other I/O operations.

These values depend on the hardware and workload. Validate them against storage latency and actual database performance; excessively high values can increase background I/O and compete with foreground operations.

Undo Log and Purge Management

innodb_undo_log_truncate = ON
innodb_max_undo_log_size = 1G
innodb_purge_rseg_truncate_frequency = 32
innodb_purge_threads = 8

Write-intensive workloads can generate significant undo data. These settings help control undo tablespace growth and reduce purge lag. Long-running transactions can still prevent old row versions from being removed, so monitor transaction duration, history list length, undo tablespace growth, and purge activity together.

Reducing Lock Contention

innodb_autoinc_lock_mode = 2 uses interleaved auto-increment locking. It can reduce contention during concurrent inserts by avoiding broader table-level auto-increment locks. Test it with the application’s insert patterns and replication requirements.

innodb_adaptive_hash_index = 0 disables the adaptive hash index. This may reduce internal contention in highly concurrent workloads where the index becomes a bottleneck rather than a benefit. The effect depends on the workload, access patterns, and MariaDB version, so confirm it through monitoring and testing.

Minimizing Resynchronizations with Gcache and IST

The fastest SST is the one that never has to happen. When a node is restarted only briefly, Galera may perform an Incremental State Transfer (IST) instead of copying the entire database. During an IST, the node receives only the transactions it missed while offline.

Galera stores recent transaction history in a ring buffer called gcache. A suitable configuration may look like this:

wsrep_provider_options = "gcache.size=8G;gcache.keep_pages_size=1G;gcache.recover=yes;gcs.fc_limit=128;gcs.fc_factor=0.8;"

Important settings include:

• gcache.size = 8G: Retains more recent transaction history on the node.
• gcache.keep_pages_size = 1G: Reserves space for gcache pages.
• gcache.recover = yes: Helps recover gcache information after a restart, making it easier to identify the last known sequence position and request only missing transactions.
• gcs.fc_limit = 128: Raises the replication queue threshold before Galera activates Flow Control. This can help absorb short write bursts, but it also allows a node to fall further behind.
• gcs.fc_factor = 0.8: Controls when Flow Control is released after the replication queue decreases.

The required gcache size depends on the write rate and the expected duration of outages. Monitor the cluster carefully when changing Flow Control thresholds.

Changing the SST Method at Runtime

During major version upgrades, Mariabackup binaries on different nodes may temporarily be incompatible. If a node fails during this period and requires an SST, the mismatch can cause a problem. In such cases, the SST method can be changed dynamically without restarting MariaDB:

SET GLOBAL wsrep_sst_method = 'rsync';

Now perform the node upgrade, then switch back to Mariabackup:

SET GLOBAL wsrep_sst_method = 'mariabackup';

Settings in the [sst] and [mariabackup] sections are read by the SST helper scripts when a transfer starts. Changes to compression settings, decompression commands, or thread counts can therefore take effect during the next SST without restarting the mariadb service.

The [mariabackup] section is used only when the active SST method is mariabackup. If rsync is selected temporarily, the SST script ignores that section; there is no need to delete or comment it out.

Network and Proxy Tuning

Let me introduce some particularly beneficial configurations when monitoring services use persistent database connections. Enterprise monitoring such as Icinga 2 (based on Icinga DB) often use persistent connection pools. When activity decreases, these connections may remain idle for a long time. If an intermediate proxy closes idle connections before the database server does, MariaDB may log repeated errors such as:

[Warning] Aborted connection ... host: 'mariadb.neteyelocal' (Got an error reading communication packets)

Application connection pools usually reconnect automatically, but frequent connection resets consume resources and create unnecessary log noise. They can also make genuine database problems harder to identify.

Resolve Timeout Mismatches

MariaDB’s default wait_timeout for non-interactive connections is 28,800 seconds, or eight hours. If NGINX Stream uses a shorter timeout, it may close an idle connection before MariaDB does. This can create unexpected TCP RST or FIN packets and leave the client connection pool out of sync.

The architectural rule is simple: Connection termination should be controlled by MariaDB, not by the load balancer. A suitable NGINX configuration could look like this:

# /neteye/local/nginx-ha/conf/conf.d/global_custom_timeout.conf
proxy_connect_timeout 5s;
proxy_timeout 28860s;  # 8 hours + 1 minute
proxy_socket_keepalive on;

proxy_timeout 28860s is slightly longer than MariaDB’s default wait_timeout. MariaDB therefore remains responsible for closing idle database sessions according to the database protocol.

proxy_socket_keepalive on enables operating-system-level TCP keepalives. Keepalives can help prevent stateful firewalls and NAT devices from silently removing connections that have been idle for an extended time. They don’t extend MariaDB’s database timeout; they only help maintain the underlying network session for as long as it remains valid.

proxy_connect_timeout 5s limits how long NGINX waits when connecting to a backend database node. If a node becomes unresponsive, NGINX can quickly failover to another healthy node.

Summary and Key Takeaways

True high availability in a 2+1 Galera architecture requires coordinated behavior across the storage, database, clustering, and network layers. The main principles are:

• Use non-blocking SST with Mariabackup and Zstandard to keep donor nodes available while a recovering node is being synchronized.
• Use Galera-aware health checks and local firewall rules to prevent traffic from reaching nodes that are not ready.
• Align database and proxy timeouts and enable TCP keepalives to reduce connection-pool desynchronization and unnecessary connection resets.
• Tune InnoDB, replication apply threads, gcache, and Flow Control settings only after validating them against the actual workload and hardware.

The aspects highlighted in this article provide practical strategies for addressing performance and availability challenges in Galera environments. The recommendations are based on research and practical experience with a high-availability cluster. Galera’s default settings are a sound starting point and work well under normal conditions – but once you put serious pressure on the environment, its behavior can change quickly.

Useful Links

• mariadb-backup SST Method
• Rapid Node Recovery with IST and the GCache
• Performance Tuning

These Solutions are Engineered by Humans

Did you find this article interesting? Does it match your skill set? Our customers often present us with problems that need customized solutions. In fact, we’re currently hiring for roles just like this and others here at Würth IT Italy.