Monitoring PXC from outside the cluster: Prometheus, alerts and failure detection

A database cluster should not be responsible for deciding whether it is healthy.

If monitoring lives only inside the same infrastructure it watches, a network failure, hypervisor problem, storage incident or site outage can take out both the service and the system that was supposed to alert you about it.

For a production Percona XtraDB Cluster deployment, I prefer a simple rule:

monitor the database from outside the database cluster.

The monitoring system does not have to be in another country or another cloud. It does need an independent failure domain and a network path that lets it observe the service from the outside.

Architecture

A practical layout looks like this:

             External monitoring host
        Prometheus + Alertmanager + Grafana
                    |
        +-----------+-----------+
        |           |           |
        v           v           v
      PXC-1       PXC-2       PXC-3
     node +       node +       node +
     mysqld       mysqld       mysqld
     exporter     exporter     exporter

The exporters expose metrics. Prometheus collects them. Alertmanager decides when a condition deserves a notification. Grafana is useful for investigation, but dashboards are not a replacement for alerts.

What I monitor on every PXC node

Host metrics are the first layer. At minimum I want visibility into CPU, load, memory, swap, filesystem capacity, disk latency, network errors and whether the host itself is reachable.

For MySQL/PXC, ordinary mysql_up is not enough. A process can accept TCP connections while the node is no longer useful to the cluster.

Important Galera signals include:

wsrep_cluster_status
wsrep_cluster_size
wsrep_local_state
wsrep_local_state_comment
wsrep_ready
wsrep_connected
wsrep_local_recv_queue
wsrep_local_send_queue
wsrep_local_cert_failures
wsrep_local_bf_aborts
wsrep_flow_control_paused

The exact metric names exposed by an exporter can vary, so I verify the actual /metrics output before writing rules.

Cluster quorum deserves its own alert

For a three-node cluster, losing one node is degraded but still operational. Losing quorum is a different class of incident.

The monitoring should distinguish these states instead of producing one generic “MySQL down” notification.

Useful conditions are:

cluster size < expected size
cluster status != Primary
node not Ready
node not Connected
node state != Synced

This matters operationally. If two nodes in a three-node PXC cluster disappear, the remaining node must not simply be treated as a healthy standalone database. The cluster has lost quorum and write availability is affected by design.

Alert on symptoms before the database disappears

A good alert set catches the path toward failure, not only the final outage.

For example:

filesystem free space is low
inode space is low
disk latency is abnormal
memory pressure / swap activity is high
MySQL exporter cannot connect
PXC node leaves Synced state
cluster size changes
cluster loses Primary status
flow control remains elevated
replication queues keep growing

Thresholds should be tuned to the workload. A fixed queue value that is dangerous for one system may be normal during a batch operation on another.

Monitor the backup, not just the database

A backup job that has not failed loudly is not the same thing as a successful backup.

I want monitoring to know at least:

time of last successful backup
backup age
backup size
job exit status
free space on backup storage
whether the expected artifact exists
whether periodic restore validation succeeded

A simple and powerful alert is backup age. If the newest valid backup is older than the recovery policy allows, the alert fires even if the backup script itself never reported an error.

This catches cron failures, broken credentials, network problems, full disks and jobs that silently stopped running.

External monitoring changes what you can detect

When Prometheus is outside the cluster, up == 0 has more meaning. The monitoring host is testing the network path and the target together instead of asking a failed environment whether it considers itself healthy.

For important services I also add black-box checks from outside the production host: TCP reachability, HTTPS availability, certificate expiration and, where appropriate, an application-level health endpoint.

This gives two different views:

inside the server:  CPU, disk, MySQL, wsrep
outside the server: can the service actually be reached?

Both are necessary.

Keep monitoring access narrow

Exporters should not become a new public attack surface. Bind them to a management address where possible and restrict access with firewall rules to the monitoring host or monitoring network.

The MySQL monitoring account should also have only the permissions required by the exporter. Monitoring does not need an administrative database account.

The result

The goal is not a wall of Grafana dashboards. The goal is to answer operational questions quickly:

Is the cluster Primary?
How many nodes are healthy?
Is every node Synced and Ready?
Are replication queues growing?
Is storage becoming dangerous?
Did the backup really happen?
Can the service be reached from outside?

If monitoring cannot answer those questions when the infrastructure is partially broken, it is not yet production monitoring.

For PXC, MySQL, Prometheus and infrastructure monitoring work, see Monitoring & Alerting → or contact 0ROOT →.