JAVA J2EE 4 U

 

Hi SRE POD,

This is to notify you about an issue reported by the application team this morning after they observed an application outage.

During the investigation, we noticed that Patroni was unable to start or update the cluster state because etcd had raised a NOSPACE alarm. The following error was observed:

etcdserver: mvcc: database space exceeded

Although the actual active data in etcd was relatively small, the etcd backend database had grown close to its configured quota due to unused or fragmented space. As a result, etcd started rejecting write operations. Patroni was therefore unable to update its leader lock and member status in etcd, which affected database availability.

The following recovery steps were performed:

1. Verified the etcd endpoint status and confirmed the NOSPACE alarm.
2. Checked the etcd database size and active data usage.
3. Compacted the older etcd revisions.
4. Defragmented the etcd backend database to reclaim unused space.
5. Cleared the NOSPACE alarm.
6. Restarted and validated the etcd and Patroni services.
7. Confirmed that the Patroni cluster was healthy and database connectivity was restored.

A possible reason for the fragmentation is the continuous creation and deletion of key revisions in etcd. Patroni regularly updates leader locks, member status, configuration and health information. Although older revisions may no longer be required, they continue occupying space in the etcd backend until compaction and defragmentation are performed.

The issue can also occur when:

• Automatic etcd compaction is not configured or is not running.
• Defragmentation has not been performed periodically.
• There is frequent key or lease churn from Patroni or other applications using the same etcd cluster.
• The etcd backend quota is relatively small compared to the rate of revision growth.
• Old deleted keys and historical revisions continue occupying physical database space.

As a preventive action, we should review the etcd auto-compaction configuration, establish periodic database-size monitoring and schedule defragmentation when the difference between the total database size and the active data size exceeds the agreed threshold.

Regards,
Nikhil