Tuesday, 28 January 2025

Subject: Database Restoration Issue - Patroni Starts in Standby Mode Post Restore

Filled under:

Description:

We have a 3-node Patroni cluster with a PostgreSQL HA setup. As part of a recovery process, we restored the database using a tape backup. However, post-restore, when we attempt to start the Patroni service on the restored site, the node consistently appears as a replica (standby mode). Conversely, if we bypass Patroni and start PostgreSQL directly using pg_ctl, the database successfully starts in read-write (primary) mode without any issues.

Given this discrepancy, we are unsure why Patroni is forcing the database into standby mode, even though the same database appears fully functional as primary with manual pg_ctl startup.

Current Status:

  • The site is currently running in standalone mode with PostgreSQL started using pg_ctl instead of Patroni.
  • We have scheduled maintenance over the weekend to reinitialize and bring all nodes back into HA with proper cluster resilience.
  • We need clear steps to ensure a seamless switch from pg_ctl startup to Patroni without errors and to reintegrate the restored site into the cluster properly.

Request for Assistance:

  1. Understanding the Issue: What could be causing Patroni to perceive the restored node as a replica (standby mode), despite it being in read-write mode when started manually? Is it possible that incorrect DCS metadata or configuration within Patroni could be the reason?
  2. Recommended Steps: What steps should we follow to safely stop the PostgreSQL instance running via pg_ctl and switch back to Patroni startup without triggering replication or standby mode?
  3. HA Restoration: Guidelines on how to reinitialize and reintegrate this site into the Patroni cluster without any inconsistency, especially post-restoration from tape backup.

We are looking forward to your assistance in diagnosing the root cause and providing a validated approach to restore HA functionality for the cluster.

Please advise on any logs or additional details you may require from our side.

0 comments:

Post a Comment