Hi Team, today’s PostgreSQL incident call was quite intense, with several questions from the application team due to a similar Sev 4 impact in May. As no PostgreSQL SRE was available at that time, I joined the call, reviewed the issue and shared the resolution within around 13 minutes.
The alert had not been generated, which added to the concern. I have shared the possible causes, relevant checks and observations over email, with Ranjit, Khozema and the relevant SREs included.
Following the discussion, a handful of SREs from other regions are now helping to analyse the condition further and explore how it can be included as part of the alerting. This also highlights the need for broader PostgreSQL SRE ownership so that incidents and follow-up actions do not continue to depend mainly on the existing SMEs.
Product Owner: Dhirendra Kumar
Redis Provisioning Policy Enablement: Updated the Deny-Redis-Create policy effect from Deny to Audit, enabling users to provision Azure Cache for Redis instances while retaining governance and compliance visibility through policy monitoring.
Cosmos DB Group-Based Access Enhancement: Enabled persistent MIM group-based access for Cosmos DB NoSQL APIs. This ensures continuity for technical-account access and reduces dependency on the role-sync pipeline through group ID-based access management.
Secure Cross-Tenant Cosmos DB Migration Enablement: Released the required changes to support cross-tenant migration from RU-based Cosmos DB accounts to vCore without enabling public endpoints. This provides a secure and controlled migration approach aligned with internal connectivity standards.
Emerging DB Documentation Migration and Governance: Completed the migration of Emerging DB documentation from the existing SharePoint location to the secure ITD SharePoint repository. Confluence content was reviewed, classified and consolidated, with the original content to be removed after final validation to prevent duplication and improve documentation governance, security and controlled access.
The cross-tenant migration and Redis provisioning policy items are the strongest because they directly enable product usage and delivery.
The Cosmos DB group-based access item is also valuable if it resolved an operational dependency or access-management risk.
The documentation migration is useful governance work, but it would normally be the first item to remove when the monthly summary needs only two or three major achievements.
#!/bin/bash
# Patroni/etcd fragmentation alert
# Reads only the latest 250 lines from the existing Patroni log.
# No state file or temporary report file is created.
PATRONI_LOG="/var/log/patroni/patroni.log"
MAIL_TO="postgres-support-dl@example.com"
MAIL_SUBJECT="WARNING: Patroni/etcd space alert on $(hostname -s)"
export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS="https://etcd1:2379,https://etcd2:2379,https://etcd3:2379"
export ETCDCTL_CACERT="/path/to/ca.crt"
export ETCDCTL_CERT="/path/to/client.crt"
export ETCDCTL_KEY="/path/to/client.key"
# Match only etcd space/quota/fragmentation-related messages.
ERROR_PATTERN='NOSPACE|database space exceeded|mvcc: database space exceeded|etcd.*(space|quota|fragment|alarm)'
if [ ! -r "$PATRONI_LOG" ]; then
echo "Cannot read Patroni log: $PATRONI_LOG" >&2
exit 1
fi
# Read only the latest 250 lines from the existing Patroni log.
NEW_ERRORS=$(tail -n 250 "$PATRONI_LOG" | grep -Ei "$ERROR_PATTERN" || true)
# No matching message found.
if [ -z "$NEW_ERRORS" ]; then
exit 0
fi
send_report() {
{
echo "Patroni/etcd space-related warning detected"
echo "Host: $(hostname -f)"
echo "Time: $(date)"
echo
echo "===== Matching entries from last 250 Patroni log lines ====="
printf '%s\n' "$NEW_ERRORS"
echo
echo "===== etcd endpoint status ====="
echo "Compare DB SIZE with SIZE IN USE to identify fragmentation."
etcdctl endpoint status --cluster -w table 2>&1
echo
echo "===== etcd active alarms ====="
etcdctl alarm list 2>&1
}
}
if command -v mailx >/dev/null 2>&1; then
send_report | mailx -s "$MAIL_SUBJECT" "$MAIL_TO"
elif command -v mail >/dev/null 2>&1; then
send_report | mail -s "$MAIL_SUBJECT" "$MAIL_TO"
else
echo "Neither mailx nor mail is installed." >&2
send_report
exit 1
fi
Hello Friends! I am aliena part time blogger from Bihar. I am a student and I manage my study as well as my blog. I am 14 years old and I want to help those people who really want to do something on internet.
Copyright © 201. Metro UI Theme. Designed by: Templateism
