Use postgres database for pg_rewind cleanly shutdown execution to avoid potential pg_rewind hang.
During testing, I encountered an incremental gprecoverseg hang issue. Incremental gprecoverseg is based on pg_rewind. pg_rewind launches a single mode postgres process and quits after crash recovery if the postgres instance was not cleanly shut down - this is used to ensure that the postgres is in a consistent state before doing incremental recovery. I found that the single mode postgres hangs with the below stack. \#1 0x00000000008cf2d6 in PGSemaphoreLock (sema=0x7f238274a4b0, interruptOK=1 '\001') at pg_sema.c:422 \#2 0x00000000009614ed in ProcSleep (locallock=0x2c783c0, lockMethodTable=0xddb140 <default_lockmethod>) at proc.c:1347 \#3 0x000000000095a0c1 in WaitOnLock (locallock=0x2c783c0, owner=0x2cbf950) at lock.c:1853 \#4 0x0000000000958e3a in LockAcquireExtended (locktag=0x7ffde826aa60, lockmode=3, sessionLock=0 '\000', dontWait=0 '\000', reportMemoryError=1 '\001', locallockp=0x0) at lock.c:1155 \#5 0x0000000000957e64 in LockAcquire (locktag=0x7ffde826aa60, lockmode=3, sessionLock=0 '\000', dontWait=0 '\000') at lock.c:700 \#6 0x000000000095728c in LockSharedObject (classid=1262, objid=1, objsubid=0, lockmode=3) at lmgr.c:939 \#7 0x0000000000b0152b in InitPostgres (in_dbname=0x2c769f0 "template1", dboid=0, username=0x2c59340 "gpadmin", out_dbname=0x0) at postinit.c:1019 \#8 0x000000000097b970 in PostgresMain (argc=5, argv=0x2c51990, dbname=0x2c769f0 "template1", username=0x2c59340 "gpadmin") at postgres.c:4820 \#9 0x00000000007dc432 in main (argc=5, argv=0x2c51990) at main.c:241 It tries to hold the lock for template1 on pg_database with lockmode 3 but it conflicts with the lock with lockmode 5 which was held by a recovered dtx transaction in startup RecoverPreparedTransactions(). Typically the dtx transaction comes from "create database" (by default the template database is template1). Fixing this by using the postgres database for single mode postgres execution. The postgres database is commonly used in many background worker backends like dtx recovery, gdd and ftsprobe. With this change, we do not need to worry about "create database" with template postgres, etc since they won't succeed, thus avoid the lock conflict. We may be able to fix this in InitPostgres() by bypassing the locking code in single mode but the current fix seems to be safer. Note InitPostgres() locks/unlocks some other catalog tables also but almost all of them are using lock mode 1 (except mode 3 pg_resqueuecapability per debugging output). It seems that it is not usual in real scenario to have a dtx transaction that locks catalog with mode 8 which conflicts with mode 1. If we encounter this later we need to think out a better (might not be trivial) solution for this. For now let's fix the issue we encountered at first. Note in this patch the code fixes in buildMirrorSegments.py and twophase.c are not related to this patch. They do not seem to be strict bugs but we'd better fix them to avoid potential issues in the future. Reviewed-by: NAshwin Agrawal <aashwin@vmware.com> Reviewed-by: NAsim R P <pasim@vmware.com> (cherry picked from commit 288908f3)
Showing
想要评论请 注册 或 登录