Fix flaky test isolation2:pg_basebackup_with_tablespaces (#10509)

Here is the diff output of the test result.

 drop database some_database_without_tablespace;
 -DROP
 +ERROR:  database "some_database_without_tablespace" is being accessed by other users
 +DETAIL:  There is 1 other session using the database.
 drop tablespace some_basebackup_tablespace;
 -DROP
 +ERROR:  tablespace "some_basebackup_tablespace" is not empty

The reason is that after client connection to the database exits, the server
needs some time (the process might be scheduled out soon, and the operation
needs to content for the ProcArrayLock lock) to release the PGPROC in
proc_exit()->ProcArrayRemove(). During dropdb() (for database drop), postgres
will call CountOtherDBBackends() to see if there are still sessions that are
using the database by checking proc->databaseId, and it will try at most 5 sec.
This test quits the db connection of some_database_without_tablespace and then
drops the database immediately. This should be mostly fine but if the system is
in slow or in heavy load, this still could lead to test flakiness.

This issue could be simulated using gdb. Let's poll until database drop
commands succeeds for the affected database.  It seems that drop database sql
command could not be in transaction block so I could not use plpgsql to
implement, instead I use dropdb utility and bash command to implement that.
Reviewed-by: NAsim R P <pasim@vmware.com>
(cherry picked from commit c8b00ac7)

src/test/isolation2/input/pg_basebackup_with_tablespaces.source

@@ -51,7 +51,10 @@ select validate_tablespace_symlink('@testtablespace@/some_basebackup_datadir', '
 -- Cleanup things we've created
 0U: select pg_drop_replication_slot('some_replication_slot');
 drop database some_database_with_tablespace;
-drop database some_database_without_tablespace;
+-- Need to poll until db drop succeeds since after 2q: happens the PGPROC
+-- on the server side might be still not released when we run database drop.
+-- This makes the test flaky.
+!\retcode bash -c 'for ((i = 0; i < 5; i++)); do dropdb some_database_without_tablespace && exit 0; done; exit 1';
 drop tablespace some_basebackup_tablespace;
 !\retcode rm -rf @testtablespace@/some_basebackup_datadir/;
 !\retcode rm -rf @testtablespace@/some_basebackup_tablespace/100;
@@ -91,7 +94,10 @@ select validate_tablespace_symlink('@testtablespace@/some_basebackup_datadir', '
 
 -- Cleanup things we've created
 0U: select pg_drop_replication_slot('some_replication_slot');
-drop database some_database_without_tablespace;
+-- Need to poll until db drop succeeds since after 2q: happens the PGPROC
+-- on the server side might be still not released when we run database drop.
+-- This makes the test flaky.
+!\retcode bash -c 'for ((i = 0; i < 5; i++)); do dropdb some_database_without_tablespace && exit 0; done; exit 1';
 drop tablespace some_basebackup_tablespace;
 !\retcode rm -rf @testtablespace@/some_basebackup_datadir/;
 !\retcode rm -rf @testtablespace@/some_basebackup_tablespace_c0/100;

src/test/isolation2/output/pg_basebackup_with_tablespaces.source

@@ -103,8 +103,14 @@ select validate_tablespace_symlink('@testtablespace@/some_basebackup_datadir', '
 (1 row)
 drop database some_database_with_tablespace;
 DROP
-drop database some_database_without_tablespace;
-DROP
+-- Need to poll until db drop succeeds since after 2q: happens the PGPROC
+-- on the server side might be still not released when we run database drop.
+-- This makes the test flaky.
+!\retcode bash -c 'for ((i = 0; i < 5; i++)); do dropdb some_database_without_tablespace && exit 0; done; exit 1';
+-- start_ignore
+
+-- end_ignore
+(exited with code 0)
 drop tablespace some_basebackup_tablespace;
 DROP
 !\retcode rm -rf @testtablespace@/some_basebackup_datadir/;
@@ -192,8 +198,14 @@ select validate_tablespace_symlink('@testtablespace@/some_basebackup_datadir', '
 --------------------------
                           
 (1 row)
-drop database some_database_without_tablespace;
-DROP
+-- Need to poll until db drop succeeds since after 2q: happens the PGPROC
+-- on the server side might be still not released when we run database drop.
+-- This makes the test flaky.
+!\retcode bash -c 'for ((i = 0; i < 5; i++)); do dropdb some_database_without_tablespace && exit 0; done; exit 1';
+-- start_ignore
+
+-- end_ignore
+(exited with code 0)
 drop tablespace some_basebackup_tablespace;
 DROP
 !\retcode rm -rf @testtablespace@/some_basebackup_datadir/;