I have seen too many "[CRITICAL]:-gpstart failed. (Reason='')
exiting..." errors, and there was nothing in the log. The reason could
be "SSH PATH", "Python modules" or some other issues.

Log the stderr to save debugging efforts.

This commit does the following:
1. Extract config_primaries_for_replication to be used by both gpaddmirrors
and gprecoverseg.

2. gprecoverseg: add replication entries for primaries

3. gprecoverseg: add support for --hba-hostnames

It didn't log the error message before if pg_rewind fails, fix that to make
DBA/field/developer's life eaisier.

Before this:
```
20201022:15:19:10:011118 gprecoverseg:earth:adam-[INFO]:-Running pg_rewind on required mirrors
20201022:15:19:12:011118 gprecoverseg:earth:adam-[WARNING]:-Incremental recovery failed for dbid 2. You must use gprecoverseg -F to recover the segment.
20201022:15:19:12:011118 gprecoverseg:earth:adam-[INFO]:-Starting mirrors
20201022:15:19:12:011118 gprecoverseg:earth:adam-[INFO]:-era is 0406b847bf226356_201022151031
```

After this:
```
20201022:15:33:31:019577 gprecoverseg:earth:adam-[INFO]:-Running pg_rewind on required mirrors
20201022:15:33:31:019577 gprecoverseg:earth:adam-[WARNING]:-pg_rewind: fatal: could not find common ancestor of the source and target cluster's timelines
20201022:15:33:31:019577 gprecoverseg:earth:adam-[WARNING]:-Incremental recovery failed for dbid 2. You must use gprecoverseg -F to recover the segment.
20201022:15:33:31:019577 gprecoverseg:earth:adam-[INFO]:-Starting mirrors
20201022:15:33:31:019577 gprecoverseg:earth:adam-[INFO]:-era is 0406b847bf226356_201022151031
```

The parameters were incorrectly passed while gprecoverseg was invoked
causing gprecoverseg to fail.

Co-authored-by: Bhuvnesh Chaudhary<bchaudhary@vmware.com>

Postgres has logic to reuse or cleanup the shared-memory from previous
unclean shut-down. Plus, also starting b0fc0df9 the System V shared
memory consumption was dramatically reduced. Hence, no need to have
this logic in utilities to clean up shared memory.

The main reason to make this change now is postmaster.pid file format
changed and postmaster status is recorded on last
line. CleanSharedMem() was coded with expectation last line will
always be shared memory key, no more holds true due to it. If we have
to keep this logic around need to change the logic to read line 7 from
file and not last line. Given the need doesn't exist, just deleting
the logic instead of fixing it.

Based on inputs from Heikki Linnakangas and Asim R P.

This was an outstanding TODO, and there is an added benefit of removing
yet another extensive shell command which is fragile.
Authored-by: NTyler Ramer <tramer@vmware.com>

This commit makes several broad changes to address conversion issues common to
multiple test files:

- Several built-in functions have been deprecated or renamed, or now need to
  use bytestrings (and associated encoding and decoding) instead of strings

- There is a "test case" run when ComputeCatalogUpdate is executed as a
  standalone program, but this should not be present in shipped code, so we
  remove it

- Some shelled-out commands in test code have been simplified due to changes
  to shell escaping, file redirection, and string manipulation, moving string
  parsing logic from shell commands to internal Python logic wherever possible
Co-authored-by: NAshwin Agrawal <aashwin@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Co-authored-by: NTyler Ramer <tramer@vmware.com>

Pickling was previously used in several utilities when shelling out commands
and/or executing commands remotely, in order to avoid needing to escape strings
when passing them back to the master.  The actual string contents were largely
or wholly ASCII, so pickling was overkill for that purpose.

The semantics of byte strings in Python 3 breaks the pickling logic, so we've
taken the opportunity to simplify that whole logic stack.  Code that formerly
pickled strings now uses shlex.quote() to escape strings where possible and
serializes strings with json where that is insufficient, removing any helper
functions that are no longer necessary.
Authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Authored-by: NTyler Ramer <tramer@vmware.com>

Removed unused or unecessary helper functions from gppylib

Shell escape function was unused and python 3 shlex.quote() function
should be used anyway.

canStringBeParsedAsInt was a silly helper function, and also failed to
actually complete the cast as string.
Authored-by: NTyler Ramer <tramer@vmware.com>

This commit makes several broad changes to address conversion issues common to
multiple utilities:

- The input and output of subprocess in Python 3 are now bytestrings instead
  of strings. Thus, some sanitizing of inputs and outputs is necessary

- Many built-in functions like raw_input and __cmp__ are deprecated in Python 3,
  and as a side effect list sorting and hashing work differently, requiring a
  different set of helper functions

- Implicit relative imports no longer work, so dbconn (in utilities code) and
  mgmt_utils (in test code) must be added to the search path and imported using
  a full path instead

- File objects require flush methods in python3, and popen2 has been deprecated
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Co-authored-by: NTyler Ramer <tramer@vmware.com>

- Update Python file shebangs to use python3 and update gp_replicate_check and
  gpversion.py to allow running under Python 3

- Use Centos 7 dev containers with Python 3 and pip3 installed for testing, as
  prod containers do not yet work with Python 3, and update Travis with Python 3

- Install dependencies with pip3 to get Python 3-compatible versions

- Copy the Python 3 version of .so files, don't unset PYTHONHOME and PYTHONPATH,
  and don't remove built files from install locations, so that the Python 2 and
  Python 3 versions of various files can coexist
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Co-authored-by: NKris Macoskey <kmacoskey@vmware.com>
Co-authored-by: NTyler Ramer <tramer@vmware.com>

The 2to3 utility is an officially-supported script to automatically convert
Python 2 code to Python 3.  It's not a complete fix by any means, but it
handles most basic syntax transformations and similar.

This commit is the result of running 2to3 against every Python file in the
gpMgmt directory, so it's quite large and fairly scattershot.  Manual updates
to any code that 2to3 can't handle will come in later commits.

Add gpreload behave test

* gpreloaqd test where a table (public.t1) and
  view (gpreload_schema.t1) have the same name.

Add gpreload CI test jobs.
Co-authored-by: NEd Espino <eespino@vmware.com>

Gphostcache has numerous issues, and has been a pain point for some
time. For this reason, we are removing it.

This commit moves the useful function of gphostcache - the hostname
deduping - to the gparray class, where a list of deduped hostnames is
returned from gpArray.get_hostlist().

There is a FIXME of correctly adding hostname to a newly added or
recovered mirror. The hostname resolution from address was incorrect and
faulty in its logic - an IP address never requires a hostname associated
with it. However, the hostname field in gp_segment_configuration should
be populated somehow - we recommend a "hostname" field addition to any
configuration files that require it. For now, we simple set the
"hostname" to "address" which ultimately delivers the same functionality
as the gphostcache implementation.
Co-authored-by: NTyler Ramer <tramer@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

During testing, I encountered an incremental gprecoverseg hang issue.
Incremental gprecoverseg is based on pg_rewind.  pg_rewind launches a single
mode postgres process and quits after crash recovery if the postgres instance
was not cleanly shut down - this is used to ensure that the postgres is in a
consistent state before doing incremental recovery. I found that the single
mode postgres hangs with the below stack.

\#1  0x00000000008cf2d6 in PGSemaphoreLock (sema=0x7f238274a4b0, interruptOK=1 '\001') at pg_sema.c:422
\#2  0x00000000009614ed in ProcSleep (locallock=0x2c783c0, lockMethodTable=0xddb140 <default_lockmethod>) at proc.c:1347
\#3  0x000000000095a0c1 in WaitOnLock (locallock=0x2c783c0, owner=0x2cbf950) at lock.c:1853
\#4  0x0000000000958e3a in LockAcquireExtended (locktag=0x7ffde826aa60, lockmode=3, sessionLock=0 '\000', dontWait=0 '\000', reportMemoryError=1 '\001', locallockp=0x0) at lock.c:1155
\#5  0x0000000000957e64 in LockAcquire (locktag=0x7ffde826aa60, lockmode=3, sessionLock=0 '\000', dontWait=0 '\000') at lock.c:700
\#6  0x000000000095728c in LockSharedObject (classid=1262, objid=1, objsubid=0, lockmode=3) at lmgr.c:939
\#7  0x0000000000b0152b in InitPostgres (in_dbname=0x2c769f0 "template1", dboid=0, username=0x2c59340 "gpadmin", out_dbname=0x0) at postinit.c:1019
\#8  0x000000000097b970 in PostgresMain (argc=5, argv=0x2c51990, dbname=0x2c769f0 "template1", username=0x2c59340 "gpadmin") at postgres.c:4820
\#9  0x00000000007dc432 in main (argc=5, argv=0x2c51990) at main.c:241

It tries to hold the lock for template1 on pg_database with lockmode 3 but
it conflicts with the lock with lockmode 5 which was held by a recovered dtx
transaction in startup RecoverPreparedTransactions(). Typically the dtx
transaction comes from "create database" (by default the template database is
template1).

Fixing this by using the postgres database for single mode postgres execution.
The postgres database is commonly used in many background worker backends like
dtx recovery, gdd and ftsprobe. With this change, we do not need to worry
about "create database" with template postgres, etc since they won't succeed,
thus avoid the lock conflict.

We may be able to fix this in InitPostgres() by bypassing the locking code in
single mode but the current fix seems to be safer.  Note InitPostgres()
locks/unlocks some other catalog tables also but almost all of them are using
lock mode 1 (except mode 3 pg_resqueuecapability per debugging output).  It
seems that it is not usual in real scenario to have a dtx transaction that
locks catalog with mode 8 which conflicts with mode 1.  If we encounter this
later we need to think out a better (might not be trivial) solution for this.
For now let's fix the issue we encountered at first.

Note in this patch the code fixes in buildMirrorSegments.py and twophase.c are
not related to this patch. They do not seem to be strict bugs but we'd better
fix them to avoid potential issues in the future.
Reviewed-by: NAshwin Agrawal <aashwin@vmware.com>
Reviewed-by: NAsim R P <pasim@vmware.com>

Commit 21a2cb27b38117cce90c4ff06d8d447842c5acf1, added in PR #10361,
updated yaml and changed yaml.load to yaml.safe_load in gpload.

gppkg uses yaml as well, but references were not updated - this commit
resolves that discrepancy.
Co-authored-by: NTyler Ramer <tramer@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

Due to refactor of dbconn and newer versions of pygresql, using
`with dbconn.connect() as conn` no longer attempts to close a
connection, even if it did prior. Instead, this syntax uses the
connection itself as context and, as noted in execSQL, overrides the
autocommit functionality of execSQL.

Therefore, close the connection manually to ensure that execSQL is
auto-commited, and the connection is closed.
Co-authored-by: NTyler Ramer <tramer@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

One reason pygresql was previously modified was that it did not handle
closing a connection very gracefully. In the process of updating
pygresql, we've wrapped the connection it provides with a
ClosingConnection function, which should handle gracefully closing the
connection when the "with dbconn.connect as conn" syntax is used.

This did, however, illustrate issues where a cursor might have been
created as the result of a dbconn.execSQL() call, which seems to hold
the connection open if not specifically closed.

It is therefore necessary to remove the ability to get a cursor from
dbconn.execSQL(). To highlight this difference, and to ensure that
future calls to this library is easy to use, I've cleaned up and
clarified the dbconn execution code, to include the following features.

- dbconn.execSQL() closes the cursor as part of the function. It returns
  no rows
- functions dbconn.query() is added, which behaves like dbconn.execSQL()
  except that it now returns a cursor
- function dbconn.execQueryforSingleton() is renamed
  dconn.querySingleton()
- function dbconn.execQueryforSingletonRow() is renamed
  dconn.queryRow()
Authored-by: NTyler Ramer <tramer@pivotal.io>

This commit updates pygresql from 4.0.0 to 5.1.2, which requires
numerous changes to take advantages of the major result syntax change
that pygresql5 implemented. Of note, cursors or query objects
automatically cast returned values as appropriate python types - list of
ints, for example, instead of a string like "{1,2}". This is the bulk of
the changes.

Updating to pygresql 5.1.2 provides numerous benfits, including the
following:

- CVE-2018-1058 was addressed in pygresql 5.1.1

- We can save notices in the pgdb module, rather than relying on importing
the pg module, thanks to the new "set_notices()"

- pygresql 5 supports python3

- Thanks to a change in the cursor, using a "with" syntax guarentees a
  "commit" on the close of the with block.

This commit is a starting point for additional changes, including
refactoring the dbconn module.

Additionally, since isolation2 uses pygresql, some pl/python scripts
were updated, and isolation2 SQL output is further decoupled from
pygresql. The output of a psql command should be similar enough to
isolation2's pg output that minimal or no modification is needed to
ensure gpdiff can recognize the output.
Co-Authored-by: NTyler Ramer <tramer@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

1. When two gppkg packages have the same dependencies, gppkg utility
   will refuse to install the second gppkg package and throw an error.
   This patch fixes this issue and the second gppkg package can install
   successfully.

2. Fix install/uninstall issue if the master and standby master use the
   same node address.

Incremental recovery and rebalance operations involve running
pg_rewind against failed primaries.  This patch changes gprecoverseg
such that pg_rewind is invoked in parallel, using the WorkerPool
interface, for each affected segment in the cluster.  There is no
reason to rewind segments one after the other.

Fixes Github issue #9466

Reviewed by: Mark Sliva and Paul Guo

This helps script handling by checking return values.
Reviewed-by: NAsim R P <apraveen@pivotal.io>

As a continued follow-up in the CVE-2018-1058, ensure that the utilities
that connect using PyGreSQL primitives also have search_path protection.

We set an empty search_path for the connection in all cases except for
segment_reconfigurer.py, where a fully-qualified SELECT statement seemed
sufficient (there is only one query being performed on the connection,
and that seems unlikely to change).
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>

The tree I was working off clearly had stale files, which led me to
include two utils which were removed some time ago: gpcheckutil.py
and gpcheck.py. Remove these two from their respective Makefiles.

Also fix a Bash error in the Stream symlink test, the logoical AND
requires [[ .. ]]; rather than [ .. ];.

Both of these spotted while repeatedly running make install with
trees in various states.

Installing the Management utilities used to be pretty brute-force
operation which copied more or less everything over blindly and then
tried to remove what shouldn't be installed. This is clearly not a
terribly clean and sustainable solution, as subsequent issues with
it has proven (editor savefiles, patch .rej/.orig files etc were
routinely copied and never purged etc).

This takes a first stab at turning installation of gpMgmt/bin, sbin
and doc into proper recursive make targets which only install the
files that were intended to be installed.

Discussion: https://github.com/greenplum-db/gpdb/pull/8179
Reviewed by Bradford Boyle, Kalen Krempely, Jamie McAtamney and
many more

* Revert "CI: skip CLI behave tests that currently fail on ubuntu18.04"

This reverts commit 6a0cb2c6.
This commit is to re-enable behave test for centos/ubuntu

* Ouput the same error message format

gppkg for rpm and deb outputs error messages when consistency is broken, but
the message is not the same, which make a pipeline fail.

* Update gppkg for behave test and fix some minor error in gppkg

Now, the binary sample.gppkg is removed, instead we add sample-*.rpm and sample-*.deb.
Because sample.gppkg is platform specific and GP major version sensitive. The uploaded
rpm/deb files are available unless the rpm/deb file is incompatible on the specific
platform. GP_MAJORVERSION is dynamically retrieved from a makefile installed in the gpdb
folder, so the gp major version in the gppkg will be always correct.
sample.gppkg will only be generated when gppkg tag is provided

Due to the number of platforms we now support and variety of customer
specific environments, it is not practical to update gpcheck to cover
all these cases.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Currently, explicit 'BEGIN' creates a full-size writer gang and starts a transaction
on it, the following 'END' will commit the transaction in a two-phase way. It can be
optimized for some cases:
case 1:
BEGIN;
SELECT * FROM pg_class;
END;

case 2:
BEGIN;
SELECT * FROM foo;
SELECT * FROM bar;
END;

case 3:
BEGIN;
INSERT INTO foo VALUES(1);
INSERT INTO bar VALUES(2);
END;

For case 1, it's unnecessary to create a gang and no need to have two-phase commit.
For case 2, it's unnecessary to have two-phase commit as the executors don't write
any XLOG.
For case 3, don't have to create a full-size writer gang and do two-phase commit on
a full-size gang.
Co-authored-by: NJialun Du <jdu@pivotal.io>

Mostly in code comments, but some in user facing messaging.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

Removes imports of modules once required, but for which usage has
been removed over the years in various refactorings.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

test_RemoteOperation_logger_debug() has been flaking out on the CI
pipeline, and there's no indication of what is going wrong. Replace the
assertTrue() call, which gives no indication of the difference between
actual and expected, with mock.assert_has_calls(), which will tell us
exactly what the calls were in case of failure.

It's possible that this will fix the flake entirely. The previous test
implementation depended on logger.debug() to be called *first* with our
expected output, but given the poor isolation of our global logger
system, it's entirely possible that some other code occasionally calls
debug(). (That this is an issue at all indicates that this isn't really
a unit test, but that's not something to tackle here.) assert_has_calls()
doesn't mind how many other calls happen as long as the one we're
looking for is eventually made, and I think that matches the intent of
the test better anyway.

Backport to 6X_STABLE.

For some areas of the ICW test framework -- isolation2 in particular --
the additional data written to stdout by gprecoverseg's progress
increased the load on the system significantly. (Some tests are
buffering stdout without bound, for instance.)  Additionally, the
updates were coming at ten times a second, which is an order of
magnitude more than the update interval we get from pg_basebackup
itself.

To help with this, we have have added a --no-progress flag that
suppresses the output of pg_basebackup.  We have also changed the
pg_basebackup progress update rate to once per second to minimize I/O.

The impacted regression/isolation2 tests utilizing gprecoverseg have
also been modified to use the --no-progress flag.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

When -s is present, show pg_basebackup progress sequentially instead
of inplace. Useful when writing to a file, or if a tty does not support
escape sequences. Defaults to showing the progress inplace.

The gprecoverseg utility runs pg_basebackup in parallel on all segments that are
being recovered.  In this commit, we are logging the progress of each
pg_basebackup on its host and displaying them to the user of gprecoverseg.  The
progress files are deleted upons successful completion of gprecoverseg.

Unit tests have also been added.
Authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NEd Espino <edespino@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This seems to have broken when we updated gprecoverseg to work with WAL
rep.

We removed the registerMirrorsInCatalog method from the
cluster-wide utilities.
Co-authored-by: NShoaib Lari <slari@pivotal.io>

Remove period from assertRaisesRegexp expectation to not be conflated
with the regex metacharacter.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

Reword exception message to be more clear and concise.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

Authored-by: NKalen Krempely <kkrempely@pivotal.io>

One of these changes appears to have possibly introduced a serious
performance regression in the master pipeline. To avoid destabilizing
work over the weekend, I'm reverting for now and we can investigate more
fully next week.

This reverts the following commits:
"gprecoverseg: Show progress of pg_basebackup on each segment"
    1b38c6e8
"Add gprecoverseg -s to show progress sequentially"
    9e89b5ad
"gpinitstandby: guide the user on single-host systems"
    c9c3c351
"gpinitstandby: rename -F to -S and document it"
    ba3eb5b4

When -s is present, show pg_basebackup progress sequentially instead
of inplace. Useful when writing to a file, or if a tty does not support
escape sequences. Defaults to showing the progress inplace.