I have seen too many "[CRITICAL]:-gpstart failed. (Reason='')
exiting..." errors, and there was nothing in the log. The reason could
be "SSH PATH", "Python modules" or some other issues.

Log the stderr to save debugging efforts.

Planner costs index-only-scan relative to index-scan. In the case of
index-only-scan, `RelOptInfo->allvisfrac` can be used to reduce the cost
when the visimap contains all-visible blocks. Thus, if index-only-scan
is possible, it will be favored.

In this commit, Orca also costs index-only-scan relative to index-scan
and scales the cost based on percentage of all-visible blocks in the
visimap. This is done by storing `pg_class.relallvisible` stats in
CDXLRelStats which is accessible during costing. In DXL this is added to
RelationStatistics:
```
<dxl:RelationStatistics ... RelPages="XX" RelAllVisible="YY"/>
```

Also in this commit, we update gpsd and minirepro to collect stats for
relallvisible. In doing this we also udpated the tools to use python3
sytnax.

And finally, it also adds a new option "index_only_scan_tests" in
cal_bitmap_test to callibrate index only scan cost/execution.

* refactor gpload test file TEST.py

1. migrate gpload test to pytest
2. new function to form config file through yaml package and make it more reasonable
3. add a case to cover gpload update_condition arggument

* migrate gpload and TEST.py to python3.6
new test case 43 to test gpload behavior when column name has capital letters and without data type
change some ans file since psql react different

* change sql to find reuseable external table to make gpload compatible in gp7 and gp6
better TEST.py to write config file with ruamel.yaml moudle
Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>

This commit does the following:
1. Extract config_primaries_for_replication to be used by both gpaddmirrors
and gprecoverseg.

2. gprecoverseg: add replication entries for primaries

3. gprecoverseg: add support for --hba-hostnames

This commit does the following:
1. Removes the calls related to gphost cache.
2. Uses ping to validate if the hostname could be resolved.

Earlier the error was sent to /dev/null and the information was lost
displaying the cause of the error. Redirect the error to log file.

Co-authored-by: NGang Xiong <gangx@vmware.com>

Commit 4bbbb381 introduced some hardening
around concurrent drop and recreate of tables while analyzedb is running but it
failed to take into account the code around updating the last operation
performed. This commit fixes it.

The error was introduced by dc96f667.
If `set -u` was called before sourcing greenplum_path.sh with bash, an
error `ZSH_VERSION: unbound variable` would be reported.
To solve the issue, use shell syntax `{:-}` which will output an empty
value if the variable doesn't exist.

Tested with zsh, bash and dash.

The generated greenplum_path.sh env file contained bash specific syntax
previously, so it errors out if the user's shell is zsh.

zsh doesn't have BASH_SOURCE. "${(%):-%x}" is the similar replacement
for zsh.
Also try to support other shells with some command combinations.
Tested with bash/zsh/dash.

Looking at GP documents, there is no indication that master dbid
must be 1. However, when CREATE_QD_DB, gpinitsystem always writes
"gp_dbid=1" into file `internal.auto.conf` even if we specify:

```
mdw~5432~/data/master/gpseg-1~2~-1
 OR
mdw~5432~/data/master/gpseg-1~0~-1
```

But catalog gp_segment_configuration can have the correct master
dbid value (2 or 0), the mismatch causes gpinitsystem hang.
Users can run into such problem for their first time to use
gpinitsystem -I.

Here we test dbid 0, because PostmasterMain() will simply check
dbid >= 0 (non-utility mode), it says:

> This value must be >= 0, or >= -1 in utility mode

It seems 0 is a valid value.

Changes:

- use specified master dbid field when CREATE_QD_DB.
- remove unused macros MASTER_DBID, InvalidDbid in C sources.
Reviewed-by: NAshwin Agrawal <aashwin@vmware.com>

If the $(UBUNTU_PLATFORM) is an empty string, the test command will fail,
double quotes it to fix.

```
--- mock for platform
/bin/sh: line 0: [: =: unary operator expected
```

Otherwise it will raise an exception "command not run yet".

The tests in commit be5d11e2 contained a typo that caused the changes
in the Scenario "gpstart starts even if the standby host is unreachable"
to not properly cleanup after itself.  Though the test feature still
passes, this leaves a bug to be found later when more tests are added.

It didn't log the error message before if pg_rewind fails, fix that to make
DBA/field/developer's life eaisier.

Before this:
```
20201022:15:19:10:011118 gprecoverseg:earth:adam-[INFO]:-Running pg_rewind on required mirrors
20201022:15:19:12:011118 gprecoverseg:earth:adam-[WARNING]:-Incremental recovery failed for dbid 2. You must use gprecoverseg -F to recover the segment.
20201022:15:19:12:011118 gprecoverseg:earth:adam-[INFO]:-Starting mirrors
20201022:15:19:12:011118 gprecoverseg:earth:adam-[INFO]:-era is 0406b847bf226356_201022151031
```

After this:
```
20201022:15:33:31:019577 gprecoverseg:earth:adam-[INFO]:-Running pg_rewind on required mirrors
20201022:15:33:31:019577 gprecoverseg:earth:adam-[WARNING]:-pg_rewind: fatal: could not find common ancestor of the source and target cluster's timelines
20201022:15:33:31:019577 gprecoverseg:earth:adam-[WARNING]:-Incremental recovery failed for dbid 2. You must use gprecoverseg -F to recover the segment.
20201022:15:33:31:019577 gprecoverseg:earth:adam-[INFO]:-Starting mirrors
20201022:15:33:31:019577 gprecoverseg:earth:adam-[INFO]:-era is 0406b847bf226356_201022151031
```

Currently, if a host is unreachable when gpstart is run, it will not report this
and will instead fail with an error that is both inaccurate and unhelpful to the
user, such as claiming that checksums are invalid for segments on a given host
when it simply can't reach that host to verify the checksums.

This commit adds a check to verify that all hosts are reachable before beginning
the startup process and, if one or more hosts are not reachable, marks segments
on those hosts down (in gparray, not in the cluster) so gpstart won't try to run
any checks against unreachable hosts and so that the cluster can still be started
in this state so long as there are otherwise enough valid segments to start it.

The parameters were incorrectly passed while gprecoverseg was invoked
causing gprecoverseg to fail.

Co-authored-by: Bhuvnesh Chaudhary<bchaudhary@vmware.com>

After testing analyzedb on a huge database with 170k tables we
have found a bottleneck while printing candidate list to analyze.
It took about 45 minutes to print all tables. The bottleneck was
in O(n^2) complexity when we validated candidates in a loop with
a list() instead of set(). The same O(n^2) validation is made while
running analyze commands on executor pool.
This commit change candidate type from list() to set() to reduce
complexity from O(n^2) to O(n).

With the ifaddrs utility, we excluded ip addresses on the loopback
interface which caused regression causing replication entries to be not
populated for such interfaces causing gpaddmirrors and gpinitstandby to
fail. Routable IP addresses can be assigned to the loopback interface,
and this case was not considered earlier.
This commit fixes the issues by allowing all loopback addresses
except 127.0.0.1 and ::1 address

In case of multi-host setup, gpinitsystem used to sort the hostnames
provided in hostfile. This logic seems guessing user intention and
hence unnecessary intelligence. Better to just use the order in which
names appear in file to deploy GPDB.

Searching though the historical code of greenplum, found previously it
used sort command but that used to yield unintended outcome and hence
logic was coded in python (because the desired outcome wished is
equivalent to what sort --version-sort would give). Though why sorting
existed in first place is no where to be found.

Input host file:
-------
sdw1-1
sdw10-1
sdw1-2
sdw10-2
-------

Sorted:
-------
sdw1-1
sdw1-2
sdw10-1
sdw10-2
-------

This logic got broken with Python3 changes, as the regex coded doesn't
work with Python3. It's still mystery for me how it worked for Python2
even. Anyway, lets just avoid sorting as we have no idea what naming
convention user is having for hostnames.

Postgres has logic to reuse or cleanup the shared-memory from previous
unclean shut-down. Plus, also starting b0fc0df9 the System V shared
memory consumption was dramatically reduced. Hence, no need to have
this logic in utilities to clean up shared memory.

The main reason to make this change now is postmaster.pid file format
changed and postmaster status is recorded on last
line. CleanSharedMem() was coded with expectation last line will
always be shared memory key, no more holds true due to it. If we have
to keep this logic around need to change the logic to read line 7 from
file and not last line. Given the need doesn't exist, just deleting
the logic instead of fixing it.

Based on inputs from Heikki Linnakangas and Asim R P.

Change sorting to use key functions, use bytestrings for interview

Co-authored-by: NAshwin Agrawal <aashwin@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

This was an outstanding TODO, and there is an added benefit of removing
yet another extensive shell command which is fragile.
Authored-by: NTyler Ramer <tramer@vmware.com>

This commit is mostly general Python 3 changes of the sort already made to the
utilities code (updating print syntax, updating imports, using bytestrings when
necessary, fixing sorting behavior, and so forth).

The major change of note in this commit is entirely replacing plpythonu with
plpython3u.  While it would technically be possible to support both versions of
PL/Python concurrently, as the version of Python used by PL/Python doesn't have
to match the version used for the utilities, we've made the decision not to try
to support both, since users will need to update to Python 3 regardless.
Co-authored-by: NAshwin Agrawal <aashwin@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Co-authored-by: NTyler Ramer <tramer@vmware.com>

This commit makes several broad changes to address conversion issues common to
multiple test files:

- Several built-in functions have been deprecated or renamed, or now need to
  use bytestrings (and associated encoding and decoding) instead of strings

- There is a "test case" run when ComputeCatalogUpdate is executed as a
  standalone program, but this should not be present in shipped code, so we
  remove it

- Some shelled-out commands in test code have been simplified due to changes
  to shell escaping, file redirection, and string manipulation, moving string
  parsing logic from shell commands to internal Python logic wherever possible
Co-authored-by: NAshwin Agrawal <aashwin@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Co-authored-by: NTyler Ramer <tramer@vmware.com>

- Certain objects in gpcheckcat can no longer be sorted with Python 3's new
  hashing logic, and the sorting was not functionally necessary, so the sorting
  has been removed.

- In Python 2, variables of type gpcatalog.GPCatalogTable were automatically
  coerced to strings when performing string comparisons.  Python 3 is stricter,
  so an explicit conversion is required.

- The reduce function is no longer built in and must be imported, and the changes
  to sorting require a separate key function in the orphan table check.
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Co-authored-by: NTyler Ramer <tramer@vmware.com>

Pickling was previously used in several utilities when shelling out commands
and/or executing commands remotely, in order to avoid needing to escape strings
when passing them back to the master.  The actual string contents were largely
or wholly ASCII, so pickling was overkill for that purpose.

The semantics of byte strings in Python 3 breaks the pickling logic, so we've
taken the opportunity to simplify that whole logic stack.  Code that formerly
pickled strings now uses shlex.quote() to escape strings where possible and
serializes strings with json where that is insufficient, removing any helper
functions that are no longer necessary.
Authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Authored-by: NTyler Ramer <tramer@vmware.com>

Removed unused or unecessary helper functions from gppylib

Shell escape function was unused and python 3 shlex.quote() function
should be used anyway.

canStringBeParsedAsInt was a silly helper function, and also failed to
actually complete the cast as string.
Authored-by: NTyler Ramer <tramer@vmware.com>

This commit makes several broad changes to address conversion issues common to
multiple utilities:

- The input and output of subprocess in Python 3 are now bytestrings instead
  of strings. Thus, some sanitizing of inputs and outputs is necessary

- Many built-in functions like raw_input and __cmp__ are deprecated in Python 3,
  and as a side effect list sorting and hashing work differently, requiring a
  different set of helper functions

- Implicit relative imports no longer work, so dbconn (in utilities code) and
  mgmt_utils (in test code) must be added to the search path and imported using
  a full path instead

- File objects require flush methods in python3, and popen2 has been deprecated
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Co-authored-by: NTyler Ramer <tramer@vmware.com>

The subprocess32 package is a backport of Python 3 subprocess functionality to
Python 2, so with the upgrade to Python 3 it is no longer necessary.

This commit deletes the package from pythonSrc and changes import statements to
import subprocess directly, instead of falling back to it only if subprocess32
is not importable.
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Co-authored-by: NTyler Ramer <tramer@vmware.com>

- Update Python file shebangs to use python3 and update gp_replicate_check and
  gpversion.py to allow running under Python 3

- Use Centos 7 dev containers with Python 3 and pip3 installed for testing, as
  prod containers do not yet work with Python 3, and update Travis with Python 3

- Install dependencies with pip3 to get Python 3-compatible versions

- Copy the Python 3 version of .so files, don't unset PYTHONHOME and PYTHONPATH,
  and don't remove built files from install locations, so that the Python 2 and
  Python 3 versions of various files can coexist
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
Co-authored-by: NKris Macoskey <kmacoskey@vmware.com>
Co-authored-by: NTyler Ramer <tramer@vmware.com>

The 2to3 utility is an officially-supported script to automatically convert
Python 2 code to Python 3.  It's not a complete fix by any means, but it
handles most basic syntax transformations and similar.

This commit is the result of running 2to3 against every Python file in the
gpMgmt directory, so it's quite large and fairly scattershot.  Manual updates
to any code that 2to3 can't handle will come in later commits.

* add double quatations when creating staging table
omit distribution key

* fix gpload fail when column names have capital letters in merge mode
Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>

In commit 5eaa5889, the --novacuum option is removed, however the help
page of gpexpand keep the -V option, which is a short option for
--novacuum.

Co-authored-by: NNing Wu <ningw@vmware.com>
Co-authored-by: NShaoqi Bai <bshaoqi@vmware.com>
Reviewed-by: NXin Zhang <zhxin@vmware.com>
Reviewed-by: NAdam Lee <adlee@vmware.com>
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>
Reviewed-by: NJesse Zhang <sbjesse@gmail.com>

Co-authored-by: NNing Wu <ningw@vmware.com>
Co-authored-by: NShaoqi Bai <bshaoqi@vmware.com>
Reviewed-by: NXin Zhang <zhxin@vmware.com>
Reviewed-by: NAdam Lee <adlee@vmware.com>
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>
Reviewed-by: NJesse Zhang <sbjesse@gmail.com>

When the standby is unreachable and the user proceeds with startup,
the standby would attempt to be started resulting in a stack trace.
Detect when the standby is unreachable and set start_standby to False to
prevent starting it later in the startup process.
Co-authored-by: NKalen Krempely <kkrempely@vmware.com>

This reverts commit 1060a425.