This reverts commit d0839b23.

It's a Bourne Again script, Not a Bourne script.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>

The logic hasn't actually changed much (we are still validating the the
postgres version and gpdb_src SHA match), but has become clearer and
easier to follow:
* avoids use of globals
* separates assert function into two to have clearer scope separation
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

As a kill may be not taken effect at "the database is killed on hosts "
immediately, so we double check the cluster and stop database if it is
still running. But if kill takes effect after checking, stop_database
will raise an exception for gpstop will fail to stop an unstarted cluster.
To fix this flaky case, we stop database first and check the database
status later. It will raise an exception only when gpstop failed and
the cluster is still running.

For some areas of the ICW test framework -- isolation2 in particular --
the additional data written to stdout by gprecoverseg's progress
increased the load on the system significantly. (Some tests are
buffering stdout without bound, for instance.)  Additionally, the
updates were coming at ten times a second, which is an order of
magnitude more than the update interval we get from pg_basebackup
itself.

To help with this, we have have added a --no-progress flag that
suppresses the output of pg_basebackup.  We have also changed the
pg_basebackup progress update rate to once per second to minimize I/O.

The impacted regression/isolation2 tests utilizing gprecoverseg have
also been modified to use the --no-progress flag.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>

When -s is present, show pg_basebackup progress sequentially instead
of inplace. Useful when writing to a file, or if a tty does not support
escape sequences. Defaults to showing the progress inplace.

The gprecoverseg utility runs pg_basebackup in parallel on all segments that are
being recovered.  In this commit, we are logging the progress of each
pg_basebackup on its host and displaying them to the user of gprecoverseg.  The
progress files are deleted upons successful completion of gprecoverseg.

Unit tests have also been added.
Authored-by: NShoaib Lari <slari@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NEd Espino <edespino@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

These files used to be installed to ${prefix}/bin/ directory, but as
they are not user documents we should not install them.
Reviewed-by: NBradford D. Boyle <bboyle@pivotal.io>
Reviewed-by: NBob Bao <bbao@pivotal.io>

The fly commands outputted by gen_pipeline needed to be updated for a
chance in the naming convention of one of the secrets files used.
Co-authored-by: NBradford Boyle <bboyle@pivotal.io>
Co-authored-by: NKris Macoskey <kmacoskey@pivotal.io>

Include gpaddmirrors in the distribution.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NMark Sliva <msliva@pivotal.io>

Corresponding ORCA commit: "Fix crash during expression to DXL
translation in correlated NLJ"

We add gpmovemirrors tests and add them to the pipeline as well.
Using behave tags, some of the tests can run locally on a demo
cluster.  We also introduce Behave tag logic to choose which
tests to run.

<Co-Authored-By> Kalen Krempely <kkrempely@pivotal.io>
<Co-Authored-By> Mark Sliva <msliva@pivotal.io>

This seems to have broken when we updated gprecoverseg to work with WAL
rep.

We removed the registerMirrorsInCatalog method from the
cluster-wide utilities.
Co-authored-by: NShoaib Lari <slari@pivotal.io>

Because set_next_rotation_time() was being called once per call to
logfile_rotate(), the timestamp used to create the log file name was
being advanced multiple times per rotation. This led to the creation of
empty log files on disk that looked like they were from "the future".

This code badly needs to be refactored -- the initial intent, for the
text and csv implementations to be independent, is no longer the case --
but for now, we can hack around this problem by pulling the call to
set_next_rotation_time() up one level. This will only be called if *all*
logfiles are correctly rotated, to match the upstream logic (on failure,
we loop around to the beginning and try to open all log files again).
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Log rotations were happening well after the timestamp on the log file
(e.g. a per-minute rotation might actually take place thirty seconds
after it was supposed to).

After the 9.2 merge, we were incorrectly shadowing the `now` variable,
so the rotation delay only got bigger and bigger. Periodic logging (such
as from FTS) caused us to break out of the wait by luck, but otherwise
the syslogger would take longer and longer to wake up to perform a
rotation.
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

tracker story here:

  https://www.pivotaltracker.com/story/show/162139588

  When we compile the gpdb, we observer a error in the stdout:

  Error Copying Documentation: /root/gpdb/gpAux/docs/release is not
  present

  We sent a email to confirm with gpdb dev about this issue. and got
  the reply is to remove it completely.
Authored-by: NBob Bao <bbao@pivotal.io>

For GP 6 beta, the release engineering team is removing the apr-util
package from the list of bundled dependencies. Users will be asked
to provide their own apr-util package, which can differ in the version
on each platform. So it is necessary to verify gpperfmon is workable
with the platform provided apr-util on each supported platform.

Originally, gpperfmon test is done in CLI test suite, only covers centos6.
This commit changed it to a dedicated suite in order to test multiple
platforms.
Note: on SLES12 does not need to install libapr-util1 to run gpmmon

With the creation of 6X_STABLE, the pipelines creating artifacts from
the master branch need to separate from 6X_STABLE in blob store
Authored-by: NKris Macoskey <kmacoskey@pivotal.io>

Gp_role is asserted to GP_ROLE_DISPATCH

In some scenarios ERROR "append-only table version -1 is invalid" is
being hit. Code inspection doens't reveal any clues why it can
happen. So, for now we thought of adding more info to figure out the
cause. Backtrace would definitely be helpful when the error hits.

Also, check the format version as soon as entry is read from
catalog. Plus also would be greatly helpful if can have corefile when
the problem happens so under the controlled environment with guc
evelate ERROR to PANIC. GUC
`debug_appendonly_print_verify_write_block` usage for same is not
ideal but wish to avoid adding special guc just for this case, hence
piggyback on this existing uinteresting debug guc for now.
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

* dblink - xref to postgres 9.4 docs

* docs - add some using and writing your own FDW info

* add quote missing from mpp_execute master option

* foreign dash data, misc edits

ci: remove `-oss` from pattern used to parse version output

Remove period from assertRaisesRegexp expectation to not be conflated
with the regex metacharacter.
Authored-by: NKalen Krempely <kkrempely@pivotal.io>

* gpexpand
- remove `-D` flag and specify expansion schema is created in postgres database
- Add gpstate -x flag

* From review: consistent formatting for gpexpand and postgres

* Edit from review

* README mentions how gpupgrade and gpbackup both use the database "regression" that remains after an ICW run in the directory "regress".
Authored-by: NLarry Hamel <lhamel@pivotal.io>

- get port from MASTER_DATA_DIRECTORY, so there is no confusion if
  PGPORT and MASTER_DATA_DIRECTORY are set to different clusters
- delete tmp status file 'gpexpand.standby.status' and copy the
  status file to standby directly
- get standby data directory from catalog instead of assuming its
  same with master
- copy gp_segment_configuration backup file to standby also, so
  standby can restore this catalog if master is down

Previously we added the code due to Coverity report,
however, we conclude that we should better align with upstream
after some further discussion. See below.

https://github.com/greenplum-db/gpdb/pull/7120

In the long run, we hope StrNCpy() is gone, by the way.
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

FirstNormalObjectId indicates the first OID for user objects, if a
relation's oid >= FirstNormalObjectId then it's an user object,
otherwise it's a catalog object.

Fixed some incorrect uses of it, they treated oid == FirstNormalObjectId
as catalog objects.

Discussion: https://groups.google.com/a/greenplum.org/d/msg/gpdb-dev/ZHy-x1DaoXE/ceyzZ4TuCgAJReviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>

Authored-by: NKris Macoskey <kmacoskey@pivotal.io>

Authored-by: NKris Macoskey <kmacoskey@pivotal.io>

Authored-by: NKris Macoskey <kmacoskey@pivotal.io>

The default is a non OSS build across CI. Explict CI jobs that do build
OSS can use `--with-extra-version` to append `-oss`.
Authored-by: NKris Macoskey <kmacoskey@pivotal.io>

The catullus script had not been used to synchronize all pg_proc
changes into the catalog, so ensure we are in sync before we cut
Beta and (mostly) freeze the catalog. Also fix a set of typos in
DESC properties.

* Doc updates for removal of filespaces.

Changes to utility refs for #4314 and #7116

- Removes references to filespaces
- Allow configuring standby and master on the same host (options to specify host, port, data directory)
- Standy master dir is specified with -F (gpinitsystem) or -S (gpinitstandby)

* Updates from reviews

The error output in gpos job is very different from other jobs. Add
ignore first to fix the pipeline. I would find the root cause and
enable the case again.

It tests heap table and index, temporary table and index created in user tablespace , still exist in  pg_basebackup output.

A replicated table has a full replica of the data on each segment, so
the 'rows' in the EXPLAIN output should not be scaled.

A partial table's 'rows' in the EXPLAIN output should be scaled with the
numsegments of itself.

We used to scale both of above cases with the cluster size in the
EXPLAIN output, so the 'rows' were incorrectly displayed.  It's only a
bug in EXPLAIN output, the cost calculation of the plan is not affected.