A user found a scenario where they had stored a -1 for a tuple's
segno. We'd like to be notified of this type of scenario because it
should not exist and we would like to track down the root
cause. Validations that throw errors will help us track the root cause
down.

Authored-by: NTingfang Bao <bbao@pivotal.io>

Commit 6ebce733 do some optimization for select statement
with locking clause for some simple cases. It also applies
such optimization to select statement with limit.

In Greenplum, the results of limit can only be known on QD,
but we can only lock tuples on QEs. So this commit fixes this.

For replicated table, select statement may only execute
on one segment, but update statement has to happen on all segments.
We should also turn off such optimization on locking clause for
replicated table.

Also, Currently, Greenplum uses two-stage sort to implement
order by, and might generate a gather motion to QD.

If the processing order is: first order-by then rowmarks,
we might put a lockrows plannode above a gather motion.
However, we cannot lock tuples on QD.

This commit changes the order. The plan for the query
`select * from t order by c for update` on a hash-distributed
table or randomly-distributed table t is:

  previous:
               QUERY PLAN
    ------------------------------------------
    LockRows
        ->Gather Motion 3:1  (slice1; segments: 3)
          Merge Key: c
          ->  Sort
              Sort Key: c
               ->  Seq Scan on t

  after this commit:
             QUERY PLAN
    ------------------------------------------
    Gather Motion 3:1  (slice1; segments: 3)
        Merge Key: c
        ->  Sort
            Sort Key: c
             ->  LockRows
                 ->  Seq Scan on t

Follow previous commits, make gpexpand tablespace input file's
separator consistent with gpexpand input file.

Build, test and install file_fdw by default.

Also for the benefit of testing more FDW functions, since this is the
first built-in FDW extension.

Reverts commit 1aef6434 and fixes
a couple typos.

This PR removes SLES11 from the gpAux build system and from Concourse CI
scripts. SLES11 will not be supported for GPDB 6+.

When the default tablespace guc is set, objects created should go into that tablespace.

FileRep was replaced a long time ago, but a few leftovers had remained
in the codebase (mainly in comments and out-of-common-codepath places).
Fix by removing.
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

fts_recovery_in_progress execute 'select gp_request_fts_probe_scan()' to
promote the mirror and then run 'gprecoverseg', it fails sometimes because the
mirror promotion hasn't finished yet.

No need LD_LIBRARY_PATH='' before wget, because it is needed in the past
because wget will complain about the openssl lib, now it is run direct
in the bash script.

No need wget option no-check-certificate, becasue validate the server's
certificate should be ok for component orca, no need to bypass.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NJesse Zhang <jzhang@pivotal.io>
Co-authored-by: NShaoqi Bai <sbai@pivotal.io>
Reviewed-by: NChris Hajas <chajas@pivotal.io>
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

Co-authored-by: NWenlin Zhang <wzhang@pivotal.io>
Co-authored-by: NRenyuan Wang <rewang@pivotal.io>

In operation tree it's possible the leaf query has a different number of
columns than the previous ones. Greenplum used to fill the smaller number
of leaf infos before realizing it's different and breaking.

It works well before until Greenplum supports zero target queries like
"SELECT FROM t1", in this case the smaller number is zero, the list of
leaf infos refer to 0x7f7f7f7f7f7f7f7e, coming from palloc0(0).

Reproducer from Asim https://github.com/greenplum-db/gpdb/issues/7613

```
create table foo (a int, b int);
insert into foo select i,i from generate_series(1,21)i;

select from foo union select * from foo;
```

This commit fixes it by bypassing filling the leaf infos if they have
different numbers of columns.

Previously, Ubuntu 16 installed libxerces-c-dev and
python-software-properties. The removal of these dependencies caused
failures in the creation of demo cluster. Adding back these dependencies
for Ubuntu 16 only. Only libxerces-c-dev is needed to make GPDEMO.

Also, updated the README.md to fix the command to create Ubuntu images.
Co-authored-by: NFrancisco Guerrero <fguerrero@pivotal.io>
Co-authored-by: NOliver Albertini <oalbertini@pivotal.io>

Earlier we didn't bother converting an in list of constants into an ArrayExpr
when then number of elements was larger than array_expansion_threshold.

We still should create an ArrayExpr because it can be useful in deriving stats
on filters even if we are not deriving constraints on it.

Bump ORCA version to v3.47.0

Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

The function `make_cluster` included a default value for
`DEFAULT_QD_MAX_CONNECT` but this value is set in the gpdemo Makefile
since commit 659be26d.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NJesse Zhang <jzhang@pivotal.io>

Authored-by: NBob Bao <bbao@pivotal.io>

* Add windows gpdb clients compile task to pipeline
Co-authored-by: NPeifeng Qiu <pqiu@pivotal.io>
Co-authored-by: NBob Bao <bbao@pivotal.io>

Improve GDD log when deadlock happens to make it more
readable.

Also, this commit rename `pg_dist_wait_status` to
`gp_dist_wait_status` and add more info the its output.

* Add WebSocket protocols to ports & protocol table

* Remove stray j

* docs - gpssh-exkeys utility reference updates due to removing paramiko

- add new 1-n passwordless ssh prerequisite

- not required to run gpssh-exkeys for root

- edits

* Edit from review

* Edit from review

* Remove comment about gpssh-exkeys creating keys

* docs - add info about config'ing pxf OOM actions

* misc edits requested by david

* clarify dump file renaming per comment from oliver

* address comments from francisco

The IMemoryPool interface was removed in ORCA to remove an unnecessary
abstraction layer and avoid costly casting.

Corresponding ORCA commit: e64a2b42

Bumps ORCA version to 3.46.0
Authored-by: NChris Hajas <chajas@pivotal.io>

We used to show the proposed and granted quota and shared memory in
gp_toolkit.gp_resgroup_status_per_host and
gp_toolkit.gp_resgroup_status_per_segment.  As we have retired the
proposed columns in the resource group catalog tables we should also
retire related columns from the views.
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>

* Remove windows support under gpAux.

Confirmed with GPClient Team, the windows support under gpAux is not
used in master and 6X_STABLE

* Remove the win32 logic from sync_tools.bash

Refer https://groups.google.com/a/pivotal.io/forum/#!searchin/gp-releng/sync_tools/gp-releng/-uu_bR3Sx2s/GvTsuxbsAAAJReviewed-by: NBradford D. Boyle <bboyle@pivotal.io>
Reviewed-by: NPeifeng Qiu <pqiu@pivotal.io>

* Create a public dockerfile for compiling greenplum on ubuntu18

* Make the Dockerfiles for ubuntu18 and ubuntu16 consistent

* Combine ubuntu16 and ubuntu18 and paraketerize the script

Add exhaustive2 option to optimizer_join_order guc.
The corresponding ORCA PR is
https://github.com/greenplum-db/gporca/pull/467.

The same change will go into 6X_STABLE. In 5X_STABLE, the new guc
option will not be available.

This reverts commit 9fa19b76.

Analyze should update the statistics on the root partition when all the
leaf partitions have been analyzed. However, analyzedb runs analyze on
the leaf partitions in parallel and does not update the stats on the
root partition. Without the correct stats on the root partition, ORCA
ends up picking a bad plan.

Additionally, add tests to analyzedb to ensure root stats are updated.
In master/6X, the relpages/reltuples of a root partition table is not
updated in pg_class, rather this is calculated during planning time.
Therefore, the tests check that columns statistics in pg_statistic are
populated.

This will be backported to 6X_STABLE.

We simplify and refactor the parsing for gpaddmirrors, gpmovemirrors,
gprecoverseg and gpexpand.  This eliminates a few hundred lines of code.
In addition, this commit changes the format of the input lines for the
input files to these routines.  The separator is now '|' instead of ':'.

<Co-Authored-By> Mark Sliva <msliva@pivotal.io>
<Co-Authored-By> Jamie McCatamney <jmcatamney@pivotal.io>

In merge work of postgres94, the subplan of updatable view check option
was not parallelized correctly. So we disable the check option and add
a FIXME.
The root cause is The root cause of this problem is that we ignored
modifytable.withCheckOptionLists in plan_tree_mutator.
Add it and enable the check option with subplan.

- remove redundant member 'distributedId' in 'DtxContextInfo'
- remove 'maxCount' in 'DistributedSnapshot' and 'maxLocalXidsCount' in 
  'DistributedSnapshotWithLocalMapping', instead, use 'GetMaxSnapshotXidCount()'
  as the size of the array.
- refactor the function GetSnapshotData
- call qsort after release ProcArrayLock

The verify_gpdb_versions task is using a base CentOS 7 image and the
version of the shared objects included are incorrect for executing the
GPDB binaries built for Ubuntu. Instead of actually executing the
binaries, we check for the presence of the commit SHA by dumping the
rodata section of `greenplum-db/bin/postgres`.
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NJesse Zhang <jzhang@pivotal.io>