Also, add retry to check for walsender is gone after walrcv_disconnect() as it
can take sometime to detect the connection drop.

For subselect_gp test, we were removing the distribution policy of a
table to see if it would do a gather motion or not. Since it's
technically a corrupted table, we should delete it after we're done
with it. We also remove a quicklz reference that should not have been
there.

For gppc test, they were using the regression database. This made our
gpcheckcat call at the end of ICW relatively useless since all our
data would have been deleted due to gppc tests recreating the
regression database.

For gpload test, some generated files were previously commited. We
should be actively cautious of this and remove them when we see them.

PL/Perl is an optional component, and the main ICW should not use
it as it may not be present. Move the tests that seem useful to
the plperl test suite instead and remove the ones which we have
ample coverage for elsewhere.

The gpmapreduce application is an optional install included via
the --enable-mapreduce configure option. The tests were however
still in src/test/regress and unconditionally included in the
ICW schedule, thus causing test failures when mapreduce wasn't
configured.

Move all gpmapreduce tests to co-locate them with the mapreduce
code and only test when configured.

Also, add a dependency on Perl for gpmapreduce in autoconf since
it's a required component.

This is part of the effort to get all backup/restore tests using the
same test suite. Since the Netbackup tests take significantly longer, we
only run a subset of the regular test suite. We also tag scenarios to
allow parallel runs on separate hosts in CI.

This suite will take 1h, 40 mins after parallelization, down from the
current 2h, 20 mins.

- Before building Index object (IMDIndex), we build LogicalIndexes
  via calling `gpdb::Plgidx(oidRel)` in which a partition tables is
  traversed and index  information (such as logicalIndexOid,
  nColumns, indexKeys, indPred, indExprs, indIsUnique, partCons,
  defaultLevels) is captured.
- For Indexes which are available on all the partitions partCons
  and defaultLevels are NULL/empty.
- Later in `CTranslatorRelcacheToDXL::PmdindexPartTable` to build
  Index object, we use the derived LogicalIndexes information and
  populates the array  holding the levels on which default
  partitions exists. But since defaultLevels is NIL in this case,
  pdrgpulDefaultLevels is set to empty i,e `default partitions on levels: {}`
- This causes an issue while trying to build the propagation expression,
  as because of wrong number of default partitions on level we mark the
  scan as partial and tries to construct a test propagation expression
  instead of a const propagation expression.
- This patch fixes the issue by marking the default partitions on
  levels for index equal to the default partitions on levels for the
  part relation if the index exists on all the parts.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

* Clarify `make behave` behavior and remove unused call

* Add flags argument to pass in custom flags to behave

Allows us to use more complex tag combinations for including/excluding
scenarios. This also removes the skip_tags argument as it only worked in
limited circumstances.
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Fix MU_check concourse task to print failing unit test on error.

This patch is to fix below defects reported by Coverity Scan.

```
New defect(s) Reported-by: Coverity Scan
Showing 3 of 3 defect(s)

** CID 169611:  Security best practices violations  (TOCTOU)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
272 in createDir()

________________________________________________________________________________________________________
*** CID 169611:  Security best practices violations  (TOCTOU)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
272 in createDir()
266
267             buildPath(group, comp, "", path, pathsize);
268
269             if (access(path, F_OK))
270             {
271                     /* the dir is not created yet, create it */
>>>     CID 169611:  Security best practices violations  (TOCTOU)
>>>     Calling function "mkdir" that uses "path" after a check
>>>     function. This can cause a time-of-check, time-of-use race
>>>     condition.
272                     if (mkdir(path, 0755) && errno != EEXIST)
273                             return false;
274             }
275
276             return true;
277     }

** CID 169610:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
412 in writeData()

________________________________________________________________________________________________________
*** CID 169610:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
412 in writeData()
406             size_t ret = write(fd, data, datasize);
407
408             /* save errno before close */
409             int err = errno;
410             close(fd);
411
>>>     CID 169610:  Control flow issues  (NO_EFFECT)
>>>     This less-than-zero comparison of an unsigned value is never
>>>     true. "ret < 0UL".
412             if (ret < 0)
413                     CGROUP_ERROR("can't write data to file '%s':
%s", path, strerror(err));
414             if (ret != datasize)
415                     CGROUP_ERROR("can't write all data to file
'%s'", path);
416     }
417

** CID 169609:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
385 in readData()

________________________________________________________________________________________________________
*** CID 169609:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
385 in readData()
379             size_t ret = read(fd, data, datasize);
380
381             /* save errno before close() */
382             int err = errno;
383             close(fd);
384
>>>     CID 169609:  Control flow issues  (NO_EFFECT)
>>>     This less-than-zero comparison of an unsigned value is never
>>>     true. "ret < 0UL".
385             if (ret < 0)
386                     CGROUP_ERROR("can't read data from file '%s':
%s", path, strerror(err));
387
388             return ret;
389     }
390
```

Remove duplicate unittest_check_gpdb in compile_gpdb.bash, it's already run by the above "if" block when needed.

Some minor documentation fixes

[ci skip]

`outputdir` here is where to place the converted files, not the
directory named `output`. PostgreSQL places them into `outputdir`
rightly.

    commit 64cdbbc4
    Author: Peter Eisentraut <peter_e@gmx.net>
    Date:   Sat Feb 14 21:33:41 2015 -0500

        pg_regress: Write processed input/*.source into output dir

        Before, it was writing the processed files into the input directory,
        which is incorrect in a vpath build.

This way we are consistent now with the upstream Postgres.

On pulse, this seemed to make no change
On concourse, this added color
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

* doc updates for variadic function arguments and type modifiers

* david s requested edits

Amazon S3 supports server side encryption to encrypt the uploaded files.
In this commit, gpcloud can support setting S3 server side encryption
(SSE-S3) in writable external table. A new option is added into gpcloud
configuration file, named server_side_encryption. User can add an option
'server_side_encryption = sse-s3' to enable server side encryption.

The default value of server_side_encryption is none if this option is not
appeared in configuration file and server side encryption will be disabled.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

ON MASTER feature is not fully supported by built-in protocols other
than s3 yet, disable it with those protocols for now.

1. Add --with-gssapi to sles configurations in gpAux/Makefile to enable kerberos build
2. Add kerberos sbin path to PATH for sles.
3. Disable psql pager to avoid concourse hang.
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

- Provide a high-level overview welcome section
- Remove exceptions for gpcloud -- now in this repo
- Newline/return more frequently in the markdown

Based on feedback from new developers to gpdb, we are improving
the top level make targets.

https://github.com/greenplum-db/gpdb/issues/2422

Just a start to have wal replication tests in ICW. This has simple protocol
functions whichs kind-of mocks walreceiver side to help validate walsender and
xlog stream. Mainly to portray something on these lines can be easily leveraged
to validate like xlog generation and stream for AO tables when done, avoiding to
fully instantiate a mirror or something on similar lines at ease.

CID 129287 Buffer not null terminated. In primaryMirrorCheckNICFailure: The
string buffers `localHost` and `peerHost` may not have a null terminator if the
source string's length is equal to the buffer size. Hence use strlcpy instead to
be same.

CID 130101 Dereference before null check. In isFilespaceInfoConsistent: All
paths that lead to `primaryFilespaceLocation` null pointer comparison already
dereference the pointer earlier. Since `primaryFilespaceLocation` cannot be null
else PersistentFilespace_GetPrimaryAndMirror() will error out itself, removing
the check for null.

CID 130130 Copy into fixed size buffer. In primaryMirrorPopulateFilespaceInfo: A
source buffer `pch` of statically unknown size is copied into a fixed-size
destination buffer `peerFilespacePath`. Hence replace strcpy with strlcpy.

CID 159071 Uninitialized scalar variable. In primaryMirrorPopulateFilespaceInfo:
Use of an uninitialized variable filespacePath. Hence initialing it in else path
to empty string.

Currently for function isFilespaceInfoConsistent() and
primaryMirrorPopulateFilespaceInfo().

* updating gppkg reference to no longer mention PL/Java, PL/Perl as separate packages

* pl\python typos; more gppkg cleanup; pgcrypto install

* fixing repeated URL

* Section on xml datatype

* Format corrections

* Fix typos

* Reorganization suggested by Dave

* Format updates

* Update headings

The support for anonymous blocks which was backported in commit
cacb2839 missed updatint the query
in getProcLangs() to correctly extract the laninline function.
Fix by updating the query to what it looks like in upstream 9.0
where anonymous blocks first appeared.

Added a small comment to explain why for when this conflicts in
a merge. There shouldn't be a conflict until we merge 9.0 since
getProcLangs() was quite stable during 8.4, but one never know.

resource_group cases were already moved to the standalone resgroup
target, however one entrance was still left in isolation2 schedule which
caused a failure in pipeline.

This commit is a cleanup to make pipeline green.

As cgroup is now required to enable resgroup on linux and cgroup itself
requires privileged permission to setup & config, so resgroup tests will
fail or at least produce extra warnings in ICW pipeline. We moved them
to the installcheck-resgroup target as there is a standalone privileged
pipeline to run this target.

Also the tests are updated as the psql output format is different
between ICW and installcheck-resgroup.