I have seen too many "[CRITICAL]:-gpstart failed. (Reason='')
exiting..." errors, and there was nothing in the log. The reason could
be "SSH PATH", "Python modules" or some other issues.

Log the stderr to save debugging efforts.

The regex used in the script was too strict.  Relax the regex so as to
match 'select distinct gp_inject_fault(...);'.  The script may be
invoked against a directory, such as src/test/fsync, which may not
contain any schedule files.  This no longer results in an error.  When
invoked against isolation directory, it should scan tests under
'specs' subdir.  Not all of input, sql and specs subdirs exist always,
so add a '-s' option to grep to mask errors reporting non-existance.
Co-authored-by: NHao Wu <gfphoenix78@gmail.com>

Some tests merely need to create an extension, let's use an extension
that's always enabled instead of gp_inject_fault, which can now be
configured with --enable-faultinjector configure option.

This change is cherry-picked from GitHub PR #9029.

If faultinjector is not enabled during configuration, this patch
identifies tests that use faultinjector and excludes them from running.
A new option "--exclude-file" is added to pg_regress.  It specifies a
file having names of tests, one per line, to be excluded from running.

The global makefile is changed so that the exclude file creation and
adding the --exclude-file option to pg_regress (and
pg_isolation*regress) command line is taken care of automatically.
However, one must perform these actions manually if pg_regress needs to
be directly invoked from command line (instead of a make target).
Running tests with faultinjector disabled is not a common developer
workflow, so this should be acceptable.

The advantage of changing global makefile is that all test frameworks
that are based on pg_regress, such as isolation and also contrib modules
need no change to their makefiles.  For example, if one creates a
isolation test with faultinjector, simply add it to the right schedule,
it will automatically be excluded from running when faultinjector is not
enabled.

Definition FAULT_INJECTOR is hardcoded in a header(pg_config_manual.h)
file.  Fault injector is useful, but it may introduce some issues in
production stage, like runtime cost, security problems.  It's better
to enable this feature in development and disable it in release.  The
--enable-debug-extensions configure option is used to control this
feature.

It makes isolation2 framework more consistent with pg_regress and
isolation in terms of how it is invoked from the build system.

We included some rough steps in 57b744c1. This patch updates
those with a higher level of details so that it's hopefully easier to
follow in a step-by-step fashion.

Previously, standby will replay checkpoint XLOG record's DTX info
in function XLogProcessCheckpointRecord. However, is some certain cases,
it will cause anomaly: When a DTX has flushed FORGET COMMITTED XLOG into
disk, but didn't change its own state to DTX_STATE_INSERTED_FORGET_COMMITTED.
If at this very moment, checkpoint process is calculating DTX info, it
will include the DTX into its XLOG record. So when standby replaying this
checkpoint XLOG record from master, it will add an already forgotten GID to
shmCommittedGxactArray again, which may cause the overflow of shmCommittedGxactArray.

Since DTX info stored in checkpoint XLOG record has been populated earlier in
ReadCheckpointRecord(), there is no need to call XLogProcessCheckpointRecord()
again during recovery.
Co-authored-by: Nwuchengwen <wcw190496@alibaba-inc.com>
Co-authored-by: NDenis Smirnov <sd@arenadata.io>

Those functionality has been handled in checkpoint during recovery in utility
mode.  The code is not needed anymore.
Reviewed-by: NAsim R P <pasim@vmware.com>

During preprocessing, Orca simplifies the query by merging/deduplicating
constraints. However, Orca did not consider the case where the passed in
constraint was a column compared with an empty array. This assumption
caused Orca to crash when dealing with predicates such as `a = ANY('{}')`.

Instead, we now explicitly return an empty constraint when dealing with an ANY('{}')
clause. For ALL('{}'), we won't process the constraint as there's no
simplification to do.

This is part of commit de7eaa9d

This is done to speed up development iterations by reducing the time the
of the "compile" job in the following pipelines, and starting ICW jobs
sooner. Runs the unit tests in parallel to ICW jobs.

Pipelines modified:
- gporca-dev
- master build pipeline
- master build pipeline without asserts
- PR pipeline
Co-authored-by: NShreedhar Hardikar <hardikar@cs.wisc.edu>
Co-authored-by: NChris Hajas <chajas@vmware.com>

Planner costs index-only-scan relative to index-scan. In the case of
index-only-scan, `RelOptInfo->allvisfrac` can be used to reduce the cost
when the visimap contains all-visible blocks. Thus, if index-only-scan
is possible, it will be favored.

In this commit, Orca also costs index-only-scan relative to index-scan
and scales the cost based on percentage of all-visible blocks in the
visimap. This is done by storing `pg_class.relallvisible` stats in
CDXLRelStats which is accessible during costing. In DXL this is added to
RelationStatistics:
```
<dxl:RelationStatistics ... RelPages="XX" RelAllVisible="YY"/>
```

Also in this commit, we update gpsd and minirepro to collect stats for
relallvisible. In doing this we also udpated the tools to use python3
sytnax.

And finally, it also adds a new option "index_only_scan_tests" in
cal_bitmap_test to callibrate index only scan cost/execution.

In order to produce an index only scan plan, all columns referenced in the
query must be covered by an index. This check is made in the 
`CXformIndexGet2IndexOnlyScan::Transform`.

Issue is that distribution columns were being marked as used even when they
were not directly referenced. This is because they may still appear in
distribution specs of derived physical properties. However, this also meant
that Orca would never consider an index only scan if the distribution column
was not covered by the index.

With reject limit, if reject limit is not reached,
there should be some messages that how many rows
are rejected as `execute on segments` does.

Co-authored-by Mingli Zhang <zmingli@vmware.com>
Co-authored-by Adam Lee <adlee@vmware.com>
Co-authored-by: NMingli Zhang <zmingli@vmware.com>

Appendoptimized tables need block directory relation for index access. Creating
and maintaining block directory is expensive. We delay the block directory
creation until it is really needed - the first index creation. Once created,
all indexes share the same block directory. We need stronger lock
(ShareRowExclusiveLock) if the block directory needs to be created. If the
block directory already exists, we can use the same lock as heap tables.

This reverts commit 6c16abca.

After we merged to PostgreSQL 12, plpython has the plpy.commit(), this
workaround is not needed then.

After commit aef99680 master build on macOS started to fail with
errors like this:

execute_pipe.c:197:6: error: implicit declaration of function 'kill' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        if (kill(pid, 0) == 0) /* process exists */
            ^
1 error generated.

POSIX only guarantees that kill(2) be provided through <signal.h>. It
turns out we were relying on <unistd.h> to indirectly include <signal.h>
for us. But that's a GNU-ism.

This also showed up in Travis CI on greenplum-db/gpdb#11121
(https://travis-ci.org/github/greenplum-db/gpdb/builds/742342058) before
it got merged into master.
Co-authored-by: NShreedhar Hardikar <hardikar@cs.wisc.edu>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

This reverts commit 9265ea6a.

popen_with_stderr() was introduced in execute external table, to
extract stderr from external program. It's later adopted by COPY
command too, replacing OpenPipeStream() from upstream.

Currently popen_with_stderr() is used both by external table and
COPY command. As we are going to move external table related code
to extension, we need to move this function to a place where both
have access.

function 'pg_logdir_ls' returns the creation time of log files by parsing the
file name, it's different from 'pg_ls_dir_files', we should keep it if we still
consider 'pg_logdir_ls' useful.

* refactor gpload test file TEST.py

1. migrate gpload test to pytest
2. new function to form config file through yaml package and make it more reasonable
3. add a case to cover gpload update_condition arggument

* migrate gpload and TEST.py to python3.6
new test case 43 to test gpload behavior when column name has capital letters and without data type
change some ans file since psql react different

* change sql to find reuseable external table to make gpload compatible in gp7 and gp6
better TEST.py to write config file with ruamel.yaml moudle
Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>

The user can adjust the ic-proxy peer addresses at runtime and reload by
sending SIGHUP, if an address is modified or removed, the corresponding
peer connection must be closed or reestablished.  The same to the peer
listener, if the listener port is changed, then must re-setup the
listener.

The peer addresses are specified with the GUC
gp_interconnect_proxy_addresses, it can be reloaded on SIGHUP, we used
to only care about the newly added ones, however it is also possible for
the user to modify them, or even remove some of them.

So now we add the logic to classify the addresses after parsing the GUC,
we can tell whether an address is added, removed, or modified.

The handling of the classified addresses will be done in the next
commit.

We used to scan the whole addr list to find my addr, now we record it
directly when parsing the addresses.

A ICProxyAddr variable is usually named as "addr", so the attribute is
referred as "addr->addr", it's confusing and sometimes ambiguous.

So renamed the attribute to "sockaddr", the function
ic_proxy_extract_addr() is also renamed to ic_proxy_extract_sockaddr().

ProcessCopyOptions checks the option list of COPY command. It's also
called by external table when text/csv format is used. It's better
not to mix external table specific options here and check them
separately.

Checking custom protocol here is not necessary because it's checked
when parsing location urls in GenerateExtTableEntryOptions anyway.

This commit does the following:
1. Extract config_primaries_for_replication to be used by both gpaddmirrors
and gprecoverseg.

2. gprecoverseg: add replication entries for primaries

3. gprecoverseg: add support for --hba-hostnames

This commit does the following:
1. Removes the calls related to gphost cache.
2. Uses ping to validate if the hostname could be resolved.

When stash a 'VisimapDelete' to the buffile, we must seek to end of the last
physical file if the buffile contains multiple files. This commit cherry-pick
part of the commit from upstream:

commit 808e13b282efa7e7ac7b78e886aca5684f4bccd3
Author: Amit Kapila <akapila@postgresql.org>
Date:   Wed Aug 26 07:36:43 2020 +0530

    Extend the BufFile interface.

    Allow BufFile to support temporary files that can be used by the single
    backend when the corresponding files need to be survived across the
    transaction and need to be opened and closed multiple times. Such files
    need to be created as a member of a SharedFileSet.

    Additionally, this commit implements the interface for BufFileTruncate to
    allow files to be truncated up to a particular offset and extends the
    BufFileSeek API to support the SEEK_END case. This also adds an option to
    provide a mode while opening the shared BufFiles instead of always opening
    in read-only mode.

    These enhancements in BufFile interface are required for the upcoming
    patch to allow the replication apply worker, to handle streamed
    in-progress transactions.

    Author: Dilip Kumar, Amit Kapila
    Reviewed-by: Amit Kapila
    Tested-by: Neha Sharma
    Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com

This is a cherry-pick of the change from PR https://github.com/greenplum-db/gporca/pull/607

Avoid costing change for IN predicates on btree indexes

Commit e5f1716 changed the way we handle IN predicates on indexes, it
now uses a more efficient array comparison instead of treating it like
an OR predicate. A side effect is that the cost function,
CCostModelGPDB::CostBitmapTableScan, now goes through a different code
path, using the "small NDV" or "large NDV" costing method. This produces
very high cost estimates when the NDV increases beyond 2, so we
basically never choose an index for these cases, although a btree
index used in a bitmap scan isn't very sensitive to the NDV.

To avoid this, we go back to the old formula we used before commit e5f1716.
The fix is restricted to IN predicates on btree indexes, used in a bitmap
scan.

Add an MDP for a larger IN list, using a btree index on an AO table

Misc. changes to the calibration test program

- Added tests for btree indexes (btree_scan_tests).
- Changed data distribution so that all column values range from 1...n.
- Parameter values for test queries are now proportional to selectivity,
  a parameter value of 0 produces a selectivity of 0%.
- Changed the logic to fake statistics somewhat, hopefully this will
  lead to more precise estimates. Incorporated the changes to the
  data distribution with no more 0 values. Added fake stats for
  unique columns.
- Headers of tests now use semicolons to separate parts, to give
  a nicer output when pasting into Google Docs.
- Some formatting changes.
- Log fallbacks.
- When using existing tables, the program now determines the table
  structure (heap or append-only) and the row count.
- Split off two very slow tests into separate test units. These are
  not included when running "all" tests, they have to be run
  explicitly.
- Add btree join tests, rename "bitmap_join_tests" to "index_join_tests"
  and run both bitmap and btree joins
- Update min and max parameter values to cover a range that includes
  or at least is closer to the cross-over between index and table scan
- Remove the "high NDV" tests, since the ranges in the general test
  now include both low and high NDV cases (<= and > 200)
- Print out selectivity of each query, if available
- Suppress standard deviation output when we execute queries only once
- Set search path when connecting
- Decrease the parameter range when running bitmap scan tests on
  heap tables
- Run btree scan tests only on AO tables, they are not designed
  for testing index scans

Updates to the experimental cost model, new calibration

1. Simplify some of the formulas, the calibration process seemed to justify
   that. We might have to revisit if problems come up. Changes:
   - Rewrite some of the formulas so the costs per row and costs per byte
     are more easy to see
   - Make the cost for the width directly proportional
   - Unify the formula for scans and joins, use the same per-byte costs
     and make NDV-dependent costs proportional to num_rebinds * dNDV,
     except for the logic in item 3.

   That makes the cost for the new experimental cost model a simple linear formula:

   num_rebinds * ( rows * c1 + rows * width * c2 + ndv * c3 + bitmap_union_cost + c4 ) + c5

   We have 5 constants, c1 ... c5:

   c1: cost per row (rows on one segment)
   c2: cost per byte
   c3: cost per distinct value (total NDV on all segments)
   c4: cost per rebind
   c5: initialization cost
   bitmap_union_cost: see item 3 below

2. Recalibrate some of the cost parameters, using the updated calibration
   program src/backend/gporca/scripts/cal_bitmap_test.py

3. Add a cost penalty for bitmap index scans on heap tables. The added
   cost takes the form bitmap_union_cost = <base table rows> * (NDV-1) * c6.

   The reason for this is, as others have pointed out, that heap tables
   lead to much larger bit vectors, since their CTIDs are more spaced out
   than those of AO tables. The main factor seems to be the cost of unioning
   these bit vectors, and that cost is proportional to the number of bitmaps
   minus one and the size of the bitmaps, which is approximated here by the
   number of rows in the table.

   Note that because we use (NDV-1) in the formula, this penalty does not
   apply to usual index joins, which have an NDV of 1 per rebind. This is
   consistent with what we see in measurements and it also seems reasonable,
   since we don't have to union bitmaps in this case.

4. Fix to select CostModelGPDB for the 'experimental' model, as we do in 5X.

5. Calibrate the constants involved (c1 ... c6), using the calibration program
   and running experiments with heap and append-only tables on a laptop and
   also on a Linux cluster with 24 segments. Also run some other workloads
   for validation.

6. Give a small initial advantage to bitmap scans, so they will be chosen over
   table scans for small tables. Otherwise, small queries will
   have more or less random plans, all of which cost around 431, the value
   of the initial cost. Added a 10% advantage of the bitmap scan.

* Port calibration program to Python 3

- Used 2to3 program to do the basics.
- Version parameter in argparse no longer supported
- Needs additional option in connection string to keep the search path
- The dbconn.execSQL call can no longer be used to get a cursor,
  this was probably a non-observable defect in the Python 2 version
- Needed to use // (floor division) in some cases
Co-authored-by: NDavid Kimura <dkimura@vmware.com>

* Improve partition elimination when indexes are present (port from 6X)

* Use original join pred for DPE with index nested loop joins

Dynamic partition selection is based on a join predicate. For index
nested loop joins, however, we push the join predicate to the inner
side and replace the join predicate with "true". This meant that
we couldn't do DPE for nested index loop joins.

This commit remembers the original join predicate in the index nested
loop join, to be used in the generated filter map for DPE. The original
join predicate needs to be passed through multiple layers.

* SPE for index preds

Some of the xforms use method CXformUtils::PexprRedundantSelectForDynamicIndex
to duplicate predicates that could be used both as index predicates and as
partition elimination predicates. The call was missing in some other xforms.
Added it.

* Changes to equivalent distribution specs with redundant predicates

Adding redundant predicates causes some issues with generating
equivalent distribution specs, to be used for the outer table of
a nested index loop join. We want the equivalent spec to be
expressed in terms of outer references, which are the columns of
the outer table.

By passing in the outer refs, we can ensure that we won't replace
an outer ref in a distribution spec with a local variable from
the original distribution spec.

Also removed the asserts in CPhysicalFilter::PdsDerive that ensure the
distribution spec is complete (consisting of only columns from the
outer table) after we see a select node. Even without my changes, the
asserts do not always hold, as this test case shows:

  drop table if exists foo, bar;
  create table foo(a int, b int, c int, d int, e int) distributed by(a,b,c);
  create table bar(a int, b int, c int, d int, e int) distributed by(a,b,c);

  create index bar_ixb on bar(b);

  set optimizer_enable_hashjoin to off;
  set client_min_messages to log;

  -- runs into assert
  explain
  select *
  from foo join bar on foo.a=bar.a and foo.b=bar.b
  where bar.c > 10 and bar.d = 11;

Instead of the asserts, we now use the new method of passing in the
outer refs to ensure that we move towards completion. We also know
now that we can't always achieve a complete distribution spec, even
without redundant predicates.

* MDP changes

Various changes to MDPs:

- New SPE filters used in plan
- New redundant predicates (partitioning or on non-partitioning columns)
- Plan space changes
- Cost changes
- Motion changes
- Regenerated, because plan switched to a hash join, so used a guc
  to force an index plan
- Fixed lookup failures
- Add mdp where we try unsuccessfully to complete a distribution spec

* ICG result changes

- Test used the 'experimental' cost model to force an index scan, but we
  now get the index scan even with the default cost model (plans currently
  fall back).

This is a cherry-pick of commit 4a7a6821 from the 6X_STABLE branch.

Earlier the error was sent to /dev/null and the information was lost
displaying the cause of the error. Redirect the error to log file.

They have the same no-op implementation as the overridden base method,
and they aren't called anywhere.

The method called doesn't actually do any printing, and we don't assert
on the output. Removing the call doesn't even change the program output.