This includes the commit on master, 2 back branches, and the standalone
gporca repository.

(cherry picked from commit 7ab6b7a4)

We used to free the workfile_set when all its files are closed.
But in some cases, we use the workfile_set for the whole node execution
process. So, we should pin the workfile_set to avoid unexcepted free
operation. And manually remove it once we are sure we don't want to use
it anymore. Otherwise, the freed workfile_set may be allocated to others.
This fixes the issue: https://github.com/greenplum-db/gpdb/issues/10855.

Also, take care of the empty workfile_set, free it when applicable.
Distinguish by the workfile_set life cycle, the workfile_set could be class
into several classes.
1. workfile_set for a temporary workfile with interXact false(see buffile.c).
Once the file closed, the workfile_set will also get freed.
And if current transaction aborted, the workfile's resource owner is
responsible for closing it if the workfile is leaked, which also removes
the workfile_set.

2. workfile_set for a temporary workfile with interXact true(see buffile.c).
Once the file closed, the workfile_set will also get removed.
Since the workfile doesn't register to a resource owner, WorkFileLocalCtl
is responsible for removing its corresponding workfile_set in
AtProcExit_WorkFile(which gets called when process exit).

3. workfile_set is pinned and saved to prevent unexpected free operation.
When the workfile_set gets pinned, after all the file in the workfile_set
gets closed, we don't free the workfile_set, cause we may register new workfile
later. Call workfile_mgr_close_set to explicitly remove the workfile_set.
To deal with aborting, since the resource owner of the workfiles in the
workfile_set will close the workfiles, and this operation will not
remove workfile_set when no file exists in the workfile_set. We have to free
these kinds of workfile_sets when transaction ends in AtEOXact_WorkFile.
If the workfile_set contains interXact workfiles, we still rely on
WorkFileLocalCtl to remove workfile_sets in AtProcExit_WorkFile. But currently,
we don't have this kind of usage.

NOTE: if a workfile_set gets created but no workfile register into it(in case of
error, RegisterFileWithSet not called), it will never have a chance to be removed
automatically through file close. Then we have to check and remove these
workfile_sets when the transaction end to prevent workfile_set overflow
in AtEOXact_WorkFile.
Co-authored-by: NGang Xiong <gxiong@pivotal.io>
(cherry picked from commit 5b147fbe)

We had several issues on 5x and 6x that shows the resource group wait queue is
somehow corrupted, this commit add some checks and PANIC at the earliest to
help debugging.

The function combines two project nodes - one underneath the other. The
assertion checks if the newly created project uses only available
colrefs. However, it didn't include the outer refs in the upper project
node.

This commit fixes the assert highlighted by the following case:

create table foo(a int, b int);
create table bar(a int, b int);

select *
from foo
where a is null or
      a = (select foo.a+1
           from (select a+2 a2, b+2 b2
                 from bar) l2
           where l2.b2 = 5);
Co-authored-by: NHans Zeller <hzeller@pivotal.io>

In a previous commit (df5f06d6), we added code to fall back gracefully
when a subquery select list contains a single outer ref that is not part
of an expression, such as in

select * from foo where a is null or a = (select foo.b from bar)

This commit adds a fix that allows us to handle such queries in ORCA
by adding a project in the translator that will echo the outer ref
from within the subquery, and using that projected value in the
select list of the subquery. This ensures that we use a NULL value for
the scalar subquery in the expression for the outer ref when the
subquery returns no rows.

Also note that this is still skipped for grouping cols in the target
list. This was done to avoid regression for certain queries, such as:

select *
from A
where not exists (select sum(C.i)
                  from C
                  where C.i = A.i
                  group by a.i);

ORCA is currently unable to decorrelate sub-queries that contain project
nodes, So, a `SELECT 1` in the subquery would also cause this
regression. In the above query, the parser adds `a.i` to the target list
of the subquery, that would get an echo projection (as described above),
and thus would prevent decorelation by ORCA. For this reason, we decided
to maintain existing behavior until ORCA is able to handle projections
in subqueries better.

Also, regenerate 6 MDPs with outer refs in their project lists

Regenerating the MDPs for 6 test cases that were identified having
outer refs in their project lists. These test cases will go through
the fix in the translator and most of them will generate an extra
project in the DXL query section. Some of them now also have an
extra project in the plan section.
Co-authored-by: NHans Zeller <hzeller@pivotal.io>
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>

With the ifaddrs utility, we excluded ip addresses on the loopback
interface which caused regression causing replication entries to be not
populated for such interfaces causing gpaddmirrors and gpinitstandby to
fail. Routable IP addresses can be assigned to the loopback interface,
and this case was not considered earlier.
This commit fixes the issues by allowing all loopback addresses
except 127.0.0.1 and ::1 address

Earlier the output filename for the test check_orphaned_toastrels
was partitioned_tables.txt, so updated that to represent toast tables

Planner was choosing a different join order for different builds.
Spreading out the values of the smaller table, so that for a hash
join, the two tables will be of different size. That should make
choosing a hash join order more clear.

(cherry picked from commit 84bff28b)
with some additional changes

* Add a CPatternNode operator

This operator can be used in patterns of xforms and it can match one of
multiple regular operators.

Which operators it matches depends on its match type. Right now, there is
only one match type, EmtMatchInnerOrLeftOuterJoin, that matches a
logical inner join or a logical left outer join.

* Add 2 new xforms for index apply

The new xforms use the new CPatternNode for a patterns of the form

CPatternNode(Leaf, Tree, Tree)

The CPatternNode matches both an inner and a left outer join.
The Tree for the right child can contain arbitrary operators.
To avoid an explosion of the search space, we add two conditions
to the xform: First, the xform is applied only once, not for all
bindings of the Tree node. Second, the xform is applied only if
the right child has a "join depth" of 1, excluding any children
that are complex and wouldn't satisfy the conditions of this xform
anyway.

* Remove 16 obsolete xforms and replace them with 2 new ones

To remove xforms, we have to add a mechanism to skip unused xform ids,
to preserve the ids of the remaining xforms that are used in trace flags.
Our array of xforms now allows for holes.

* Changes to unit test programs

Updated the test programs to tolerate holes in the xform array.
Used a new xform instead of one of the removed ones.

* MDP changes

* ICG changes

* Fixes for ICG failures in join and aggregates tests

The new xform allows additional plans that are chosen in the explain
output. It also surfaced a bug where we can't eliminate a groupby that
sits on top of a CLogicalIndexGet, because the index get doesn't derive
a key set.

* Support for project nodes in index nested loop joins

When generating required distribution specs for its children,
CPhysicalInnerIndexNLJoin will start with its inner child and send it
an ANY required distribution spec. It will then force the outer child
to match the inner's distribution spec (or require a broadcast on the outer).

Now, assume we have CPhysicalComputeScalar as the inner child. This
node, in CPhysicalComputeScalar::PdsRequired will currently require
its child to be REPLICATED (opt request 1) or SINGLETON (opt request
0), if the expression has any outer references. This won't work, since
the underlying table has neither of these distribution schemes and
since we don't want any motions between the index join and the
index get.

This commit changes the behavior of a CPhysicalComputeScalar. If it
senses that it is part of an index nested loop join, it will just
propagate the required distribution spec from the parent.
How does it sense that? By the required ANY distribution spec that
allows outer references. This request is generated in only two places:
CPhysicalInnerIndexNLJoin::PdsRequired and
CPhysicalLeftOuterIndexNLJoin::PdsRequired, so it is only used in the
context of an index NLJ. This behavior is similar to what the CPhysicalFilter
node does, the other node allowed between an index NLJ and the get.

* Adding a check for grouping columns

Allow groupby under the right side of an index join only if the groupby
can be co-located with the get underneath. This happens when the grouping
columns are a superset of the distribution columns, since then every group
will be entirely contained in a single segment.

* History of this commit

This is the second attempt at this commit, after having to revert earlier
commits 24671b5b and 17da1026. In addition to the first commit,
we had to add a check for the groupby columns to be a superset of the
distribution columns (see item above) and we also fixed several diffs in tests.
Co-authored-by: NDavid Kimura <dkimura@vmware.com>
Co-authored-by: NHans Zeller <hzeller@vmware.com>
(cherry picked from commit 84bff28b)

For CTAS creating tables with non-legacy hashop distribution from tables
with legacy hashops, CTAS with Orca would distribute the data according
to the value of gp_use_legacy_hashops; however, it would set the table's
distribution policy hashop to the value of the original table. This
caused queries to give incorrect results as the distribution policy
mismatched the data distribution.

The data was being distributed correctly as we used the correct opfamily
internally in Orca when determining the distribution spec. However, when
populating the distribution policy value in the table in
CTranslatorDXLToPlStmt.cpp, we were trying to derive the opclass from
the opfamily. This doesn't work, and instead we need to pass the opclass
from the query through Orca and to the plan statement. Hence we use the
opfamily internally within Orca, and simply pass through the opclass to
populate the gp_distribution_policy value correctly.

Additionally, CTAS minidumps have been failing for quite a while, and
subsequent changes made this a bit difficult to untangle. Since we only
have ~10 mdp files, I simply regenerated most of these files and made
the opfamily/opclasses arrays required as part of CMDRelationCTAS,
LogicalCTAS, and PhysicalCTAS.

Also, remove the `GPOS_ASSERT(NULL == mdhtacc.Find());` assert in
CMDAccessor.cpp. We've been hitting this with newly created mdps since
at least a couple of years, but haven't added new CTAS mdps since then.
Removing the assert doesn't seem to cause any problems and it's only
relevant for mdps.

(cherry picked from commit a8eb84a2)

The canonical config file is in src/backend/gporca/.clang-format, I've
created two symlinks, one for GPOPT headers, one for GPOPT.

This is spiritually a cherry-pick of commit 2f7dd76c, but with
the actual code of this branch (6X_STABLE) formatted, of course.

(cherry picked from commit 2f7dd76c)

Add a stage called "check-format" that runs before the fan-out of
installcheck-small. While we're at it, also enforce that the full config
file is generated from the intent file.  This should be fairly quick. On
my laptop, the `fmt chk` step takes 3 seconds.

Enforcement in Concourse is forthcoming.

(cherry picked from commit 54273fdd)

Otherwise it seems clang-format recognizes this as a multi-line
preprocessor directive, strange, but preventable.

(cherry picked from commit 3c673965)

This is intended for both local developer use and for CI.

This depends on GNU parallel. One-time install:

macOS: brew install parallel clang-format
Debian: apt install parallel clang-format-10

To format all ORCA / GPOPT code:

$ src/tools/fmt fmt

To check for formatting conformance:

$ src/tools/fmt chk

To modify the configuration, you'll need two steps:
1. Edit clang-format.intent.yml
2. Generate the expanded configuration file:

$ src/tools/fmt gen

This commit also adds a formatting README, To document some of the
rationale behind tooling choice. Also mention the new `README.format.md`
from both the style guide and ORCA's main README.

(cherry picked from commit 57b744c1)

Generated using clang-format-10

(cherry picked from commit 16b48d24)

  ICW concourse jobs consume lots of computing resources and make the 
  system load is very high. and the concourse performance is very bad
  for the user. Creating the external concourse worker pool
  for icw jobs can reduce the whole concourse system workload.

  Add the tags attribution to the pipeline, and ensure the icw job will
  be placed at the external worker pool.

  This commit will affect the gpdb6 icw jobs on centos6/centos7/oracle7/ubuntu18.04
  platforms
Authored-by: NTingfang Bao <baotingfang@gmail.com>

When gp_use_legacy_hashops is set, for plans that have an Aggregate,
ORCA might crash when computing the distribution spec. This occurs when
the grouping columns are of a type that have default hash distribution
opfamilies, but do not have legacy hash distribution opfamilies.

This happens because the distribution is done only for types for which
"IsRedistributable()" returns true. This is (incorrectly) determined by
looking (only) at the default opfamily (independent of the GUC).
Subsequently, CDistributionSpecHashed::PopulateDefaultOpfamilies() tries
to assemble opfamilies for all redistributable columns, which shouldn't
include such types (when the GUC is set), resulting in dereferencing a
nullptr.

For example, int[] type has a default distr opfamily, but no
legacy distr opfamily.

This commit fixes the implementation of IsRedistributable() to avoid the
above error.
Co-authored-by: NShreedhar Hardikar <hardikar@cs.wisc.edu>

Co-authored-by: NXin Zhang <zhxin@vmware.com>
Co-authored-by: NBrent Doli <bdoil@vmware.com>

Commit 0cf1c0d8 defined a global variable in a header. This
resulted in a ton of "unused variable" warnings.

The `static step_timer timer` is only used in contrib/pg_upgrade/util.c.
So move it here.

Function `RelationGetIndexList()` does not filter out invalid indexes.
That responsiblity is left to the caller (e.g. `get_relation_info()`).
Issue is that Orca was not checking index validity.

This commit also introduces an optimization to Orca that is already used
in Planner whereby we first check relhasindex before checking pg_index.

(cherry picked from commit b011c351)

For context, the registry-image resource was introduced more than two
years ago as a drop-in replace for the docker-image resource (especially
when you're only fetching, not building). It's leaner, less
resource-intense, faster, and doesn't rely on spawning a Docker daemon.

Also swept up in this patch are four unused files that are left behind
by previous changes.

This is spiritually a cherry-pick of commit be6ff30f
(greenplum-db/gpdb#10818).

See https://github.com/concourse/registry-image-resource for more.

This is a backport of the changes in #10799 to 6X_STABLE.

When we have an outer ref in a subquery, like this

select * from foo where foo.a is null or foo.a = (select foo.b from bar)

then we can't simply use the outer reference for the condition when
we unnest the subquery into an apply. This is because if the subquery
returns no rows, then we must be using a NULL instead of the outer
reference.

We have code to handle this for quantified subqueries, but not for
scalar subqueries.

When we translate the generated DXL to a plan, we assert when we find
an outer reference in the project list of a subquery. In rare cases,
we might also crash, when the subquery contained a project with multiple
values below the outer reference (see added test in gporca.sql).

The "fix" (more a workaround) is to force a fallback when we detect
this situation during unnesting of a scalar subquery.

Previously after the 9.2 merge, we had a hack that short-circuits the
execution of the join when ORCA is on. This hack stops working with the
Postgres 12 merge because the FIXME subquery gets executed on the QEs as
well rather than just on the QD. This patch simply turns ORCA off for
the query.
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
(cherry picked from commit 891e77d1)

Commit df5d3389 attempted to backport e79b81e5 but we
missed a critical chunk while resolving a conflict. This (of course)
breaks Travis macOS build. Put it back here.

(cherry picked from commit e79b81e5)

ccache 3.2 (from Ubuntu Xenial) defaults CCACHE_CPP2 to off (this
setting defaults to on starting from ccache 3.3). That default leads to
unlegible compiler warning outputs because GCC and Clang will emit
warnings using the preprocessed output. Turn it on here.

(cherry picked from commit 44e6bfec)

While working on https://github.com/greenplum-db/gpdb/pull/9937 ("Build
ORCA with C++14."), I realized we took out ccache from Travis CI. This
patch set brings it back with some additional TLC.

Highlights:

* Remove deprecated [1][2] key "sudo: false"

* Set Travis language to C++

  Travis propagates the environment variables CXX (in addition to CC,
  which is exported when the language is C) when the language is C++.
  This prepares us for a forthcoming change to enable ORCA build in
  Travis.

* Care is taken to ensure ccache is actually used on macOS.

  The default settings in Travis somehow neglected it
  (travis-ci/travis-build#655) so we compensate for it.

* Similar care is taken to ensure we use ccache with Clang.

  The version of ccache used in Travis is just slightly older than when
  automatic Clang symlink supported was added.

* Show ccache stats (hit rate and more) for each build

Reference:
[1] https://blog.travis-ci.com/2018-11-19-required-linux-infrastructure-migration
[2] Job config linting warning from Travis

(cherry picked from commit e79b81e5)

This reverts commit 4c8332b5.

In commit 5eaa5889, the --novacuum option is removed, however the help
page of gpexpand keep the -V option, which is a short option for
--novacuum.

(cherry picked from commit cb2afaf9)

Commit 889ba39e fixed the query string truncation in dispatcher
to make it locale-aware.  This patch refactors that change so as to
avoid accessing a string beyond its length.

Reviewed by: Heikki, Ning Yu and Polina Bungina

(cherry picked from commit ef09a3a6)

Execution of a long enough query containing multi-byte characters can cause incorrect truncation of the query string. Incorrect truncation implies an occasional cut of a multi-byte character and (with log_min_duration_statement set to 0 ) subsequent write of an invalid symbol to segment logs. Such broken character present in logs produces problems when trying to fetch logs info from gp_toolkit.__gp_log_segment_ext  table - queries fail with the following error: «ERROR: invalid byte sequence for encoding…».
This is caused by buildGpQueryString function in `cdbdisp_query.c`, which prepares query text for dispatch to QE. It does not take into account character length when truncation is necessary (text is longer than QUERY_STRING_TRUNCATE_SIZE).

(cherry picked from commit 889ba39e)

Here is part of the diff output.

@@ -14,11 +14,11 @@
  role | preferred_role | content | mode | status
  ------+----------------+---------+------+--------
  m    | m              | -1      | s    | u
- m    | m              | 0       | s    | u
+ m    | m              | 0       | n    | u

The root cause has nothing to do with this test case.  It's because test
prepared_xact_deadlock_pg_rewind finally calls gprecoverseg to recover the
cluster but does not wait until the cluster state restores.
Reviewed-by: NJunfeng(Jerome) Yang <jeyang@pivotal.io>
(cherry picked from commit 07c594ed)

The terminate_timeout and recv_timeout parameters are defined based on
incoming quantum value before its invalidation/normalization phase. As a
consequence those parameters can take on values that lead to unstable
interaction between gpmmon and gpsmon.

Current fix moves the assignment of terminate_timeout and recv_timeout
behind invalidation/normalization block.  Furthermore, as
terminate_timeout is passed to gpsmon as startup option, the guard that
disallow zero timeout value for tcp_event is added.

Using exit() requires stdlib.h, which is not included.  Use return
instead.  Also add return type for main().
Reviewed-by: NHeikki Linnakangas <hlinnaka@iki.fi>
Reviewed-by: NThomas Munro <thomas.munro@enterprisedb.com>
(cherry picked from commit 1c0cf52b)
(cherry picked from commit 6d3c99bb)

This file will be used to record commits to be ignored by default by
git-blame (user still has to opt in). This is intended to include
large (generally automated) reformatting or renaming commits.

(cherry picked from commit b19e6abb)

The GUC gp_interconnect_proxy_addresses is used to set the listener
addresses and ports of all the proxy bgworkers, only IP addresses were
supported previously, which is inconvenient to use.

Now we add the support for hostnames too, the IP addresses are also
supported.

Note that if a hostname is bound to a different IP at runtime, we must
reload the setting with the "gpstop -u" command.
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>
(cherry picked from commit 2a1794bc)

A typical mistake on allocating typed memory is as below:

    int64 *ptr = malloc(sizeof(int32));

To prevent this, now we make ic_proxy_new() a typed allocator, it always
return a pointer of the specified type, for example:

    int64 *p1 = ic_proxy_new(int64); /* good */
    int64 *p2 = ic_proxy_new(int32); /* bad, gcc will raise a warning */
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>
(cherry picked from commit a3ef623d)

We used to have `Assert` to check `RegisterFileWithSet` never register
file to a non-active workfile_set. But in production, there could be
some corner cases that caller register file to a non-active workfile_set.
It'll cause inconsistent `workfile_shared->num_active` with the real
active workfile_sets numbers under some situations.
For example,
1. `RegisterFileWithSet` a file to a created work_set. (current
`work_set->num_files` is 1)
2. `FileClose` closes the file and causes `WorkFileDeleted` to detele
the work_set since current `work_set->num_files` is 0 after detele file.
Which also decrease `workfile_shared->num_active`.
3.  `RegisterFileWithSet` another file to the created work_set(which
actually is not active now, but we dont't prevent that, only uses
`Assert` to check).
4. `FileClose` closes the file and causes `WorkFileDeleted` to detele
the work_set again. The `workfile_shared->num_active` gets decreased
again.

Raise PANIC to expose the coner cases.
Normally the caller of `RegisterFileWithSet` should ensure the correctness.
But we lack of the check in the `RegisterFileWithSet`.

(cherry picked from commit c23980cb)