The canonical config file is in src/backend/gpopt/.clang-format (instead
of under the non-existent src/backend/gporca), I've created one (instead
of two) symlink, for GPOPT headers. Care has been taken to repoint the
symlink to the canonical config under gpopt, instead of gpopt as it is
under HEAD.

This is spiritually a cherry-pick of commit 2f7dd76c.
(cherry picked from commit 2f7dd76c)

Function `RelationGetIndexList()` does not filter out invalid indexes.
That responsiblity is left to the caller (e.g. `get_relation_info()`).
Issue is that Orca was not checking index validity.

This commit also introduces an optimization to Orca that is already used
in Planner whereby we first check relhasindex before checking pg_index.

(cherry picked from commit b011c351)

This commit updates the following functions names
- CHistogram::Buckets ->  CHistogram::GetNumBuckets
- CHistogram::ParseDXLToBucketsArray -> CHistogram::GetBuckets
for clarity on the GPORCA side.

Orca uses this property for cardinality estimation of joins.
For example, a join predicate foo join bar on foo.a = upper(bar.b)
will have a cardinality estimate similar to foo join bar on foo.a = bar.b.

Other functions, like foo join bar on foo.a = substring(bar.b, 1, 1)
won't be treated that way, since they are more likely to have a greater
effect on join cardinalities.

Since this is specific to ORCA, we use logic in the translator to determine
whether a function or operator is NDV-preserving. Right now, we consider
a very limited set of operators, we may add more at a later time.

This commit enables the MCVs for text related types such as varchar, name etc
to be passed to ORCA so that it can estimate the cardinalities for columns
containing text related types. Prior to this commit, ORCA would estimate the
cardinality to be 40% of the tuples which would cause mis-estimation for
certain queries.
Co-authored-by: NAbhijit Subramanya <asubramanya@pivotal.io>
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>

Also make minor interface changes needed for ORCA version 3.84.

Send btree indexes on AO tables as type btree

Previously, GPDB sent btree indexes on AO tables to ORCA as type bitmap
as btree indexes were not allowed on AO tables. This was a bit hacky, so
instead, send it over as type btree and let ORCA handle it properly.
Co-Authored-By: NAshuka Xue <axue@pivotal.io>
Co-Authored-By: NHans Zeller <hzeller@pivotal.io>

This commit corresponds to the ORCA commit "Implement Full Merge Join"

In GPDB 5, merge join is disabled, but the following changes were made
to continue allowing compilation of GPDB 5 with ORCA.

1. Translator changes for Merge Join.
2. Add IsOpMergeJoinable() and GetMergeJoinOpFamilies()  wrappers.

The IMemoryPool interface was removed in ORCA to remove an unnecessary
abstraction layer and avoid costly casting.

Corresponding ORCA commit: e64a2b42

Bumps ORCA version to 3.46.0
Authored-by: NChris Hajas <chajas@pivotal.io>

Prior to this patch, the MCVs for text columns was not being passed to ORCA.
Hence the cardinality estimation for predicates involving text was inaccurate
and led to sub optimal plans being picked. This patch allows the MCVs to be
passed in to ORCA so that it can now estimate the cardinality using MCVs equal
and not equal operators for text columns.
Co-authored-by: NAbhijit Subramanya <asubramanya@pivotal.io>
Co-authored-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>

Related to https://github.com/greenplum-db/gpdb/issues/5981.

In pg_statistic catalog table, stadistinct indicates the (approximate)
number of distinct non-null data values in the column. When < 0, it
holds the negative of multiplier for number of rows.

In the relcache translator, stadistinct is used to calculate
num_distinct, which is the absolute (approx) number of non-null values.
But, the formula used num_rows directly, without discounting for null
values. This would occasionally trip an assertion in ORCA for bool
column types, as it would appear there are 3 distinct bool values, since
it would include the NULL values in num_distinct.

This reverts commit ac6ebca7.

This caused a regression in TPC-DS query 24. It seems we miscalcuated
the num_distinct in CTranslatorRelcacheToDXL::RetrieveColStats since
stadistinct only counts number of distinct non-null values.

Related to https://github.com/greenplum-db/gpdb/issues/5981

In the relcache translator we incorrectly count the distinct because of
the null value into the histogram. This makes an assert fail inside
GPORCA.

No test added since the issue was discovered during an existing tests.

These are the warnings I saw:

CTranslatorRelcacheToDXL.cpp: In static member function ‘static gpmd::IMDIndex* gpdxl::CTranslatorRelcacheToDXL::RetrieveIndex(gpos::IMemoryPool*, gpopt::CMDAccessor*, gpmd::IMDId*)’:
CTranslatorRelcacheToDXL.cpp:1152:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for (ULONG ul = 0; ul < form_pg_index->indnatts; ul++)
                      ~~~^~~~~~~~~~~~~~~~~~~~~~~~~
CTranslatorRelcacheToDXL.cpp: In static member function ‘static gpmd::IMDIndex* gpdxl::CTranslatorRelcacheToDXL::RetrievePartTableIndex(gpos::IMemoryPool*, gpopt::CMDAccessor*, LogicalIndexInfo*, gpmd::IMDId*, const gpmd::IMDRelation*)’:
CTranslatorRelcacheToDXL.cpp:1309:24: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  for (ULONG ul = 0; ul < index_info->nColumns; ul++)
                     ~~~^~~~~~~~~~~~~~~~~~~~~~

The access to the members of 'Relation' struct after heap_close() will
cause crash when '-DRELCACHE_FORCE_RELEASE' is set.
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NGang Xiong <gxiong@pivotal.io>

As part of moving away from Hungarian notation in the GPORCA codebase,
the integration points between GPORCA and GPDB in the translator have
been renamed to the new convention used in GPORCA. The libraries
currently updated to the new notation in GPORCA are Naucrates and GPOS.
The new naming convention is a custom version of common C++ naming
conventions. The style guide for this convention can be found in the
GPORCA repository.

Also bump ORCA version to 2.69.0
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
Co-authored-by: NEkta Khanna <ekhanna@pivotal.io>
Co-authored-by: NAbhijit Subramanya <asubramanya@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io&gt;&lt;Paste>
Co-authored-by: NDhanashree Kashid <dkashid@pivotal.io>
Co-authored-by: NOmer Arap <oarap@pivotal.io>

Prior to this commit, there was no support for GiST indexes in GPORCA.
For queries involving GiST indexes, ORCA was selecting Table Scan paths
as the optimal plan. These plans could take up to 300+ times longer than
Planner, which generated a index scan plan using the GiST index.

Example:
```
CREATE TABLE gist_tbl (a int, p polygon);
CREATE TABLE gist_tbl2 (b int, p polygon);
CREATE INDEX poly_index ON gist_tbl USING gist(p);

INSERT INTO gist_tbl SELECT i, polygon(box(point(i, i+2),point(i+4,
i+6))) FROM generate_series(1,50000)i;
INSERT INTO gist_tbl2 SELECT i, polygon(box(point(i+1, i+3),point(i+5,
i+7))) FROM generate_series(1,50000)i;

ANALYZE;
```
With the query `SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE
gist_tbl.p <@ gist_tbl2.p;`, we see a performance increase with the
support of GiST.

Before:
```
EXPLAIN SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
                                                     QUERY PLAN
---------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=0.00..171401912.12 rows=1 width=8)
   ->  Gather Motion 3:1  (slice2; segments: 3)  (cost=0.00..171401912.12 rows=1 width=8)
         ->  Aggregate  (cost=0.00..171401912.12 rows=1 width=8)
               ->  Nested Loop  (cost=0.00..171401912.12 rows=335499869 width=1)
                     Join Filter: gist_tbl.p <@ gist_tbl2.p
                     ->  Table Scan on gist_tbl2  (cost=0.00..432.25 rows=16776 width=101)
                     ->  Materialize  (cost=0.00..530.81 rows=49997 width=101)
                           ->  Broadcast Motion 3:3  (slice1; segments: 3)  (cost=0.00..525.76 rows=49997 width=101)
                                 ->  Table Scan on gist_tbl  (cost=0.00..432.24 rows=16666 width=101)
 Optimizer status: PQO version 2.65.1
(10 rows)

Time: 170.172 ms
SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
 count
-------
 49999
(1 row)

Time: 546028.227 ms
```

After:
```
EXPLAIN SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
                                                  QUERY PLAN
---------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=0.00..21749053.24 rows=1 width=8)
   ->  Gather Motion 3:1  (slice2; segments: 3)  (cost=0.00..21749053.24 rows=1 width=8)
         ->  Aggregate  (cost=0.00..21749053.24 rows=1 width=8)
               ->  Nested Loop  (cost=0.00..21749053.24 rows=335499869 width=1)
                     Join Filter: true
                     ->  Broadcast Motion 3:3  (slice1; segments: 3)  (cost=0.00..526.39 rows=50328 width=101)
                           ->  Table Scan on gist_tbl2  (cost=0.00..432.25 rows=16776 width=101)
                     ->  Bitmap Table Scan on gist_tbl  (cost=0.00..21746725.48 rows=6667 width=1)
                           Recheck Cond: gist_tbl.p <@ gist_tbl2.p
                           ->  Bitmap Index Scan on poly_index  (cost=0.00..0.00 rows=0 width=0)
                                 Index Cond: gist_tbl.p <@ gist_tbl2.p
 Optimizer status: PQO version 2.65.1
(12 rows)

Time: 617.489 ms

SELECT count(*) FROM gist_tbl, gist_tbl2 WHERE gist_tbl.p <@ gist_tbl2.p;
 count
-------
 49999
(1 row)

Time: 7779.198 ms
```

GiST support was implemented by sending over GiST index information to
GPORCA in the metadata using a new index enum specifically for GiST.
Signed-off-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>
Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>

When there is no stats available for any table, ORCA was treating it as an
empty table while planning. On the other hand planner is utilizing a guc
`gp_enable_relsize_collection` to obtain the estimated size of the table, but
no other statistics. This commit enables ORCA to have the same behavior as
planner when the guc is set.
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>

If the column statistics in `pg_statistic` has values with type
different than column type, metadata accessor should not translate the
stats and create a dummy stats instead.

This commit also reorders stats collection from the `pg_statistic` to
align with how analyze generates stats. MCV and Histogram translation is
moved to the end after NDV, nullfraction and column width extraction.
Signed-off-by: NMelanie Plageman <mplageman@pivotal.io>

Fixes greenplum-db/gporca#358

For text, varchar, char and bpchar, ORCA does not collect the
MCV and Histogram information, so the calculation of NDVRemain and
FreqRemain must be updated to account for it.

For such columns, NDVRemain is the stadistinct as available in the
pg_statistic, and FreqRemain is everything except the NULL frequency.

Earlier, NDVRemain and FreqRemain for such columns would yield 0
resulting in poor cardinality estimation and suboptimal  plans.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>
(cherry picked from commit 4a5c58a5)

ORCA has historically ignored type modifiers from databases that support
them, noticeably Postgres and Greenplum. This has led to surprises in a
few cases:

1. The output description over the wire (for Postgres protocol) will
lose the type modifier information, which often meant length. This
surprises code that expects a non-default type modifier, e.g. a JDBC
driver.

2. The executor in some cases -- notably DML -- expects a precise type
modifier. Because ORCA always erases the type modifiers and presents a
default, the executor is forced to find that information elsewhere.

After this commit, ORCA will be aware of type modifiers in table
columns, scalar identifiers, constants, and length-coercion casts.
Signed-off-by: NShreedhar Hardikar <shardikar@pivotal.io>
(cherry picked from commit 2d907526)

In scenarios where pg_statistic contains wrong statistic entry for an
attribute, or when the statistics on a particular attribute are broken,
for e.g the type of elements stored in stavalues<1/2/3> is different
than the actual attribute type or when there are holes in the attribute
numbers due to adding/dropping columns; following two APIs fail because
they relied on the attribute type sent by the caller:

- get_attstatsslot() : Extracts the contents (numbers/frequency array and
values array) of the requested statistic slot (MCV, HISTOGRAM etc). If the
attribute is pass-by-reference or if the attribute is of toastable type
(varlena types)then it returns a copy allocated with palloc()
- free_attstatsslot() : Frees any palloc'd data by get_attstatsslot()

This problem was fixed in upstream 8.3
(8c21b4e9) for get_attstatsslot(),
wherein the actual element type of the array will be used for
deconstructing it rather that using caller passed OID.
free_attstatsslot() still depends on the type oid sent by caller.

However the issue still exists for free_attstatsslot() where it crashes while
freeing the array. The crash happened because the caller sent type OID was of
type TEXT meaning this a varlena type and hence free_attstatsslot() attempted
to free the datum; however due to the broken slot the datums extracted from
values array were of fixed length type such as int. We considered the int value
as memory address and crashed while freeing it.

This commit brings in a following fix from upstream 10 which redesigns
get_attstatsslot()/free_attstatsslot() such than they robust to scenarios like
these.

commit 9aab83fc
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Sat May 13 15:14:39 2017 -0400

    Redesign get_attstatsslot()/free_attstatsslot() for more safety and speed.

    The mess cleaned up in commit da075960 is clear evidence that it's a
    bug hazard to expect the caller of get_attstatsslot()/free_attstatsslot()
    to provide the correct type OID for the array elements in the slot.
    Moreover, we weren't even getting any performance benefit from that,
    since get_attstatsslot() was extracting the real type OID from the array
    anyway.  So we ought to get rid of that requirement; indeed, it would
    make more sense for get_attstatsslot() to pass back the type OID it found,
    in case the caller isn't sure what to expect, which is likely in binary-
    compatible-operator cases.

    Another problem with the current implementation is that if the stats array
    element type is pass-by-reference, we incur a palloc/memcpy/pfree cycle
    for each element.  That seemed acceptable when the code was written because
    we were targeting O(10) array sizes --- but these days, stats arrays are
    almost always bigger than that, sometimes much bigger.  We can save a
    significant number of cycles by doing one palloc/memcpy/pfree of the whole
    array.  Indeed, in the now-probably-common case where the array is toasted,
    that happens anyway so this method is basically free.  (Note: although the
    catcache code will inline any out-of-line toasted values, it doesn't
    decompress them.  At the other end of the size range, it doesn't expand
    short-header datums either.  In either case, DatumGetArrayTypeP would have
    to make a copy.  We do end up using an extra array copy step if the element
    type is pass-by-value and the array length is neither small enough for a
    short header nor large enough to have suffered compression.  But that
    seems like a very acceptable price for winning in pass-by-ref cases.)

    Hence, redesign to take these insights into account.  While at it,
    convert to an API in which we fill a struct rather than passing a bunch
    of pointers to individual output arguments.  That will make it less
    painful if we ever want further expansion of what get_attstatsslot can
    pass back.

    It's certainly arguable that this is new development and not something to
    push post-feature-freeze.  However, I view it as primarily bug-proofing
    and therefore something that's better to have sooner not later.  Since
    we aren't quite at beta phase yet, let's put it in.

    Discussion: https://postgr.es/m/16364.1494520862@sss.pgh.pa.us

Most of the changes are same as the upstream commit with following additional
changes:
- Relcache translator changes in ORCA.
- Added a test that simulates the crash due to broken stats
- get_attstatsslot() contains an extra check for empty slot array which existed
in master but is not there in upstream.
Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>
(cherry picked from commit ae06d7b0)

To support that, this commit adds 2 new ORCA APIs:
- SignalInterruptGPOPT(), which notifies ORCA that an abort is requested
  (must be called from the signal handler)
- ResetInterruptsGPOPT(), which resets ORCA's state to before the
  interruption, so that the next query can run normally (needs to be
  called only on the QD)

Also check for interrupts right after ORCA returns.

GPORCA should not spend time extracting column statistics that are not
needed for cardinality estimation. This commit eliminates this overhead
of requesting and generating the statistics for columns that are not
used in cardinality estimation unnecessarily.

E.g:
`CREATE TABLE foo (a int, b int, c int);`

For table foo, the query below only needs for stats for column `a` which
is the distribution column and column `c` which is the column used in
where clause.
`select * from foo where c=2;`

However, prior to that commit, the column statistics for column `b` is
also calculated and passed for the cardinality estimation. The only
information needed by the optimizer is the `width` of column `b`. For
this tiny information, we transfer every stats information for that
column.

This commit and its counterpart commit in GPORCA ensures that the column
width information is passed and extracted in the `dxl:Relation` metadata
information.

Preliminary results for short running queries provides up to 65x
performance improvement.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

Currently ORCA does not support index scan on leaf partitions. It only supports
index scan if we query the root table. This commit along with the corresponding
ORCA changes adds a support for using indexes when leaf partitions are queried
directly.

When a root table that has indexes (either homogenous/complete or
heterogenous/partial) is queried; the Relcache Translator sends index
information to ORCA.  This enables ORCA to generate an alternative plan with
Dynamic Index Scan on all partitions (in case of homogenous index) or a plan
with partial scan i.e. Dynamic Table Scan on leaf partitions that  don’t have
indexes + Dynamic Index Scan on leaf partitions with indexes (in case of
heterogeneous index).

This is a two step process in Relcache Translator as described below:

Step 1 - Get list of all index oids

`CTranslatorRelcacheToDXL::PdrgpmdidRelIndexes()` performs this step and it
only retrieves indexes on root and regular tables; for leaf partitions it bails
out.

Now for root, list of index oids is nothing but index oids on its leaf
partitions. For instance:

```
CREATE TABLE foo ( a int, b int, c int, d int) DISTRIBUTED by (a) PARTITION
BY RANGE(b) (PARTITION p1 START (1) END (10) INCLUSIVE, PARTITION p2 START (11)
END (20) INCLUSIVE);

CREATE INDEX complete_c on foo USING btree (c); CREATE INDEX partial_d on
foo_1_prt_p2 using btree(d);
```
The index list will look like = { complete_c_1_prt_p1, partial_d }

For a complete index, the index oid of the first leaf partitions is retrieved.
If there are partial indexes, all the partial index oids are retrieved.

Step 2 - Construct Index Metadata object

`CTranslatorRelcacheToDXL::Pmdindex()` performs this step.

For each index oid retrieved in Step #1 above; construct an Index Metadata
object (CMDIndexGPDB) to be stored in metadata cache such that ORCA can get all
the information about the index.
Along with all other information about the index, `CMDIndexGPDB` also contains
a flag `fPartial` which denotes if the given index is homogenous (if yes, ORCA
will apply it to all partitions selected by partition selector) or heterogenous
(if yes, the index will be applied to only appropriate partitions).
The process is as follows:
```
	Foreach oid in index oid list :
		Get index relation (rel)
		If rel is a leaf partition :
			Get the root rel of the leaf partition
			Get all	the indexes on the root (this will be same list as step #1)
			Determine if the current index oid is homogenous or heterogenous
			Construct CMDIndexGPDB based appropriately (with fPartial, part constraint,
			defaultlevels info)
		Else:
			Construct a normal CMDIndexGPDB object.
```

Now for leaf partitions, there is no notion of homogenous or heterogenous
indexes since a leaf partition is like a regular table. Hence in `Pmdindex()`
we should not got for checking if index is complete or not.

Additionally, If a given index is homogenous or heterogenous needs to be
decided from the perspective of relation we are querying(such as root or a
leaf).

Hence the right place of `fPartial` flag is in the relation metadata object
(CMDRelationGPDB) and not the independent Index metadata object (CMDIndexGPDB).
This commit makes following changes to support index scan on leaf partitions
along with partial scans :

Relcache Translator:

In Step1, retrieve the index information on the leaf partition and create a
list of CMDIndexInfo object which contain the index oid and `fPartial` flag.
Step 1 is the place where we know what relation we are querying which enable us
to determine whether or not the index is homogenous from the context of the
relation.

The relation metadata tag will look like following after this change:

Before:
```
	<dxl:Indexes>
		<dxl:Index Mdid="0.17159874.1.0"/>
		<dxl:Index Mdid="0.17159920.1.0"/>
	</dxl:Indexes>
```

After:
```
	<dxl:IndexInfoList>
		<dxl:IndexInfo Mdid="0.17159874.1.0" IsPartial="true"/>
		<dxl:IndexInfo Mdid="0.17159920.1.0" IsPartial="false"/>
	</dxl:IndexInfoList>

```

A new class `CMDIndexInfo` has been created in ORCA which contains index mdid
and `fPartial` flag.  For external tables, normal tables and leaf partitions;
the `fPartial` flag will always be false.

Hence at the end, relcache translator will provide list of indexes defined on
leaf partitions when they are queried directly with `fPartial` being false
always. And when root table is queried; the `fPartial` will be set
appropriately based on the completeness of the index.  ORCA will refer to
Relation Metadata for fPartial information and not to the indepedent Index
Metadata Object.

[Ref ##120303669]

(cherry picked from commit dae6849f)

While gathering metadata information about a partitioned relation, the
relcache translator in GPDB constructs and sends partition
constraints in `CMDPartConstraintGPDB` object.
CMDPartConstraintGPDB contains following:
1. m_pdrgpulDefaultParts = List of partition levels that have default
                           partitions
2. m_fUnbounded          = indicate if the constraint is unbounded
3. m_pdxln               = Part Constraint expression

When you have more partitioning levels, the part constraint expression
can grow large and we end up spending significant amount of time in
translating this expression to DXL format.
For instance, for the following query, relcache translator spends
`18895.444000 ms` in fetching the metadata information on debug build:
```
DROP TABLE IF EXISTS date_parts;
CREATE TABLE DATE_PARTS (id int, year int, month int, day int, region text)
DISTRIBUTED BY (id) PARTITION BY RANGE (year) SUBPARTITION BY LIST
(month) SUBPARTITION TEMPLATE ( SUBPARTITION Q1 VALUES (1, 2, 3),
SUBPARTITION Q2 VALUES (4 ,5 ,6), SUBPARTITION Q3 VALUES (7, 8, 9),
SUBPARTITION Q4 VALUES (10, 11, 12), DEFAULT SUBPARTITION other_months )
SUBPARTITION BY RANGE(day) SUBPARTITION TEMPLATE ( START (1) END (31)
EVERY (10), DEFAULT SUBPARTITION other_days) ( START (2002) END (2012)
EVERY (1), DEFAULT PARTITION outlying_years );

INSERT INTO date_parts SELECT i, EXTRACT(year FROM dt), EXTRACT(month
FROM dt), EXTRACT(day FROM dt), NULL FROM (SELECT i, '2002-01-01'::DATE
+ i * INTERVAL '1 day' day AS dt FROM GENERATE_SERIES(1, 3650) AS i) AS
t;

EXPLAIN SELECT * FROM date_parts WHERE month BETWEEN 1 AND 4;
```

At ORCA end, however, we do not use this part constraints expression
unless the relation has indices built on it.  This is evident from
CTranslatorDXLToExpr.cpp + L565 (`CTranslatorDXLToExpr::PexprLogicalGet`).
Hence we do not need to send the PartConstraint expression from GPDB
Recache Translator since:
A. It will not be consumed if there are no indices
B. The DXL translation is expensive.

This commit fixes the Relcache Translator to send the Partition Constraint
Expression to ORCA only in the cases listed below :

	IsPartTable     Index 	DefaultParts   ShouldSendPartConstraint
	NO		-	-	       -
	YES		YES	YES	       YES
	YES		NO 	NO	       NO
	YES		NO 	YES            YES (but only default levels info)

After fix the metadata fetch and translation time is reduced to `87.828000ms`.

We also need changes in ORCA ParseHandler since we will be sending the part
constraint expression in some cases only.

[Ref #149769559]
Signed-off-by: NOmer Arap <oarap@pivotal.io>
(cherry picked from commit 752e06f6)

Fix Relcache Translator to send CoercePath info

Currently ORCA crashes while executing following query:
```
CREATE TABLE FOO(a integer NOT NULL, b double precision[]);
SELECT b FROM foo
UNION ALL
SELECT ARRAY[90, 90] as Cont_features;
```

In the query, we are appending an integer array (ARRAY[90, 90]) to a double
precision array (foo.b) and hence we need to apply a cast on ARRAY[90, 90] to
generate ARRAY[90, 90]::double precision[].
In gpdb5 there is not direct function available that can cast array of any type
to array of any other type.
So in relcache to dxl translator we look into the array elements and get their type
and try to find a cast function for them.  For this query, source type is 23 i.e.
integer and destination type is 701 i.e. double precision and we try to find if
we have a conversion function for 23 -> 701. Since that is available we send
that function to ORCA as follows:
```
<dxl:MDCast Mdid="3.1007.1.0;1022.1.0"
Name="float8" BinaryCoercible="false" SourceTypeId="0.1007.1.0"
DestinationTypeId="0.1022.1.0" CastFuncId="0.316.1.0"/>
```
Here we are misinforming ORCA by specifying that function with id 316 is available
to convert type 1007 i.e. integer array to 1022 i.e. double precision array.
However Function id 316 is simple int4 to float8 conversion function and it CAN NOT
convert an array of int4 to array of double precision. ORCA generates a plan
using this function but executor crashes while executing this function because
this function can not handle arrays.

This commit fixes this issue by passing a ArrayCoercePath info to ORCA.
In Relcache Translator, The appropriate cast function is retrieved in `gpdb::FCastFunc()`
which relies on `find_coercion_pathway()` to provide the cast function oid given the src
and dest types.

`find_coercion_pathway()` does not just determines the cast function to be used but
also determines the coercion path; however we ignored the coercision path
and generate a simple Cast Metadata Object.

With this commit, we now pass the pathtype to relcache translator and
generate ArrayCoerceCast Metadata object depending on the coercion path.

In ORCA, when the dxl is translated to expression, we check the path type along with
the cast function and generate `CScalarArrayCoerceExpr` if the path type is
array coerce path; otherwise we generate simple `CScalaraCast`.

Please check the corresponding ORCA PR.

Bump ORCA version to 2.40
Signed-off-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>

In ORCA, we donot process interrupts during planning stage, however
if there are elog/ereport (which further calls errfinish) statements to
print additional messages we prematurely exit out the planning stage
without cleaning up the memory pools leading to inconsistent memory pool
state. This results in crashes for the subsequent queries.

This commit fixes the issue by handling interrupts while
printing messages using elog/ereport in ORCA.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

If gpdb and orca is build with debugging enabled, there is an assert in
orca to check if the upper and lower bound are both closed.
`GPOS_ASSERT_IMP(FSingleton(), fLowerClosed && fUpperClosed);`

The histogram that is stored in pg_statistics might lead to have a
singleton buckets as follows: `10, 20, 20, 30, 40` which will lead to have
buckets in this format: `[0,10), [10, 20), [20,20), [20,30), [30,40]`

This will cause assert to fail since [20,20) is a singleton bucket but
its upper bound is open.

With this fix, the generated buckets will look like below:
`[0,10], [10,20), [20,20], (20,30), [30,40]`
Signed-off-by: NShreedhar Hardikar <shardikar@pivotal.io>

Instead we should maintain NDVRemain and NullFreq to do Cardinality
Estimation.

Adding function to check if we need to create stats bucket in DXL

Function `FCreateStatsBucket` returns true if column data type is
not a text/varchar/char/bpchar type.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

- Before building Index object (IMDIndex), we build LogicalIndexes
  via calling `gpdb::Plgidx(oidRel)` in which a partition tables is
  traversed and index  information (such as logicalIndexOid,
  nColumns, indexKeys, indPred, indExprs, indIsUnique, partCons,
  defaultLevels) is captured.
- For Indexes which are available on all the partitions partCons
  and defaultLevels are NULL/empty.
- Later in `CTranslatorRelcacheToDXL::PmdindexPartTable` to build
  Index object, we use the derived LogicalIndexes information and
  populates the array  holding the levels on which default
  partitions exists. But since defaultLevels is NIL in this case,
  pdrgpulDefaultLevels is set to empty i,e `default partitions on levels: {}`
- This causes an issue while trying to build the propagation expression,
  as because of wrong number of default partitions on level we mark the
  scan as partial and tries to construct a test propagation expression
  instead of a const propagation expression.
- This patch fixes the issue by marking the default partitions on
  levels for index equal to the default partitions on levels for the
  part relation if the index exists on all the parts.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

This reverts commit 83a2f870.

With cd2cfa7d (disabling heterogeneous index scans) , now we can make
stronger assumption about the usefulness of partial logical indices and
return only complete logical indices for a partitioned table.

GPDB supports range and list partitions. Range partitions are represented as a set of rules. Each rule defines the boundaries of a part. E.g., a rule might say that a part contains all values between (0, 5], where left bound is 0 exclusive, but the right bound is 5, inclusive. List partitions are defined by a list of values that the part will contain. 

ORCA uses the above rule definition to generate expressions that determine which partitions need to be scanned. These expressions are of the following types:

1. Equality predicate as in PartitionSelectorState->levelEqExpressions: If we have a simple equality on partitioning key (e.g., part_key = 1).

2. General predicate as in PartitionSelectorState->levelExpressions: If we need more complex composition, including non-equality such as part_key > 1.

Note:  We also have residual predicate, which the optimizer currently doesn't use. We are planning to remove this dead code soon.

Prior to  this PR, ORCA was treating both range and list partitions as range partitions. This meant that each list part will be converted to a set of list values and each of these values will become a single point range partition.

E.g., consider the DDL:

```sql
CREATE TABLE DATE_PARTS (id int, year int, month int, day int, region text)
DISTRIBUTED BY (id)
PARTITION BY RANGE (year)
    SUBPARTITION BY LIST (month)
       SUBPARTITION TEMPLATE (
        SUBPARTITION Q1 VALUES (1, 2, 3), 
        SUBPARTITION Q2 VALUES (4 ,5 ,6),
        SUBPARTITION Q3 VALUES (7, 8, 9),
        SUBPARTITION Q4 VALUES (10, 11, 12),
        DEFAULT SUBPARTITION other_months )
( START (2002) END (2012) EVERY (1), 
  DEFAULT PARTITION outlying_years );
```

Here we partition the months as list partition using quarters. So, each of the list part will contain three months. Now consider a query on this table:

```sql
select * from DATE_PARTS where month between 1 and 3;
```

Prior to this ORCA generated plan would consider each value of the Q1 as a separate range part with just one point range. I.e., we will have 3 virtual parts to evaluate for just one Q1: [1], [2], [3]. This approach is inefficient. The problem is further exacerbated when we have multi-level partitioning. Consider the list part of the above example. We have only 4 rules for 4 different quarters, but we will have 12 different virtual rule (aka constraints). For each such constraint, we will then evaluate the entire subtree of partitions.

After this PR, we no longer decompose rules into constraints for list parts and then derive single point virtual range partitions based on those constraints. Rather, the new ORCA changes will use ScalarArrayOp to express selectivity on a list of values. So, the expression for the above SQL will look like 1 <= ANY {month_part} AND 3 >= ANY {month_part}, where month_part will be substituted at runtime with different list of values for each of quarterly partitions. We will end up evaluating that expressions 4 times with the following list of values:

Q1: 1 <= ANY {1,2,3} AND 3 >= ANY {1,2,3}
Q2: 1 <= ANY {4,5,6} AND 3 >= ANY {4,5,6}
...

Compare this to the previous approach, where we will end up evaluating 12 different expressions, each time for a single point value:

First constraint of Q1: 1 <= 1 AND 3 >= 1
Second constraint of Q1: 1 <= 2 AND 3 >= 2
Third constraint of Q1: 1 <= 3 AND 3 >= 3
First constraint of Q2: 1 <= 4 AND 3 >= 4
...

The ScalarArrayOp depends on a new type of expression PartListRuleExpr that can convert a list rule to an array of values. ORCA specific changes can be found here: https://github.com/greenplum-db/gporca/pull/149

Previously gporca translator was only pruning the non-visible system columns from
the table descriptor for non-partition `appendonly` tables or if the
paritition table is marked as `appendonly` at the root level.

If one of the leaf partitions in is marked as `appendonly` but the root
is not, the system columns still appears in the table descriptor.

This commit fixes the issue by checking if the root table has
`appendonly` paritions and pruning system columns if it has.

These warnings are not enabled by default, but you'll see them with -Wall.

With PostgreSQL 8.3, there's a new concept called "operator families".
An operator class is now part of an operator family, which can contain
cross-datatype operators that are "compatible" with each other.

ORCA doesn't know anything about that. This commit updates the
Translator files to refer to OpFamily instead of 'OpClasses'.

ORCA still doesn't take advantage of this, but at least we are using
operator families in operator classes' stead to make indexes work.
Signed-off-by: NHaisheng Yuan <hyuan@pivotal.io>

A bunch of functions and classes that are not used anywhere.