This reverts commit 1060a425.

* fix gpload fail when capital letters in column name in merge mode

add double quotations in column names when create staging tables
omit distribution key
Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>

fix match column condition to resovle primary key conflict when using the gpload
merge mode to import data to the Multi-level partition table
fix fail when special char and capital letters in column names
Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>

This commit updates pygresql from 4.0.0 to 5.1.2, which requires
numerous changes to take advantages of the major result syntax change
that pygresql5 implemented. Of note, cursors or query objects
automatically cast returned values as appropriate python types - list of
ints, for example, instead of a string like "{1,2}". This is the bulk of
the changes.

Updating to pygresql 5.1.2 provides numerous benfits, including the
following:

- CVE-2018-1058 was addressed in pygresql 5.1.1

- We can save notices in the pgdb module, rather than relying on importing
the pg module, thanks to the new "set_notices()"

- pygresql 5 supports python3

- Thanks to a change in the cursor, using a "with" syntax guarentees a
  "commit" on the close of the with block.

This commit is a starting point for additional changes, including
refactoring the dbconn module.

Additionally, since isolation2 uses pygresql, some pl/python scripts
were updated, and isolation2 SQL output is further decoupled from
pygresql. The output of a psql command should be similar enough to
isolation2's pg output that minimal or no modification is needed to
ensure gpdiff can recognize the output.
Co-Authored-by: NTyler Ramer <tramer@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

while gpload is loading data if the configure file contains "error_table" and doesn't contain "preload", an error of no attribute "staging_table" or "fast_path"  occurs.

External tables now use relkind='f', like all foreign tables. They have
an entry in pg_foreign_table, as if they belonged to a special foreign
server called "exttable_server". That foreign server gets special
treatment in the planner and executor, so that we still plan and execute
it the same as before.

* ALTER / DROP EXTERNAL TABLE is now mapped to ALTER / DROP FOREIGN TABLE.
  There is no "OCLASS_EXTTABLE" anymore. This leaks through to the user
  in error messages, e.g:

    postgres=# drop external table boo;
    ERROR:  foreign table "boo" does not exist

  and to the command tag on success:

    postgres=# drop external table boo;
    DROP FOREIGN TABLE

* psql \d now prints external tables as Foreign Tables.

Next steps:
* Use the foreign table API routines instead of special casing
  "exttable_server" everywhere.

* Get rid of the pg_exttable table, and store the all the options in
  pg_foreign_table.ftoptions instead.

* Get rid of the extra fields in pg_authid to store permissions to
  create different kinds of external tables. Store them as ACLs in
  pg_foreign_server.

* Ship modified python module subprocess32 again

subprocess32 is preferred over subprocess according to python documentation.
In addition we long ago modified the code to use vfork() against fork() to
avoid some "Cannot allocate memory" kind of error (false alarm though - memory
is actually sufficient) on gpdb product environment that is usually with memory
overcommit disabled.  And we compiled and shipped it also but later it was just
compiled but not shipped somehow due to makefile change (maybe a regression).
Let's ship it again.

* Replace subprocess with our own subprocess32 in python code.

The installation target in gpMgmt/ blindly copied Perl programs and
modules in order for the gpload test to be able to run gpdiff from
PATH. This causes non-production code to be erroneously installed
and required an elaborate dance to remove before packaging.

This removes the installation and instead installs the required
Perl code from regress as symlinks just like how other test suites
are doing it.

A careful observer might note that TEST_REMOTE.py still call the
Perl programs from PATH, and that's Ok (for now) as it's only to
be invoked via the clients package CI and that CI performs a copy
step of gpdiff much like how this patch does a symlink. There are
probably parts of that which can be cleaned up and simplified but
for now we are focusing on the core server parts.

Removing the non-defunct NON_PRODUCTION_FILES step remains as a
task after this.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NAmil Khanzada <akhanzada@pivotal.io>

In the past several weeks, gpload test case 22 failed two times.
It may report: Fatal Python error: GC object already tracked.
We have reproduced this issue locally and in dev pipeline for 3
days, but it can't be reproduced.
So we decide to disable it now, in order to not blocking other PRs.

While == None works for comparison, it's a wasteful operation as it
performs type conversion and expansion. Instead move to using the
"is" operator which is the documented best practice for Python code.

Reviewed-by: Jacob Champion

* Add external table encoding option as condition of finding reusable table

Get database default encoding if ENCODING is not set in config file.
Find encoding code by encoding string and then add encoding code as one of
conditions of finding reusable table.

- The results of fast_match SQL don't include shema name, so we need
add shema name to extSchemaTable for fast_match
- Remove locationStr which is unused.

- add fast_match option in gpload config file. If both reuse_tables
and fast_match are true, gpload will try fast match external
table(without checking columns). If reuse_tables is false and
fast_match is true, it will print warning message.

"False is not true" is not particularly helpful when looking at the CI.
Dump the diff file as part of the assertion message.

Diff the actual against the expected, instead of the other way around,
and use three lines of unified context.

- if the data file contains "\N" as the delimiter, it would not be
recognized properly by gpload
- root cause: gpload replace the quote in nullas option as well as
replace '\' as '\\'
- solution: add quote_no_slash function to handle nullas option

to definite the reusing external table by themselves instead of
searching in gpload, which may have bad performance if there are
too many external tables.

- Fix gpload.py, add schema prefix to every external table name
  when EXTERNAL.SCHEMA is set
- Add new test cases

gpload ERROR_TABLE configuration is forbidden in GPDB5 and use
LOG_ERRORS instead. That's why Informatica 9.x connector can't
load data into GPDB5.

So we accept ERROR_TABLE, if ERROR_TABLE is set, we will not
create it as before, but set LOG_ERRORS and REUSE_TABLE to true
instead.

gpload error count is incorrect when more than one segment has format
error, for the cmdtime is different, and only errors with the newest
cmdtime is counted.

So we add startTime which will be used for counting all the errors
occured during the same gpload operation.

0, default database doesn't exist in some env.
1, reports good even the cases failed.
2, init_file ignores too many lines.

This commit fixes these.

Signed-off-by: NMarbin Tan <mtan@pivotal.io>

1. fix an error if the quote and escape options are not specified in the
CSV format.

2. add codes for delimiter, escape and quote options to process a
ASCII, unicode-encoded or escaped single character in order to find
the correct table to reuse when load external data.

The old cdbfast test suite had these, good to move to ICW.