optarg is a global and is not required to be passed to
process_greenplum_option. Also, for bool parameters pg_strdup must not
be called else it will fail with cannot duplicate null pointer (internal error)

- creates GreenplumClusterInfo data structure.

We need to populate DbInfo during checks and normal upgrade with the location of a tablespace.

The upgrade workflow for Greenplum works like this:

init-new-cluster -> upgrade-dispatcher -> copy-from-master ->
upgrade-segments

pg_upgrade is only responsible for the upgrade-dispatcher and
upgrade-segments steps, but it depends on the ordering of these steps.

This commit adds support for tablespaces during the copy-from-master step,
as well as adding logic inside of pg_upgrade necessary to map files
from GPDB5's filespaces to GPDB6's tablespaces. Described here:

During the upgrade-dispatcher step, gpupgrade can run pg_upgrade
--mode=dispatcher which will generate a file (old_tablespaces.txt)
into the current directory containing the tablespace information
inside of the GPDB6 cluster. This file can then be used to perform the
copy-from-master step and the upgrade-segments step.

This is a backport of the below commit from upstream PostgreSQL,
which was originally written for Greenplum and submitted as an
upstream-first feature. The commit didn't cherrypick as upstream
has moved pg_upgrade to src/bin and Greenplum has yet to merge
that.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

  commit 2d34ad84
  Author: Tom Lane <tgl@sss.pgh.pa.us>
  Date:   Sat Dec 1 15:45:11 2018 -0500

    Add a --socketdir option to pg_upgrade.

    This allows control of the directory in which the postmaster sockets
    are created for the temporary postmasters started by pg_upgrade.
    The default location remains the current working directory, which is
    typically fine, but if it is deeply nested then its pathname might
    be too long to be a socket name.

    In passing, clean up some messiness in pg_upgrade's option handling,
    particularly the confusing and undocumented way that configuration-only
    datadirs were handled.  And fix check_required_directory's substantially
    under-baked cleanup of directory pathnames.

    Daniel Gustafsson, reviewed by Hironobu Suzuki, some code cleanup by me

    Discussion: https://postgr.es/m/E72DD5C3-2268-48A5-A907-ED4B34BEC223@yesql.se

The --progress, --add-checksum, and --remove-checksum options weren't
being recognized because their option.vals were set to the characters
'2', '3', and '4' (integer values 50, 51, and 52, respectively), while
the option handling was comparing to the literal values 2, 3, and 4.
Switch back to integers.

Previously each QE would dump/restore the schema before upgrading
the datafiles, which consumes a lot of time on large databases.
Instead, use the datadir which was created from the dump/restore
on the QD for the QEs as well and bootstrap the segment upgrade
to jump straight to copying files instead.

This is a first stab at implementing this model of distribution,
follow-up commits will be required to finalize the patch.

The previous commit backported the PostgreSQL 9.3 pg_upgrade code
base and overwrote the hacks to allow for upgrading a Greenplum
cluster. This commit brings back the patchsets that were lost, and
adapts them to the new pg_upgrade code as well as mildly refactors
them to take advantage of new opportunities.

This overwrites the Greenplum pg_upgrade version with the upstream
9.3.22 sources. No attempt is made to make this compile or at all
be even remotely useable. Follow-up commits will re-introduce all
functionality which was lost.

When upgrading the cluster, allow the addition or removal of data
checksums on the data pages as they are copied across from the
old cluster. This allows for adding checksums without requiring a
full dump/restore cycle.

The test_gpdb.sh script is extended to support adding/removal of
checksums. In the process, also extend to remove gpdemo config
files generated when setting up the new cluster, and put them in
gitignore as well to keep git status happy during testing.

This also fixes a previously incorrect check for checksum version
and aligns variable names better with upstream.

Instead of meticulously recording the OIDs of each object in the pg_dump
output, dump and load all OIDs as a separate steps in pg_upgrade.

We now only preserve OIDs of types, relations and schemas from the old
cluster. Other objects are assigned new OIDs as part of the restore.
To ensure the OIDs are consistent between the QD and QEs, we dump the
(new) OIDs of all objects to a file, after upgrading the QD node, and use
those OIDs when restoring the QE nodes. We were already using a similar
mechanism for new array types, but we now do that for all objects.

In order for higher-level wrappers around pg_upgrade to know what
pg_upgrade does, this adds a trivial file based queue which reports
information on the operations performed. When started with -X, or
--progress, pg_upgrade will write progress as a set of operations
to a logfile. At intervals of either X number of operations, or Y
number of seconds, the file will be released and a new created such
that the released file can be consumed by a wrapper program. The
files will be named <sequence>.inprogress until released when they
are renamed to <sequence>.done. The sequence is gapless. Once the
file is renamed to <sequence>.done, pg_upgrade will never touch it
again and it can at that point be removed.

The format of the progress file is a simple semicolon delimited
line-based protocol:

  <timestamp>;<cluster>;<operation>;<message>;

timestamp is the number of microseconds since epoch; cluster defines
either "new", "old" or "none"; operation defines what actually did
happen and message contains the user defined message per report. The
timestamp is monotonically increasing for each operation such that
reading files in order is less important, operations can still be
individually sorted.

This commit includes a small set of progress report operations, while
not likely to be the final set it's enough to show how more can be
added.

Discussion:
  https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/CVgK36oU5Oo

This commit just brings in the upstream code for pg_upgrade and
pg_upgrade_support as is, with no changes. Subsequent work will change
it to work for Greenplum. The goal is to make it usable for upgrading
from Greenplum 4.3 to 5.0.

pg_upgrade_support is a crucial part of pg_upgrade, but lives in a
separate directory to make the build easier.

Also fix getErrorText() to return the right error string on failure.
This behavior now matches that of other operating systems.

Report by Noah Misch

Backpatch through 9.1

This includes removing tabs after periods in C comments, which was
applied to back branches, so this change should not effect backpatching.

The files were already included by pg_upgrade.h.

We used to have externs for getopt() and its API variables scattered
all over the place.  Now that we find we're going to need to tweak the
variable declarations for Cygwin, it seems like a good idea to have
just one place to tweak.

In this commit, the variables are declared "#ifndef HAVE_GETOPT_H".
That may or may not work everywhere, but we'll soon find out.

Andres Freund

Per report from Jackie Chang

Update all files in head, and files COPYRIGHT and legal.sgml in all back
branches.

Setting default_transaction_read_only=true could prevent pg_upgrade from
completing, so prepend default_transaction_read_only=false to
PGOPTIONS.

This allows decorating pg_fatal() with noreturn compiler hints, leading
to better diagnostics.
Reviewed-by: NMarko Tiikkaja <marko@joh.to>

If -U (user) is specified, pass the username into the created analyze
script.
Per request from Ray Stell

-h (help) is not needed;  pg_upgrade already supports --help and -?,
which is consistent with other tools.

Previous code had old/new prefixes on option values, e.g.
--old-datadir=OLDDATADIR.  Remove them, for simplicity;  now:
--old-datadir=DATADIR.  Also update docs to do the same.

Change -u (user) option to -U, for consistency with other tools like
pg_dump and psql.  Also expand --user to --username, again for
consistency.
BACKWARD INCOMPATIBILITY

This is the first run of the Perl-based pgindent script.  Also update
pgindent instructions.

libpgcommon is a new static library to allow sharing code among the
various frontend programs and backend; this lets us eliminate duplicate
implementations of common routines.  We avoid libpgport, because that's
intended as a place for porting issues; per discussion, it seems better
to keep them separate.

The first use case, and the only implemented by this patch, is pg_malloc
and friends, which many frontend programs were already using.

At the same time, we can use this to provide palloc emulation functions
for the frontend; this way, some palloc-using files in the backend can
also be used by the frontend cleanly.  To do this, we change palloc() in
the backend to be a function instead of a macro on top of
MemoryContextAlloc().  This was previously believed to cause loss of
performance, but this implementation has been tweaked by Tom and Andres
so that on modern compilers it provides a slight improvement over the
previous one.

This lets us clean up some places that were already with
localized hacks.

Most of the pg_malloc/palloc changes in this patch were authored by
Andres Freund. Zoltán Böszörményi also independently provided a form of
that.  libpgcommon infrastructure was authored by Álvaro.

Fully update git head, and update back branches in ./COPYRIGHT and
legal.sgml files.

Add pg_upgrade --jobs, which allows parallel dump/restore of databases,
which improves performance.

socket location.  Also, prevent putting the socket in the current
directory for pre-9.1 servers in live check and non-live check mode,
because pre-9.1 pg_ctl -w can't handle it.

Backpatch to 9.2.

When starting either an old or new postmaster, force it to place its Unix
socket in the current directory.  This makes it even harder for accidental
connections to occur during pg_upgrade, and also works around some
scenarios where the default socket location isn't usable.  (For example,
if the default location is something other than "/tmp", it might not exist
during "make check".)

When checking an already-running old postmaster, find out its actual socket
directory location from postmaster.pid, if possible.  This dodges problems
with an old postmaster having a configured location different from the
default built into pg_upgrade's libpq.  We can't find that out if the old
postmaster is pre-9.1, so also document how to cope with such scenarios
manually.

In support of this, centralize handling of the connection-related command
line options passed to pg_upgrade's subsidiary programs, such as pg_dump.
This should make future changes easier.

Bruce Momjian and Tom Lane