1. add global error message variable to report to gpdb console.
2. add try/catch block around fetchData() call to catch error message.
3. refactor shared error mechanism between ChunkBuffers.
4. fix unit tests.

Also fixed a memleak thanks to Kuien.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>
Signed-off-by: NKuien Liu <kliu@pivotal.io>

Using new GPReader, and make it clean.
Signed-off-by: NKuien Liu <kliu@pivotal.io>

1. add s3common_reader to detect the format of file
   if the file is compressed call decompressReader
   otherwise call S3KeyReader directly.
2. add checkCompressionType() into S3Interface.
3. fix a bug if the file size is less than S3_MAGIC_BYTES_NUM
   and add a unit test
4. fix a bug if chunkSize is zero.
5. add unit tests
6. rename uncompress* to decompress*
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

Two major problems this commit fixes:
credential should be copied to keep the data.
keyReader must be reset before every new file downloading.
Signed-off-by: NKuien Liu <kliu@pivotal.io>

The clang in mac does not check the -std=c++98 flag strictly.
Hence when compiling on the other platform like redhat 7 (using gcc),
the source code may not be compiled correctly. The compiling errors have
been fixed and the tests have been passed on redhat7 using c++98.

1. std::vector<T>::shrink_to_fit is added in C++11.
2. need to include <stdexcept> to use std::runtime_error.
3. In C++98, if a class has reference member, then it can't be copy assigned by default,
   we need to implement operator= explicitly.
4. stoi is added in C++11.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

1. write the GPReader class to connect all readers

2. add unit tests on Open, Construct, Close, Read,
   multi-thread Read, and failed response handling

3. and a few cleanup
  3.1. clean source codes in s3ext
  3.2. refactor debug switch method
  3.3. move mockRESTfulService to mock_classes.h
  3.4. reset QueryCancelPending to FALSE after tests
Signed-off-by: NKuien Liu <kliu@pivotal.io>

Put Makefile in each subdirectory, instead of using one Makefile at top
directory to do all the things, and extract the common parts.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

== race condition

Without this fix, ChunkBuffer::read() might be waiting for the signal
from ChunkBuffer::fill(), which has already been terminated in some
certain cases.

== exception bug

When a FAIL response from KeyReader is given, an exception
will be throw from S3Service::fetchData(), and this exception
will crash the downloader threads and process (core dumped).

We replace exception with error return, and it will be handled
by the caller.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NKuien Liu <kliu@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

Run all cases:
    $ make test

Run a subset cases using pattern match, run all tests in S3KeyReaderTest suite:
    $ make test gtest_filter=S3KeyReaderTest.*

Run a subset cases using pattern match, run all MTRead tests:
    $ make test gtest_filter=S3KeyReaderTest.MTReadWith*

Also fix s3conf test cases' issue exposed by this change.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

1. add gpcommon.h to keep 'GpId' data structures;
2. remove s3reader.h because s3key_reader is ready;
3. move two pthread related functions to gpreader.h
   because they are invoked by gpreader only.
4. move XMLParserCallback and XMLInfo to s3uploader.cpp
   because they are deprecated already for readers.
Signed-off-by: NKuien Liu <kliu@pivotal.io>

Also retire DEBUG_S3_CURL, make it configurable.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

Separate key downloading and decompressing functions, make it more
flexible and robust.
Signed-off-by: NAdam Lee <ali@pivotal.io>
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

* change code style to avoid AllowShortFunctionsOnASingleLine

* add a few cases for uncompression

* gzip reader phase 2: read() with the buffer <= size of chuck

* gzip reader phase 3: read() with the size of chuck < internal read buffer

* gzip reader phase 4: read() with the buffer >> size of chuck
Signed-off-by: NKuien Liu <kliu@pivotal.io>

s3interface is to handle all s3 restful api wrapper, like listBucket()
and fetchData(), s3restful is to handle HTTP GET, and PUT in future.

Also introducing google mock for unit tests.

refactor, optimize and rename them to gpreader and s3reader.

Currently, gpdemo does not build mirrors for Darwin and Sparc. This was a
restriction from the early 2010s and should not be an issue today with
modern systems.

Saving the error message into the static buffer can lead to overflow
since the string copy isn't bounded. Move to using a bounded copy and
also indicate truncated messages by ending them with ".."

Considering that this doesn't seem to have happened indicates that it's
not worth dynamically extending/shrinking the buffer to cope with the
input, simply ensuring to not overflow is probably enough.

Fixed the function return value declarations to be consistent while
in there fixing another thing. There are more style fixes that could
be applied but one thing at a time.

Spotted when working on related code.

If we are unable to unlink the temporary files in the shareinput
cleanup it seems quite aggressive to insist_log() and forcibly
exit. This cleanup can be called as part of transaction abort.

When bfz_close() is called in the codepath during the abortion of a
transaction we must avoid throwing even more errors unless the
situation calls for it. For bfz_close() it's fine to lower the
ereport level to WARNING in this case. Longer term we should move
this, and other, codepaths away from calling unlink() directly and
instead use the API provided but this closes a current issue in
ICG so better to close this immediately and refactor all callsites
when having a clean ICG.

The LOG_GANG_DEBUG statements that include members in the Gang struct
should only run after we've validated that the Gang struct pointers
aren't NULL to avoid segfaulting.

Found this README while reading some code and rather than fixing
the incorrect path specification it seemed equally useful to just
remove it since it carries very little information.

When create gang failed and all segment DB are alive,
we could retry when any of the following conditions is met:
1) This is the writer gang.
2) This is the first reader gang and the writer gang is healthy.
3) All failed connections are due to segment recovery.

Signed-off-by: NShreedhar Hardikar <shardikar@gmail.com>

This closes #836
Signed-off-by: NMarc Spehlmann <cramja@gmail.com>

reorder header file

This closes #858