1. 17 11月, 2020 2 次提交
    • J
      Fix stale references to pipeline task files · 15f1e992
      Jesse Zhang 提交于
      The build's been broken since Thursday, and the reason is not unobvious:
      Following greenplum-db/greenplum-database-release@4daf1694421cc46e , we
      got a dangling reference to the old underscore named yaml file instead
      of the dash version. This patch fixes that.
      15f1e992
    • J
      Fix up CI unit_tests_gporca job reference, take two · 722da0a2
      Jesse Zhang 提交于
      Commit de7eaa9d erroneously added as a passing condition the new
      job "unit_tests_gporca" to the compiled binary bin_gpdb_*. This led to a
      pipeline validation failure because the orca unit test job does not
      really do anything to the compiled binary. Commit 1bc0a548
      attempted to fix that by doubling down on pretending the unit test job
      *does* depend on the binary as an input. But it changed only the
      generated pipeline YAML, not the template.
      
      This patch fixes the pipeline validation with the right fix: we only
      need the unit tests job as a passing condition for the GPDB source code
      resource, not the binary.
      722da0a2
  2. 14 11月, 2020 2 次提交
    • J
      Fix multiple definition linker error in unit tests · 108504ff
      Jesse Zhang 提交于
      In the same spirit as ee7eb0e8, this removes the extra
      definition in postgres_test.c.
      
      This is uncovered by building with GCC 10, where -fno-common is the new
      default [1][2] (vis a vis -fcommon). I could also reproduce this by
      turning on "-fno-common" in older releases of GCC and Clang.
      
      Backpatch to 6X_STABLE.
      108504ff
    • A
      gpstart logs commands' stderr · 717f0c47
      Adam Lee 提交于
      I have seen too many "[CRITICAL]:-gpstart failed. (Reason='')
      exiting..." errors, and there was nothing in the log. The reason could
      be "SSH PATH", "Python modules" or some other issues.
      
      Log the stderr to save debugging efforts.
      717f0c47
  3. 13 11月, 2020 9 次提交
    • A
      Fix the script to check flaky test schedules · 20d7f010
      Asim R P 提交于
      The regex used in the script was too strict.  Relax the regex so as to
      match 'select distinct gp_inject_fault(...);'.  The script may be
      invoked against a directory, such as src/test/fsync, which may not
      contain any schedule files.  This no longer results in an error.  When
      invoked against isolation directory, it should scan tests under
      'specs' subdir.  Not all of input, sql and specs subdirs exist always,
      so add a '-s' option to grep to mask errors reporting non-existance.
      Co-authored-by: NHao Wu <gfphoenix78@gmail.com>
      20d7f010
    • H
      Avoid using faultinjector extension when possible · ba7dc5a4
      Hao Wu 提交于
      Some tests merely need to create an extension, let's use an extension
      that's always enabled instead of gp_inject_fault, which can now be
      configured with --enable-faultinjector configure option.
      
      This change is cherry-picked from GitHub PR #9029.
      ba7dc5a4
    • A
      Exclude tests that inject faults when faultinjector is disabled · 8b897b12
      Asim R P 提交于
      If faultinjector is not enabled during configuration, this patch
      identifies tests that use faultinjector and excludes them from running.
      A new option "--exclude-file" is added to pg_regress.  It specifies a
      file having names of tests, one per line, to be excluded from running.
      
      The global makefile is changed so that the exclude file creation and
      adding the --exclude-file option to pg_regress (and
      pg_isolation*regress) command line is taken care of automatically.
      However, one must perform these actions manually if pg_regress needs to
      be directly invoked from command line (instead of a make target).
      Running tests with faultinjector disabled is not a common developer
      workflow, so this should be acceptable.
      
      The advantage of changing global makefile is that all test frameworks
      that are based on pg_regress, such as isolation and also contrib modules
      need no change to their makefiles.  For example, if one creates a
      isolation test with faultinjector, simply add it to the right schedule,
      it will automatically be excluded from running when faultinjector is not
      enabled.
      8b897b12
    • H
      Make the macro of FAULT_INJECTOR configurable · 4111340a
      Hao Wu 提交于
      Definition FAULT_INJECTOR is hardcoded in a header(pg_config_manual.h)
      file.  Fault injector is useful, but it may introduce some issues in
      production stage, like runtime cost, security problems.  It's better
      to enable this feature in development and disable it in release.  The
      --enable-debug-extensions configure option is used to control this
      feature.
      4111340a
    • A
      Extract common parts from isolation2 makefile into global makefile · 085c3b25
      Asim R P 提交于
      It makes isolation2 framework more consistent with pg_regress and
      isolation in terms of how it is invoked from the build system.
      085c3b25
    • J
      Update Instructions for formatting in-flight PR's · ce9f5253
      Jesse Zhang 提交于
      We included some rough steps in 57b744c1. This patch updates
      those with a higher level of details so that it's hopefully easier to
      follow in a step-by-step fashion.
      ce9f5253
    • D
      Fix overflow of shmCommittedGxactArray on standby (#11071) · afcf30be
      dreamedcheng 提交于
      Previously, standby will replay checkpoint XLOG record's DTX info
      in function XLogProcessCheckpointRecord. However, is some certain cases,
      it will cause anomaly: When a DTX has flushed FORGET COMMITTED XLOG into
      disk, but didn't change its own state to DTX_STATE_INSERTED_FORGET_COMMITTED.
      If at this very moment, checkpoint process is calculating DTX info, it
      will include the DTX into its XLOG record. So when standby replaying this
      checkpoint XLOG record from master, it will add an already forgotten GID to
      shmCommittedGxactArray again, which may cause the overflow of shmCommittedGxactArray.
      
      Since DTX info stored in checkpoint XLOG record has been populated earlier in
      ReadCheckpointRecord(), there is no need to call XLogProcessCheckpointRecord()
      again during recovery.
      Co-authored-by: Nwuchengwen <wcw190496@alibaba-inc.com>
      Co-authored-by: NDenis Smirnov <sd@arenadata.io>
      afcf30be
    • P
      Remove gpdb specific dtx recovery code for utility mode. (#11117) · 71043c5b
      Paul Guo 提交于
      Those functionality has been handled in checkpoint during recovery in utility
      mode.  The code is not needed anymore.
      Reviewed-by: NAsim R P <pasim@vmware.com>
      71043c5b
    • C
      Fix handling of empty constraint array in Orca (#11139) · f97a3e23
      Chris Hajas 提交于
      During preprocessing, Orca simplifies the query by merging/deduplicating
      constraints. However, Orca did not consider the case where the passed in
      constraint was a column compared with an empty array. This assumption
      caused Orca to crash when dealing with predicates such as `a = ANY('{}')`.
      
      Instead, we now explicitly return an empty constraint when dealing with an ANY('{}')
      clause. For ALL('{}'), we won't process the constraint as there's no
      simplification to do.
      f97a3e23
  4. 12 11月, 2020 5 次提交
    • D
      12296f0f
    • D
      Fix up CI unit_tests_gporca job reference · 1bc0a548
      David Kimura 提交于
      This is part of commit de7eaa9d
      1bc0a548
    • S
      Extract test_orca as a separate job outside of compile step (#10819) · de7eaa9d
      Shreedhar Hardikar 提交于
      This is done to speed up development iterations by reducing the time the
      of the "compile" job in the following pipelines, and starting ICW jobs
      sooner. Runs the unit tests in parallel to ICW jobs.
      
      Pipelines modified:
      - gporca-dev
      - master build pipeline
      - master build pipeline without asserts
      - PR pipeline
      Co-authored-by: NShreedhar Hardikar <hardikar@cs.wisc.edu>
      Co-authored-by: NChris Hajas <chajas@vmware.com>
      de7eaa9d
    • D
      Add Orca index only scan cost formula (#11106) · bc7ab7bd
      David Kimura 提交于
      Planner costs index-only-scan relative to index-scan. In the case of
      index-only-scan, `RelOptInfo->allvisfrac` can be used to reduce the cost
      when the visimap contains all-visible blocks. Thus, if index-only-scan
      is possible, it will be favored.
      
      In this commit, Orca also costs index-only-scan relative to index-scan
      and scales the cost based on percentage of all-visible blocks in the
      visimap. This is done by storing `pg_class.relallvisible` stats in
      CDXLRelStats which is accessible during costing. In DXL this is added to
      RelationStatistics:
      ```
      <dxl:RelationStatistics ... RelPages="XX" RelAllVisible="YY"/>
      ```
      
      Also in this commit, we update gpsd and minirepro to collect stats for
      relallvisible. In doing this we also udpated the tools to use python3
      sytnax.
      
      And finally, it also adds a new option "index_only_scan_tests" in
      cal_bitmap_test to callibrate index only scan cost/execution.
      bc7ab7bd
    • D
      Allow Orca to choose index only scan on non-distribution columns (#11129) · fb92ac3d
      David Kimura 提交于
      In order to produce an index only scan plan, all columns referenced in the
      query must be covered by an index. This check is made in the 
      `CXformIndexGet2IndexOnlyScan::Transform`.
      
      Issue is that distribution columns were being marked as used even when they
      were not directly referenced. This is because they may still appear in
      distribution specs of derived physical properties. However, this also meant
      that Orca would never consider an index only scan if the distribution column
      was not covered by the index.
      fb92ac3d
  5. 10 11月, 2020 4 次提交
  6. 09 11月, 2020 11 次提交
    • H
      Revert "ic-proxy: refresh peers on demand" · 7f558913
      Hubert Zhang 提交于
      This reverts commit 9265ea6a.
      7f558913
    • P
      Move popen_with_stderr to separate file (#11121) · aef99680
      Peifeng Qiu 提交于
      popen_with_stderr() was introduced in execute external table, to
      extract stderr from external program. It's later adopted by COPY
      command too, replacing OpenPipeStream() from upstream.
      
      Currently popen_with_stderr() is used both by external table and
      COPY command. As we are going to move external table related code
      to extension, we need to move this function to a place where both
      have access.
      aef99680
    • G
      Mashes error severity DEBUG1-DEBUG5 into DEBUG · aa0c3735
      Gang Xiong 提交于
      aa0c3735
    • G
      Remove a fixme of pg12 merge · bb320766
      Gang Xiong 提交于
      function 'pg_logdir_ls' returns the creation time of log files by parsing the
      file name, it's different from 'pg_ls_dir_files', we should keep it if we still
      consider 'pg_logdir_ls' useful.
      bb320766
    • G
      Adopt the upstream's default value of 'log_line_prefix' · bd3e9fe9
      Gang Xiong 提交于
      bd3e9fe9
    • X
      compatible gpload (#11103) · f7174966
      xiaoxiao 提交于
      * refactor gpload test file TEST.py
      
      1. migrate gpload test to pytest
      2. new function to form config file through yaml package and make it more reasonable
      3. add a case to cover gpload update_condition arggument
      
      * migrate gpload and TEST.py to python3.6
      new test case 43 to test gpload behavior when column name has capital letters and without data type
      change some ans file since psql react different
      
      * change sql to find reuseable external table to make gpload compatible in gp7 and gp6
      better TEST.py to write config file with ruamel.yaml moudle
      Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>
      f7174966
    • N
      ic-proxy: refresh peers on demand · 9265ea6a
      Ning Yu 提交于
      The user can adjust the ic-proxy peer addresses at runtime and reload by
      sending SIGHUP, if an address is modified or removed, the corresponding
      peer connection must be closed or reestablished.  The same to the peer
      listener, if the listener port is changed, then must re-setup the
      listener.
      9265ea6a
    • N
      ic-proxy: classify peer addresses · 854c4b84
      Ning Yu 提交于
      The peer addresses are specified with the GUC
      gp_interconnect_proxy_addresses, it can be reloaded on SIGHUP, we used
      to only care about the newly added ones, however it is also possible for
      the user to modify them, or even remove some of them.
      
      So now we add the logic to classify the addresses after parsing the GUC,
      we can tell whether an address is added, removed, or modified.
      
      The handling of the classified addresses will be done in the next
      commit.
      854c4b84
    • N
      ic-proxy: optimize looking up of my addr · 40facdb1
      Ning Yu 提交于
      We used to scan the whole addr list to find my addr, now we record it
      directly when parsing the addresses.
      40facdb1
    • N
      ic-proxy: rename ICProxyAddr.addr to sockaddr · 2c2ca626
      Ning Yu 提交于
      A ICProxyAddr variable is usually named as "addr", so the attribute is
      referred as "addr->addr", it's confusing and sometimes ambiguous.
      
      So renamed the attribute to "sockaddr", the function
      ic_proxy_extract_addr() is also renamed to ic_proxy_extract_sockaddr().
      2c2ca626
    • P
      Separate external table options out of copy option check (#11104) · e898755e
      Peifeng Qiu 提交于
      ProcessCopyOptions checks the option list of COPY command. It's also
      called by external table when text/csv format is used. It's better
      not to mix external table specific options here and check them
      separately.
      
      Checking custom protocol here is not necessary because it's checked
      when parsing location urls in GenerateExtTableEntryOptions anyway.
      e898755e
  7. 07 11月, 2020 3 次提交
  8. 05 11月, 2020 4 次提交
    • X
      Correctly seek to the end of buffile that contains multiple physical files · 5028054e
      xiong-gang 提交于
      When stash a 'VisimapDelete' to the buffile, we must seek to end of the last
      physical file if the buffile contains multiple files. This commit cherry-pick
      part of the commit from upstream:
      
      commit 808e13b282efa7e7ac7b78e886aca5684f4bccd3
      Author: Amit Kapila <akapila@postgresql.org>
      Date:   Wed Aug 26 07:36:43 2020 +0530
      
          Extend the BufFile interface.
      
          Allow BufFile to support temporary files that can be used by the single
          backend when the corresponding files need to be survived across the
          transaction and need to be opened and closed multiple times. Such files
          need to be created as a member of a SharedFileSet.
      
          Additionally, this commit implements the interface for BufFileTruncate to
          allow files to be truncated up to a particular offset and extends the
          BufFileSeek API to support the SEEK_END case. This also adds an option to
          provide a mode while opening the shared BufFiles instead of always opening
          in read-only mode.
      
          These enhancements in BufFile interface are required for the upcoming
          patch to allow the replication apply worker, to handle streamed
          in-progress transactions.
      
          Author: Dilip Kumar, Amit Kapila
          Reviewed-by: Amit Kapila
          Tested-by: Neha Sharma
          Discussion: https://postgr.es/m/688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com
      5028054e
    • H
      Experimental cost model update (port from 6X) (#11115) · 9363718d
      Hans Zeller 提交于
      This is a cherry-pick of the change from PR https://github.com/greenplum-db/gporca/pull/607
      
      Avoid costing change for IN predicates on btree indexes
      
      Commit e5f1716 changed the way we handle IN predicates on indexes, it
      now uses a more efficient array comparison instead of treating it like
      an OR predicate. A side effect is that the cost function,
      CCostModelGPDB::CostBitmapTableScan, now goes through a different code
      path, using the "small NDV" or "large NDV" costing method. This produces
      very high cost estimates when the NDV increases beyond 2, so we
      basically never choose an index for these cases, although a btree
      index used in a bitmap scan isn't very sensitive to the NDV.
      
      To avoid this, we go back to the old formula we used before commit e5f1716.
      The fix is restricted to IN predicates on btree indexes, used in a bitmap
      scan.
      
      Add an MDP for a larger IN list, using a btree index on an AO table
      
      Misc. changes to the calibration test program
      
      - Added tests for btree indexes (btree_scan_tests).
      - Changed data distribution so that all column values range from 1...n.
      - Parameter values for test queries are now proportional to selectivity,
        a parameter value of 0 produces a selectivity of 0%.
      - Changed the logic to fake statistics somewhat, hopefully this will
        lead to more precise estimates. Incorporated the changes to the
        data distribution with no more 0 values. Added fake stats for
        unique columns.
      - Headers of tests now use semicolons to separate parts, to give
        a nicer output when pasting into Google Docs.
      - Some formatting changes.
      - Log fallbacks.
      - When using existing tables, the program now determines the table
        structure (heap or append-only) and the row count.
      - Split off two very slow tests into separate test units. These are
        not included when running "all" tests, they have to be run
        explicitly.
      - Add btree join tests, rename "bitmap_join_tests" to "index_join_tests"
        and run both bitmap and btree joins
      - Update min and max parameter values to cover a range that includes
        or at least is closer to the cross-over between index and table scan
      - Remove the "high NDV" tests, since the ranges in the general test
        now include both low and high NDV cases (<= and > 200)
      - Print out selectivity of each query, if available
      - Suppress standard deviation output when we execute queries only once
      - Set search path when connecting
      - Decrease the parameter range when running bitmap scan tests on
        heap tables
      - Run btree scan tests only on AO tables, they are not designed
        for testing index scans
      
      Updates to the experimental cost model, new calibration
      
      1. Simplify some of the formulas, the calibration process seemed to justify
         that. We might have to revisit if problems come up. Changes:
         - Rewrite some of the formulas so the costs per row and costs per byte
           are more easy to see
         - Make the cost for the width directly proportional
         - Unify the formula for scans and joins, use the same per-byte costs
           and make NDV-dependent costs proportional to num_rebinds * dNDV,
           except for the logic in item 3.
      
         That makes the cost for the new experimental cost model a simple linear formula:
      
         num_rebinds * ( rows * c1 + rows * width * c2 + ndv * c3 + bitmap_union_cost + c4 ) + c5
      
         We have 5 constants, c1 ... c5:
      
         c1: cost per row (rows on one segment)
         c2: cost per byte
         c3: cost per distinct value (total NDV on all segments)
         c4: cost per rebind
         c5: initialization cost
         bitmap_union_cost: see item 3 below
      
      2. Recalibrate some of the cost parameters, using the updated calibration
         program src/backend/gporca/scripts/cal_bitmap_test.py
      
      3. Add a cost penalty for bitmap index scans on heap tables. The added
         cost takes the form bitmap_union_cost = <base table rows> * (NDV-1) * c6.
      
         The reason for this is, as others have pointed out, that heap tables
         lead to much larger bit vectors, since their CTIDs are more spaced out
         than those of AO tables. The main factor seems to be the cost of unioning
         these bit vectors, and that cost is proportional to the number of bitmaps
         minus one and the size of the bitmaps, which is approximated here by the
         number of rows in the table.
      
         Note that because we use (NDV-1) in the formula, this penalty does not
         apply to usual index joins, which have an NDV of 1 per rebind. This is
         consistent with what we see in measurements and it also seems reasonable,
         since we don't have to union bitmaps in this case.
      
      4. Fix to select CostModelGPDB for the 'experimental' model, as we do in 5X.
      
      5. Calibrate the constants involved (c1 ... c6), using the calibration program
         and running experiments with heap and append-only tables on a laptop and
         also on a Linux cluster with 24 segments. Also run some other workloads
         for validation.
      
      6. Give a small initial advantage to bitmap scans, so they will be chosen over
         table scans for small tables. Otherwise, small queries will
         have more or less random plans, all of which cost around 431, the value
         of the initial cost. Added a 10% advantage of the bitmap scan.
      
      * Port calibration program to Python 3
      
      - Used 2to3 program to do the basics.
      - Version parameter in argparse no longer supported
      - Needs additional option in connection string to keep the search path
      - The dbconn.execSQL call can no longer be used to get a cursor,
        this was probably a non-observable defect in the Python 2 version
      - Needed to use // (floor division) in some cases
      Co-authored-by: NDavid Kimura <dkimura@vmware.com>
      9363718d
    • H
      Improve partition elimination when indexes are present (port from 6X) (#11107) · bfcc63e1
      Hans Zeller 提交于
      * Improve partition elimination when indexes are present (port from 6X)
      
      * Use original join pred for DPE with index nested loop joins
      
      Dynamic partition selection is based on a join predicate. For index
      nested loop joins, however, we push the join predicate to the inner
      side and replace the join predicate with "true". This meant that
      we couldn't do DPE for nested index loop joins.
      
      This commit remembers the original join predicate in the index nested
      loop join, to be used in the generated filter map for DPE. The original
      join predicate needs to be passed through multiple layers.
      
      * SPE for index preds
      
      Some of the xforms use method CXformUtils::PexprRedundantSelectForDynamicIndex
      to duplicate predicates that could be used both as index predicates and as
      partition elimination predicates. The call was missing in some other xforms.
      Added it.
      
      * Changes to equivalent distribution specs with redundant predicates
      
      Adding redundant predicates causes some issues with generating
      equivalent distribution specs, to be used for the outer table of
      a nested index loop join. We want the equivalent spec to be
      expressed in terms of outer references, which are the columns of
      the outer table.
      
      By passing in the outer refs, we can ensure that we won't replace
      an outer ref in a distribution spec with a local variable from
      the original distribution spec.
      
      Also removed the asserts in CPhysicalFilter::PdsDerive that ensure the
      distribution spec is complete (consisting of only columns from the
      outer table) after we see a select node. Even without my changes, the
      asserts do not always hold, as this test case shows:
      
        drop table if exists foo, bar;
        create table foo(a int, b int, c int, d int, e int) distributed by(a,b,c);
        create table bar(a int, b int, c int, d int, e int) distributed by(a,b,c);
      
        create index bar_ixb on bar(b);
      
        set optimizer_enable_hashjoin to off;
        set client_min_messages to log;
      
        -- runs into assert
        explain
        select *
        from foo join bar on foo.a=bar.a and foo.b=bar.b
        where bar.c > 10 and bar.d = 11;
      
      Instead of the asserts, we now use the new method of passing in the
      outer refs to ensure that we move towards completion. We also know
      now that we can't always achieve a complete distribution spec, even
      without redundant predicates.
      
      * MDP changes
      
      Various changes to MDPs:
      
      - New SPE filters used in plan
      - New redundant predicates (partitioning or on non-partitioning columns)
      - Plan space changes
      - Cost changes
      - Motion changes
      - Regenerated, because plan switched to a hash join, so used a guc
        to force an index plan
      - Fixed lookup failures
      - Add mdp where we try unsuccessfully to complete a distribution spec
      
      * ICG result changes
      
      - Test used the 'experimental' cost model to force an index scan, but we
        now get the index scan even with the default cost model (plans currently
        fall back).
      
      This is a cherry-pick of commit 4a7a6821 from the 6X_STABLE branch.
      bfcc63e1
    • B
      Redirect the error to log message · fc572100
      Bhuvnesh Chaudhary 提交于
      Earlier the error was sent to /dev/null and the information was lost
      displaying the cause of the error. Redirect the error to log file.
      fc572100