1. 28 8月, 2020 4 次提交
    • B
    • S
    • M
      [broker] Fix deadlock when adding consumer (#7841) · 23dc5c74
      Masahiro Sakamoto 提交于
      ### Motivation
      
      A deadlock occurred on our Pulsar 2.4.2 broker server. The cause is the following two threads:
      ```
      "prometheus-stats-36-1" #410 prio=5 os_prio=0 tid=0x00007f4b70019800 nid=0x30ca waiting for monitor entry [0x00007f4bbe3b7000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at org.apache.pulsar.broker.service.persistent.PersistentSubscription.getNumberOfEntriesDelayed(PersistentSubscription.java:1013)
              - waiting to lock <0x00007f913d098dd0> (a org.apache.pulsar.broker.service.persistent.PersistentSubscription)
              at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$getTopicStats$8(NamespaceStatsAggregator.java:129)
              at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$523/2109257042.accept(Unknown Source)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:385)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159)
              at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.getTopicStats(NamespaceStatsAggregator.java:122)
              at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$null$0(NamespaceStatsAggregator.java:64)
              at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$521/1017174654.accept(Unknown Source)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:385)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159)
              at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$null$1(NamespaceStatsAggregator.java:63)
              at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$520/1098830264.accept(Unknown Source)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:385)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159)
              at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.lambda$generate$2(NamespaceStatsAggregator.java:62)
              at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator$$Lambda$316/212211274.accept(Unknown Source)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:385)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159)
              at org.apache.pulsar.broker.stats.prometheus.NamespaceStatsAggregator.generate(NamespaceStatsAggregator.java:59)
              at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generate(PrometheusMetricsGenerator.java:73)
              at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet.lambda$doGet$0(PrometheusMetricsServlet.java:70)
              at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsServlet$$Lambda$315/1221766138.run(Unknown Source)
              at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32)
              at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      
      "ForkJoinPool.commonPool-worker-104" #953 daemon prio=5 os_prio=0 tid=0x00007f4dc8030800 nid=0x3b87 waiting on condition [0x00007f48f6ce1000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007f913d08b5c8> (a org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section)
              at java.util.concurrent.locks.StampedLock.acquireRead(StampedLock.java:1215)
              at java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:377)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:159)
              at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.values(ConcurrentOpenHashMap.java:174)
              at org.apache.pulsar.broker.service.persistent.PersistentTopic.getNumberOfConsumers(PersistentTopic.java:1227)
              at org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers.isConsumersExceededOnTopic(PersistentDispatcherMultipleConsumers.java:178)
              at org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers.addConsumer(PersistentDispatcherMultipleConsumers.java:144)
              - locked <0x00007f91120dc258> (a org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers)
              at org.apache.pulsar.broker.service.persistent.PersistentSubscription.addConsumer(PersistentSubscription.java:238)
              - locked <0x00007f913d098dd0> (a org.apache.pulsar.broker.service.persistent.PersistentSubscription)
              at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$subscribe$11(PersistentTopic.java:590)
              at org.apache.pulsar.broker.service.persistent.PersistentTopic$$Lambda$451/1414070467.accept(Unknown Source)
              at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
              at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683)
              at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010)
              at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:584)
              at org.apache.pulsar.broker.service.ServerCnx.lambda$null$10(ServerCnx.java:699)
              at org.apache.pulsar.broker.service.ServerCnx$$Lambda$408/1168861154.apply(Unknown Source)
              at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
              at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
              at org.apache.pulsar.broker.service.ServerCnx.lambda$null$13(ServerCnx.java:682)
              at org.apache.pulsar.broker.service.ServerCnx$$Lambda$406/1351396211.apply(Unknown Source)
              at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
              at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
              at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
              at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
              at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:943)
              at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457)
              at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
              at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
              at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
              at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163)
      ```
      
      `prometheus-stats-36-1` was trying to lock `PersistentSubscription` (and `PersistentDispatcherMultipleConsumers`) after locking `PersistentTopic#subscriptions`, an instance of `ConcurrentOpenHashMap`.
      https://github.com/apache/pulsar/blob/v2.4.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java#L122-L129
      
      On the other hand, `ForkJoinPool.commonPool-worker-104` was trying to lock these instances in reverse order.
      https://github.com/apache/pulsar/blob/v2.4.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java#L176-L237
      https://github.com/apache/pulsar/blob/v2.4.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L127-L144
      https://github.com/apache/pulsar/blob/v2.4.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L159-L175
      https://github.com/apache/pulsar/blob/v2.4.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1212-L1214
      
      `PersistentSubscription#getNumberOfEntriesDelayed()` is no longer used in the master code, but it seems that this deadlock has not yet been resolved.
      https://github.com/apache/pulsar/blob/e06e8726847584700d9e4fc98fd56a495eb05a23/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L1528-L1529
      https://github.com/apache/pulsar/blob/17ae233a5d0fa364048b7c30ec90b8f7291d0d07/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java#L967-L1000
      https://github.com/apache/pulsar/blob/6e7d1a83c3c2737610f01cb372f61e2b830a62f7/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L809
      
      ### Modifications
      
      Moved the `isConsumersExceededOnTopic()` method to check the number of connected consumers when adding a consumer to the topic from the `Dispatcher` classes to the `AbstractTopic` class. Avoid the deadlock mentioned above by calling `PersistentTopic#getNumberOfConsumers()` before locking the `PersistentSubscription` instance.
      23dc5c74
    • M
      Split message ranges by ledger ID and store them in individualDeletedMessages (#7861) · 0b7f034e
      Masahiro Sakamoto 提交于
      Fixes #7554
      
      ### Motivation
      
      As mentioned in #7554, the class of `individualDeletedMessages` is different between Pulsar 2.3.2 (and earlier) and 2.4.0 (and later). This causes some of ranges contained in `individualDeletedMessages` to be lost when the version of Pulsar is upgraded, and a large number of messages that have already been acked can be redelivered to consumers.
      
      Also, even if the Pulsar version is 2.4.0 or later, the same phenomenon occurs when the value of `managedLedgerUnackedRangesOpenCacheSetEnabled` is switched from false to true.
      
      ### Modifications
      
      If the list of individually deleted message ranges loaded from ZK contains ranges that span different ledgers, split the ranges by ledger ID and store them in `individualDeletedMessages`.
      
      As a result, information about deleted message ranges is never lost and messages that have already been acked will not be redelivered.
      0b7f034e
  2. 27 8月, 2020 1 次提交
    • R
      [Pulsar SQL] Fix Pulsar SQL CI test (#7898) · 656b1306
      ran 提交于
      ### Motivation
      
      Currently, the Pulsar SQL CI test is blocked. 
      
      ### Modifications
      
      1. Build a new docker image before CI test to make sure the latest changes are covered.
      2. Remove the config `distributed-joins-enabled` in the `$PULSAR_HOME/conf/presto/config.properties`, this config was discarded in presto version `332`.
      3. Add the config `-Dpresto-temporarily-allow-java8=true` in `$PULSAR_HOME/conf/presto/jvm.config`, this config is needed if use Java 8 environment.
      4. The ClientBuilder will be loaded by two different ClassLoaders, in the Pulsar SQL CI test the ClientBuilder will be created by unexpected ClassLoader, we need to specify the ClassLoader for it.
      656b1306
  3. 26 8月, 2020 1 次提交
  4. 25 8月, 2020 2 次提交
  5. 24 8月, 2020 3 次提交
  6. 21 8月, 2020 10 次提交
    • C
      Some clarification of docs for partition and ordering key (#7745) · cc89e0c1
      Caleb Epstein 提交于
      Motivation
      As a Pulsar newbie, the role of "partition key" and "ordering key" aren't entirely clear from the Doxygen comments. These names are not used in the public Pulsar documentation that I saw, so expanding the comments felt useful to me and may help other users.
      
      Modifications
      Expanded doc strings for MessageBuilder.setPartitionKey and setOrderingKey
      
      Verifying this change
      This change is a trivial rework / code cleanup without any test coverage.
      
      Documentation
      This change improves the C++ Doxygen output.
      cc89e0c1
    • Y
      Remove the unused CI job (#7853) · cdce3325
      Yong Zhang 提交于
      ---
      
      Fixes #7754
      
      This CI job already be removed. And the check require is removed. So let's
      remove this file.
      cdce3325
    • B
      Fix: close() never gets called for BatchSource (#7866) · 2ee109ee
      Boyang Jerry Peng 提交于
      Motivation
      close() method never gets called in BatchSource
      2ee109ee
    • A
      [Client]Add autoPartitionsUpdateInterval for producer and consumer (#7840) · 7d9319d8
      Aaron Robert 提交于
      Motivation
      Add auto partitions update interval setting for producer and consumer.
      
      Modifications
      add autoUpdatePartitionsInterval to partitioned producer and consumer
      7d9319d8
    • M
      Return more informative error message when trying to create subscription on... · 7b60e2d4
      Marvin Cai 提交于
      Return more informative error message when trying to create subscription on non-persistent throug Rest API or pulsar-admin CLI. (#7831)
      
      Fixes #7397
      
      Motivation
      When use pulsar-admin to create a subscription on a non-persistent topic, get the server error
      This change return more informative error message when trying to create subscript ion on non-persistent through Rest API or pulsar-admin CLI.
      
      Modifications
      Currently when creating subscription is called with non-persistent topic service will try to create the subscription which will fail with casting exception when trying to cast NonPersistentSubscription to PersistentSubscription and client will see internal error.
      Add check if create subscription command is called for a non-persistent topic before actually
      
      Verifying this change
      This change added tests and can be verified as follows:
      
      Added unit test
      Verified with local standalone
      7b60e2d4
    • J
      cpp: fix race condition caused by consumer seek and close (#7819) · dcc84c92
      Jia Zhai 提交于
      ## Motivation
      User try a loop of create-exclusive-consumer, seek, consume and close of a consumer with cpp client, and some times will meet “consumer busy” errors, which means the broker side consumer still alive while creating new a consumer. 
      
      Here are suspicion logs.  
      ```
      INFO:Client(88)Subscribing on Topic :public/default/jeff-test-21-partition-0Tue Aug 11 11:38:38 2020
      INFO:HandlerBase(53)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Getting connection from poolTue Aug 11 11:38:38 2020
      INFO:ConsumerImpl(175)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Created consumer on broker [10.88.109.77:44648 -> 10.88.109.71:31051] Tue Aug 11 11:38:38 2020
      INFO:HandlerBase(130)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Schedule reconnection in 0.1 sTue Aug 11 11:38:38 2020
      INFO:ConsumerImpl(1047)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Seek successfullyTue Aug 11 11:38:38 2020
      INFO:HandlerBase(53)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Getting connection from poolTue Aug 11 11:38:38 2020
      INFO:ConsumerImpl(175)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Created consumer on broker [10.88.109.77:44648 -> 10.88.109.71:31051] Tue Aug 11 11:38:40 2020
      INFO:HandlerBase(130)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Schedule reconnection in 0.1 sTue Aug 11 11:38:41 2020
      INFO:HandlerBase(53)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Getting connection from poolTue Aug 11 11:38:41 2020
      INFO:ConsumerImpl(848)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Closing consumer for topic persistent://public/default/jeff-test-21-partition-0Tue Aug 11 11:38:42 2020
      INFO:ConsumerImpl(175)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Created consumer on broker [10.88.109.77:44646 -> 10.88.109.71:31051] Tue Aug 11 11:38:42 2020
      INFO:ConsumerImpl(104)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Destroyed consumer which was not properly closedTue Aug 11 11:38:42 2020
      ```
      
      -  there is reconnection caused by seek command;
      -  close operation happens at the same time of seek.
      -  consumer destroyed with log `Destroyed consumer which was not properly closed`.
      
      The race condition happens like this:
      1. seek command triggered consumer disconnect; 
      ```
      subscription.resetCursor
      \
      disconnectFuture = dispatcher.disconnectActiveConsumers(true);
      \
      disconnectAllConsumers(boolean isResetCursor)
      consumerList.forEach(consumer -> consumer.disconnect(isResetCursor)); 
      ```
      
      2. client trigger disconnectConsumer, and triggered `connection_.reset();`
      
      ```
      case BaseCommand::CLOSE_CONSUMER: {
          		consumer->disconnectConsumer();  LOG_INFO("Broker notification of Closed consumer: " << consumerId_);	   
      
      \
      void ConsumerImpl::disconnectConsumer() {
          LOG_INFO("Broker notification of Closed consumer: " << consumerId_);
          Lock lock(mutex_);
          connection_.reset();   < === 
          lock.unlock();
      }
      ```
      
      3. connection not ready, and close consumer happened, then it leaked send CloseConsumer command to broker.
      
      ```
      void ConsumerImpl::closeAsync(ResultCallback callback) {
      ...
          LOG_INFO(getName() << "Closing consumer for topic " << topic_);
          state_ = Closing;
          ClientConnectionPtr cnx = getCnx().lock();    < === the seek operation caused cnx reset
          if (!cnx) {     < === goes into this if, and set to Closed and returned directly without closeConsumer sent to broker
              state_ = Closed;  
              lock.unlock();
              // If connection is gone, also the consumer is closed on the broker side
              if (callback) {
                  callback(ResultOk);
              }
              return;
          }
          ... 
          Future<Result, ResponseData> future =
              cnx->sendRequestWithId(Commands::newCloseConsumer(consumerId_, requestId), requested);  < ====
         .... 
      }
      ```
      
      
      ### Modifications
      
      when consumer destroy, try to send another closeConsumer command if suitable.
      
      
      * fix race condition caused by consumer seek and close
      
      * fix format
      dcc84c92
    • [Python Build] Add python-dev to pulsar standalone docker image (#7871) · 54dd24c7
      冉小龙 提交于
      Motivation
      Follow #7857 In the pulsar standalone Dockerfile, we also need to do the same operation
      
      Modifications
      Add python2.7-dev to pulsar standalone docker image
      54dd24c7
    • Add release 2.6.1 blog for website (#7756) · f53c9d33
      冉小龙 提交于
      Signed-off-by: Nxiaolong.ran <rxl@apache.org>
      
      ### Modifications
      
      Add release 2.6.1 blog for the website.
      f53c9d33
    • update the version json file for website (#7868) · ee728db5
      冉小龙 提交于
      Signed-off-by: Nxiaolong.ran <rxl@apache.org>
      
      
      ### Modifications
      
      Use the 2.6.1 version of the docs as the latest stable version
      ee728db5
    • Update swagger files for release 2.6.1 (#7867) · cea9fe50
      冉小龙 提交于
      Signed-off-by: Nxiaolong.ran <rxl@apache.org>
      cea9fe50
  7. 20 8月, 2020 2 次提交
    • M
      Add python-dev to pulsar docker image (#7857) · b5ffac30
      Masahiro Sakamoto 提交于
      ### Motivation
      
      Recently, building Pulsar Docker image in CI jobs has continued to fail. It seems that the header file "Python.h" does not exist.
      ```
      [INFO]   building 'fastavro._read' extension
      [INFO]   creating build/temp.linux-x86_64-2.7
      [INFO]   creating build/temp.linux-x86_64-2.7/fastavro
      [INFO]   x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c fastavro/_read.c -o build/temp.linux-x86_64-2.7/fastavro/_read.o
      [INFO]   fastavro/_read.c:4:10: fatal error: Python.h: No such file or directory
      [INFO]    #include "Python.h"
      [INFO]             ^~~~~~~~~~
      [INFO]   compilation terminated.
      [INFO]   error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
      [INFO]   ----------------------------------------
      [INFO]
      [INFO]   ERROR: Failed building wheel for fastavro
      ```
      
      ### Modifications
      
      I'm not sure why the build has recently failed, but added `python2.7-dev` and `python3.7-dev` to the Docker image.
      b5ffac30
    • F
      Support deduplication on topic level (#7821) · b4102745
      feynmanlin 提交于
      ### Motivation
      Support set `DeduplicationEnabled` on topic level
      
      ### Modifications
      Support set/get/remove `DeduplicationEnabled` policy on topic level.
      
      ### Verifying this change
      Added Unit test to verify set/get/remove `DeduplicationEnabled` policy at Topic level work as expected when Topic level policy is enabled/disabled
      
      `org.apache.pulsar.broker.service.persistent.TopicDuplicationTest`
      b4102745
  8. 19 8月, 2020 3 次提交
    • Update release.json and versions.json (#7847) · 6cae4afa
      冉小龙 提交于
      Signed-off-by: Nxiaolong.ran <rxl@apache.org>
      
      ### Motivation
      
      Currently, when the official website 2.6.1 document is released, we should first let 2.6.0 continue as the default version, wait for version 2.6.1 to appear in the historical version, and then try to use version 2.6.1 as the default version. If we directly publish the 2.6.1 version of the document, the search function will be unavailable.
      
      ### Modifications
      
      - Update versions.json
      6cae4afa
    • Add 2.6.2 docs (#7835) · 04afe234
      冉小龙 提交于
      Signed-off-by: Nxiaolong.ran <rxl@apache.org>
      
      ### Motivation
      
      Currently, the 2.6.2 docs are a complete copy based on 2.6.1. In the next pull request, if there is a fix for the 2.6.2 document, we can modify it based on the current 2.6.2, so A release manager can directly reuse the contents of the 2.6.2 document when releasing 2.6.2
      
      
      ### Modifications
      
      - Add docs of 2.6.2
      - Add sidebars of 2.6.2
      04afe234
    • A
      Limit cpu count for proxy unit test cases (#7845) · 809c6be3
      Ali Ahmed 提交于
      * Limit cpu count for proxy unit test cases
      
      * Reduce logging in github workflows
      Co-authored-by: NAli Ahmed <alia@splunk.com>
      809c6be3
  9. 18 8月, 2020 3 次提交
  10. 17 8月, 2020 5 次提交
    • J
      add oauth2 wrapper for python (#7813) · 58704f9e
      Jia Zhai 提交于
      Motivation
      There was already cpp oauth2 client provided, this or tries to provide a wrapper around it for Python client.
      
      Modifications
      add wrapper on cpp to support python client oauth2.
      58704f9e
    • H
      [ISSUE 7757] Support Persistence Policies on topic level (#7817) · ff1780c0
      Hao Zhang 提交于
      Link [https://github.com/apache/pulsar/issues/7757](https://github.com/apache/pulsar/issues/7757) and master issue [https://github.com/apache/pulsar/issues/2688](https://github.com/apache/pulsar/issues/2688)
      
      ### Motivation
      
      Support set/get/remove persistence  policies on topic level.
      
      ### Verifying this change
      new unit test added.
      ff1780c0
    • F
      Support acknowledging a list of messages (#7688) · e06e8726
      feynmanlin 提交于
      Fixes #7626
      
      
      ### Motivation
      Expose `MessagesImpl` ,so that we can ack list of messages by using `CompletableFuture<Void> acknowledgeAsync(Messages<?> messages)`
      
      
      ### Modifications
      Change the visibility level of the method from protect to public
      
      ### Verifying this change
      unit test:
      org.apache.pulsar.client.api.ConsumerBatchReceiveTest#testBatchAck
      e06e8726
    • F
      Support MaxUnackedMessagesOnConsumer on topic level (#7818) · 39280055
      feynmanlin 提交于
      ### Motivation
      support set MaxUnackedMessagesOnConsumer on topic level
      
      ### Modifications
      Support set/get/remove MaxUnackedMessagesOnConsumer policy on topic level.
      
      ### Verifying this change
      Added Unit test to verify set/get/remove MaxUnackedMessagesOnConsumer policy at Topic level work as expected when Topic level policy is enabled/disabled
      
      - org.apache.pulsar.broker.admin.MaxUnackedMessagesTest#testMaxUnackedMessagesOnConsumerApi
      - org.apache.pulsar.broker.admin.MaxUnackedMessagesTest#testMaxUnackedMessagesOnConsumer
      39280055
    • Y
      [pulsar-client] Avoid subscribing the same topic again (#7823) · e1b76a32
      Yunze Xu 提交于
      ### Motivation
      
      The current key of `MultiTopicsConsumerImpl.topics` is the topic name passed by user. The `topicNameValid` method checks if the name is valid and `topics` doesn't contain the key.
      
      However, if a multi topics consumer subscribed a partition of a subscribed partitioned topic,  `subscribeAsync` succeed and a new `ConsumerImpl` of the same partition was created, which is redundant.
      
      Also, if a multi topics consumer subscribed `public/default/topic` or `persistent://public/default/topic`, while the initial subscribed topic is `topic`, the redundant consumers would be created.
      
      ### Modifications
      
      - Use full topic name as key of `MultiTopicsConsumerImpl.topics`
      - Check both full topic name and full partitioned topic name not exist in `MultiTopicsConsumerImpl.topics` when `subscribeAsync` is called
      - Throw a different exception to differ topic is invalid and topic is already subscribed
      - Add a unit test for subscribing a partition of a subscribed partitioned topic or the same topic with prefix
      e1b76a32
  11. 15 8月, 2020 1 次提交
  12. 14 8月, 2020 5 次提交
    • Add 2.6.1 release notes to repo (#7739) · bfa36aeb
      冉小龙 提交于
      Signed-off-by: Nxiaolong.ran <rxl@apache.org>
      
      ### Modifications
      
      - Add 2.6.1 release notes to repo
      bfa36aeb
    • S
      update contact page and resources page (#7808) · ba98d5bf
      sijia-w 提交于
      Motivation:
      The resource page is out of date.
      
      Modification:
      1. The tech blog of the same series are put next to each other.
      2. The shared slides are updated.
      3. The Bot WeChat ID is added on contact page.
      ba98d5bf
    • M
      Implement toString() method for TopicMessageIdImpl class (#7807) · dc8609f1
      Masahiro Sakamoto 提交于
      ### Motivation
      
      Currently, the `TopicMessageIdImpl` class does not override the `toString()` method. Therefore, even if the ID of a message received from a partitioned topic is output to the log, we can only know the ID of the Java object.
      ```java
      LOG.info("Received: {} (ID: {})", new String(msg.getData()), msg.getMessageId());
      ```
      ```
      15:57:17.759 [main] INFO  SampleConsumer - Received: msg0 (ID: org.apache.pulsar.client.impl.TopicMessageIdImpl@e35239b1)
      15:57:17.760 [main] INFO  SampleConsumer - Received: msg1 (ID: org.apache.pulsar.client.impl.TopicMessageIdImpl@e3523a0e)
      15:57:17.761 [main] INFO  SampleConsumer - Received: msg2 (ID: org.apache.pulsar.client.impl.TopicMessageIdImpl@e3523d72)
      15:57:17.762 [main] INFO  SampleConsumer - Received: msg3 (ID: org.apache.pulsar.client.impl.TopicMessageIdImpl@e3524133)
      15:57:17.762 [main] INFO  SampleConsumer - Received: msg4 (ID: org.apache.pulsar.client.impl.TopicMessageIdImpl@e3523dcf)
      ```
      
      ### Modifications
      
      Added the `toString()` method to the `TopicMessageIdImpl` class. This method returns the result of converting the `MessageId` instance held internally to a string.
      dc8609f1
    • Fix security vulnerabilities of Pulsar (#7801) · 855ee939
      冉小龙 提交于
      ### Motivation
      
      Based on the scan results of `Black Duck`, we found that there are security vulnerabilities in the components currently used by pulsar, some are directly referenced by pulsar, and some are indirectly referenced by the pulsar.
      855ee939
    • R
      [Transaction] Support consume transaction messages. (#7781) · 6e7d1a83
      ran 提交于
      Master Issue: #2664 
      
      Fix https://github.com/streamnative/pulsar/issues/1304
      
      ### Motivation
      
      Currently, the consumer can't receive transaction messages.
      
      ### Modifications
      
      Support process the commit marker in the topic partition and fetch transaction messages from TransactionBuffer. 
      6e7d1a83