1. 09 9月, 2020 2 次提交
    • Y
      Fix comment: pulsar.Client.create_producer() (#7972) · 7154185b
      Yuku Takahashi 提交于
      Correct argument name is `send_timeout_millis` not `send_timeout_seconds`
      7154185b
    • F
      Fix dispatchRate is overwritten (#8004) · 8933d8dd
      feynmanlin 提交于
      Fixes #7863
      
      ### Motivation
      1) Topic level policy will be overwritten by namespace level
      2) After removing the topic level policy, `DispatchLimiter` does not take effect,  the namespace level policy should be used
      
      ### Modifications
      1)If there is a topic-level policy, it will not be overwritten by the namespace-level policy when the policy is updated
      2)When removing topic-level policy, namespace-level policy will be used. If the namespace-level policy does not exist, the broker level will be used
      
      ### Verifying this change
      unit test:
      TopicPoliciesTest#testPolicyOverwrittenByNamespaceLevel
      8933d8dd
  2. 08 9月, 2020 11 次提交
    • Y
      Remove python2 from the pulsar images build (#7936) · ed356add
      Yong Zhang 提交于
      
      The pulsar docker and pulsar standalone docker image build failed by:
      ```
      Processing ./pulsar-client/pulsar_client-2.6.0-cp27-cp27mu-manylinux1_x86_64.whl
      2020-08-25T02:23:28.9567844Z [INFO] Collecting fastavro
      2020-08-25T02:23:29.0080316Z [INFO]   Downloading fastavro-1.0.0.tar.gz (658 kB)
      2020-08-25T02:23:29.3108074Z [INFO]     ERROR: Command errored out with exit status 1:
      2020-08-25T02:23:29.3109489Z [INFO]      command: /usr/bin/python2.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jvZueR/fastavro/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jvZueR/fastavro/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-iH1mMy
      2020-08-25T02:23:29.3110061Z [INFO]          cwd: /tmp/pip-install-jvZueR/fastavro/
      2020-08-25T02:23:29.3110303Z [INFO]     Complete output (5 lines):
      2020-08-25T02:23:29.3110520Z [INFO]     Traceback (most recent call last):
      2020-08-25T02:23:29.3110741Z [INFO]       File "<string>", line 1, in <module>
      2020-08-25T02:23:29.3111190Z [INFO]       File "/tmp/pip-install-jvZueR/fastavro/setup.py", line 58, in <module>
      2020-08-25T02:23:29.3111437Z [INFO]         if sys.implementation.name != "pypy":
      2020-08-25T02:23:29.3111886Z [INFO]     AttributeError: 'module' object has no attribute 'implementation'
      ```
      
      I remove the python2.7 and using the python3 to build images.
      
      * Remove python2 in images
      
      * Remove python2 stuff
      
      * Update the default python version
      ed356add
    • Exclude vertx from bookkeeper-http package (#7997) · 98233c09
      冉小龙 提交于
      Signed-off-by: Nxiaolong.ran <rxl@apache.org>
      
      ### Motivation
      
      ```
      $ mvn dependency:tree|grep vertx
      ```
      
      Output:
      ```
      [INFO] +- org.apache.bookkeeper.http:vertx-http-server:jar:4.10.0:compile
      [INFO] +- io.vertx:vertx-core:jar:3.5.3:compile
      [INFO] +- io.vertx:vertx-web:jar:3.5.3:compile
      [INFO] |  +- io.vertx:vertx-auth-common:jar:3.5.3:compile
      [INFO] |  \- io.vertx:vertx-bridge-common:jar:3.5.3:compile
      [INFO] |  +- org.apache.bookkeeper.http:vertx-http-server:jar:4.10.0:provided
      [INFO] |  +- io.vertx:vertx-core:jar:3.5.3:provided
      [INFO] |  \- io.vertx:vertx-web:jar:3.5.3:provided
      [INFO] |     +- io.vertx:vertx-auth-common:jar:3.5.3:provided
      [INFO] |     \- io.vertx:vertx-bridge-common:jar:3.5.3:provided
      ```
      
      ### Modifications
      
      - exclude `vertx` from bookkeeper-http package and include vertx of `3.5.3` to `vertx-http-server`
      98233c09
    • Upgrade protobuf version to 3.11.4 (#7998) · 7488b8e5
      冉小龙 提交于
      Signed-off-by: Nxiaolong.ran <rxl@apache.org>
      
      ### Modifications
      
      - Upgrade `protobuf` version to `3.11.4`
      7488b8e5
    • A
      update from zookeeper for available broker one more time if availableBrokers is empty (#7975) · 9ddafdf1
      Aloys 提交于
      Fixes #7476 
      
      ### Motivation
      
      In some situations,  discovery service can't find active broker while all brokers are running well.  This PR will update from zookeeper if available broker list is empty to avoid  "No active broker is available"  `RestException` happen.
      
      
      ### Modifications
      
      double check from zookeeper if availableBrokers is empty
      9ddafdf1
    • F
      [DOCS] Small improvements and clarifications around AuthN / AuthZ (#7976) · 18a54c41
      Frank J Kelly 提交于
      * Small documentation improvements and clarifications around AuthN / AuthZ
      
      * PR feedback and added some clarifications
      18a54c41
    • A
      Fix publish buffer of one ServerCnx caculated multi-times when check broker publish buffer (#7926) · c9ede6ed
      Aloys 提交于
      Fixes #7925 
      
      Master Issue: #<xyz>
      
      ### Motivation
      
       This PR fix publish buffer of one ServerCnx caculated multi-times when check broker's publish buffer.
      
      ### Modifications
      
       check publish buffer of broker by adding messagePublishBufferSize of a ServerCnx for only one time.
      c9ede6ed
    • [Security] Upgrade the snakeyaml verion to 1.26 (#7994) · 5adbc1f9
      冉小龙 提交于
      Fixes #7928
      
      ### Motivation
      
      As https://nvd.nist.gov/vuln/detail/CVE-2017-18640 said, the `snakeyaml` < 1.26
      
      ### Modifications
      
      In `pulsar-functions` model:
      
      - The `snakeyaml` 1.19 appears to be included from dependency on org.apache.pulsar:pulsar-functions-secrets:jar:2.6.1 based on included dependency of io.kubernetes:client-java-api:jar:2.0.0:compile Fixed in 9.0.2
      
      - The `snakeyaml` 1.16 appears to be included from the dependency on org.apache.pulsar:pulsar-functions-instance:jar:2.6.1 based on io.prometheus.jmx:collector:jar:0.12.0 Fixed in 0.13.0
      
      - The 1.17 org.apache.pulsar.tests:integration:test-jar:tests:2.6.1:test depends on org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:6.3.2:test Fixed in elasticsearch >= 7.7.1 (7.9.1 current)
      5adbc1f9
    • R
      [Transaction] Txn ack abort process on subscription (#7979) · 8f2540fc
      ran 提交于
      Fixes https://github.com/streamnative/pulsar/issues/1314
      
      ### Motivation
      
      Currently, the transaction ack abort process is not well.
      
      ### Modifications
      
      Make some changes for the transaction ack abort process.
      8f2540fc
    • H
      Remove stateful functions note (#7956) · 37a1d87e
      HuanliMeng 提交于
      Motivation
      The state is not production-ready.
      
      Modifications
      Remove the note about stateful function from the Deploy a cluster on bare metal document.
      
      updated docs for releases: master ~ 2.1.0
      37a1d87e
    • H
    • M
      Report compacted topic ledger info when calling get internal stats. (#7988) · 97ba09e6
      Marvin Cai 提交于
      Fixes #7895
      
      ### Motivation
      For get-internal-stats of persistent topic admin cli: https://pulsar.apache.org/docs/en/2.6.0/admin-api-persistent-topics/#get-internal-stats, we can also return the compacted topic ledger id if compaction is enabled. So we'll able to read from ledger without creating additional subscription, it can benefit like querying compacted topic from Pulsar SQL.
      
      ### Modifications
      Expose CompactedTopicContext from CompactedTopicImpl, try to get ledger information of compacted topic ledger if exist in PersistentTopic.
      
      ### Verifying this change
      
      This change added tests and can be verified as follows:
      - Added unit test to verify correct compacted ledger info is returned after compaction.
      
      
      * Report compacted topic ledger info when calling get internal stats.
      
      * Update documentation to add information about returning compacted topic ledger when get-internal-stats.
      97ba09e6
  3. 07 9月, 2020 1 次提交
    • M
      Add bookkeeperClientMinNumRacksPerWriteQuorum,... · 7b5301d9
      Michal Koziorowski 提交于
      Add bookkeeperClientMinNumRacksPerWriteQuorum, bookkeeperClientEnforceMinNumRacksPerWriteQuorum conf options (#7977)
      
      Added bookkeeperClientMinNumRacksPerWriteQuorum and bookkeeperClientEnforceMinNumRacksPerWriteQuorum to be able to configure rack and region aware bookie selection policy more precisely.
      
      Motivation
      We have Apache Pulsar setup running on Azure in 3 different Availability Zones (2 bookkeepers per zone). We wanted to store copy of each pulsar message in all that AZs. After configuring rack aware bookie selection policy, we found that messages are usually stored only in 2 AZs. After digging, we've found that it's caused by default bookeeper client minNumRacksPerWriteQuorum=2 setting. We've added bookkeeperClientMinNumRacksPerWriteQuorum and also bookkeeperClientEnforceMinNumRacksPerWriteQuorum to be able to configure this behaviour.
      
      Modifications
      Added bookkeeperClientMinNumRacksPerWriteQuorum and bookkeeperClientEnforceMinNumRacksPerWriteQuorum configuration option, modified tests, config files and documentation.
      
      Verifying this change
      We've updated existing unit tests, deployed modified pulsar docker image on AKS kubernetes cluster using helm charts and checked that stored messages were written always to bookkeepers in all AZs.
      
      
      * Added `bookkeeperClientMinNumRacksPerWriteQuorum` and `bookkeeperClientEnforceMinNumRacksPerWriteQuorum` broker configuration options to be able set `minNumRacksPerWriteQuorum` and `enforceMinNumRacksPerWriteQuorum` bookkeeper client options.
      
      * Reverted accidental change in ServiceConfigurationTest.
      Co-authored-by: NMichal Koziorowski <Michal.Koziorowski@tomtom.com>
      7b5301d9
  4. 06 9月, 2020 1 次提交
  5. 05 9月, 2020 2 次提交
  6. 04 9月, 2020 4 次提交
    • Add docs of package Pulsar Functions (#7904) · 9e2aca8c
      冉小龙 提交于
      * Add docs of package Pulsar Functions
      Signed-off-by: Nxiaolong.ran <rxl@apache.org>
      9e2aca8c
    • R
      [Transaction] Handle Acknowledge In The Transaction (#7856) · 1fdffe75
      ran 提交于
      Fix https://github.com/streamnative/pulsar/issues/1307
      
      Master Issue: #2664 
      
      ### Motivation
      
      Handle acknowledge in the transaction.
      
      ### Modifications
      
      1. Register subscription to the transaction metadata store.
      2. Make the ack command carry the transaction information.
      3. When committing a transaction, commit every subscription in the transaction.
      1fdffe75
    • F
      Support topicl level offload (#7883) · 62e3df39
      feynmanlin 提交于
      ### Motivation
      Support set Offload Policy on topic level
      
      ### Modifications
      Support set/get/remove Offload policy on topic level.
      
      ### Verifying this change
      Added Unit test to verify set/get/remove Offload policy at Topic level work as expected when Topic level policy is enabled/disabled
      `org.apache.pulsar.broker.admin.AdminApiOffloadTest#testOffloadPoliciesApi`
      `org.apache.pulsar.broker.admin.AdminApiOffloadTest#testTopicLevelOffload`
      62e3df39
    • B
      Various fixes for Batch Source (#7965) · 1632dcaf
      Boyang Jerry Peng 提交于
      * Various fixes for Batch Source
      1. Create intermediate topic/subscription prior to function running in case auto topic creation is turned off
      2. Fix possible NPE in CronTrigger when calling stop()
      3. Stop all producers created in ContextImpl
      4. Delete intermediate topic for batch source
      Co-authored-by: NJerry Peng <jerryp@splunk.com>
      1632dcaf
  7. 03 9月, 2020 9 次提交
  8. 02 9月, 2020 10 次提交
    • Y
      [pulsar-broker] Stop to dispatch when skip message temporally since Key_Shared... · c7ac08b1
      Yuri Mizushima 提交于
      [pulsar-broker] Stop to dispatch when skip message temporally since Key_Shared consumer stuck on delivery (#7553)
      
      ### Motivation
      In some case of Key_Shared consumer, messages ordering was broken.
      Here is how to reproduce(I think it is one of case to reproduce this issue).
      
      1. Connect Consumer1 to Key_Shared subscription `sub` and stop to receive
         - receiverQueueSize: 500
      2. Connect Producer and publish 500 messages with key `(i % 10)`
      3. Connect Consumer2 to same subscription and start to receive
         - receiverQueueSize: 1
         - since https://github.com/apache/pulsar/pull/7106 , Consumer2 can't receive (expected)
      4. Producer publish more 500 messages with same key generation algorithm
      5. After that, Consumer1 start to receive
      6. Check Consumer2 message ordering
         - sometimes message ordering was broken in same key
      
      Consumer1:
      ```
      Connected: Tue Jul 14 09:36:39 JST 2020
      [pulsar-client-io-1-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
      [pulsar-timer-4-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - [persistent://public/default/key-shared-test] [sub0] [820f0] Prefetched messages: 499 --- Consume throughput received: 0.02 msgs/s --- 0.00 Mbit/s --- Ack sent rate: 0.00 ack/s --- Failed messages: 0 --- batch messages: 0 ---Failed acks: 0
      Received: my-message-0 PublishTime: 1594687006203 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-1 PublishTime: 1594687006243 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-2 PublishTime: 1594687006247 Date: Tue Jul 14 09:37:46 JST 2020
      ...
      Received: my-message-498 PublishTime: 1594687008727 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-499 PublishTime: 1594687008731 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-500 PublishTime: 1594687038742 Date: Tue Jul 14 09:37:46 JST 2020
      ...
      Received: my-message-990 PublishTime: 1594687040094 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-994 PublishTime: 1594687040103 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-995 PublishTime: 1594687040105 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-997 PublishTime: 1594687040113 Date: Tue Jul 14 09:37:46 JST 2020
      ```
      
      Consumer2:
      ```
      Connected: Tue Jul 14 09:37:03 JST 2020
      [pulsar-client-io-1-1] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
      Received: my-message-501 MessageId: 4:1501:-1 PublishTime: 1594687038753 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-502 MessageId: 4:1502:-1 PublishTime: 1594687038755 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-503 MessageId: 4:1503:-1 PublishTime: 1594687038759 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-506 MessageId: 4:1506:-1 PublishTime: 1594687038785 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-508 MessageId: 4:1508:-1 PublishTime: 1594687038812 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-901 MessageId: 4:1901:-1 PublishTime: 1594687039871 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-509 MessageId: 4:1509:-1 PublishTime: 1594687038815 Date: Tue Jul 14 09:37:46 JST 2020
      ordering was broken, key: 1 oldNum: 901 newNum: 511
      Received: my-message-511 MessageId: 4:1511:-1 PublishTime: 1594687038826 Date: Tue Jul 14 09:37:46 JST 2020
      Received: my-message-512 MessageId: 4:1512:-1 PublishTime: 1594687038830 Date: Tue Jul 14 09:37:46 JST 2020
      ...
      ```
      
      I think this issue is caused by https://github.com/apache/pulsar/pull/7105.
      Here is an example.
      1. dispatch messages
      2. Consumer2 was stuck and `totalMessagesSent=0`
         - Consumer2 availablePermits was 0
      3. skip redeliver messages temporally
         - Consumer2 availablePermits was back to 1
      4. dispatch new messages
         - new message was dispatched to Consumer2
      5. back to redeliver messages
      4. dispatch messages
         - ordering was broken
      
      ### Modifications
      Stop to dispatch when skip message temporally since Key_Shared consumer stuck on delivery.
      c7ac08b1
    • R
      [Transaction] Support transaction abort on partition (#7953) · 3ac98d8d
      ran 提交于
      Fixes https://github.com/streamnative/pulsar/issues/1312
      
      # Motivation
      
      Currently, the transaction abort on partitions operation is not getting through.
      
      ### Modifications
      
      Make the transaction abort on partitions operation get through.
      3ac98d8d
    • M
      [Issue 7864][docs]Update document for how to pick active consumer for failover... · 6a2b5745
      Marvin Cai 提交于
      [Issue 7864][docs]Update document for how to pick active consumer for failover subscription. (#7917)
      
      Update document to specify how active consumer is picked for partitioned and non-partitoned topic for failover subscription mode. The change was push in 2.4.2 but document has not been updated.
      
      Fixes #7864
      6a2b5745
    • Y
      Add python oauth2 document (#7946) · cd8b7c77
      Yong Zhang 提交于
      **Motivation**
      
      After #7813, the python client supports the oauth2 authentication as well. Add an example for it.
      cd8b7c77
    • F
      [Issue 7933][pulsar-broker-common] Add some logging to improve Authentication... · 7ab29d85
      Frank J Kelly 提交于
      [Issue 7933][pulsar-broker-common] Add some logging to improve Authentication debugging and Fix typos in code "occured" -> "occurred" (#7934)
      
      Fixes #7933 
      
      ### Motivation
      Newbie trying to make his first contribution to this projection :-)
      
      ### Modifications
      
      Added some logging to Authentication Service to help debugging when there are more than one AuthN provider.
      When I did that I noticed a typo ('occured' should have two r's) so I decided to try to fix all of them
      7ab29d85
    • H
      support message publish rate on topic level (#7948) · 66688d15
      hangc0276 提交于
      Modifications
      Support set publish rate on topic level.
      Support get publish rate on topic level.
      Support remove publish rate on topic level.
      66688d15
    • 5cee214a
    • J
      [blog] Add image and update file name (#7958) · cacded24
      Jennifer Huang 提交于
      cacded24
    • J
      [blog] Add Pulsar Summit Asia 2020 CFP blog (#7886) · 2248a2ef
      Jennifer Huang 提交于
      * Add Pulsar Summit CFP blog
      
      * update
      
      * Update 2020-08-25-pulsar-summit-asia-2020-cfp.md
      2248a2ef
    • Fix the time of 2.6.1 release (#7954) · ea0f4a63
      冉小龙 提交于
      Signed-off-by: Nxiaolong.ran <rxl@apache.org>
      
      ### Modifications
      
      Fix the time of release 2.6.1 to `08/21`
      ea0f4a63