Motivation
Add auto partitions update interval setting for producer and consumer.

Modifications
add autoUpdatePartitionsInterval to partitioned producer and consumer

Return more informative error message when trying to create subscription on non-persistent throug Rest API or pulsar-admin CLI. (#7831)

Fixes #7397

Motivation
When use pulsar-admin to create a subscription on a non-persistent topic, get the server error
This change return more informative error message when trying to create subscript ion on non-persistent through Rest API or pulsar-admin CLI.

Modifications
Currently when creating subscription is called with non-persistent topic service will try to create the subscription which will fail with casting exception when trying to cast NonPersistentSubscription to PersistentSubscription and client will see internal error.
Add check if create subscription command is called for a non-persistent topic before actually

Verifying this change
This change added tests and can be verified as follows:

Added unit test
Verified with local standalone

## Motivation
User try a loop of create-exclusive-consumer, seek, consume and close of a consumer with cpp client, and some times will meet “consumer busy” errors, which means the broker side consumer still alive while creating new a consumer. 

Here are suspicion logs.  
```
INFO:Client(88)Subscribing on Topic :public/default/jeff-test-21-partition-0Tue Aug 11 11:38:38 2020
INFO:HandlerBase(53)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Getting connection from poolTue Aug 11 11:38:38 2020
INFO:ConsumerImpl(175)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Created consumer on broker [10.88.109.77:44648 -> 10.88.109.71:31051] Tue Aug 11 11:38:38 2020
INFO:HandlerBase(130)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Schedule reconnection in 0.1 sTue Aug 11 11:38:38 2020
INFO:ConsumerImpl(1047)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Seek successfullyTue Aug 11 11:38:38 2020
INFO:HandlerBase(53)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Getting connection from poolTue Aug 11 11:38:38 2020
INFO:ConsumerImpl(175)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Created consumer on broker [10.88.109.77:44648 -> 10.88.109.71:31051] Tue Aug 11 11:38:40 2020
INFO:HandlerBase(130)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Schedule reconnection in 0.1 sTue Aug 11 11:38:41 2020
INFO:HandlerBase(53)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Getting connection from poolTue Aug 11 11:38:41 2020
INFO:ConsumerImpl(848)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Closing consumer for topic persistent://public/default/jeff-test-21-partition-0Tue Aug 11 11:38:42 2020
INFO:ConsumerImpl(175)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Created consumer on broker [10.88.109.77:44646 -> 10.88.109.71:31051] Tue Aug 11 11:38:42 2020
INFO:ConsumerImpl(104)[persistent://public/default/jeff-test-21-partition-0, 0_itom_di_scheduler_bat_prod_jeff-test-21, 1612] Destroyed consumer which was not properly closedTue Aug 11 11:38:42 2020
```

-  there is reconnection caused by seek command;
-  close operation happens at the same time of seek.
-  consumer destroyed with log `Destroyed consumer which was not properly closed`.

The race condition happens like this:
1. seek command triggered consumer disconnect; 
```
subscription.resetCursor
\
disconnectFuture = dispatcher.disconnectActiveConsumers(true);
\
disconnectAllConsumers(boolean isResetCursor)
consumerList.forEach(consumer -> consumer.disconnect(isResetCursor)); 
```

2. client trigger disconnectConsumer, and triggered `connection_.reset();`

```
case BaseCommand::CLOSE_CONSUMER: {
    		consumer->disconnectConsumer();  LOG_INFO("Broker notification of Closed consumer: " << consumerId_);	   

\
void ConsumerImpl::disconnectConsumer() {
    LOG_INFO("Broker notification of Closed consumer: " << consumerId_);
    Lock lock(mutex_);
    connection_.reset();   < === 
    lock.unlock();
}
```

3. connection not ready, and close consumer happened, then it leaked send CloseConsumer command to broker.

```
void ConsumerImpl::closeAsync(ResultCallback callback) {
...
    LOG_INFO(getName() << "Closing consumer for topic " << topic_);
    state_ = Closing;
    ClientConnectionPtr cnx = getCnx().lock();    < === the seek operation caused cnx reset
    if (!cnx) {     < === goes into this if, and set to Closed and returned directly without closeConsumer sent to broker
        state_ = Closed;  
        lock.unlock();
        // If connection is gone, also the consumer is closed on the broker side
        if (callback) {
            callback(ResultOk);
        }
        return;
    }
    ... 
    Future<Result, ResponseData> future =
        cnx->sendRequestWithId(Commands::newCloseConsumer(consumerId_, requestId), requested);  < ====
   .... 
}
```


### Modifications

when consumer destroy, try to send another closeConsumer command if suitable.


* fix race condition caused by consumer seek and close

* fix format

Motivation
Follow #7857 In the pulsar standalone Dockerfile, we also need to do the same operation

Modifications
Add python2.7-dev to pulsar standalone docker image

Signed-off-by: Nxiaolong.ran <rxl@apache.org>

### Modifications

Add release 2.6.1 blog for the website.

Signed-off-by: Nxiaolong.ran <rxl@apache.org>


### Modifications

Use the 2.6.1 version of the docs as the latest stable version

Signed-off-by: Nxiaolong.ran <rxl@apache.org>

### Motivation

Recently, building Pulsar Docker image in CI jobs has continued to fail. It seems that the header file "Python.h" does not exist.
```
[INFO]   building 'fastavro._read' extension
[INFO]   creating build/temp.linux-x86_64-2.7
[INFO]   creating build/temp.linux-x86_64-2.7/fastavro
[INFO]   x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c fastavro/_read.c -o build/temp.linux-x86_64-2.7/fastavro/_read.o
[INFO]   fastavro/_read.c:4:10: fatal error: Python.h: No such file or directory
[INFO]    #include "Python.h"
[INFO]             ^~~~~~~~~~
[INFO]   compilation terminated.
[INFO]   error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
[INFO]   ----------------------------------------
[INFO]
[INFO]   ERROR: Failed building wheel for fastavro
```

### Modifications

I'm not sure why the build has recently failed, but added `python2.7-dev` and `python3.7-dev` to the Docker image.

### Motivation
Support set `DeduplicationEnabled` on topic level

### Modifications
Support set/get/remove `DeduplicationEnabled` policy on topic level.

### Verifying this change
Added Unit test to verify set/get/remove `DeduplicationEnabled` policy at Topic level work as expected when Topic level policy is enabled/disabled

`org.apache.pulsar.broker.service.persistent.TopicDuplicationTest`

Signed-off-by: Nxiaolong.ran <rxl@apache.org>

### Motivation

Currently, when the official website 2.6.1 document is released, we should first let 2.6.0 continue as the default version, wait for version 2.6.1 to appear in the historical version, and then try to use version 2.6.1 as the default version. If we directly publish the 2.6.1 version of the document, the search function will be unavailable.

### Modifications

- Update versions.json

Signed-off-by: Nxiaolong.ran <rxl@apache.org>

### Motivation

Currently, the 2.6.2 docs are a complete copy based on 2.6.1. In the next pull request, if there is a fix for the 2.6.2 document, we can modify it based on the current 2.6.2, so A release manager can directly reuse the contents of the 2.6.2 document when releasing 2.6.2


### Modifications

- Add docs of 2.6.2
- Add sidebars of 2.6.2

* Limit cpu count for proxy unit test cases

* Reduce logging in github workflows
Co-authored-by: NAli Ahmed <alia@splunk.com>

### Motivation

The pending batchIndexAcks bitSet batchSize is not correct.

### Modifications

Fix the bitSet batchSize.

Signed-off-by: Nxiaolong.ran <rxl@apache.org>

### Motivation

Update 2.6.1 docs

### Modifications

- update 2.6.1 docs
- add 2.6.1 sidebars.json
- update releases.json
- update versions.json

Fixes #7604

Master Issue: #7604

### Motivation

We should redirect Get message by id request when a broker doesn't serve for the topic

### Modifications

* Redirect Get message by id request when a broker doesn't serve for the topic
* ~~Test log config use log4j2.xml not logback.xml
     config by parent pom: -Dlog4j.configurationFile=log4j2.xml~~

### Verifying this change

This change is a trivial change without any test coverage.

Motivation
There was already cpp oauth2 client provided, this or tries to provide a wrapper around it for Python client.

Modifications
add wrapper on cpp to support python client oauth2.

Link [https://github.com/apache/pulsar/issues/7757](https://github.com/apache/pulsar/issues/7757) and master issue [https://github.com/apache/pulsar/issues/2688](https://github.com/apache/pulsar/issues/2688)

### Motivation

Support set/get/remove persistence  policies on topic level.

### Verifying this change
new unit test added.

Fixes #7626


### Motivation
Expose `MessagesImpl` ，so that we can ack list of messages by using `CompletableFuture<Void> acknowledgeAsync(Messages<?> messages)`


### Modifications
Change the visibility level of the method from protect to public

### Verifying this change
unit test:
org.apache.pulsar.client.api.ConsumerBatchReceiveTest#testBatchAck

### Motivation
support set MaxUnackedMessagesOnConsumer on topic level

### Modifications
Support set/get/remove MaxUnackedMessagesOnConsumer policy on topic level.

### Verifying this change
Added Unit test to verify set/get/remove MaxUnackedMessagesOnConsumer policy at Topic level work as expected when Topic level policy is enabled/disabled

- org.apache.pulsar.broker.admin.MaxUnackedMessagesTest#testMaxUnackedMessagesOnConsumerApi
- org.apache.pulsar.broker.admin.MaxUnackedMessagesTest#testMaxUnackedMessagesOnConsumer

### Motivation

The current key of `MultiTopicsConsumerImpl.topics` is the topic name passed by user. The `topicNameValid` method checks if the name is valid and `topics` doesn't contain the key.

However, if a multi topics consumer subscribed a partition of a subscribed partitioned topic,  `subscribeAsync` succeed and a new `ConsumerImpl` of the same partition was created, which is redundant.

Also, if a multi topics consumer subscribed `public/default/topic` or `persistent://public/default/topic`, while the initial subscribed topic is `topic`, the redundant consumers would be created.

### Modifications

- Use full topic name as key of `MultiTopicsConsumerImpl.topics`
- Check both full topic name and full partitioned topic name not exist in `MultiTopicsConsumerImpl.topics` when `subscribeAsync` is called
- Throw a different exception to differ topic is invalid and topic is already subscribed
- Add a unit test for subscribing a partition of a subscribed partitioned topic or the same topic with prefix

Co-authored-by: NJerry Peng <jerryp@splunk.com>

Signed-off-by: Nxiaolong.ran <rxl@apache.org>

### Modifications

- Add 2.6.1 release notes to repo

Motivation:
The resource page is out of date.

Modification:
1. The tech blog of the same series are put next to each other.
2. The shared slides are updated.
3. The Bot WeChat ID is added on contact page.

### Motivation

Currently, the `TopicMessageIdImpl` class does not override the `toString()` method. Therefore, even if the ID of a message received from a partitioned topic is output to the log, we can only know the ID of the Java object.
```java
LOG.info("Received: {} (ID: {})", new String(msg.getData()), msg.getMessageId());
```
```
15:57:17.759 [main] INFO  SampleConsumer - Received: msg0 (ID: org.apache.pulsar.client.impl.TopicMessageIdImpl@e35239b1)
15:57:17.760 [main] INFO  SampleConsumer - Received: msg1 (ID: org.apache.pulsar.client.impl.TopicMessageIdImpl@e3523a0e)
15:57:17.761 [main] INFO  SampleConsumer - Received: msg2 (ID: org.apache.pulsar.client.impl.TopicMessageIdImpl@e3523d72)
15:57:17.762 [main] INFO  SampleConsumer - Received: msg3 (ID: org.apache.pulsar.client.impl.TopicMessageIdImpl@e3524133)
15:57:17.762 [main] INFO  SampleConsumer - Received: msg4 (ID: org.apache.pulsar.client.impl.TopicMessageIdImpl@e3523dcf)
```

### Modifications

Added the `toString()` method to the `TopicMessageIdImpl` class. This method returns the result of converting the `MessageId` instance held internally to a string.

### Motivation

Based on the scan results of `Black Duck`, we found that there are security vulnerabilities in the components currently used by pulsar, some are directly referenced by pulsar, and some are indirectly referenced by the pulsar.

Master Issue: #2664 

Fix https://github.com/streamnative/pulsar/issues/1304

### Motivation

Currently, the consumer can't receive transaction messages.

### Modifications

Support process the commit marker in the topic partition and fetch transaction messages from TransactionBuffer. 

* Fix admin-api-namespaces page grammatical errors

* Resolve review comments

This PR fixes admin-api-partitioned-topics page grammatical errors.

### Motivation
In #7699, i update doc for the master only. The contents are approved and merged.

This PR is for updating docs for releases 2.6.1 --2.1.0



### Modifications

Update the following docs:

Pulsar IO > get started
Connector Admin CLI

Motivation
support set MaxUnackMessagesPerSubscription on topic level

Modifications
Support set/get/remove MaxUnackMessagesPerSubscription policy on topic level.

Verifying this change
Added Unit test to verify set/get/remove MaxUnackMessagesPerSubscription policy at Topic level work as expected when Topic level policy is enabled/disabled

org.apache.pulsar.broker.admin.MaxUnackedMessagesTest#testMaxUnackedMessagesOnSubscriptionApi
org.apache.pulsar.broker.admin.MaxUnackedMessagesTest#testMaxUnackedMessagesOnSubscription

* Fix allowTopicOperationAsync logic (#1355)

*Modifications*

- We should use the original role to verify if it is allowed for a given topic operation
- use the original authentication data
- Authz provider doesn't have to be aware of proxyRole
- Fix authorization test

* Refactor authorize logic to provide a uniform authorization behavior

Master Issue: #2688 

### Motivation
support topic level delayed delivery policy

### Modifications
Support set/get/remove delayed delivery policy on topic level.

### Verifying this change

Added Unit test to verify set/get/remove delayed delivery policy at Topic level work as expected when Topic level policy is enabled/disabled

- org.apache.pulsar.broker.admin.AdminApiDelayedDelivery#testEnableAndDisableTopicDelayedDelivery
- org.apache.pulsar.broker.admin.AdminApiDelayedDelivery#testEnableTopicDelayedDelivery

### Motivation

Adding a subdirectory associated with current time willmake it easier to process HDFS files in batch.

For example, user can create multiple running sink instances with `yyyy-MM-dd-hh` pattern. Then stop all instances at next hour. Eventually, files of the subdirectory will contain all messages consumed during this hour.

### Modifications

- Add a `subdirectoryPattern` field to `HdfsSinkConfig`
- Update some simple tests for `HdfsSinkConfig`
- Update the doc of HDFS2 sink

### Documentation

  - Does this pull request introduce a new feature? (yes)
  - If yes, how is the feature documented? (docs)

### Motivation
User reports a valgrind error for `client::createReader` method:

```
==23308== 284,826 (160 direct, 284,666 indirect) bytes in 1 blocks are definitely lost in loss record 113 of 113
==23308==    at 0x4C2A593: operator new(unsigned long) (vg_replace_malloc.c:344)
==23308==    by 0x5303B4A: allocate (new_allocator.h:104)
==23308==    by 0x5303B4A: allocate (alloc_traits.h:351)
==23308==    by 0x5303B4A: __shared_count<pulsar::InternalState<pulsar::Result, pulsar::Reader>, std::allocator<pulsar::InternalState<pulsar::Result, pulsar::Reader> > > (shared_ptr_base.h:499)
==23308==    by 0x5303B4A: __shared_ptr<std::allocator<pulsar::InternalState<pulsar::Result, pulsar::Reader> > > (shared_ptr_base.h:957)
==23308==    by 0x5303B4A: shared_ptr<std::allocator<pulsar::InternalState<pulsar::Result, pulsar::Reader> > > (shared_ptr.h:316)
==23308==    by 0x5303B4A: allocate_shared<pulsar::InternalState<pulsar::Result, pulsar::Reader>, std::allocator<pulsar::InternalState<pulsar::Result, pulsar::Reader> > > (shared_ptr.h:598)
==23308==    by 0x5303B4A: make_shared<pulsar::InternalState<pulsar::Result, pulsar::Reader> > (shared_ptr.h:614)
==23308==    by 0x5303B4A: Promise (Future.h:91)
==23308==    by 0x5303B4A: pulsar::Client::createReader(std::string const&, pulsar::MessageId const&, pulsar::ReaderConfiguration const&, pulsar::Reader&) (Client.cc:142)
==23308==    by 0x401DDB: main (pulsarReader.cpp:92)
==23308== 
```
It seems the `ReaderImpl` has been tracked twice when call WaitForCallbackValue. this PR is to fix the issue.

### Modifications

- fix WaitForCallbackValue which is changed in PR #3484.
- add test for the reference issue.

### Verifying this change
ut passed.
valgrind found no issue:
```
==14758== LEAK SUMMARY:
==14758==    definitely lost: 0 bytes in 0 blocks
==14758==    indirectly lost: 0 bytes in 0 blocks
==14758==      possibly lost: 0 bytes in 0 blocks
==14758==    still reachable: 12,621 bytes in 145 blocks
==14758==         suppressed: 0 bytes in 0 blocks
==14758== 
==14758== For lists of detected and suppressed errors, rerun with: -s
==14758== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
```

### Motivation
The PR #4400 introduced new params on the pulsar-client consume command but it broked some old ones.

### Modifications

This PR fixes this problem to restore the good param behavior.

### Motivation
The zk cache executor thread pool size is hard code to 10 when pulsar service start, it should be configurable in broker.conf.
```
private final ScheduledExecutorService cacheExecutor = Executors.newScheduledThreadPool(10,
            new DefaultThreadFactory("zk-cache-callback"));
```
#### Changes
make the zk cache executor threads pool size configurable in broker.conf

* update source create description

* update as per advice from sijia

### Motivation
The orderedExecutor number is hard code to 8 when pulsar service start, it should be configurable in broker.conf.
```
orderedExecutor = OrderedExecutor.newBuilder().numThreads(8).name("pulsar-ordered")
```
### Changes
make the orderedExecutor threads number configurable in broker.conf

### Motivation
Support set retention quota on topic level.
Based on the system topic function.
### Modifications
Support get-retention on topic level.
Support set-retention on topic level.
Support remove-retention on topic level.

Motivation:
Update the resources page.

Modification:
Some blogs are added.