### Motivation

Fixes #2743 

NPE occurs when there are assignments for workers not currently in the membership.  This could happen if the worker crashed before failure detector unassigned the functions assign to it and the scheduler is triggered by other means.

### Modifications

Filter out the assignments for worker not in membership.  

*Motivation*

base64 is non compatible between python2 and 3

*Changes*

Provides a python 2/3 compatible method

*Motivation*

We need an upgrade/backward compatibility story for schema enforcement. Especially around:

- `Producers cannot connect without a schema to topics with a schema`

*Changes*

- provide a flag on brokers to enable schema validation (and disabled it by default). this allows a smooth upgrade on brokers,
  otherwise, it will break all non-java producers on topics with schema immediately when upgrade to the new version

### Motivation

Expose metrics in Prometheus regarding the number of log messages printed by each process.

This will add metrics like: 

```
# TYPE log4j2_appender_total counter
log4j2_appender_total{cluster="standalone",level="debug"} 0.0
log4j2_appender_total{cluster="standalone",level="warn"} 1.0
log4j2_appender_total{cluster="standalone",level="trace"} 0.0
log4j2_appender_total{cluster="standalone",level="error"} 0.0
log4j2_appender_total{cluster="standalone",level="fatal"} 0.0
log4j2_appender_total{cluster="standalone",level="info"} 181.0
```

* The log4j should be prefixed by pulsarRootDir

* User importlib because use of __import__ is discouraged

*Motivation*

Sometime when we run worker service as part of broker, some pods can be hanging on creating producers to assignment topics.
It is unknown whether is it a k8s problem or not. But in general, timeout to fail fast to allow k8s to reschedule the pods.

*Changes*

Add a timeout logic at creating producers.

*Motivation*

Ideally we should run all integration tests on both cluster mode and standalone
mode. However the apache ci can't really afford to do so. so we run all the integration
tests on cluster mode. We only run basic validation and test new features (e.g. state)
on standalone.

*Changes*

Add PulsarStandalone related test base and suite and a simmple smoke test.
This would set the framework for integration tests for state

* Add a config to allow user code dependencies to be installed at runtime.
Enable it only for Kubernetes runtime

* Fix bug

* Fixed indentation issue

* Specify the right cmd line

* Install dep only in the temp dir

* Queue is not in python3. Thus add a try catch block

* Fixed unittest

*Motivation*

getFunctionStat call can happen before spanwer starts the runtime

*Changes*

Add null checks and also change the order of closing sequence in RuntimeSpawner

*Motivation*

People shouldn't be looking into code to understand how to configure a function worker.
We should have all the possible settings available in the configuration file.

*Changes*

reorganize the function worker configuration file and add missing fields

*Motivation*

k8s runtime doesn't support generating function stats at worker service right now.
so skip it for now until that feature is added.

*Changes*

skip function stats for k8s runtime

### Motivation

If pulsar service starts broker and worker in the same process then worker should not use broker-advertised address but use localhost address to connect. 

* [functions][stats] NPE in FunctionStatsGenerator when worker service is not ready

*Motivation*

NullPointerException was thrown when function worker is running as part of broker and metrics collection kicks in
before worker service completes initialization

*Changes*

Only generate functions when worker service is ready

* Fix FunctionSTatsGeneratorTest

* Download the user code into pulsarRootDir

* Apply it everywhere

* Fix unittest

* Removed shading relocations for Circe-checksum and lz4 libraries

* Also remove the jni shading in pulsar-broker-shaded

* Removed remaining relocations directives

* Moved Kafka wrapper tests to integration tests

* Do not shade io.netty.buffer.ByteBuf since it's passed between components

* Fixed presto dependencies license file

- Tick doesn't implement equals
- TestBasicPresto doesn't run

For a topic with metadata similar to https://github.com/apache/pulsar/pull/2673, IllegalArgumentException may occur in the following line:
https://github.com/apache/pulsar/blob/b2484d92d5068d4f0699eb9c3d31640cb48f9dd0/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java#L655

This is the broker log when the exception has occurred:
[invalid_range_error.txt](https://github.com/apache/pulsar/files/2442924/invalid_range_error.txt)

It is because `readPosition` is ahead of `ledger.getLastPosition().getNext()`, so `ManagedCursorImpl#getNumberOfEntries()` should return 0 as the number of entries to read in that case.

I think this issue and https://github.com/apache/pulsar/pull/2673 are the result of that `ledger.getLastPosition()` is no longer the real last of the managed ledger because of https://github.com/apache/pulsar/pull/1550.

* First cut of the serverside valiation

* Revert unneeded changes

* Do the easier checks first

* Check for null

* Simplify

* Do the actual validation

* Check for null runtime

* Update Test

* Fix build

* Fix build

* misc fixes

* Add explicit check for serde

* Changed for a better condition

* Better serde check

* Fixed some unittests

* Fixed more tests

* Fixed unittest

* Fixed unittests

* Fixed unittest

* Stricter checks

* adding metrics to presto pulsar connector

* rename batch size

* adding comments

* refactoring metrics

* modifying bytes read metric

* fixing tests

* deleting tmp file

* adding jars to LICENSE

* Clarify version match requirements between Go and C++ client librarires

* Document how to see local changes

* Link to docs README

* Move warning out from `go get` paragraph

*Motivation*

Currently trigger function is broken due to the schema enforcement we added recently:
A producer without schema can't produce messages into a topic with schema.

*Changes*

- Rename `AUTO` to `AUTO_CONSUME`
- Introduce `AUTO_PRODUCE` schema. The schema produces `byte[]`, but it will validate
  the bytes are compatible with the schema associated with the topic before producing.
- Change trigger function to use `AUTO_PRODUCE` schema.

### Motivation

This pr will allow functions being deployed as kubernetes jobs have installation defined labels.

* Do not register each DispatchRateLimiter for policies notifications

* Fixed updates on per-subscriptions limits

* Fixed testClusterPolicyOverrideConfiguration

* Fix stats generator

* Set the response to text/plain

* fix TopicStats and NamespaceStats

### Motivation

add kafka source connect adaptor for debezium
This will save data from kafka source connect into Pulsar.

### Modifications

add class and test 

### Result

ut pass

### Motivation

Create consumers before producing will make sure consumers received all the messages that they expect to receive.

* Avoid scheduling heartbeat function if owner-worker not available

* fix: heartbeat function place

* add heartbeat check at scheduler-manager

* create seaparate list for heatbeat functions