* Various fixes for Batch Source
1. Create intermediate topic/subscription prior to function running in case auto topic creation is turned off
2. Fix possible NPE in CronTrigger when calling stop()
3. Stop all producers created in ContextImpl
4. Delete intermediate topic for batch source
Co-authored-by: NJerry Peng <jerryp@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>

Motivation
close() method never gets called in BatchSource

If a function has many threads who are doing context.newOutputMessage().send(), there could be a lot of contention due to big synchronization blocks in the ProducerImpl. By allowing functions to use thread local producers, this synchronization can be avoided leading to increased performance.
This pr adds the configurability of using thread local producers in functions and sources
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

* Added ability to specify producer config for functions and sources

* Fixed test

* Fix test

* Add generated function proto

* Add header

* Address comments
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>

* [Issue 7489] Remove timestamp from exception metrics for functions and connectors

* [Issue 7489] Remove timestamp from exception metrics for Go functions

* [Issue 7489] Remove unused import in go stats

* [Issue 7489] Remove timestamp from metrics in python stats

* Change to v2 of go github actions

* Update go github actions

* Remove the version from the go test command

* Rename github jobs for go
Co-authored-by: NMatteo Merli <mmerli@splunk.com>

* Allow ability to specify retain key ordering in functions

* Address comments
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

## Motivation
Provide support to configure different docker images for go, python, java function

## Modifications
Add javaFunctionDockerImageName, pythonFunctionDockerImageName, and goFunctionDockerImageName in KubernetesRuntimeFactoryConfig.
Add unit test case
Update docs of functions-runtime.md

Fixes #5538

### Motivation



### Modifications
In function mode and sink mode, PulsarSource can read compacted topic。
By `inputSpecs` parameter, each topic can independently decide whether to read compacted

### Verifying this change
unit test:
org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedFunction
org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedSink

1）
Change the value of the CLI tool parameter "--custom-schema-input" from
"TopicName-> schemaType" 
 to
"topicName-> {" schemaType ":" type"," jsr310ConversionEnabled ": true," alwaysAllowNull ": true}"

2）
Modify Function.proto, add properties "jsr310ConversionEnabled", "alwaysAllowNull"。So that we can receive the above 2 parameters

3）
Modify the "FunctionConfigUtils#convert" method ， put the two parameters in "CustomSchemaInputs" into ConsumerSpec

4）
In “JavaInstanceRunnable#setupInput” method, put the above 2 parameters into "ConsumerConfig" and pass it to "PulsarSource", let it set the parameters into "schema" when creating the consumer of Source。
So that，The “function” can get these 2 parameters from the message of 
”currentRecord“

Fixes #7032

### Motivation
Source data flow:
Custom Source-> Function-> Pulsar producer-> Broker

Sink data flow:
Broker-> Pulsar consumer-> Function-> Custom Sink

Pulsar consumer is only used in sink mode, so it is necessary to add DLQ parameters for sink mode

* Moved ClassLoading and Reflections Helper functions to common

* Fix tests

* Fix test
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

* Fix null pointer when getting function instance metrics.

* Made more functions sync

* Made the remaining public interface synchronized

* Made stats class public method synchronized so they are thread safe.

* Made setup synchronized so that it and close won't run together

* Undo making stats sync until we resolve differences

* Incorporated feedback
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

* Make Nar Extraction Directory configurable

* Fixed unittests
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Fixes #6519

### Motivation

Currently, Pulsar Functions not support Async mode, e.g. user passed in a Function in format :
```
Function<I, CompletableFuture<O>>
```
This kind of function is useful if the function might use RPCs to call external systems.

e.g.
```java
public class AsyncFunction implements Function<String, CompletableFuture<O>> {
    CompletableFuture<O> apply (String input) {
        CompletableFuture future = new CompletableFuture();
        ...function compute...
        future.whenComplete(() -> {
            ...  call external system  ...
        });
        return future;
    }
```

### Modifications
- add support for Async Functions support.

### Verifying this change
current ut passed.



* support func: Function<I, CompletableFuture<O>>

* add 2 examples

* add limit to the max outstanding items

Fixes #5633 


### Motivation

Currently, some users want to support Avro schema in debezium, so this pr supports this feature.
For Kafka's Avro schema, it depends on the Avro 1.8 version, but Avro version has just been upgraded to 1.9 in pulsar, so shade is needed to avoid finding `addProp` function

### Modifications

* Add a package `kafka-connect-avro-converter-shaded`
* Add class KafkaSchema to converter Kafka's Avro schema to pulsar's schema

### Verifying this change 

Unit test and integration tests

Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Fixes #5503 
From #6445 

# Motivation
In functions, it will encounter ```ClassCastException``` when using the Avro schema for topics.

```
Exception in thread "main" java.lang.ClassCastException: org.apache.pulsar.shade.org.apache.avro.generic.GenericData$Record cannot be cast to io.streamnative.KeyValueSchemaTest$Foo2
	at io.streamnative.KeyValueSchemaTest.testConsumerByPythonProduce(KeyValueSchemaTest.java:412)
	at io.streamnative.KeyValueSchemaTest.main(KeyValueSchemaTest.java:305)
```

# Modifications
In functions, when using Avro schema specific the ClassLoader for ReflectDatumReader.

Add integration test ```testAvroSchemaFunction``` in class ```PulsarFunctionsTest```.

This PR aims to fix or simplify `Optional` type usage logic:
- Null value used for optional type.
- Optional.isPresent() can be replaced with functional style expression
- Optional call chain that can be simplified.

Fixes #6170

### Motivation
Allow users set custom system properties while submitting functions. This can be used to pass credentials via system property.

### Modifications

- pulsar-admin client tool to accept multiple `--custom-property` options while creating the function
- function's java runtime to append these provided properties.

### Motivation

Master Issue: #5454 

When one Consumer subscribe multi topic, setSchemaInfoPorvider() will be covered by the consumer generated by the last topic.

### Modification
clone schema for each consumer generated by topic.
### Verifying this change
Add the schemaTest for it.

[Issue 4175] [pulsar-function-go] Create integration tests for Go Functions for production-readiness (#6104)

This PR is to provide integration tests that test execution of Go functions that are managed by the Java FunctionManager. This will allow us to test things like behavior during function timeouts, heartbeat failures, and other situations that can only be effectively tested in an integration test. 

Master issue: #4175
Fixes issue: #6204 

### Modifications

We must add Go to the integration testing logic. We must also build the Go dependencies into the test Dockerfile to ensure the Go binaries are available at runtime for the integration tests. 

Fixes #5116 

### Motivation
Based on the request in #5116, adding this flag can help reduce ambiguity and increase flexibility. By default, the function passes source message properties.

### Modifications
- add flags in pulsar admin client tools
- update functions proto and functionConfig to contain the flag
- update the pulsar sink to pass source message property if the flag is set

### Motivation

Some of the classes in the pulsar-functions module had a mixture of the following lombok annotations:

```
@data
@Setter	
@Getter	
@EqualsAndHashCode	
@ToString
```

The [@data](https://projectlombok.org/features/Data) annotation includes all other annotations:

> All together now: A shortcut for @ToString, @EqualsAndHashCode, @Getter on all fields, @Setter on all non-final fields, and @RequiredArgsConstructor!


### Modifications

Removed `@Setter`, `@Getter`, `@EqualsAndHashCode`, and '@ToString' if the `@data` annotation was also present

Fixes #5726 

### Motivation

When running pulsar-io connectors and functions from the Intellij IDE some actions fail
due to uncaught class-not-found throwables.
The expectation being that the class is being dynamically loaded and only the ClassNotFoundException will occur if the class is not found.
When the function is created or run with https://pulsar.apache.org/docs/en/functions-deploying/#local-run-mode this is indeed the case.
When running under the control https://pulsar.apache.org/docs/en/functions-debug/#debug-with-localrun-mode as a gradle plugin the class may already be known and throw a NoClassDefFoundError.
It seems to me that any time ClassNotFoundException is handled then NoClassDefFoundError should also be caught.

### Modifications

Wherever there was a `catch (ClassNotFoundException ` I replaced it with 
`catch (ClassNotFoundException | NoClassDefFoundError ` .
There were multiple cases where the ClassNotFoundException were handled e.g. the jar loader failed so the nar loader was used to handle the jar loader's failure.

### Motivation
With [PIP-3](https://github.com/apache/pulsar/wiki/PIP-3:-Message-dispatch-throttling) , Pulsar broker already supports to configure dispatch rate-limiting for a given topic. Dispatch-throttling feature allows user to configure absolute dispatch rate based on current publish-rate for a given topic or subscriber, and broker will make sure to dispatch only configured number of messages to the consumers regardless current publish-rate or backlog on that topic.

Current dispatch-rate limiting doesn't consider change in publish-rate so, increasing publish-rate on the topic might be larger than configured dispatch-rate which will cause backlog on the topic and consumers will never be able to catch up the backlog unless user again reconfigured the dispatch-rate based on current publish-rate. Reconfiguring dispatch-rate based on publish-rate requires human interaction and monitoring. Therefore, we need a mechanism to configure dispatch rate relative to the current publish-rate on the topic.

### Modification
`set-dispatch-rate` cli have a flag `--relative-to-publish-rate` to enable relative dispatch throttling.

```
pulsar-admin namespaces <property/cluster/namespace> set-dispatch-rate --msg-dispatch-rate 1000 --relative-to-publish-rate
```

### Note:
I will add broker-level configuration and documentation into separate PR.

Signed-off-by: Nxiaolong.ran <ranxiaolong716@gmail.com>

### Motivation

- Adjust code style

### Modifications

- Add the missing placeholder in log statement
- Delete redundant `return clause` for void method 

* Add subscribe position param for consumer of sink
Signed-off-by: Nxiaolong.ran <ranxiaolong716@gmail.com>

Fixes #5350

### Motivation

When using the `--output-serde-classname` option, `functionClassLoader` is not set correctly.

### Motivation

Adding support for state deletion for Functions.  Ran into a use-case where it would be nice to be able to let go of state wherein the key is changing but the data itself is ephemeral. 

### Modifications

I've reflected the changes in the bookkeepr API back through in the context and state context API's so that you can delete on a put by passing null or call the new api for delete.

### Verifying this change

This change added tests and can be verified as follows:
 - Running the unit tests available in the ContextImplTest and ContextStateImplTest