* Various fixes for Batch Source
1. Create intermediate topic/subscription prior to function running in case auto topic creation is turned off
2. Fix possible NPE in CronTrigger when calling stop()
3. Stop all producers created in ContextImpl
4. Delete intermediate topic for batch source
Co-authored-by: NJerry Peng <jerryp@splunk.com>

[Issue 7933][pulsar-broker-common] Add some logging to improve Authentication debugging and Fix typos in code "occured" -> "occurred" (#7934)

Fixes #7933 

### Motivation
Newbie trying to make his first contribution to this projection :-)

### Modifications

Added some logging to Authentication Service to help debugging when there are more than one AuthN provider.
When I did that I noticed a typo ('occured' should have two r's) so I decided to try to fix all of them

Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>

* Added upgrade notes

* Add new end point to verify leader readiness

* Address feedback

* Address feedback

* Add new end point to verify leader readiness

* Address feedback

* Address feedback
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>
Co-authored-by: NMatteo Merli <mmerli@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>

When a function worker becomes the leader, start producer to assignment topic first before stopping the assignment tailer (#7525)
Co-authored-by: NJerry Peng <jerryp@splunk.com>

* Added upgrade notes

* Cleanup the code to avoid deadlocks

* Fix test

* Fix test

* Fix test
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>

* Differentiate between source/sink/function operations

* Added release notes
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Co-authored-by: NJerry Peng <jerryp@splunk.com>

Fixes #7379 

### Motivation

Rename some settings to make those broker-client TLS settings clearer.

### Modifications

- replace `clientAuthenticationParameters` with `brokerClientAuthenticationParameters`
- replace `clientAuthenticationPlugin` with `brokerClientAuthenticationPlugin`
- replace `tlsHostnameVerificationEnable` with `tlsEnableHostnameVerification`

Co-authored-by: NJerry Peng <jerryp@splunk.com>

* Function workers re-direct call update requests to the leader

* Fixed test

* tests pass

* Working version

* Fix test

* Short circuit update

* Fix test

* Fix test

* Fix tests

* Added one more catch

* Added one more catch

* Seperated internal and external errors

* Fix test

* Address feedback

* Do not expose updateOnLeader to functions

* hide api

* hide api

* removed duplicate comments

* Do leadership changes in function metadata manager

* make the function sync

* Added more comments

* Throw error

* Changed name

* address comments

* Deleted unused classes

* Rework metadata manager

* Working

* Fix test

* A better way for test

* Address feedback

* Added an option to compact function metadata topic

* Address feedback

* Incorporate feedback
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

* Function workers re-direct call update requests to the leader

* Fixed test

* tests pass

* Working version

* Fix test

* Short circuit update

* Fix test

* Fix test

* Fix tests

* Added one more catch

* Added one more catch

* Seperated internal and external errors

* Fix test

* Address feedback

* Do not expose updateOnLeader to functions

* hide api

* hide api

* removed duplicate comments

* Do leadership changes in function metadata manager

* make the function sync

* Added more comments

* Throw error

* Changed name

* address comments

* Deleted unused classes

* Rework metadata manager

* Working

* Fix test

* A better way for test

* Address feedback
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

* Add an endpoint to check whether function worker service is initialized
Co-authored-by: NJerry Peng <jerryp@splunk.com>

Fix race condition in which exitFuture in FunctionAssignmentTailer never gets completed even though the tailer thread has exited (#7351)

* fix race condition in which exitFuture in FunctionAssignmentTailer never gets completed even though the tailer thread has exited

* fix
Co-authored-by: NJerry Peng <jerryp@splunk.com>

1. FunctionRuntimeManager initialize() should return messageId of last read message which is the position assignment tailer should start reading from
2. Inefficient use of data structures in processAssignment()
3. Wait for leader to finish init routine before allowing scheduler to compute new assignments
4. Start leader service early so that the worker joins the worker membership sooner and its assignments don't get re-scheduled because initialize() routines for function runtime manager and metadata manager may take a while
Co-authored-by: NJerry Peng <jerryp@splunk.com>

* Fix leader/scheduler assignment processing lag problem

* add license header

* adding more comments

* improving impl

* fixing bugs

* improving impl

* fixing tests

* adding comments

* add more testing

* addressing comments

* cleaning up

* refactoring implementation

* addressing comments
Co-authored-by: NJerry Peng <jerryp@splunk.com>

* Fix logic while updating functions. Cleanup should only happen for temp files that were generated

* Fix integration tests

* Fix test

* Fix logic for parallelism > 1

* Fix test

* Address comments
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

* Have metadata tailer use its own thread for processing

* Merged with master

* Address comments

* Address comments
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

### Motivation

When running the worker service as part of the broker, fatal errors encountered in the worker should also trigger the broker to shutdown

* Add ability for function services to notify errors

The broker and proxy both allow for hitting the metrics endpoint without auth. The functions
worker should allow that to be configurable as well. This adds an option to
allow for metrics endpoint to allow the endpoint to be hit without auth

Additionally, the functions worker doesn't expose the default prometheus
metrics (such as JVM info, etc).

This commit implements and adds an option to support that
Co-authored-by: NAddison Higham <ahigham@instructure.com>

* FunctionAssignmentTailer should use its own thread
Co-authored-by: NJerry Peng <jerryp@splunk.com>

fix typo: follwing => following

* Attach names for all producers/readers in worker service

* Fix tests
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Master Issue: #6003

### Motivation

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors). 

The builtin function must include a `pulsar-io.yml` file with the following content

```yml
name: <function-name>
description: <function-desciption>
functionClass: <function-class>
```

e.g.
```yml
name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction
```

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala
```scala
val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

```

Function folder to be specified in the `conf/functions_worker.yml` conf file

e.g.
`functionsDirectory: ./functions`

Function package must be in `*.nar` format like for source/sink connectors

### Modifications

I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation.
Also `Function.proto` has been modified in order to include the `builtin` property

#### What this MR does not include

- modification of pulsar-admin to fetch the available buildin functions
- the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.

* Seperate out FunctionMetaData related functions into a utility class

* Fixed bug
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

* Extennd PulsarIO to support Batch Sources. This pr implements PIP-65

* Added license

* Fix headers
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

* Moved ClassLoading and Reflections Helper functions to common

* Fix tests

* Fix test
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

* remove function files stored in BK when function is de-registered
Co-authored-by: NJerry Peng <jerryp@splunk.com>

* Added sourceConfigClass and sinkConfigClass

* Add Validator annotation helpers to validate class parameters

* Fix build errors

* Take feedback into account

* Connected with validation

* Fix bugs

* Added tests

* Fix class name

* Address feedback
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Fixes #4803 

### Motivation
Allow the typed consumer receive messages with `null` value if the producer sends message without payload.

### Modifications
- add a flag in `MessageMetadata` to indicate if the payload is set when the message is created
- check and return `null` if the flag is not set when reading data from a message