* Various fixes for Batch Source
1. Create intermediate topic/subscription prior to function running in case auto topic creation is turned off
2. Fix possible NPE in CronTrigger when calling stop()
3. Stop all producers created in ContextImpl
4. Delete intermediate topic for batch source
Co-authored-by: NJerry Peng <jerryp@splunk.com>

1. FunctionRuntimeManager initialize() should return messageId of last read message which is the position assignment tailer should start reading from
2. Inefficient use of data structures in processAssignment()
3. Wait for leader to finish init routine before allowing scheduler to compute new assignments
4. Start leader service early so that the worker joins the worker membership sooner and its assignments don't get re-scheduled because initialize() routines for function runtime manager and metadata manager may take a while
Co-authored-by: NJerry Peng <jerryp@splunk.com>

Master Issue: #6003

### Motivation

This pull request implements the possibility to add builtin functions (in the same way of the build in connectors). 

The builtin function must include a `pulsar-io.yml` file with the following content

```yml
name: <function-name>
description: <function-desciption>
functionClass: <function-class>
```

e.g.
```yml
name: test-function
description: test function description
functionClass: it.oncode.pulsar.functions.TestFunction
```

it is possible to create a builtin function in the same way of the builtin sinks/sources.

Example in scala
```scala
val functionConfigBuilder: FunctionConfigBuilder = FunctionConfig.builder()
    val function =
      functionConfigBuilder
        .tenant("public")
        .namespace("default")
        .jar("builtin://test-function")
        .name("test-function-name")
        .className("it.oncode.pulsar.functions.TestFunction")
        .inputs(Seq("channel_in").asJava)
        .output("channel_out")
        .runtime(FunctionConfig.Runtime.JAVA)
        .build()

Pulsar.admin.functions
      .createFunction(function, null)

```

Function folder to be specified in the `conf/functions_worker.yml` conf file

e.g.
`functionsDirectory: ./functions`

Function package must be in `*.nar` format like for source/sink connectors

### Modifications

I modified the `pulsar-function-utils`, `pulsar-functions-worker` and `pulsar-common` modules on the basis of the built in connectors implementation.
Also `Function.proto` has been modified in order to include the `builtin` property

#### What this MR does not include

- modification of pulsar-admin to fetch the available buildin functions
- the related documentation

This is a feature that is critical for us, I think we could open an issue for the remaining points and consider to merge this PR.

* Extennd PulsarIO to support Batch Sources. This pr implements PIP-65

* Added license

* Fix headers
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

* Make Nar Extraction Directory configurable

* Fixed unittests
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>

Fixes #6519

### Motivation

Currently, Pulsar Functions not support Async mode, e.g. user passed in a Function in format :
```
Function<I, CompletableFuture<O>>
```
This kind of function is useful if the function might use RPCs to call external systems.

e.g.
```java
public class AsyncFunction implements Function<String, CompletableFuture<O>> {
    CompletableFuture<O> apply (String input) {
        CompletableFuture future = new CompletableFuture();
        ...function compute...
        future.whenComplete(() -> {
            ...  call external system  ...
        });
        return future;
    }
```

### Modifications
- add support for Async Functions support.

### Verifying this change
current ut passed.



* support func: Function<I, CompletableFuture<O>>

* add 2 examples

* add limit to the max outstanding items

### Motivation

Some of the classes in the pulsar-functions module had a mixture of the following lombok annotations:

```
@data
@Setter	
@Getter	
@EqualsAndHashCode	
@ToString
```

The [@data](https://projectlombok.org/features/Data) annotation includes all other annotations:

> All together now: A shortcut for @ToString, @EqualsAndHashCode, @Getter on all fields, @Setter on all non-final fields, and @RequiredArgsConstructor!


### Modifications

Removed `@Setter`, `@Getter`, `@EqualsAndHashCode`, and '@ToString' if the `@data` annotation was also present

### Motivation

This commit adds a new argument for functions, customRuntimeOptions, which is passed to
funcions (as well as sources/sinks) that enables the ability to
customize the runtime.

This is added primarily to support the `KubernetesManifestCustomizer`
interface. This interface is intended, as the name indicates, allows for
customizing how the kubernetes manifests are generated before they
are sent off to the k8s cluster. 

This interface has a default implementation, which allows for
changing the namespace, labels, nodeSelector labels, and toleratations per function.

This interface is also pluggable, allowing for more customized
implementations. For example, the functions for a given tenant could be
mapped to different pools of compute for isolation.

### Modifications

For the CLI and protobufs, the modifications just involve plumbing through the new option, `customRuntimeOptions` through the relevant code. 

For kubernetes runtime, the modifications are fairly straight forward, adding a new configuration option, `kubernetesManifestCustomizerClassName` and `kubernetesManifestCustomizerConfig` which are options under the kubernetes runtime.

### Motivation

- Adjust code style

### Modifications

- Add the missing placeholder in log statement
- Delete redundant `return clause` for void method 

* Make Function Authentication Provider pluaggable

* add config comments

* cleaning up

* cleaning up

* fix test

Fixes #4969

### Motivation

close `FileOutputStream` after `WorkerUtils.downloadFromBookkeeper`

### Motivation

Currently golang function needs to be compiled before deploy to pulsar, so the executable permission is required when function package is downloaded to local node from bookkeeper. This PR is intent to make golang function package executable after download from bookkeeper, to make sure the function is ok to run.

* Don't create new instances of user classes during validation

* fixing tests

* uncomment functions

* allow users to update auth data during function update

### Motivation

Master Issue: #3767 

support  local-run and cluster mode for go function.

in go function, we can use:

```
./bin/pulsar-admin functions localrun/create  
--go /Users/wolf4j/github.com/apache/pulsar/pulsar-function-go/examples/outputFunc.go 
--inputs persistent://public/default/my-topic 
--output persistent://public/default/test 
--tenant public 
--namespace default 
--name pulsarfunction 
--classname hellopulsar 
--log-topic logtopic
```

Different from `--jar` or `--py`, `--go` uploads a complete executable file(including: instance file + user code file)

* fix bug in source and sink cli update

* fix import

* fix logic

* fix bug in auth when spec is not null but auth is turned off

* clean up debug

* cleanup

* fix bug when function auth is not enabled

* add comment and tests

* fix bug

* improving tests

* Cleanup temporary files and refactor code

* Consolidating utils classes

* further refactoring

* adding missing class

* add license header

* Implementing authentication for Pulsar Functions

* delete unnecessary changes

* cleaning up

* improving implementation

* fixing tests

* cleaning up

* add no op implementation

* cleaning up unnecessary changes

* refactoring based on comments

* adding comments

* change data from string type to bytes

* add proto file

* addressing comments

* up merging

* refactoring get token code

* cleaning up

* fix bugs and add tests

* add tests

* remove service account creation

* cleanup unused imports

* add field for auth provider

* adding comments

* Retry creation of assignment topic a few times before giving up

* Use Action based retry mechanism

* Fix build

* Catch interrupted exception

* Fix unittest

* Added header

* Simplified the workflow of functionruntime manager

* Fix unittest

* Took feedback into account

* added missing imports

* fix: add retry and wait for kubernetes creation/deletion

* cleaning up

* adding retry logic for cleaning up subscriptions

* adjust number of retries

* correcting namespace

fix: RuntimeInfo doesn't get set when transferring ownership of function from another worker to this worker (#3523)

* fix: RuntimeInfo doesn't get set when transferring ownership of function from another worker to this worker

* fix and add test

* fix bug and adding tests

* fixing kubernetes runtime

* add debugging message

* Cleanup consumer subscriptions and fix graceful shutdown for functions

* cleaning up

* removing testing files

* add unit tests

* adding integration testing

* refactoring

* refactoring and adding tests

* cleaning up

* fix bugs in python metrics

* instance expose metrics

* remove commented out code

* fix unit tests

* remove commented out code

* fixing test

* fix python instance test

* removing old code

* fix bug

* refactoring java metrics

* refactoring python metrics

* cleaning up code

* removing unneccessary code

* improving metrics format

* fixing test

* fix bugs and revising format

* fix bug

* fix for python3

* change user defined metric to summary

* renaming labels

* change back python

* Remove functions-util dependency from pulsar-client-admin

* Simplified

* reverted accidental change

* fixed unittest

* Updated license

* Handle builtin/functionpkgurl properly in externally managed schedulers

* Correct filename

* Handle giving out the right location for function pacakge

* Reverted changes

* Fixed unittest

* Support submitting pulsar functions to kubernetes

* Adjusted pom

* Added helper function

* Added port to instance config

* Made things public

* More fixes

* Made changes to kubernetes controller

* refactored jobname

* Removed resource

* Bumped to 2.1.0

* Fix compilation bug

* Fix compile bugs

* Fixed compile

* Compile fix

* Some remant

* Unnecessary imports

* Added dep

* Default values

* Fix

* Use kubectl proxy stuff

* Corrected the path of pulsar-amin

* Fixed bugs

* No longer required

* Download first

* make shard id work

* Removed using runtime

* Fixed npe

* Corrected command name

* Fixed bugs

* Removed commented sections

* Uncommented some stuff

* Address Jerry's comments

* Change to 2.2

* Use non camel case style for passing arguments

* removed cmdline changes

* Second set of changes

* Next set of canges

* no such module

* Code complete

* Fixed logic wrt packages

* Do not stop externally managed functions upon exit

* Modified to use AppV1 instead of beta

* use lower case

* Unpretty the function details

* If the job exists, don't flag as error

* Give ability to submit from inside a pod

* remove encode/decode

* Escape function details

* Working copy with getstatus and stop. Restart needs some work

* removed unused function

* Fixed bugs

* Fixed unittest

* Fixed unittest

* Fixed unittest

* Fixed tests

* Reverted conf changes

* Minor nit

* Exlcude logback

* Updated comments

* Changed jobname to include namespace/tenant

* Fix unittest

* Updated licenses

* Use 2.0.0 instead of beta1

* [functions] change instance id from string to int and expose number of instances in context

 ### Motivation

When writing a connector reading from a list of sources, it is hard for the connector implementation to decide how
to distribute the list of sources across the function instances. because there is no way to tell how many function
instances is running.

 ### Changes

- change instance id from string to integer (since the implementation is already assuming instance id is an integer)
- add getNumInstances in the context
- expose both interfaces in source and sink connector context

* Fix compilation

### Motivation

When we submit a function, currently we don't note down the original filename that was uploaded from the client. For regular java functions this works fine. For python workers we manage by doing tricks wrt tricks like filename using classnames. However when we add things like wheel support, the filename encodes information that cannot be discarded away.
This pr preserves that filename

### Motivation

Right now, at few places, function is not logging function-reference along with exception which makes hard to debug the issue. so, added more function-reference into logs for troubleshooting. 

* Add support to restart function

fix: pulsar function restart

* add support to restart all function instances

* FunctionActioner should set the type class name for both source and sink

*Motivation*

The type class names are not set correctly for builtin connectors by FunctionActioner.

*Changes*

Set the type class name inferred from NAR package in FunctionActioner.

* Add an integration for connectors

* Fix ProcessRuntimeTest

* Submit and run locally builtin connectors

* Fixed test class constructor

* Fixed null check

* Fixed function package update

* Fixed sourceClassName when submiting custom archive

* Add function-package-url support in function cli

add url support to function cli

* file-url comment

* add server-side validation for source/sink class and arg-type

remove cmd-line args

* Introduce kinesis sink on function

add pulsarSinkE2E test

* remove kinesis test and dep

* fix pkg location init

* test download-dir with permission